Convex-Concave Reinforcement Learning
arXiv preprint, 2026
Yaswanth Chittepu
PhD Student, Computer Science
University of Massachusetts Amherst
I am a PhD student in Computer Science at UMass Amherst, advised by Scott Niekum in the SCALAR Lab. My research focuses on designing safe and efficient reinforcement learning algorithms. To this end, I have worked on making learning from human feedback safe and reliable: alignment methods for language models that come with high-confidence guarantees, control over risk beyond the average case, and a clearer picture of how reward optimization breaks down.
I am also broadly interested in offline RL, imitation learning, inverse reinforcement learning, and generative modeling. Before the PhD, I received an M.S. in Computer Science from UMass Amherst and a B.Tech. in Mechanical Engineering from IIT Kharagpur.
Reinforcement learning Offline RL Imitation learning Inverse RL RLHF & alignment Safe RL Reward overoptimization Generative modeling LLM agents
News
- New preprint: Convex-Concave Reinforcement Learning, which recasts each policy-improvement step as a difference-of-convex program that recovers CPI, NPG, TRPO, PPO, and AWR as special cases.
- Safe Inference-Time Alignment via Lagrangian Reward Augmentation will appear at COLM 2026 in San Francisco. Come see the poster (#109) on Thu, Oct 8! Project page.
- Research intern at Adobe Research, working on learning compact state representations for LLM agents.
- New preprint: Safe RLHF Beyond Expectation, on controlling spectral risk measures with stochastic dominance.
- Released ML-Tool-Bench, a benchmark for tool-augmented planning on ML tasks.
- New preprint: Adaptive Margin RLHF via Preference over Preferences.
- Presented High-Confidence Safe RLHF at RLC 2025.
- Research intern at Adobe Research, working on tool-calling agents for solving ML problems, which led to ML-Tool-Bench.
- Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms at NeurIPS 2024.
Publications
*Equal contribution. See Google Scholar for the full list.
TRIBE: Trust-Region Policy Improvement from Batch Experience
Under review, 2026
Learning Compact State Abstractions for LLM Agents
Under review, 2026
Safe Inference-Time Alignment via Lagrangian Reward Augmentation
Conference on Language Modeling (COLM), 2026
ML-Tool-Bench: Tool-Augmented Planning for ML Tasks
arXiv preprint, 2025
Adaptive Margin RLHF via Preference over Preferences
arXiv preprint, 2025
Reinforcement Learning from Human Feedback with High-Confidence Safety Guarantees
Reinforcement Learning Conference (RLC), 2025 · Reinforcement Learning Journal, vol. 6
Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms
Advances in Neural Information Processing Systems (NeurIPS), 2024. Also at the MHFAIA workshop @ ICML 2024 and the RLBrew workshop @ RLC 2024
Service
- RecognitionGold Reviewer, ICML 2026
- Conference reviewerNeurIPS 2026, ICML 2026, ICLR 2026, COLM 2026
- Holistic reviewerRLC 2026
- Journal reviewerTransactions on Machine Learning Research (TMLR)
- Workshop reviewerRLBrew @ RLC 2024, 2025; Models of Human Feedback for AI Alignment @ ICML 2024, 2025
Education
- Ph.D., Computer Science
University of Massachusetts Amherst - M.S., Computer Science
University of Massachusetts Amherst - B.Tech., Mechanical Engineering
Indian Institute of Technology Kharagpur