Published at RLC 2025 Reinforcement Learning Conference · Reinforcement Learning Journal, vol. 6

Reinforcement Learning from Human Feedback with High-Confidence Safety Guarantees

  • Yaswanth Chittepu1*
  • Blossom Metevier1*
  • Will Schwarzer1
  • Austin Hoag2
  • Scott Niekum1†
  • Philip S. Thomas1†
1 University of Massachusetts Amherst  ·  2 Sony AI  ·  * Equal contribution  ·  † Equal advising
\(\Pr\big(\C{g}(\mathcal{A}(D)) \le 0\big) \ge \K{1-\delta}\)

Safe with high probability, not just on the training prompts. HC-RLHF maximizes helpfulness under a deliberately pessimistic harmlessness constraint, then runs a held-out statistical safety test. A model is returned only if it passes, so the chance of shipping an unsafe model is at most \(\K{\delta}\).

The problem

Satisfying a safety constraint in training is not a guarantee

Safe RLHF separates helpfulness from harmlessness and maximizes a reward model subject to a limit on a cost model. That gives much better control over the trade-off than standard RLHF. But the constraint is only enforced on the prompts seen during training, and nothing bounds the probability that the final model is unsafe once it is deployed. In medical, legal or educational settings, that probability is exactly what matters.

Read the abstract

Existing approaches to language model alignment often treat safety as a tradeoff against helpfulness, which can lead to unacceptable responses in sensitive domains. To ensure reliable performance in such settings, we propose High-Confidence Safe Reinforcement Learning from Human Feedback (HC-RLHF), a method that provides high-confidence safety guarantees while maximizing helpfulness. Similar to previous methods, HC-RLHF explicitly decouples human preferences regarding helpfulness and harmlessness (safety) and trains separate reward and cost models, respectively. It then employs a two-step process to find safe solutions. In the first step, it optimizes the reward function while ensuring that a specific upper-confidence bound on the cost constraint is satisfied. In the second step, the trained model undergoes a safety test to verify whether its performance satisfies a separate upper-confidence bound on the cost constraint. We provide a theoretical analysis of HC-RLHF, including a proof that it will not return an unsafe solution with a probability greater than a user-specified threshold. For our empirical analysis, we apply HC-RLHF to align three different language models (Qwen2-1.5B, Qwen2.5-3B, and LLaMa3.2-3B) with human preferences. Our results demonstrate that HC-RLHF produces safe models with high probability while also improving helpfulness and harmlessness compared to previous methods.

Safe RLHF: constraint on the mean
\(\max_\theta\ \mathbb{E}[\R{r}] \ \ \text{s.t.}\ \ \mathbb{E}[\C{c}] \le \tau\)

The constraint holds on average over training prompts. Whether it still holds on new prompts depends on the dataset, the threshold and luck, and there is no way to know in advance.

HC-RLHF: constraint with a confidence bound
\(\max_\theta\ \mathbb{E}[\R{r}] \ \ \text{s.t.}\ \ \hat{\mathbb{E}}[\C{c}] + \K{K(\delta)}\,\hat{\mathbb{S}}[\C{c}] \le \tau\)

Train against an upper confidence bound on cost, not the mean, then verify on data the optimizer never saw. The guarantee comes from the Seldonian framework.

How it works

Train pessimistically, then test on held-out data

HC-RLHF is the first Seldonian algorithm applied to RLHF. It uses the same decoupled preference data as Safe RLHF: a reward model \(\R{r_\phi}\) is trained on helpfulness labels and a cost model \(\C{C_\psi}\) on harmfulness labels, both with the standard Bradley–Terry loss. What changes is how the policy is optimized and what happens before it is returned.

Step 1

Partition the data

\(D \to D_c,\ D_s\)

Split the prompts into a candidate-selection set used for training and a safety set held out for the final test.

Step 2

Candidate selection

RLOO with a variance-aware augmented reward

Maximize reward under the inflated cost bound, using Lagrangian dual ascent. Inflating the bound accounts for the many models compared during optimization, and makes it more likely that the candidate will pass the test.

Step 3

Safety test

Student's t upper bound on \(D_s\)

Score the candidate's responses on the held-out prompts with the cost model and compute a (1 − δ) upper confidence bound on its expected cost.

\(U_{\text{ttest}} \le \tau\)  →  return the model
otherwise  →  return No Solution Found
The guarantee
\[\Pr\big(\C{g}(\mathcal{A}(D)) \le 0\big) \ge \K{1-\delta}\]\[\C{g}(\theta) = \mathbb{E}_{x\sim\mathcal{D}_x,\; y\sim\pi_\theta(\cdot\mid x)}\big[\C{C_\psi(x,y)}\big] - \tau\]
This is the Seldonian condition. Returning No Solution Found counts as safe, so the algorithm can decline instead of shipping a model it cannot vouch for.
Pessimistic training constraint
\[\hat{\mathbb{E}}\big[\C{C_\psi(x,y)}\big] + \K{K(\delta)}\,\hat{\mathbb{S}}\big[\C{C_\psi(x,y)}\big] \le \tau\]\[\K{K(\delta)} = \rho_1\,\frac{t_{1-\delta,\,B-1}}{\sqrt{B}} + \rho_2\,\frac{t_{1-\delta,\,n_s-1}}{\sqrt{n_s}}\]
The first term reflects the batch size \(B\) seen at each step. The second scales with the safety set size \(n_s\), so training anticipates the test the model will face. We use \(\rho_1 = 4\) and \(\rho_2 = 2\).
Policy gradient
\[\begin{aligned}\hat R(x,y) &= \R{\tilde r(x,y)} - \lambda\,\C{C_\psi(x,y)} \\ &\quad - \lambda\,\K{K(\delta)}\,\frac{\C{C_\psi(x,y)}^2 - 2\,\mathbb{E}[\C{C_\psi}]\,\C{C_\psi(x,y)}}{2\,\mathbb{S}[\C{C_\psi}]}\end{aligned}\]\[\nabla_\theta \mathcal{L}(\theta,\lambda) = \mathbb{E}_{x,\,y\sim\pi_\theta}\big[\hat R(x,y)\,\nabla_\theta \log \pi_\theta(y\mid x)\big]\]
Differentiating the Lagrangian gives a REINFORCE-style gradient with this augmented reward, where \(\tilde r\) is the KL-regularized reward. The mean and standard deviation of cost are tracked with running estimates. We use the leave-one-out baseline (RLOO) with two responses per prompt.
Safety test
\[U_{\text{ttest}}(\hat g) = \frac{1}{m}\sum_{i=1}^{m} \hat g_i + \frac{\sigma(\hat g)}{\sqrt{m}}\; t_{1-\delta,\,m-1}\]\[\Pr\big(g(\theta_c) \le U_{\text{ttest}}(\hat g)\big) \ge \K{1-\delta}\]
Each \(\hat g_i = C_\psi(x_i, y_i)\) is the cost of one response on a held-out prompt, so the estimates are i.i.d. and unbiased, and the t-test bound holds with probability at least \(1-\delta\).

Result 1 · Harmfulness

Far fewer responses cross the harm threshold

Each point is one test-set response, scored by the trained reward and cost models. Responses to the right of the dashed line (cost above \(\tau = 0\)) are judged harmful. All three models share the same reward and cost models; the only difference is the safety constraint used during RL.

x: cost (harmfulness) · y: reward (helpfulness) · red line: τ = 0
Reward versus cost for SFT responses, Llama-3.2-3B.
SFT
Reward versus cost for Safe RLHF responses, Llama-3.2-3B.
Safe RLHF
Reward versus cost for HC-RLHF responses, Llama-3.2-3B.
HC-RLHF
Reward against cost on the test set. HC-RLHF shifts the cloud left, away from the threshold, and leaves far fewer responses on the harmful side than Safe RLHF.

Result 2 · Helpfulness

Safer, and more helpful at the same time

For each test prompt we compare the HC-RLHF and Safe RLHF responses, bucket the pair by whether each response is safe, and count how often HC-RLHF earns the higher reward.

Safe RLHF ↓   HC-RLHF →
HC-RLHF safe
HC-RLHF unsafe
Safe RLHF
safe
Safe RLHF
unsafe

Cells show the HC-RLHF reward win rate and, in parentheses, the number of prompt pairs in that bucket.

Result 3 · GPT-4 as judge

An independent judge prefers HC-RLHF too

To check that the gains are not an artifact of our own reward and cost models, GPT-4 compared responses pairwise for Llama-3.2-3B. "Overall" uses the categorized red-teaming prompts from the Safe RLHF repository. Helpfulness and harmlessness use 100 unseen test prompts.

WinTieLose

HC-RLHF vs. Safe RLHF

Overall
Helpfulness
Harmlessness

HC-RLHF vs. SFT

Overall
Helpfulness
Harmlessness

For reference, Safe RLHF beats SFT 62.7% overall, 76% on helpfulness and 77% on harmlessness, below HC-RLHF on all three.

Horizontal stacked bars of HC-RLHF lose, tie and win rates against Safe RLHF for eight harm categories: Immoral, Insult, Physical Harm, Emotional Harm, Privacy, Crime, Pornographic, Social Bias.
Win, tie and lose rates against Safe RLHF across the eight harm categories of the red-teaming prompts. HC-RLHF wins at least as often as it loses in every category.

Result 4 · The guarantee in practice

No unsafe model was ever returned

To test the guarantee empirically, we repeatedly ran HC-RLHF on Qwen2-1.5B (the smallest model, so many trials were affordable) and checked each returned model against a large held-out dataset.

Allowed failure rate δ
0.10

At most 10% of runs may return an unsafe model, at τ = 0.

Trials
30

Bootstrap-resampled training sets of 1,000 prompts each.

Observed failure rate
0 / 30

Every candidate passed the safety test, and every returned model was safe.

Tightening the threshold

Lower τ means a stricter harm budget. Both methods trained on 72,000 prompts with 4,000 held out, Llama-3.2-3B, one trial per threshold.

τ
0
−4
−7
−9
−12
Safe RLHF
Safe
Safe
Safe
Unsafe
Unsafe
HC-RLHF
Safe
Safe
Safe
Safe
Safe

A single trial per threshold cannot estimate Safe RLHF's failure rate. The point is that Safe RLHF has no guarantee, so there is no way to tell in advance which thresholds or dataset sizes it will handle. HC-RLHF's guarantee holds regardless.

Citation

BibTeX

@article{chittepu2025reinforcement,
  title   = {Reinforcement Learning from Human Feedback with High-Confidence Safety Guarantees},
  author  = {Chittepu, Yaswanth and Metevier, Blossom and Schwarzer, Will and Hoag, Austin and Niekum, Scott and Thomas, Philip S.},
  journal = {Reinforcement Learning Journal},
  volume  = {6},
  pages   = {713--736},
  year    = {2025}
}