Skip to content

CISPO

CISPO (Chen et al., 2024; Khatri et al., 2024) is a policy-gradient variant that, unlike PPO, does not clip the objective. Instead it clips the importance ratio and uses the clipped value purely as a coefficient on the policy gradient — so no token's gradient is ever zeroed, only its weight is bounded.

The objective:

[ \mathcal{L}{\text{CISPO}}(\theta) = \mathbb{E}\right)\right) \cdot \log p_\theta(x) \cdot A(x)\right] ]}\left[\mathbf{sg}!\left(\text{clip}\left(\frac{p_\theta(x)}{q(x)},\, 1-\epsilon_{\text{low}},\, 1+\epsilon_{\text{high}

where (\mathbf{sg}) is stop-gradient (the clipped ratio is detached).

Equivalent code:

prob_ratio = torch.exp(target_logprobs - sampling_logprobs)
clipped_ratio = torch.clamp(prob_ratio, clip_low_threshold, clip_high_threshold)
cispo_objective = clipped_ratio.detach() * target_logprobs * advantages
loss = -cispo_objective.sum()

Inputs

  • target_tokens: array[(N,), int] — IDs sampled by (q).
  • logprobs: array[(N,), float] — sampling_logprobs.
  • advantages: array[(N,), float] — per-token advantage.

Outputs

  • logprobs — target_logprobs.

Diagnostics

  • loss:sum — total CISPO loss.

Choosing the clip bounds

Since the clipped ratio only acts as a detached coefficient weighting (\log p_\theta) — instead of clamping the objective the way PPO does — the thresholds matter differently than in PPO. In particular the lower bound is delicate.

The default is one-sided. MinT leaves clip_low_threshold=0.0 and only caps the upper side at clip_high_threshold=4.0, so you get this without passing any config. Set it explicitly like so:

fb = training_client.forward_backward(
    data=data,
    loss_fn="cispo",
    loss_fn_config={"clip_low_threshold": 0.0, "clip_high_threshold": 4.0},
)
result = fb.result()

Why drop the lower bound? With no floor, the worst case is that the coefficient simply decays toward the ordinary importance-sampling weight, which stays well-behaved. A positive lower bound (say 0.8) instead floors the weight even for tokens whose ratio has already fallen well below 1 — the stale tokens the sampler has moved away from — removing the natural attenuation that keeps off-policy training stable. On-policy this rarely matters; off-policy (async training, where (q) lags (p_\theta)) a positive floor can create a feedback loop: sampler/trainer KL climbs, more tokens slip out of the band, bias rises, and KL climbs again.

This matches both source papers. MiniMax-M1 disabled the lower IS bound and only tuned the upper; ScaleRL found CISPO largely insensitive to the upper bound (values of 4, 5, and 8 performed identically), so any upper bound around that band is fine as a default.