Importance Sampling

In RL the rollout policy (q) usually differs from the learner (p_\theta) (because sampling lags training, sampling is non-deterministic, etc.). The plain on-policy objective

[ \mathcal{L}(\theta) = \mathbb{E}{x\sim p\theta}[A(x)] ]

is then biased, because the samples come from (q), not (p_\theta). Importance sampling corrects the bias by reweighting each sample with the probability ratio:

[ \mathcal{L}{\text{IS}}(\theta) = \mathbb{E} A(x)\right] ]}\left[\frac{p_\theta(x)}{q(x)

which recovers the correct expected reward.

The two log-prob tensors that appear:

• (\log p_\theta(x)) — target_logprobs, computed by the learner on the forward half of forward_backward.
• (\log q(x)) — sampling_logprobs, recorded by the sampler during rollout and passed in as the correction term.

Equivalent code:

prob_ratio = torch.exp(target_logprobs - sampling_logprobs)
loss = -(prob_ratio * advantages).sum()

Inputs

• target_tokens: array[(N,), int] — IDs sampled by (q).
• logprobs: array[(N,), float] — sampling_logprobs for those tokens.
• advantages: array[(N,), float] — per-token advantage (positive reinforces, negative discourages).

Outputs

• logprobs — target_logprobs for the tokens.

Diagnostics

• loss:sum — total importance-weighted policy-gradient loss.