Custom Environment

Concept

An RL environment in Mint Recursive implements the RLAdapter interface:

class MyEnvironment(RLAdapter):
    def __call__(self, prompt: ModelInput, response: ModelInput) -> Reward:
        # Take the model's response and compute a reward
        # Return: Reward(score=float, metadata=dict)

The environment receives: - prompt — the input context (the tokens given to the model). - response — the model's completion (the tokens it produced).

The environment returns: - score — a scalar reward (higher is better). - metadata — an optional dict with debug info such as error messages or intermediate scores.

Common environment patterns: - Verification — check whether the response is correct (for example, a math solution). Return 1.0 or 0.0. - Scoring — apply a metric function (BLEU, edit distance, semantic similarity). Return 0.0–1.0. - External system — call an API or subprocess (code execution, fact-checking). Return the score along with error info. - Multi-step — simulate a conversational environment and check whether the model's response achieves a goal.

Pattern

from mint.rl_core import RLAdapter, Reward

class MathVerificationEnv(RLAdapter):
    """Verify math problem solutions via symbolic computation."""

    def __call__(self, prompt: mint.types.ModelInput, response: mint.types.ModelInput) -> Reward:
        tokenizer = self.get_tokenizer()

        # Decode tokens to text
        response_text = tokenizer.decode(response.to_ints())

        # Extract the numeric answer from the response
        import re
        match = re.search(r'\d+', response_text)
        if not match:
            return Reward(score=0.0, metadata={"error": "No number found"})

        predicted_answer = int(match.group())
        expected_answer = 42  # (in practice, extract from prompt or ground truth)

        # Reward: 1.0 if correct, 0.0 if incorrect
        is_correct = predicted_answer == expected_answer
        return Reward(
            score=1.0 if is_correct else 0.0,
            metadata={"predicted": predicted_answer, "expected": expected_answer},
        )

class ToolUseEnv(RLAdapter):
    """Reward model for learning to use tools."""

    def __call__(self, prompt: mint.types.ModelInput, response: mint.types.ModelInput) -> Reward:
        tokenizer = self.get_tokenizer()
        response_text = tokenizer.decode(response.to_ints())

        # Check if model called a tool (e.g., contains <tool_call>)
        used_tool = "<tool_call>" in response_text
        tool_score = 1.0 if used_tool else 0.0

        # Check if response is coherent
        coherence_score = 1.0 if len(response_text) > 20 else 0.0

        # Combine scores
        total_score = 0.7 * tool_score + 0.3 * coherence_score

        return Reward(
            score=total_score,
            metadata={"used_tool": used_tool, "coherence": coherence_score},
        )

# Usage during RL training
training_client = service_client.create_lora_training_client(base_model="Qwen/Qwen3-0.6B", rank=16)
env = MathVerificationEnv(training_client.get_tokenizer())

# Sample from the model and evaluate
sampler = service_client.create_sampling_client(base_model="Qwen/Qwen3-0.6B")
sampler.sample(prompt=..., sampling_params=...)  # Get response tokens

# Evaluate with environment
reward = env(prompt_tokens, response_tokens)
print(f"Reward: {reward.score}, Metadata: {reward.metadata}")

View full source: https://github.com/MindLab-Research/mint-quickstart/blob/main/demos/rl/adapters/environment_tooluse.py

API Surface

Class	Method	Purpose
RLAdapter	__call__(prompt, response)	Compute reward for a (prompt, response) pair
Reward	score: float	Scalar reward value
Reward	metadata: dict	Optional debug/logging info

Helper methods in RLAdapter:

• get_tokenizer() — retrieve the tokenizer for decoding tokens.
• get_model_info() — query the model family and configuration.

Caveats & Pitfalls

• Reward scale: normalize rewards to a reasonable range (0–1 or -1 to 1). Extremely large rewards destabilize gradient updates.
• Sparse rewards: binary rewards (correct/incorrect) give weak training signals. When possible, provide denser feedback — partial credit, semantic similarity scores, and so on.
• Execution safety: be cautious with code execution environments. Sandboxing is essential to prevent malicious code from damaging your system.
• Latency: if your environment calls external APIs (web search, code execution), use async patterns and timeouts. A slow environment becomes a training bottleneck.
• Determinism: if your environment has random elements, consider seeding for reproducible training. Log the sources of randomness in metadata.