Evaluation

The Evaluation page runs evaluations against a model version and reports pass rates, step rates, and a heatmap.

Chinese-only UI

MindForge's web UI is localised in Chinese only. The field and column names below are given in English; the labels you see in the product will be Chinese. For a labelled screenshot see the Chinese evaluation page.

Official benchmark

The Official benchmark card evaluates against the official SciCode benchmark.

• Official eval set — split selector: validation or test.
• Model version — the version to evaluate, e.g. Qwen/Qwen3.6-35B-A3B or a trained version.
• Temperature, Concurrency, Repeats.
• Use background prompt — include the problem's background context.
• Enable thinking — enable the model's thinking/reasoning mode.

Click expand full command to preview scripts/eval.sh, then Start official eval.

Official results are evaluation-only

Official benchmark results are not used as training data. Training data comes from model trajectories on your self-built question bank.

Problem preview

Problem preview shows the official validation/test set's size, domain distribution, and step counts — expand it to inspect what you are evaluating against.

Evaluation records

Evaluation records lists past runs under tabs validation / test / smoke. Each row shows ID, model version, background-prompt flag, thinking flag, repeats, problem pass rate, step pass rate, pass count, time, and actions. Official results never enter training.

Heatmap

Evaluation heatmap renders a per-problem pass/fail heatmap across model versions so you can see which problems each version solved. Click Load heatmap to render it for the selected records.

Question-bank evaluation

Evaluating against your own question bank (each problem scored by its own test_data.h5) is done from the Question Bank page; passing trajectories there can be turned into SFT training data.