README.md

metadata

title: Leaderboard
emoji: 🏆
colorFrom: green
colorTo: blue
sdk: static
pinned: false
license: apache-2.0

Model performance across Bench Labs' benchmark suite — Effortless, Easy, Mid, and AGI. Reads from models.json, no backend required. Defaults to the latest benchmark generation (7-2026, dual-mode); legacy 6-2026 runs stay one click away.

Evaluate a model

Easiest path: open cookbook.ipynb — a ready-to-run notebook (local or Colab) that benchmarks Qwen2.5-0.5B end to end: install → run → score tables → per-answer inspection → submit to the review queue. Change one variable to benchmark your own model.

Or the CLI — one script, every benchmark, per-category/subcategory results:

curl -sLO https://huggingface.co/spaces/bench-labs/BenchLabs-Leaderboard/resolve/main/script.py
pip install torch transformers
python script.py --model your/model --publish

Writes results.json (full breakdown), per-sample CSVs, and leaderboard.json (this run's entry) — then --publish submits that entry straight to the BenchLabs review queue, no PR or manual file upload needed. If the machine running the script has no browser, you'll be shown a short code to approve from any browser (phone, laptop, wherever) — same device-login flow as gh auth login or docker login; that approval never exposes your Hugging Face token to the machine running the script, only a scoped, revocable session token cached locally for next time. Prefer to authenticate directly? Pass --hf-token <token> (or set HF_TOKEN) to skip the device flow.

A bench-labs moderator reviews every submission — checking the diff against what's already on the board — before it goes live. You can also submit through the browser instead of the CLI at benchlabs.ereneksi.com/submit, or review the moderation queue yourself if you're a bench-labs org member at benchlabs.ereneksi.com/admin.

Run python script.py --help for all options.

7-2026 dual-mode tiers

The bench-*-7-2026 datasets use schema v2: every item carries a gold answer + aliases and target_scores choices. One run scores each tier both ways — generative (exact_match, per-item-routed hybrid_score) and log-likelihood (acc, acc_norm, soft_score, soft_score_norm) — and writes a per-item samples_<id>.jsonl with the raw generation, extracted answer, and per-choice log-probs. The default run covers exactly the latest generation (--benchmarks latest = the three 7-2026 tiers); use --benchmarks all for both generations, or name tiers directly (effortless7,easy7,mid7 / effortless,easy,mid).

Reasoning / CoT models

Models that think in <think>…</think> blocks are supported: the script strips the reasoning block and scores only the text after the final </think>. Two things to know:

• Raise the generation budget — --max-new-tokens 2048 or more. The default 32 is sized for direct-answer models; a thinking model burns it inside the block, and a generation cut off mid-think (unclosed <think>) scores as an empty answer.
• bench-mid-6-2026 is scored by log-likelihood over the answer choices with no generation, so thinking never runs there.

Provenance

leaderboard.json pins the run to exact bytes: model_revision is the Hub commit of the weights the script actually loaded (recorded by from_pretrained, not re-resolved after the fact), and script_sha256 is the hash of the script file that produced the numbers. Local checkpoints have no Hub revision and honestly report null.

all rights reserved @bench-labs