---
title: Leaderboard
emoji: 🏆
colorFrom: green
colorTo: blue
sdk: static
pinned: false
license: apache-2.0
---

Model performance across Bench Labs' benchmark suite — Effortless, Easy, Mid, and AGI.
Reads from `models.json`, no backend required. Defaults to the latest benchmark
generation (**7-2026**, dual-mode); legacy 6-2026 runs stay one click away.

## Evaluate a model

**Easiest path:** open [`cookbook.ipynb`](https://huggingface.co/spaces/bench-labs/BenchLabs-Leaderboard/blob/main/cookbook.ipynb)
— a ready-to-run notebook (local or Colab) that benchmarks Qwen2.5-0.5B end to end:
install → run → score tables → per-answer inspection → submit to the review queue.
Change one variable to benchmark your own model.

Or the CLI — one script, every benchmark, per-category/subcategory results:

```bash
curl -sLO https://huggingface.co/spaces/bench-labs/BenchLabs-Leaderboard/resolve/main/script.py
pip install torch transformers
python script.py --model your/model --publish
```

Writes `results.json` (full breakdown), per-sample CSVs, and `leaderboard.json`
(this run's entry) — then `--publish` submits that entry straight to the
BenchLabs review queue, no PR or manual file upload needed. If the machine
running the script has no browser, you'll be shown a short code to approve
from any browser (phone, laptop, wherever) — same device-login flow as `gh
auth login` or `docker login`; that approval never exposes your Hugging Face
token to the machine running the script, only a scoped, revocable session
token cached locally for next time. Prefer to authenticate directly? Pass
`--hf-token <token>` (or set `HF_TOKEN`) to skip the device flow.

A bench-labs moderator reviews every submission — checking the diff against
what's already on the board — before it goes live. You can also submit
through the browser instead of the CLI at
[benchlabs.ereneksi.com/submit](https://benchlabs.ereneksi.com/submit/), or
review the moderation queue yourself if you're a bench-labs org member at
[benchlabs.ereneksi.com/admin](https://benchlabs.ereneksi.com/admin/).

Run `python script.py --help` for all options.

### 7-2026 dual-mode tiers

The `bench-*-7-2026` datasets use schema v2: every item carries a gold answer +
aliases **and** `target_scores` choices. One run scores each tier both ways —
generative (`exact_match`, per-item-routed `hybrid_score`) and log-likelihood
(`acc`, `acc_norm`, `soft_score`, `soft_score_norm`) — and writes a per-item
`samples_<id>.jsonl` with the raw generation, extracted answer, and per-choice
log-probs. The default run covers exactly the latest generation
(`--benchmarks latest` = the three 7-2026 tiers); use `--benchmarks all` for both
generations, or name tiers directly (`effortless7,easy7,mid7` / `effortless,easy,mid`).

### Reasoning / CoT models

Models that think in `<think>…</think>` blocks are supported: the script strips the
reasoning block and scores only the text after the final `</think>`. Two things to know:

- Raise the generation budget — `--max-new-tokens 2048` or more. The default 32 is sized
  for direct-answer models; a thinking model burns it inside the block, and a generation
  cut off mid-think (unclosed `<think>`) scores as an empty answer.
- `bench-mid-6-2026` is scored by log-likelihood over the answer choices with no
  generation, so thinking never runs there.

### Provenance

`leaderboard.json` pins the run to exact bytes: `model_revision` is the Hub commit of the
weights the script actually loaded (recorded by `from_pretrained`, not re-resolved after
the fact), and `script_sha256` is the hash of the script file that produced the numbers.
Local checkpoints have no Hub revision and honestly report `null`.

all rights reserved @bench-labs