Dwarf-Bench is a small LLM benchmark that tests how well models recall obscure, specific facts about dwarves across fantasy media — Tolkien, Warhammer, Dungeons & Dragons, Discworld, Dragon Age, World of Warcraft, and friends.
Each model is prompted with tools, search, and browsing disabled, and is told to answer from internal knowledge only or to reply "I don't know". Free-form responses are then graded against a gold-standard answer key by an LLM-as-judge using a 1.0 / 0.5 / 0.0 rubric (fully correct / partially correct / wrong-or-unknown).
It's a half-joke benchmark, but it's also a real one — mostly an excuse to build an eval pipeline from scratch (provider calls, async runner, judge, reporting) without leaning on a heavy framework.
Latest results
| # | Model | Accuracy | Correct | Partial | Wrong | N |
|---|---|---|---|---|---|---|
| 1 | claude-opus-4-6 | 84.6% | 11 | 0 | 2 | 13 |
| 1 | gpt-5.5 | 84.6% | 11 | 0 | 2 | 13 |
| 3 | gemini-2.5-pro | 83.3% | 10 | 0 | 2 | 12 |
| 4 | claude-opus-4-7 | 76.9% | 10 | 0 | 3 | 13 |
| 4 | claude-sonnet-4-6 | 76.9% | 10 | 0 | 3 | 13 |
| 6 | gpt-5 | 69.2% | 9 | 0 | 4 | 13 |
| 6 | gpt-5.4 | 69.2% | 9 | 0 | 4 | 13 |
| 8 | claude-haiku-4-5 | 38.5% | 5 | 0 | 8 | 13 |
Source
Code, dataset, judge prompt, and the leaderboard JSON this page consumes all live on GitHub: github.com/nocount/dwarf-bench.