LegoBench - AI agents building LEGO cars

I have spent the past few weeks watching the release of many new LLM models, looking great on benchmarks, but then in practice, they feel dumb and/or expensive. Sometimes I still have no feel for whether the model understood the thing it was changing. LEGO models do not let you hide like that.

So I built another benchmark, called LegoBench, and I am releasing it today. An agent gets one of three goals. Follow the official instruction booklet. Replicate a finished set from photos. Or build a new model from photos of a real car. The harness runs in a sandbox with LeoCAD, and it has to hand back an LDraw file of the finished model, plus renders of the front, back, left and right. The prompt asks for four hours. That is a budget, not a hard stop. One published Porsche run went for 8 hours.

Under the hood it uses a custom benchmark runner I built called Metabench. Its purpose is to keep the sandbox and the judge the same for every attempt. It is simple, but built exactly for these types of benchmark experiments. I define the task as a prompt, build a sandbox image, and it runs the agent and collects the output. A separate judge grades whatever comes back, in its own sandbox. Claude Code, Codex, Grok and Antigravity each run as their own harness, which means the result is affected by the harness too, not only the model. The judge is boring on purpose. It counts the parts, checks where they sit, and marks overlaps and pieces that are not actually connected. When a real, verified LDraw model of the set is available, the score is a match of brick positions against that file. When there is no such file, three LLM judges and human ratings score how much the build looks like the car. That likeness number is separate from the build score.

To keep the benchmarks interesting and small enough, I decided to use LEGO Speed Champions cars as the first benchmarks.

The run I keep opening is the Chiron above. GPT 6 Astra, through Codex, at 89.1%. It cost $19.14 and finished in 56 minutes.

Photos are a different problem. A Golf R does not come with an official model. The judge can still say whether the model is a legal LEGO model. Whether it looks like the car is a different question.

One Golf R attempt kept 70% of the provided example that ships in the workspace. The render is just a blue Nissan LEGO car. It is not a Golf. The judge recorded the overlap and the build score went to zero. The three likeness models read that note and each gave it 3%. GPT 6 Luna did the same kind of copy on the Audi RS 3, not on this Golf. That run overlapped the Nissan example by 97.4%.

The sets in the benchmark are the Porsche 911 GT3 RS, the Nissan Skyline GT-R, the Audi Sport quattro S1 and the Bugatti Chiron, from the booklet and from photos of the finished model. The Toyota GR Supra booklet has no reference model to match against. The Audi RS3 and the Golf R are photos of real cars, so they are scored with LLMs and reviewed by humans.

The methodology is written out on the methodology page in detail.

If you want me to put another harness or LLM on these benchmarks, let me know at [email protected].