Open, reproducible evaluation

Benchmark the code that cannot afford to guess.

A dependency-free harness for measuring how language models reason about interrupts, memory, timing, protocols, recovery, and safety at the firmware boundary.

Run the benchmark Explore all tasks

Pinned environments
No-network validation
Public rubrics

validation / run-01 evidence ready

$ npm run fixture:validate -- --task embedded-ring-buffer

TARGET PROFILE

c11-lock-free-spsc

compile / C11PASS

public behaviorPASS

controlled mutationsREJECTED

sandbox policyATTESTED

EXECUTABLE EVIDENCE 4 checks captured

RUBRIC SCOREPending

Tests provide evidence. The published task rubric determines the score.

—total tasks

—firmware tasks

—active fixtures

—validation profiles

Method, not vibes

From identical prompt to reviewable evidence.

Every compared model sees the same checked-in prompt. The harness preserves provenance; fixture-owned validators and public rubrics turn the answer into evidence a reviewer can audit.

01
Select a task

Choose exact task IDs or a whole suite from the versioned registry.
02
Run the matrix

Send identical prompts across selected models, providers, and runs.
03
Validate safely

Extract the answer and run bounded commands in a pinned, isolated fixture.
04
Grade & disclose

Score the public rubric, blind identities, then export only sanitized results.

Embedded realism

Interrupts, MMIO boundaries, RTOS semantics, stack budgets, DMA, boot flows, and recovery behavior—without requiring physical hardware.

Reproducible validation

Toolchains, dependencies, environment revisions, resource limits, and sandbox policy are machine-readable and fingerprinted.

Evaluation integrity

Prompt hashes prevent stale reuse. Raw provider output stays private; reviewed allowlisted projections are the publication boundary.

Task registry

Explore what the benchmark exercises.

Read tasks.json

0 tasks match your filters

Transparent grading

Ten points. Seven firmware dimensions. No hidden criteria.

Each task owns a ten-point rubric. Firmware tasks use the shared firmware-v1 profile, with every prompt requirement assigned to one dimension so a single defect is not counted twice.

Functional correctness

Interfaces, state transitions, successful paths, and outputs.

Bounded resource use

Memory, storage, allocation, alignment, and buffer limits.

Timing behavior

Deadlines, wraparound, latency, lock-freedom, and blocking.

Concurrency safety

Ownership, atomics, ordering, interrupts, RTOS interaction, and races.

Fault recovery

Invalid input, partial failure, retries, timeouts, and safe states.

Portability

Target assumptions, language rules, alignment, and undefined behavior.

Clarity & validation

Reviewable code, explicit assumptions, and focused tests.

Optional hardware labs

Take representative firmware onto real boards.

Hardware-in-the-loop runs add reviewable lab evidence for representative STM32, NXP, and TI targets. They are optional and supplemental: deterministic host-mock validation remains the required scoring path.

Supplemental onlyHIL evidence cannot change benchmark scores.

Network isolatedExecution runs with network access disabled.

Sanitized reportsOnly reviewed, allowlisted evidence is published.

Quick start

Run your first firmware comparison.

Node.js 22+ runs the dependency-free harness. Generating answers also requires one configured provider runtime or OpenAI-compatible endpoint; each validator declares its own language toolchain.

Clone and configure

Create a private model configuration. It is ignored by Git and must never be committed.

Choose and authenticate a provider

The example uses Codex and gpt-5.6-luna. Claude Code, OpenCode, and OpenAI-compatible endpoints are also supported; see the provider setup instructions.

Run one constrained sample

Start with one run, one model, and one or two task IDs. Run 1 writes one record per pair directly under results/.

Extract and validate

The answer is written into fixture-owned ignored paths, then compiled and tested under its declared contract.

01 / setup — Codex example

# Node.js 22+
git clone https://github.com/arjun7965/llm-firmware-benchmark.git
cd llm-firmware-benchmark
cp models.example.json models.local.json
codex login

02 / benchmark

npm run benchmark -- \
  --models gpt-5.6-luna \
  --suites firmware \
  --tasks embedded-ring-buffer \
  --tasks firmware-state-machine \
  --runs 1 --concurrency 2

03 / validate one answer

npm run fixture:extract -- \
  --result results/embedded-ring-buffer--gpt-5.6-luna.json

npm run fixture:validate -- \
  --task embedded-ring-buffer

Repeat the same matrix

Generate runs 2 and 3 with the same model and task filters. Repeated records are written under results/run-2/ and results/run-3/.

npm run benchmark:repeats -- \
  --models gpt-5.6-luna \
  --suites firmware \
  --tasks embedded-ring-buffer \
  --tasks firmware-state-machine \
  --runs 2,3 --concurrency 2

Score, then summarize

Copy the score template, then record each 0–10 total under its task ID for every completed model run.

cp repeat-scores.example.json repeat-scores.json
# Edit repeat-scores.json, then:
npm run summarize

Publish safely

Never publish raw results. Export the allowlisted projection and inspect every file marked reviewRequired.

npm run export:public -- \
  --input results \
  --output public-results

Published calibration

MPU containment cohort · September 26, 2026

All nine attempts produced answers. AI-assisted scores were reviewed and approved by the user: GPT-5.6 Luna and Kimi K3 each averaged 9.5/10; GLM-5.3 averaged 7.5/10 across three answers per model.

All nine passed the original validator; six passed the strengthened audit. Blinded review found a missed duplicate-containment defect in all three GLM answers. The new regression was applied to every unchanged answer before scores were frozen. The initial diagnostic cohort and later audit are disclosed separately; different provider budgets and this small sample do not establish a model-family ranking.

Read results and methodology

Inspect every assumption

Built for results people can challenge—and reproduce.

Open the repository Browse technical docs