Snowflake’s data-eng-bench · Run
Dex passes 57.3% of data-eng-bench, and 94.7% of what the verifiers check.
The highest published figure for any Claude Sonnet 5 configuration on Snowflake's 103-task suite, measured at k=1: one attempt per task, all 103 tasks attempted, zero errored trials, and no best-of selection.
- k=1, one attempt per task
- 103 of 103 tasks
- 0 errored trials
- 0 retries
- Claude Sonnet 5 at xhigh
- Dex on 98% of trials
59 / 103 tasks resolved · 95% CI 47.6 to 66.4
2,699 of 2,850 · 90.6% task-weighted
Pass rate, k=1
57.3%59 of 103 tasks
95% CI 47.6 to 66.4
Verifier assertions
90.6%task-weighted mean
95% CI 86.4 to 94.3
Dex trigger rate
98%101 of 103 trials
Cost per trial
$2.74list pricing, $1.82 introductory
Errored trials
0and zero retries
Against the published board
Snowflake define Pass@1 as the mean single-attempt pass rate over a k=3 run, which is the same quantity a k=1 pass rate estimates, measured on three times the sample. Pass^3 counts a task only when all three attempts succeed, and a k=1 run cannot produce one.
| Harness | Model | Attempts | Pass@1 | Pass^3 | Tokens / trial |
|---|---|---|---|---|---|
| Snowflake CoCo | Opus 5 | k=3 | 73.8% | 64.1% | 1.07 M |
| Claude Code | Opus 5 | k=3 | 69.6% | 60.2% | 4.81 M |
| Snowflake CoCo | GPT 5.6 Sol | k=3 | 64.1% | 55.3% | 0.44 M |
| Codex | GPT 5.6 Sol | k=3 | 60.5% | 49.5% | 0.81 M |
| Claude Code + dex | Claude Sonnet 5 | k=1 | 57.3% | — | 5.39 M |
| Snowflake CoCo | Claude Sonnet 5 | k=3 | 56.6% | 40.8% | 2.88 M |
| Claude Code | Claude Sonnet 5 | k=3 | 56.6% | 40.8% | 7.91 M |
This run is first among every published Sonnet 5 configuration, and fifth of seven overall. The margin over the two Snowflake published at 56.6% is 0.7 points, a single task on a 103-task suite, so the honest reading is that all three are statistically indistinguishable and this one is nominally at the top. A genuinely 56.6% agent produces a draw at least this good about half the time. Snowflake publish identical figures for both of their Sonnet 5 configurations, so treat those two rows as one reference point rather than two.
Snowflake engineering blog, 6 August 2026. Their rows are k=3; ours is k=1.
Where Dex shines
What the run shows about the tool, separated from what it shows about the model. Every figure here carries its bound in the same place as the number.
Dex fired on 98% of trials
This is a measurement of Dex rather than of a tool that happened to be installed. The CLI carried 90% of the trials and the Skill wrapper 18%, for roughly 1,200 invocations in total. Every one of them was dex explore: the agent used it to verify real columns, types, null rates, grain and join keys before writing SQL.
NOTE — Usage is counted from the agent transcript, not from the installed skill files. A naive scan of the agent directory reports 100% on any run where the skills were merely present.
The more Dex ran, the more tasks passed
Trials that made twenty or more dex calls passed at 71%, against 50% for trials that made none at all. The gradient is monotonic from five calls upward, and the heaviest-usage bucket is 67% hard or harder yet still passed above the run average.
NOTE — Observational, and not a causal claim. There is no control arm in this run, the strata are small, and the arrow could point either way: an agent that has already lost the thread may also stop reaching for tools, which would make low usage a symptom rather than a cause.
On hard tasks the gap is widest
Stratified by difficulty, so the gradient is not simply easier tasks attracting more tool use. Among hard tasks, trials that leaned on Dex passed at 70% against 20% for trials that barely touched it. Among medium tasks it is 63% against 44%.
NOTE — Same caution as above. The thin cell here is three passes out of fifteen, so the 20% figure is a very small denominator.
Nearly right almost everywhere
2,699 of 2,850 individual verifier assertions passed. Even across the 44 tasks that failed, 87.6% of assertions still passed, 19 of them were blocked by exactly one assertion, and 29 by two or fewer. marketing-campaigns-harbor passed 218 of 219: the single failure compared a date against a timestamp.
The failure surface is semantic, not structural
In 38 of the 44 failed tasks the agent produced a buildable, correctly shaped model and then lost on a boundary condition or a reading of an ambiguous spec. Only six never produced a working deliverable at all. Getting the shape right is the part that depends on knowing what is actually in the warehouse.
32% fewer tokens for the same score
Against the identical harness and model on the published board, this run used 5.39 M tokens per trial against 7.91 M, while running at reasoning_effort: xhigh, a setting that pushes token use up rather than down. 97.8% of the prompt volume was cache reads.
NOTE — Snowflake's published cost and token columns do not reconcile with each other, so at least one of them is on a basis we cannot see. We state our own basis and make no cost comparison.
Each bar is a bucket of the 103 trials, split by how many dex subcommand invocations appear in the agent transcript.
n = 10
n = 18
n = 27
n = 27
n = 21
NOTE — Bucket sizes are small and the 1-to-4 bucket sits below the zero-call bucket, so read the overall shape rather than any single pair.
Two scales, one run
The reward is binary and total: a task scores only when pytest exits with zero failures and zero skips. Because the verifiers are written as many independent components, the run also produced a much finer record underneath the binary one, and reading it changes what the 57.3% means.
59 of 103 tasks. What a leaderboard records.
2,699 of 2,850 individual verifier assertions.
Mean of per-task assertion fractions, each task counted once.
Across the 44 failed tasks alone, 87.6% of assertions still passed. Nineteen of the 44 were blocked by exactly one assertion and 29 by two or fewer. The binary reward is doing what it is designed to do, but it compresses “built the wrong thing” and “got one boundary condition wrong on a 219-assertion suite” into the same zero.
NOTE — Two cautions on reading this. These are not free points: a near-miss only converts if its cause is stochastic, and most of these look systematic, so the assertion rate describes where the remaining work is rather than a pass rate in waiting. And prefer the task-weighted 90.6% to the pooled 94.7% when quoting a single figure, because pooling overweights the one task that carries 219 assertions.
Where the other 44 lost
Every failed trial was classified from its verifier output. The distribution is lopsided in a useful direction: the agent almost always produced a buildable, correctly shaped model and then lost on semantics.
| Bucket | Failure mode | Trials | Near misses |
|---|---|---|---|
| E | Classification threshold or tie-breakScores correct, the bucket they land in is not. Percentile edges, tier waterfalls, ties. | 12 | 9 of 12 |
| D | Population boundary (row count)Right shape, wrong row count. Off by an edge case, not by a formula. | 8 | 8 of 8 |
| A | Deliverable missing or unbuildableThe model, or a model it depends on, was never produced. Nothing about the SQL was judged. | 6 | 0 of 6 |
| B | Order-status population semanticsCancelled, returned, or failed orders included or excluded against what the verifier expects. | 6 | 2 of 6 |
| C | Grain and join fan-outA join multiplied rows, so totals inflate by a roughly integer factor. | 4 | 4 of 4 |
| F | NULL handlingA NULL survived into a column the verifier requires to be populated. | 4 | 3 of 4 |
| G | Type, precision, or formatNumerically right, typed or truncated wrong. | 2 | 2 of 2 |
| H | Stateful algorithmFIFO and LIFO layer allocation. Sequential state the SQL has to carry. | 2 | 1 of 2 |
Buckets D and E together are 20 of the 44, and both are boundary problems rather than modeling problems. Bucket B deserves its own attention: six separate tasks turned on whether cancelled, returned, or failed orders belong in the population. The tickets frequently do not say, and the verifiers always have an opinion. Bucket A is the only group where nothing about the SQL was judged, and it is six tasks.
How we ran it
data-eng-bench drops an agent into a 2,356-model dbt project with a ticket, then scores whether the task's own pytest verifier runs clean afterwards. This was one full pass over all 103 tasks, one attempt each, with no re-runs and no best-of selection.
- Run
- full-dex-20260810-221947. Started 2026-08-10 22:19:47 UTC, main pass finished 2026-08-11 09:20:36 UTC.
- Tasks
- All 103, one attempt each (k=1). 84 build tasks and 19 fix tasks; 3 easy, 47 medium, 45 hard, 8 very hard.
- Model
- anthropic/claude-sonnet-5 at reasoning_effort: xhigh
- Agent
- claude-code 2.1.226, median 49 steps per trial
- Dex
- v1.6.3 at commit 1554d248, exmergo-dex-core 1.6.3, warm cache in the image
- Image
- ghcr.io/snowflake-labs/data-eng-bench-base:1.0.0, dex arm
- Dataset
- snowflake-labs/data-eng-bench@v1.0, task digests unmodified
- Throughput
- 11.01 h wall clock at n=2 concurrency, 9.4 trials per hour, 22 core-hours total
- Headroom
- Longest agent execution 2,026 s against a 4,000 s ceiling. Nothing was truncated.
- Reliability
- Zero errored trials and zero retries. One trial hit a host-side network failure during agent setup, 38 s in and before any tokens were spent, and was re-run to completion on a provably identical configuration.
- This is a k=1 run: one attempt per task. The 57.3% carries a Wilson 95% interval of 47.6% to 66.4%, roughly plus or minus 9.4 points, and a single task is 0.97 points. Any comparison narrower than about ten points is noise at this sample size.
- The assertion-level interval is about two and a half times tighter than the binary one on the same run, 7.9 points against 19.4. Where a single figure is needed, the task-weighted 90.6% is the more sensitive instrument.
- The benchmark's own difficulty labels barely separate the outcomes. Medium tasks passed at 60% and hard at 53%, a seven-point spread inside a plus or minus nine-point interval. The failure taxonomy above predicts this model's misses far better than the label does.
- Build and fix tasks scored 58.3% (49 of 84) and 52.6% (10 of 19). Snowflake characterise the suite as 84 build and 19 fix but do not publish the split, so this is a first. At n=19 the fix-task interval runs from 31.7% to 72.7% and carries almost no information.
- Cost per trial was essentially flat between passes ($2.64) and failures ($2.87), so the failures are not agents burning budget while flailing.
- Every trial's assertion tally, cost, step count and dex call count is in the table at the foot of this page, and the raw verifier output is committed alongside the harness configuration.
Reading this run
The arithmetic here is easy for anyone to check, so everything that qualifies the number sits on this page rather than in a footnote.
- This is a k=1 run: one attempt per task, below the leaderboard's k=3 minimum. It sizes a submission; it is not one.
- There is no baseline arm in this run, so no delta is attributable to Dex. Nothing here measures what the same loop would have scored without it, and the usage gradient above is observational.
- Snowflake's published rows are k=3 and ours is k=1. Both estimate the same quantity, but ours is the noisier estimate on roughly a third of the sample.
- We publish no Pass^3 figure. A k=1 run cannot produce one, and no reanalysis will conjure it.
- Our arm modifies the base image. It adds uv, a warm dex-core cache, the dex CLI on PATH, and an /app/CLAUDE.md, reproduced verbatim below.
- The model ran at reasoning_effort: xhigh. Snowflake do not state their setting.
- Different day, different hardware, different operator, and no shared control arm. This is a cross-report comparison, not a controlled experiment.
- Task digests match snowflake-labs/data-eng-bench@v1.0. Nothing under tasks/ was modified.
- Cost is reported at list pricing ($3 and $15 per MTok, $3.75 cache write, $0.30 cache read), which reconstructs the harness figure of $282.12 exactly. The same token ledger at the introductory pricing running through 2026-08-31 comes to $187.95.
- Snowflake's published cost and token columns do not reconcile with each other, so we make no cost comparison against the board.
94 of the 103 tickets already instruct the agent to run echo $DB_TYPE, and 100 of 103 already instruct it to create a profiles.yml with profile name retail_dw_master. Steps 1, 2 and 5 therefore restate what the tickets already say, and the marginal prompt content is step 3.
# Project setup sequence
Every task in this project follows the same order. Do not skip a step.
1. `echo $DB_TYPE` to determine the backend.
2. Write `profiles.yml` for that backend, profile name `retail_dw_master`.
3. **`dex explore profile <table>` for every source table the ticket names.**
This returns real columns, types, null rates, grain and verified join keys.
4. Write or edit the model SQL.
5. `dbt run -s <model>` to verify it builds.
Profiling tells you what the data *is*. It does not tell you what the ticket
wants. Where the two seem to disagree, the ticket and the existing model are the
specification: implement the change that was asked for, and leave the surrounding
logic alone.
## dex commands
dex explore profile <table> columns, types, null rates, grain, PII
dex explore inventory --rank what exists, ranked by relevance
dex explore relationships --verify join keys, verified against the data
dex explore query "<SELECT ...>" guarded ad-hoc probe
The connector and database path default from `$DB_TYPE` and `$DUCKDB_PATH`, so
no flags are needed. Output is JSON on stdout. It is read-only and writes only a
`.dex/` cache, which dbt ignores.All 103 trials
Every trial in the run, with its assertion tally, cost, step count and dex call count. Failures also carry their taxonomy bucket, how many assertions stood between the trial and a pass, and the first assertion that failed.
103 of 103 trials shown
Letters mark the failure taxonomy above. “Short by” is the number of assertions between the trial and a clean pass; n/a where a verifier emitted no parseable summary or no tests ran. Raw verifier output ↗