# Graff evaluations

Explore Graff coding-agent evaluations: completed terminal tasks, DeepSWE patch verification, recorded cost per pass, and comparisons with the published FrontierHarness field.

Canonical: https://codegraff.com/evaluations

Original article published: 2026-09-06

Report updated: 2026-09-12. These are publication dates, not dates of a new evaluation run.

Results are repository-reported evaluations checked against the tracked summaries and protocol at revision b4bd80c, not a fresh rerun or an independent audit. The Graff terminal runs shown here used additional evaluation instructions. Models and execution environments differ across some rows. Published Exo costs are taken from FrontierHarness revision 8f11b13; the terminal subtotal is calculated from its public task-level records.

## Interpreting the results

The 21 selected Terminal-Bench 2.1 tasks and nine DeepSWE patch tasks test different outcomes. Both Graff DeepSWE runs passed KaTeX multicolumn-array-spans; the other eight tasks did not pass. Missing verifier results count as failures.

Graff received additional terminal instructions and used a local Docker runtime. The published board used Kimi K3 with shared runtime and instructions. These configurations do not isolate a harness-only effect. Costs include failed attempts; Graff uses historical list-price estimates while the published board uses first-turn-cold pricing. Partial costs are lower bounds, not zero. Graff median attempt times are not reported.

### Find your tradeoff.

Twelve published Kimi K3 configurations plus two recorded Graff runs. Graff used extra terminal instructions and a local Docker runtime; its full-suite totals combine the two evaluated slices. Published costs use cost_first_cold_usd; Graff costs use historical list-price summaries. All recorded failures count toward spend. Partial published costs are lower bounds. Frontier badges consider only fully costed published configurations with at least one pass. Median attempt time includes successes and failures where task timings are available; Graff summary maxima are not medians.

#### All 30 tasks
| Configuration | Evaluation conditions | Passes | Recorded total USD | Cost per pass USD | Costed tasks | Median attempt time | Published complete-cost frontier |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Codex | Published Kimi K3 board; shared runtime and instructions. | 20/30 | $69.3659 | $3.47 | 30/30 | 9m 22s | Yes |
| DSH Creator | Published Kimi K3 board; shared runtime and instructions. | 19/30 | ≥ $62.4132 | ≥ $3.28 | 27/30 | 12m 29s | No |
| Claude Code | Published Kimi K3 board; shared runtime and instructions. | 19/30 | ≥ $348.3998 | ≥ $18.34 | 29/30 | 18m 29s | No |
| Pi | Published Kimi K3 board; shared runtime and instructions. | 18/30 | $43.7934 | $2.43 | 30/30 | 10m 39s | Yes |
| DSH PTC | Published Kimi K3 board; shared runtime and instructions. | 18/30 | ≥ $82.3941 | ≥ $4.58 | 27/30 | 8m 54s | No |
| DSH Standard | Published Kimi K3 board; shared runtime and instructions. | 18/30 | ≥ $62.2600 | ≥ $3.46 | 27/30 | 9m 53s | No |
| Oh My Pi | Published Kimi K3 board; shared runtime and instructions. | 17/30 | $80.6742 | $4.75 | 30/30 | 9m 56s | No |
| Kimi Code | Published Kimi K3 board; shared runtime and instructions. | 17/30 | $62.0005 | $3.65 | 30/30 | 9m 20s | No |
| DSH Minimal | Published Kimi K3 board; shared runtime and instructions. | 17/30 | ≥ $80.2052 | ≥ $4.72 | 26/30 | 14m 54s | No |
| Exo Harness | Published Kimi K3 board; shared runtime and instructions. | 16/30 | $16.7227 | $1.05 | 30/30 | 10m 04s | Yes |
| OpenCode | Published Kimi K3 board; shared runtime and instructions. | 15/30 | ≥ $48.6652 | ≥ $3.24 | 29/30 | 11m 41s | No |
| Hermes | Published Kimi K3 board; shared runtime and instructions. | 15/30 | ≥ $43.5646 | ≥ $2.90 | 28/30 | 13m 06s | No |
| Graff / Grok 4.6 | Recorded Graff evaluation; extra terminal instructions and local Docker runtime. Full-suite totals combine the terminal and DeepSWE slices. Historical evaluator list prices. Excluded from the published-board frontier. Median attempt time is not reported in the stored summaries. | 21/30 | $31.7951 | $1.51 | 30/30 | Not reported | No |
| Graff / Kimi K3 | Recorded Graff evaluation; extra terminal instructions and local Docker runtime. Full-suite totals combine the terminal and DeepSWE slices. Historical evaluator list prices. Excluded from the published-board frontier. Median attempt time is not reported in the stored summaries. | 18/30 | $25.5611 | $1.42 | 30/30 | Not reported | No |

#### 21 terminal tasks
| Configuration | Evaluation conditions | Passes | Recorded total USD | Cost per pass USD | Costed tasks | Median attempt time | Published complete-cost frontier |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Codex | Published Kimi K3 board; shared runtime and instructions. | 15/21 | $4.8584 | $0.32 | 21/21 | 6m 51s | No |
| DSH Creator | Published Kimi K3 board; shared runtime and instructions. | 15/21 | ≥ $3.3749 | ≥ $0.22 | 18/21 | 7m 33s | No |
| Claude Code | Published Kimi K3 board; shared runtime and instructions. | 16/21 | $15.5788 | $0.97 | 21/21 | 9m 38s | No |
| Pi | Published Kimi K3 board; shared runtime and instructions. | 16/21 | $5.4599 | $0.34 | 21/21 | 8m 06s | No |
| DSH PTC | Published Kimi K3 board; shared runtime and instructions. | 16/21 | ≥ $3.0198 | ≥ $0.19 | 18/21 | 7m 55s | No |
| DSH Standard | Published Kimi K3 board; shared runtime and instructions. | 16/21 | ≥ $8.1457 | ≥ $0.51 | 19/21 | 7m 04s | No |
| Oh My Pi | Published Kimi K3 board; shared runtime and instructions. | 16/21 | $5.4428 | $0.34 | 21/21 | 7m 24s | Yes |
| Kimi Code | Published Kimi K3 board; shared runtime and instructions. | 14/21 | $9.3918 | $0.67 | 21/21 | 7m 56s | No |
| DSH Minimal | Published Kimi K3 board; shared runtime and instructions. | 15/21 | ≥ $6.2995 | ≥ $0.42 | 17/21 | 6m 55s | No |
| Exo Harness | Published Kimi K3 board; shared runtime and instructions. | 16/21 | $6.4250 | $0.40 | 21/21 | 6m 51s | No |
| OpenCode | Published Kimi K3 board; shared runtime and instructions. | 15/21 | $3.4771 | $0.23 | 21/21 | 7m 43s | Yes |
| Hermes | Published Kimi K3 board; shared runtime and instructions. | 13/21 | $11.3272 | $0.87 | 21/21 | 9m 08s | No |
| Graff / Grok 4.6 | Recorded Graff evaluation; extra terminal instructions and local Docker runtime. Full-suite totals combine the terminal and DeepSWE slices. Historical evaluator list prices. Excluded from the published-board frontier. Median attempt time is not reported in the stored summaries. | 20/21 | $6.2335 | $0.31 | 21/21 | Not reported | No |
| Graff / Kimi K3 | Recorded Graff evaluation; extra terminal instructions and local Docker runtime. Full-suite totals combine the terminal and DeepSWE slices. Historical evaluator list prices. Excluded from the published-board frontier. Median attempt time is not reported in the stored summaries. | 17/21 | $3.6463 | $0.21 | 21/21 | Not reported | No |

#### 9 DeepSWE tasks
| Configuration | Evaluation conditions | Passes | Recorded total USD | Cost per pass USD | Costed tasks | Median attempt time | Published complete-cost frontier |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Codex | Published Kimi K3 board; shared runtime and instructions. | 5/9 | $64.5075 | $12.90 | 9/9 | 43m 07s | Yes |
| DSH Creator | Published Kimi K3 board; shared runtime and instructions. | 4/9 | $59.0383 | $14.76 | 9/9 | 37m 12s | No |
| Claude Code | Published Kimi K3 board; shared runtime and instructions. | 3/9 | ≥ $332.8210 | ≥ $110.94 | 8/9 | 59m 56s | No |
| Pi | Published Kimi K3 board; shared runtime and instructions. | 2/9 | $38.3334 | $19.17 | 9/9 | 39m 11s | No |
| DSH PTC | Published Kimi K3 board; shared runtime and instructions. | 2/9 | $79.3743 | $39.69 | 9/9 | 50m 21s | No |
| DSH Standard | Published Kimi K3 board; shared runtime and instructions. | 2/9 | ≥ $54.1143 | ≥ $27.06 | 8/9 | 47m 16s | No |
| Oh My Pi | Published Kimi K3 board; shared runtime and instructions. | 1/9 | $75.2314 | $75.23 | 9/9 | 49m 57s | No |
| Kimi Code | Published Kimi K3 board; shared runtime and instructions. | 3/9 | $52.6087 | $17.54 | 9/9 | 44m 29s | No |
| DSH Minimal | Published Kimi K3 board; shared runtime and instructions. | 2/9 | $73.9058 | $36.95 | 9/9 | 41m 25s | No |
| Exo Harness | Published Kimi K3 board; shared runtime and instructions. | 0/9 | $10.2977 | No passes | 9/9 | 15m 55s | No |
| OpenCode | Published Kimi K3 board; shared runtime and instructions. | 0/9 | ≥ $45.1881 | No passes | 8/9 | 49m 05s | No |
| Hermes | Published Kimi K3 board; shared runtime and instructions. | 2/9 | ≥ $32.2374 | ≥ $16.12 | 7/9 | 28m 48s | No |
| Graff / Grok 4.6 | Recorded Graff evaluation; extra terminal instructions and local Docker runtime. Full-suite totals combine the terminal and DeepSWE slices. Historical evaluator list prices. Excluded from the published-board frontier. Median attempt time is not reported in the stored summaries. | 1/9 | $25.5616 | $25.56 | 9/9 | Not reported | No |
| Graff / Kimi K3 | Recorded Graff evaluation; extra terminal instructions and local Docker runtime. Full-suite totals combine the terminal and DeepSWE slices. Historical evaluator list prices. Excluded from the published-board frontier. Median attempt time is not reported in the stored summaries. | 1/9 | $21.9148 | $21.91 | 9/9 | Not reported | No |

## Read the experiment

[Full evaluation article and methodology](https://codegraff.com/blog/graff-frontier-harness-evals)

## Sources

- [FrontierHarness results and suite definitions](https://github.com/justrach/codegraff/blob/b4bd80c0dbf4332f249b4a1a36fbeb136ed65611/graff-evals/frontier-harness/README.md)
- [Evaluation protocol and comparison limitations](https://github.com/justrach/codegraff/blob/b4bd80c0dbf4332f249b4a1a36fbeb136ed65611/graff-evals/frontier-harness/PROTOCOL.md)
- [Graff / Grok terminal result summary](https://github.com/justrach/codegraff/blob/b4bd80c0dbf4332f249b4a1a36fbeb136ed65611/graff-evals/frontier-harness/results.maxima.json)
- [Graff / Kimi K3 terminal result summary](https://github.com/justrach/codegraff/blob/b4bd80c0dbf4332f249b4a1a36fbeb136ed65611/graff-evals/frontier-harness/kimi-results.maxima.json)
- [Published FrontierHarness task costs and Exo results](https://github.com/frontier-harness-eval/eval/blob/8f11b130c30bbf76ca1f3edeea70abc773bd8d2c/results/eval-data.json)
- [Published cost accounting and comparison method](https://github.com/frontier-harness-eval/eval/blob/8f11b130c30bbf76ca1f3edeea70abc773bd8d2c/skills/frontierharness-eval/reference.md#cost-comparability)
- [Graff / Grok DeepSWE cost summary](https://github.com/justrach/codegraff/blob/b4bd80c0dbf4332f249b4a1a36fbeb136ed65611/graff-evals/frontier-harness/swe-results.maxima.json)
- [Graff / Kimi K3 DeepSWE cost summary](https://github.com/justrach/codegraff/blob/b4bd80c0dbf4332f249b4a1a36fbeb136ed65611/graff-evals/frontier-harness/swe-kimi-results.maxima.json)