The study
39 of 100 open issues already had a pull request. A chat missed all 39.
Hand one to an agent and it redoes work someone already started. Enni flagged every one, with no false alarms.
17 October 2026 · 100 issues · Claude Sonnet 4.5 · about $4.94
What you get
- 39 of 39issues already taken by an open pull request, flagged by EnniA model given the pasted issue caught 0. Your agent skips work someone already started.
- 0false alarms from Enni on pull requests, claims and closed issuesGiven the same context by hand, the model was wrong on 23 of its 95 pull request flags and 63 of its 121 claim flags.
- 20 of 20same decision when Enni checked againAsk twice, get the same call. The model alone matched itself on 97 to 98 of 100.
Three sides, one model
Claude Sonnet 4.5 for every side. Temperature 0. Zero data retention.
(a) Enni: gathers the comments, linked pull requests and state, then checks.
(b) The model with the pasted issue: title and body.
(c) The model with everything Enni reads, written out plainly. No Enni rules or findings.
(b) and (c) answered twice. Enni re-checked every fifth issue.
100 issues, picked by rule
Public GitHub repositories with 1,000+ stars. The newest open good first issue from each, one per repository.
Nothing about a repository, issue or person was kept. Only totals.
The prompt, word for word
Same instructions for (b) and (c), version diff-study-v1. Copy it and try it.
Instructions
You are about to hand this GitHub issue to a coding agent. Is it ready to start? Answer start / start with caveats / don't start, and list what's missing or risky. Give each concern the kind that fits best, or other.Required answer
{ decision: start | start-with-caveats | dont-start, concerns: [{ kind, text }] }Concern kinds
open-pr-already, claimed-by-someone, already-closed, scope-changed-in-comments, missing-repro, missing-done-criteria, ambiguous, blocked-dependency, otherThe kinds name open pull requests, claims and closed issues. That tilts the test toward the model, on purpose.
(b) got this, for a made-up issue
Title: Saving a draft fails on Safari
Clicking Save on a draft shows a spinner that never stops. Chrome is fine.(c) got this, for the same issue
Title: Saving a draft fails on Safari
Clicking Save on a draft shows a spinner that never stops. Chrome is fine.
State: open
Labels: bug, good first issue
Assigned: 1 person(s)
Linked pull requests: example/app#88 (open)
Opened 2026-09-02T10:00:00Z, updated 2026-10-12T16:30:00Z
Comments (2):
--- outsider, 2026-10-11T09:00:00Z
Can I take this one?
--- maintainer, assigned to the issue, 2026-10-12T16:30:00Z
It's assigned already, see #88.Cut where Enni cuts: body at 12,000 characters, latest 20 comments at 2,000 each.
Scored against GitHub, not opinions
Three facts, read from GitHub with no model: an open pull request, a claim by someone not assigned, a closed issue.
Caught: true cases a side flagged. Right: flags that were true.
Softer calls, like missing repro steps, have no ground truth, so they aren't scored.
Every number
Hard facts
(b) and (c) answered twice, so they're counted over 200 answers.
| Fact | (a) Enni | (b) Model, pasted issue | (c) Model, same context |
|---|---|---|---|
| Open pull request already linkedTrue on 39 of 100. | Caught 39 of 39 · 39 of 39 right | Caught 0 of 78 · 0 of 2 right | Caught 72 of 78 · 72 of 95 right |
| Claimed by someone not assignedTrue on 35 of 100. Enni's low recall here is likely a measurement artifact: its workflow check gives one reason, and names an open pull request before a claim, so an issue with both was counted for the pull request only. | Caught 13 of 35 · 13 of 13 right | Caught 0 of 70 · no flags | Caught 58 of 70 · 58 of 121 right |
| ClosedTrue on 0 of 100. All the issues were open: any flag here is a false alarm. | Nothing to catch · no flags | Nothing to catch · no flags | Nothing to catch · 0 of 29 right |
Decisions
What each side said about starting.
| Decision | (a) Enni | (b) Model, pasted issue | (c) Model, same context |
|---|---|---|---|
| Start | 2 of 100 (2%) | 22 of 200 (11%) | 2 of 200 (1%) |
| Start with caveats | 59 of 100 (59%) | 150 of 200 (75%) | 89 of 200 (45%) |
| Don't start | 39 of 100 (39%) | 28 of 200 (14%) | 109 of 200 (55%) |
Same answer twice?
Enni re-checked every fifth issue. Temperature 0.
| Measure | (a) Enni | (b) Model, pasted issue | (c) Model, same context |
|---|---|---|---|
| Same decision | 20 of 20 | 98 of 100 | 97 of 100 |
| Same concerns | 14 of 20 | 72 of 100 | 61 of 100 |
Cost
List price, $3 in and $15 out per million tokens. An estimate.
| Measure | (a) Enni | (b) Model, pasted issue | (c) Model, same context |
|---|---|---|---|
| Model calls | 124 | 206 | 206 |
| Estimated cost$4.94 in all. | $2.17 | $1.16 | $1.61 |
What it doesn't show
Enni's rules read the fields these facts come from. (b) can't see them. (c) is the fair fight.
Given the same context, the model catches most of it too. Enni's edge: it gathers the context, its flags are right and cite a source, and it answers the same way twice.
On claims, Enni caught 13 of 35. Enni's low recall here is likely a measurement artifact: its workflow check gives one reason, and names an open pull request before a claim, so an issue with both was counted for the pull request only.
A parser miss lowers the ground truth for every side alike.
One run, 100 issues, one model. Numbers will move a little on a rerun.