Can a coding task be graded fairly?

Most coding benchmarks are built the same way. Scrape merged pull requests, keep the ones that close an issue and touch a test file, keep the ones where at least one test flips from failing to passing. The issue becomes the prompt, the repository at the pre-fix commit becomes the environment, and the PR's tests become the hidden grader.

That pipeline checks one thing: the task runs. It says nothing about whether the task is fair to grade someone on.

Two ways a task goes wrong

A task is unfair when a competent engineer could do the right thing and still fail. In practice that comes down to two defects.

1. The issue is under-specified. The text is too vague to know what to build without asking a question.
2. The tests are over-narrow. They accept only the original PR's choices, such as a parameter name, an error message or a helper the gold patch introduced, none of which the issue mentions.

These are not edge cases. When OpenAI had engineers annotate 1,699 SWE-bench tasks on exactly these two questions, 68% were filtered out. In February 2026 they stopped reporting SWE-bench Verified at all, after finding material issues in 59.4% of the hardest problems they audited.

An unscreened task is worse than a missing one. It penalises a solver for guessing a name wrong, which corrupts rankings and, in reinforcement learning, rewards the wrong behaviour.

Why a prompt is not enough

The obvious screener is one prompt: give a model the issue and the diff, ask for a verdict. That is roughly what an ensemble of LLM judges does.

The trouble is that the judgement depends on the codebase. Whether mask_invalid is "the obvious name" depends on whether the repository already uses that convention. Whether an issue is ambiguous depends on whether the surrounding code resolves the ambiguity. A model that never opens the repository is guessing.

So fairtask is an agent pipeline. A judge dispatches two read-only probes into the repository at the base commit, one per question. A deterministic verifier then checks every line the probes quote against the source, and anything that does not exist where claimed is thrown out before the verdict.

git clone https://github.com/mnkprs/fairtask && cd fairtask && npm ci

# screen any SWE-bench task by its instance id
npm run screen -- --swebench django__django-11099

What the numbers say

I measured every variant against OpenAI's public human annotations on the same 30 cases, with the same deciding model.

On the headline metric, agreement with the human usable-or-flag decision, the agent does not reliably beat the one-prompt baseline. The baseline scores 67% on two independent runs. The agent variants land between 60% and 70%, with a run-to-run spread of about two cases.

What the agent changes shows up elsewhere:

• Cited evidence is verifiable 100% of the time. For the baseline, 8 to 11% of quoted lines do not exist where it says they do.
• Recall of human-flagged tasks rises from 70% to between 75% and 85%, depending on configuration.
• Running the probes on a smaller model brings the cost to $0.55 per task with the same decisions, within noise.

Those 30 cases are a development set. I used them to diagnose failures and design each iteration, and there is no held-out evaluation. The numbers should be read with that in mind.

The part I did not expect

Five of the thirty cases are wrong for every system I built. When I went through them with the evidence in hand, they traced back to the labels, not the agent.

That is the finding I would keep if I had to drop the rest. Human annotation is the ground truth these screeners are scored against, and it is itself imperfect. A screener that returns quotes a reviewer can check is more useful than one that returns a higher score, because the disagreement becomes something you can adjudicate instead of something you average away.

The code, the evaluation set and the agent trajectories are all in the fairtask repository.