The best AI hiring tool tells you the least about who to hire — and the most about what it actually knows

What "let the model do everything" actually costs

A disqualified candidate, ranked number one

On 22 May 2026, Job Hunter did a real production run for a Senior Full-Stack Engineer role at a NSW museum. The role had hard eligibility gates: ten-plus years of post-graduation tenure, currently living in NSW or Sydney, most recent role in Australia, full work rights. In the original design, every one of those gates was written into the prompt as an instruction, and the same single LLM pass did everything — gating, parsing, date arithmetic, redaction, stack classification, and final judgement — in one shot.

It ranked a candidate first who failed three of those gates at once.

The candidate should have been filtered out before any model saw a single token of his résumé. Instead he came back at the top of the list, with a fluent rationale explaining why he was the strongest applicant in the pool.

The instinct at that moment is to blame the model. So I re-ran the identical batch the next day on a different provider — DeepSeek instead of the original. It ranked him nineteenth instead of first. Better! But look closer: DeepSeek also didn't treat the gate failures as disqualifying. It just quietly logged them as "gaps" and moved on. Different model, same architectural bug, smaller blast radius. The problem was never which model I picked. The problem was that I had asked a model to enforce rules at all.

The post-mortem put its finger on exactly why, and this is the sentence the whole rebuild hangs from:

The hard gates are written as prompt instructions, not as deterministic predicates. An instruction-tuned model trying to be helpful weighs the self-summary against the structured evidence and resolves the conflict in the candidate's favour. That is the wrong disposition for a screening tool. — post-mortem, 2026-05-22

That "wrong disposition" showed up in three distinct, repeatable ways across the run — worth naming precisely, because each one is a trap any AI hiring tool can fall into:

1. Self-claim beats evidence. The résumé prose said "6+ years"; the employment dates said five. Asked to reconcile them, the model believed the prose.
2. "Willing to relocate" read as "already there." A candidate who wrote "Willing to relocate to Sydney" was treated by the model as Sydney-based — an aspiration silently promoted to a fact.
3. Hard rules and soft judgement, blended in one prompt. A "HARD GATES" section sat in the same prompt as "be fair, consider every candidate." The model treated the gates as one more weight to balance, not a binary knockout.

The same run had company. One candidate's most recent role was in China and wasn't filtered. Another appeared twice — an ingestion dedup bug, not a model failure, but proof the pipeline itself was leaky. Another was ranked above Australia-resident candidates despite failing the "two of the last three roles in Australia" check. And in a quieter, more disturbing register: the one-shot design leaked protected characteristics — nationality, visa status, age proxies — into fifteen candidates' rationales, and at one point scored a fourteen-and-a-half-year career as 2.33 years, because it was asked to estimate tenure from prose instead of computing it from dates. Both failures are now documented in the code's own docstrings as the specific bugs the rebuild exists to kill.

The half of the story nobody tells

Then the fix threw out two-thirds of the good candidates

The fix, decided the same day, was clean and obvious: take the hard gates out of the prompt entirely and make them deterministic Python predicates that run before the LLM ever sees a résumé. Non-negotiable. If a candidate fails a gate, no model gets a turn. This is the right move, and I'd make it again.

It also over-corrected — badly.

Around 10 June, I ran a "Human-versus-AI" validation: take the gates the new system was enforcing and check them against a real recruiter's real decisions on a real role. The strict, code-enforced gates were filtering out 29 of the 45 candidates that recruiter had actually shortlisted or interviewed. A rule strict enough to catch a hallucinated number-one candidate was also strict enough to silently discard two-thirds of the people a human genuinely wanted to talk to.

This is the twist that keeps the story honest, and it's the more important half. The first failure — a disqualified candidate ranked first — is loud. It embarrasses you immediately. The second failure is silent. It looks like the system working. And it is, I'd argue, the failure mode that most AI hiring tools are quietly shipping right now, because it never generates a complaint.

The resolution wasn't to loosen everything. It was to separate what is actually binary from what is merely a strong signal. Of the original seven gates, only two survived as hard, code-enforced knockouts:

Tenure and location — the gates that had caused both failures — were demoted from filters to surfaced signals. They're still computed. They're still shown to the recruiter. They still feed into ranking. But they can no longer silently remove a human being from consideration. The machine gets to say "this person has five years, not the ten you asked for." It no longer gets to decide that five years ends the conversation.

One failure was hallucination because the system was too permissive. The next was over-filtering because it was too strict. The real problem lives in the space between them — and no bigger model closes that gap for you.

Subtraction, not a smarter prompt

Nine stages, and only two of them touch a model

The rebuilt system is a nine-stage pipeline. The governing rule is stated plainly in the build brief, and it's the entire philosophy in two sentences:

Deterministic work runs in code. The LLM is used for exactly two things: faithful extraction and capability judgement. Nothing else. Gates, date math, tenure, dedup, stack labelling, redaction, capping, and normalisation are all code. — staged_pipeline_build_brief.md

The fix, in other words, was not a bigger model or a cleverer prompt. It was removing judgement from every place that didn't need it. The model reads a résumé faithfully (Stage 2), and it judges capability against evidence it's handed (Stage 6). Everything else — every place the first version got into trouble — is now ordinary, testable, deterministic code.

Two structural rules are worth pulling out. First, every stage writes its output as a persisted, structured artefact that the next stage reads — no stage reaches back to raw input it wasn't handed. The whole pipeline is replayable stage by stage, and a reviewer can open any intermediate artefact and see exactly what that stage decided and why. Second, the scorer is capped at five out of ten on any dimension it can't back with a verbatim quote from the résumé. The model cannot score confidently on vibes. If it can't cite it, it can't sell it.

None of this needs heavy infrastructure. Pipeline runs are backgrounded — a Mongo doc plus asyncio.create_task — so a run returns a job ID immediately, gets polled for progress, and can be cancelled mid-flight. Fan-out across candidates is concurrency-limited with a semaphore of five. The whole thing runs comfortably on a free tier. The frontend is Next.js and Tailwind on Vercel; the backend is FastAPI and MongoDB on Render; the LLM layer is provider-agnostic — OpenAI, Anthropic, Gemini, DeepSeek, or Bedrock, swappable per step by an admin rather than hardcoded to one vendor.

Why "redact before judge" beats "please don't discriminate"

The scorer cannot cite what it never sees

Recall the quiet failure from the first run: protected characteristics leaking into fifteen candidates' rationales. The obvious fix is to add a line to the prompt — "never mention nationality, visa status, age, marital status, gender, or years in country." Job Hunter does say exactly that. But that instruction is backup, not the mechanism. It's the same category of thing that failed the first time: a rule a helpful model can route around.

The actual guarantee is structural. Stage 5 strips protected attributes out of the résumé text before the judgement model in Stage 6 receives anything at all. The redaction sits, physically, between the résumé and the judge.

This is what stops the leakage at the source — the scorer cannot cite what it never sees. The prompt's prohibition remains as defence-in-depth. — redact.py

The categories handled are specific: national origin and country of birth, visa / residency / citizenship status, date of birth and age, "N years old", marital status, gender, photo references, "years in country", and header labels like Nationality: or DOB:. And crucially, every redaction is logged with its category and the original span it replaced — so it's auditable, not silent. You can prove, after the fact, exactly what the model was and wasn't allowed to see. That distinction — a rule made true in the data versus a rule asked for in a prompt — is the same lesson as the very first failure, just pointed at fairness instead of eligibility. Don't ask a model to enforce a rule it can be "helpful" around. Make the rule true before the model gets a turn.

Real numbers, honestly reported

Does the score beat "most years wins"?

It's easy to claim a hiring score is good. It's harder to measure it against a real recruiter's real decisions — and harder still to report the result when it's modest. Job Hunter's evaluation harness does both, and it's PII-safe by construction: every gold-set entry is an opaque ATS item ID plus a 0/1 "advanced" label and optional years of experience. No names, no résumé text, ever.

The harness asks the two questions any AI hiring tool should be able to answer honestly:

Is the score any good? Precision and recall at k, and average precision, measured against the recruiter's actual "advanced" set. And does it add anything beyond what a recruiter already gets for free? It ranks the same pool by years of experience alone — the "most-years-wins" baseline everyone falls back to — and reports the delta. A score that merely re-derives tenure has a rank correlation near 1 and a delta near zero: worthless. A score that adds real signal diverges from tenure and beats the baseline.

Thirty-nine percent. That is modest, and it is reported as modest — no rounding up, no vibes. The point is not that the number is impressive. The point is that it's real, it beats the free baseline by a measurable margin, and the harness is explicit about its own limits: it measures, it does not rewrite the rubric. In the harness's own words, "the rubric is recruiter-owned." Which is the throughline to the next question — who, exactly, stays in control.

Calibration, not automation

The recruiter owns the rubric — even in the code's own comments

The most important design decisions in Job Hunter are the ones that keep a human holding the wheel. An org can set per-dimension weight multipliers, zero to two times, that re-rank already-scored candidates without re-running the model. The system documents this explicitly as "NOT a rubric rewrite": the six scored dimensions — job requirements, production support, seniority fit, technical depth, cultural and soft skills, career trajectory — and the evidence behind them stay the model's job. How heavily an org weighs each one is a human dial.

Transparency runs both directions. A self-scoped endpoint lets a candidate see their own anonymised score breakdown — raw score and per-dimension detail only, no rank, no other candidates, no recruiter-internal narrative. The system deliberately strips itself down before showing a candidate their own evaluation. Evidence-based scoring cuts both ways: if you're going to hold a number over someone, they should be able to see the receipts behind it.

And no org is locked into a single vendor's judgement. The same rubric can run against OpenAI, Anthropic, Google, DeepSeek, or Bedrock, admin-configurable per pipeline step — because "which model judged me" shouldn't be an accident of what was hardcoded a year ago.

The honest part

What isn't finished

This isn't a finished product, and pretending otherwise would undercut the whole argument. A recent security review flagged that while the new candidate-self-view endpoint got a proper identity-ownership guard, several other pre-existing candidate-scoped routes still don't — a real, IDOR-shaped gap being closed route by route. It's worth saying plainly: a system that makes consequential decisions about people needs the same security discipline as a system that moves their money, and that discipline is a continuous practice, not a one-time checklist. Separately, org-level LLM budget quotas — a monthly cap and a 429 guard — are planned but not yet shipped, a recognition that giving orgs a multi-provider dial also means giving them a spending dial.

An earned point of view, not a prediction

What two failures taught me about AI in recruiting

I don't have a forecast about AI transforming hiring. I have a thesis that fell out of two specific failures, and I think it generalises well past résumé screening.

The industry's default instinct — "let the model do everything, it's smart enough" — is precisely the instinct that ranked a disqualified candidate first. Language models are unreliable arbiters of facts that are actually binary — is this person legally eligible, does this date arithmetic check out — and genuinely valuable at judgement once those facts are pinned down and handed to them as evidence. The discipline the whole field needs is knowing which is which, on every single feature, and refusing to let a model freelance the wrong one — even when adding another line to the prompt would be so easy.

The opposite instinct — "hard-code the eligibility rules and you're safe" — is exactly as wrong, just quieter about it. Twenty-nine of forty-five real shortlisted candidates, silently discarded. Over-automation doesn't arrive with a hallucinated headline; it just removes good people from consideration and generates no complaint. A hiring system's rule layer needs the same audit discipline as its judgement layer, because the rules are just as capable of being wrong — they simply fail politely.

Bias-safety has to be an engineering property of the data, not a promise made in a prompt. "The scorer cannot cite what it never sees" is a guarantee. "Please don't discriminate" is a hope. As AI hiring tools meet growing regulatory scrutiny — more jurisdictions every year requiring some form of AI-in-hiring disclosure — the tools that survive it will be the ones that can show, structurally, what the model was and wasn't allowed to see, rather than pointing at a system prompt.

Evidence-bound scoring is what makes a decision defensible to the person it's about. A dimension score with no verbatim quote gets capped, not trusted. Small mechanical rule, large consequence: every number the system produces comes with a receipt. That's exactly what candidate self-transparency requires under the hood — you can't show someone their own evidence trail if the system never had to keep one.

And the recruiter keeps the dial, not the model. The realistic future of AI in recruiting isn't "the AI decides who gets hired." It's AI doing the exhausting first-pass reading of 138 résumés at a consistency and speed no human sustains, producing an auditable, evidence-cited shortlist, and handing the actual judgement call back to a person who can see exactly why the machine ranked who it ranked — and override it when the machine is wrong. Which, per this project's own honestly-reported 39% precision, it still is, disturbingly often, no matter how good the underlying model gets. Measuring against real outcomes rather than trusting a confident-sounding summary is the only way to know how often.

The system doesn't get more trustworthy by sounding more confident — that's exactly the failure that ranked a disqualified candidate first. It gets more trustworthy by being smaller, more auditable, and more honest about the edges of what it can verify.

Which is why I've come to think the best AI hiring tool is the one that tells you the least about who to hire, and the most about what it actually knows. The confidence was never the product. The receipts are.