Certain,
or silent.
Onlyanswersyoucanactuallytrust.
DownloadUnder a 5% promise, abstain answered 84.6% of passage questions with 4.2% wrong. The model alone was wrong 9.7% of the time.
Four signals, read from inside the model.
Ten samples grouped by meaning, a self-check, a second reader for the passage, and a probe on the hidden state. One learned score joins them.
The real certainty.
A cut is chosen on questions with known answers, then tested on 50 held-out splits. At the 5% promise, none of the 50 was statistically contradicted.
Say less. Mean every word.
A more honest model starts here.
Install it, generate once, certify, and ask. It runs on Qwen2.5-1.5B-Instruct on a 4 GB GPU, and nothing leaves your machine after the first run.
How abstain decides.
abstain doesn't ask the model how sure it is. It collects four kinds of evidence, combines them into one score, and answers only when that score is under a certified line.
Four run once. One runs live.
One greedy answer, ten samples at temperature 1, a Yes or No self-check, and the hidden state at two layers.
bench/generate_all.pyA logistic regression fitted on 5,000 separate questions joins the signals into one number. Higher means less sure.
scorers/Questions are lined up by score. The cut goes where the 95% upper bound on error stays under target.
conformal/selective.pyThe cut is chosen on 60% of questions and judged on the 40% it never saw, fifty times over.
experiment.pyEach new question gets one score. At or under the threshold it answers. Over it, it refuses.
abstain askTwo small files go live: the fitted score and a certificate holding one number. The certificate records the score file's fingerprint and refuses to load with any other.
Agreement is counted by meaning.
Two samples belong together when each one entails the other, judged by a small language-inference model with the question attached. The score is the entropy over those groups.
On 940 questions where all ten samples were identical, 28 answers were still wrong. Inside that group agreement ranks at chance, so the other signals have to do the work.
How well each signal ranks.
Measured inside the 940 questions where all samples agreed. With 28 errors in that group, the figures are approximate.
Where the cut goes.
The refused group would have been wrong 39.4% of the time, about ten times the answered group. Promising 5% with zero observed mistakes still takes at least 59 answered questions. Promising 1% takes 299.
Three questions about Hubble.
All ten samples match, and the passage supports the answer.
The passage does not say. Samples named the telescope's namesake and an astronaut who never existed.
Right, but phrased six ways, so the score lands near the line. Both earlier versions refused it.
Measured on fresh questions.
The headline numbers come from 7,029 passage questions the model had never seen when the rules of the test were written down. Other question sets are named where they appear. Figures are averaged over 50 random splits unless stated.
Pick a target. See the cost.
{{ dsNote }}
Every kind of question, certified on its own.
With no passage, abstain finds one with a plain keyword search. Questions written in the dataset's own words flatter that search, and each library searched was gathered for those questions, so expect less coverage on your own wording.
From 21% to 84.6%.
About half the early gain came from fixing the measurement: a grader that marked right answers wrong, answers cut off mid-sentence, and three times the data. The other half came from the fused score. Most of the jump to the fresh test came from a kinder draw of questions, not from changes to the system.
What breaks it.
A certificate says nothing about questions unlike the ones it was calibrated on. Refusals cost something too. At a 5% target, about six refusals in ten turn away a right answer. The best score needs the model's hidden state. With sample agreement alone, coverage drops from 84.6% to 72.0%.
It watches scores in blocks of 50. When they stop matching the calibration set, abstain refuses every question until you recalibrate. When passage questions turned into science questions with no passage, wrong answers let out fell from 813 to 16. It also trips on unchanged traffic in 5 runs of 50.
Install
Needs Python 3.10 or newer and a GPU with 4 GB of memory. The first run downloads Qwen2.5-1.5B-Instruct. After that, nothing needs the network. The code is not on a public host yet, so these commands start from a copy of the project folder.
Commands
Protocols
Each round's rules and expectations were written down before any question was scored.
Once a test set was studied to explain a weak number, it moved to practice data and a fresh set replaced it.
The cut is set on 60% and judged on 40%, redrawn 50 times. A promise may fail in 1 draw in 20.
The grader stayed frozen from version 2. One narrow rule was added later, tested on unseen answers, and shown beside the old grading.
Two readers working separately reread confident answers the grader failed. Only labels both agreed on changed.
Frozen certificates were tried on questions that had no part in setting them.
What was silently wrong
The repo lists 38 things that were silently wrong at some point. A few that changed the numbers:
Changelog
License
A license has not been chosen yet. Until one is added to the repository, all rights are reserved by the author.
Privacy
abstain runs on your machine. After the model download, questions, passages and answers stay local, and this site sets no tracking cookies.