Well documented,
rarely picked

Scope

What was read

Every block is one venue; its area is the number of full texts coded there. Both coders worked blind on the clinical / life-science label and on an equal random sample of other papers from the same venue-year.

Step 1 — on paper

Every documentation axis favours the medical papersprediction reversed

Rate differences, life-science minus other, Mantel–Haenszel over the nine venue-years, with 95% confidence intervals. The protocol pre-registered the opposite direction for all three labelled fields (L9.1–L9.3: life-science papers report open data, release weights and report variance less often). None of those labels was assigned, because each difference came out positive.

The same four axes as levels rather than differences: the share of coded papers on each side that met the field, and the count behind it, from the final codes the analysis uses (adjudicated for variance). The difference printed on the right of each axis is the crude one: it is close to, but not the same as, the venue-year–stratified estimate above.

Read it as: nothing here supports the premise that medical papers withhold their materials. Variance did not meet the agreement gate, so coder 2 coded it for every paper and all 1,404 disagreements on variance and compute were adjudicated before these numbers were computed.

Step 2 — in practice

The same papers are settled less oftengap −19 pts

Attempted papers in the ICML 2026 challenge, by whether any team reached a verified or falsified verdict. Permutation p = 0.0005.

The same question on the matched set, one square per attempted paper: the 656 coded ICML 2026 papers only, so both groups come from one venue and one leaderboard snapshot. Settled squares are grouped first, so the comparison is the length of the two blocks. Gap -17.0 points, 95% CI [-29.7, -4.4] — the bars above compare the same 110 medical papers against every other attempted paper in the challenge, which is why the two gaps differ.

Read it as: being attempted is not the problem and the judge is not the problem — within the same team, medical and other claims are decided about equally often. Something about which teams and which claims is.

Step 3 — where the gap goes

Two factors absorb itCI reaches zero

The full-verification gap, then the same gap adjusted for the first team's settle rate on its other papers, then for that and the claims' check cost. Post-hoc; each reading rule frozen before computing.

Read it as: once you know who picked the paper up and what its claims cost to check, the medical penalty is no longer distinguishable from zero.

Step 4 — the two factors

Weaker first teams, costlier claims

First: how often the team that took a paper first settled its other papers. Then: how the papers' claims distribute over check cost, from two blind coders on 1,083 claims (kappa 0.72).

Read it as: medical papers went first to teams that settled 40% of what else they touched, against 60% elsewhere, and 56.7% of their claims needed retraining or a large-scale run, against 39.1% (+17.6 points, 95% CI [+9.7, +25.8]).

Step 5 — the caveat on any artifact map

What a paper states barely predicts what a reproducer getskappa 0.13–0.55

Agreement between the coded artifact statement and what the reproduction logbook reports having actually used, on adjudicated pairs.

Read it as: a declared code link is close to uninformative about whether anyone ran that code. Any figure that paints a stated artifact signal as reproduction evidence is painting the wrong variable — which is why the keyword-selected layer this page used to show was retired.

Provenance

Every number on this page, and the file it comes from

Paths are relative to verdict_audit/readiness/results/, except map_label_check.json (audit/). Regenerate with python tools/build_medical_map.py.