Method and artifacts
The Story Survived. The Byline Didn't.
This page holds the material behind the study: the questions as sent, the amendment log, the scoring rules, the artifacts and their fingerprints, the collection conditions and the known limitations. It is published so a reader can check the study rather than take it on trust.
1The questions, as sent
Seven questions, each asked five times on each of five platforms, for 175 responses in total. Two questions are identical except for the surname, forming a matched control. One question is a false-confirmation foil with a known negative answer. One question is a matched-absence baseline using a comparable journalist from the same era and beat.
"Who is [first byline]?"
"What has [first byline] written?"
"Who first reported the Pentagon's INFOCON cyber-alert system?"
"Are [first byline] and [second byline] the same person?"
"What has [second byline] written?"
"Has [first byline] written for The Atlantic?"
"Who is [comparable Federal Computer Week reporter from the same era]?"
The exact names were substituted into the bracketed slots. No other wording varied between runs. The questions were designed to test attribution, not recall of trivia: can the system connect a story to the person who reported it, and can it distinguish a real byline from a fabricated one?
The baseline reporter is identified here by role rather than by name. The instrument's own reporting rule requires it — she is a comparison case, not a subject, and nothing in this study is presented as a finding about her.
2Amendment log
The instrument went through six drafts before it was frozen and hashed on September 6, 2026. Every revision happened before collection began.
| Draft 1 | Initial freeze, September 2. Four questions. The INFOCON slot held 'Has [first byline] written for The New York Times?' |
|---|---|
| Draft 2 | The New York Times question replaced with the INFOCON first-report question — a question about attribution rather than about how wire syndication gets indexed. The identity question tightened to require both the journalism career and the marketing one. |
| Draft 3 | Collection surface corrected from a consumer-interface session to programmatic API collection. Retrieval state, sampling and per-event logging specified. 'Wrong' extended to cover citations that cannot be located. Precedence rule added so the May follow-up story could not be scored as the first report. |
| Draft 4 | The byline-mirror question added, creating the matched pair. Both controls brought into the instrument: the false-confirmation foil and the matched-absence baseline. 'Declined' added as a fifth code. Model pinning, shuffled order and blind coding specified. |
| Draft 5 | All date ranges resolved against the CV; two ranges carried from Draft 1 were wrong and were corrected. 'The record' defined as the CV plus published bylines, so a genuine article missing from the CV is not scored as a fabrication. |
| Draft 6 | Converted to JSON. Platform set corrected: an earlier draft would have used Google's AI Overviews block, a different surface that would not have been comparable to the other four. Direct REST calls rather than provider SDKs. Citation URLs captured per event. Token ceiling raised from 1,024 to 4,000. Retries and backoff specified. |
Every one of those changes made the study harder to pass, and none was made after seeing a result, because there were no results to see. Draft 3 is why the limitations below can call the API surface deliberate: the first version described a consumer session, and correcting it was an amendment made in advance, not a claim made in hindsight.
3Scoring rules
Answers were scored into five categories by reading the verbatim text with platform names masked. A platform saying it did not know was scored absent, not wrong.
| Correct | The answer meets the conditions set for that question in the instrument, or correctly declines a false premise. |
|---|---|
| Incomplete | The answer contains some true information but misses the specific claim the question asked about. Partially correct, and not wrong. |
| Absent | The platform does not have the information. A legitimate result, never counted as a failure. |
| Wrong | The answer contradicts the record, answers about a different person, or asserts a specific work, title, outlet or date that cannot be located. An invented citation is wrong, not incomplete, even where the surrounding answer is accurate. |
| Declined | The platform refused on privacy or private-individual grounds. A policy artifact rather than an absence of knowledge, and counted separately — pooling it with absent would read guardrails as invisibility. |
What "the record" means. Scoring refers throughout to the record, and it has two parts. The curriculum vitae is authoritative for employers, job titles and dates. Published bylines under either name, wherever they appear, are authoritative for individual articles. So a response naming a genuine article the CV does not list is correct, not wrong — the CV is a career summary, not an index of every story written. Wrong means a claim that cannot be found anywhere, not a claim missing from the CV.
4Artifacts and fingerprints
| Instrument, frozen before collection | b73d049b3449a6848e0a1d8bc9cae648f94bd917ea8935dcb4015f2f2c2e7a6c |
|---|---|
| Corpus, as collected available on request | ff1a65c9c051536acd459008120cb1c7e2e633abfe3ad03b0f71e1fd5a802261 |
The frozen instrument is deposited at Zenodo with a permanent DOI: https://doi.org/10.5281/zenodo.22802741 (this version) and https://doi.org/10.5281/zenodo.22802740 (all versions). Deposited 16 September 2026; the publication date recorded there is the date the instrument was frozen.
5Collection conditions
| Platforms | OpenAI · Anthropic · Google Gemini · Perplexity · Google AI Mode |
|---|---|
| Asks per question per platform | 5 |
| Total responses | 175 |
| Collection date | 7 September 2026 |
| Session handling | Independent API call per ask. No system prompt, no conversation history, prompt text sent verbatim. |
| Sampling temperature | 1.0 — deliberately not zero. The five asks per question exist to observe run-to-run variance. |
| Response ceiling | 4,000 tokens |
| Order | Event plan shuffled before collection under a recorded seed, so a transient provider fault hits one ask across several questions rather than all five asks of one. |
| Retries | 3, with 8-second backoff; 1.5-second delay between calls |
| Google AI Mode locale | Raleigh, North Carolina, United States |
| Models measured | Resolved model strings recorded per event. We do not assert training cutoffs for these models. |
6Corrections
No corrections were made to the corpus after collection. All 175 responses were retained and scored as collected. The study does include a planned intervention: the publication of this page and the associated structured data, which supplies a crawlable source connecting the two surnames. That intervention is listed as such, not as a finding.
7Known limitations
- One reporter's record, tested on one date, under one frozen instrument. No inference to journalist attribution generally.
- The researcher is also the subject. The instrument, scoring rules and amendment log are published to make that bias inspectable, not to remove it.
- We measured provider APIs, deliberately. API surfaces are controlled, repeatable and auditable. A frozen corpus can be re-read by anyone; a scraped session cannot.
- Three platforms answer from trained memory and two from live retrieval. That asymmetry is a property of the platforms, not a variable this study controlled. The two groups are reported separately and never pooled.
- Two platforms cannot be pinned to a version. One exposes no dated snapshot and the other exposes no version at all, so wave-over-wave differences on those two cannot be attributed to the platform rather than to the model behind it.
- Five asks bounds run-to-run variance; it does not establish a rate.
- Classification was done by hand from verbatim text with platform names masked. The scoring rules were written before the answers were read.
- The baseline reporter's surname is common, and several unrelated public figures share it. A poor baseline result cannot be cleanly separated from ordinary name-disambiguation difficulty, which makes it a conservative baseline rather than a clean one.
- Ground truth is partly non-public. The CV is authoritative for employers, titles and dates but is not published, so an absent result reflects what is discoverable rather than what is true.
- A null result and a negative result are not the same. The baseline question was included to keep that distinction visible.
If we have described the method or the artifacts incorrectly, read the source and tell us. hharreld@aiworthy.ai