Model eval: Medqa Accuracy is the shared direct-receipt signal

Selected angle: source

One-sentence thesis

Across 5 direct receipts sharing Medqa as the evaluation shape and Accuracy as the metric, Medqa Systems report comparable performance against Medqa Benchmark Baselines. Reported values include 67.6%, 67.6%, 90.0%, 72.6%, 60.3%.

Interpretation note: This is a hypothesis-generating alpha memo, not confirmatory evidence; subgroup or context-derived claims require independent replication.

Why this is surprising

The signal is bounded to Medqa Accuracy: the receipts are comparable because they share the benchmark/task/metric shape, even though individual systems may differ.

Evidence Landscape

Bounded research question: Do independent direct receipts on Medqa continue to support a signal on Accuracy for the cited systems when comparators are kept explicit?

Evidence receipts

• fact_id=llm_evaluation/auto/2022/medqa_207573 (A_core) — Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA, MedMCQA, PubMedQA, MMLU clinical topics), including 67.6% accuracy on MedQA (US Medical License Ex doi=10.48550/arxiv.2212.13138
• fact_id=llm_evaluation/auto/2023/medqa_325097 (A_core) — Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA 3 , MedMCQA 4 , PubMedQA 5 and Measuring Massive Multitask Language Understanding (MMLU) clinical t doi=10.1038/s41586-023-06291-2
• fact_id=llm_evaluation/auto/2024/accuracy_326755 (A_core) — Under specific prompts, GPT-4 has achieved over 90% accuracy on the MedQA dataset, surpassing ordinary medical practitioners. doi=10.1145/3718391.3718410
• fact_id=llm_evaluation/auto/2024/mmlu_207616 (A_core) — The model achieved 72.6% accuracy on MedQA, outperforming the previous SOTA by 2.4%, and 81.7% accuracy on MMLU medical-subset, establishing itself as the first OS LLM to surpass 80% accuracy on this benchmark. doi=10.1038/s41598-024-64827-6
• fact_id=model_eval/auto/2026/accuracy_218254 (A_core) — , web browsing, code development and execution, and text file editing) agent systems yielded only modest accuracy gains over baseline LLMs, reaching 60.3% and 28.0% in AgentClinic MedQA and MIMIC, 30.3% on MedAgentsBench, and 8.6% on HLE te doi=10.1038/s41746-026-02443-6

What this changes

Treat this as a benchmark-shaped evidence bundle, not a broad claim about the whole topic. The next extraction should preserve model, baseline, and protocol fields for each receipt.

Limitations

• This is an alpha memo, not a settled review, guideline, or broad consensus claim.
• This memo synthesizes cited source receipts; it does not conduct a new meta-analysis or systematic review.
• Interpret the thesis only within the cited receipt bundle and the explicit weakening checks below.
• Independent receipts fail to reproduce the claimed contrast.
• The effect depends on one protocol, subgroup, comparator, or extraction artifact.

What would weaken this

• Independent receipts fail to reproduce the claimed contrast.
• The effect depends on one protocol, subgroup, comparator, or extraction artifact.

Strongest counter-evidence

• No direct opposing receipt was selected by this run. Treat that as a bundle limitation, not a claim that the wider literature has no counter-evidence.