CHEMRXN-BENCH / PUBLIC SNAPSHOT
Reaction extraction, measured figure by figure.
603 figures. 8,547 normalized reference reactions. Single-step schemes, multistep routes, Markush scopes, condition tables, and complex layouts.
23 lanes. Every scored lane uses all 603 figures; pending lanes are not ranked.
| # | Method | N | Reaction P | Reaction R | Reaction F1 | Molecule F1 | Avg. cost (USD) | Avg. time (s) |
|---|---|---|---|---|---|---|---|---|
| 1 |  RxnMiner · Claude Opus 5 | 603 | 77.28 | 93.60 | 77.68 | 94.06 | 22.391 | 865.2 |
| 2 |  RxnMiner · Qwen3.8-Flash-Next | 603 | 69.93 | 89.20 | 70.81 | 88.82 | 0.588 | 1343.6 |
| 3 |  RxnMiner · Kimi K3 | 603 | 68.60 | 86.03 | 69.02 | 87.92 | 1.932 | 1622.9 |
| 4 |  RxnMiner · DeepSeek V4.1 Flash | 603 | 62.90 | 82.15 | 63.62 | 83.43 | 0.668 | 1915.2 |
| 5 |  RxnMiner · GPT-5.6-Sol | 603 | 58.00 | 80.47 | 59.04 | 82.72 | 1.026 | 218.1 |
| 6 | Claude Opus 5 | 603 | 53.82 | 68.66 | 53.97 | 77.18 | 0.497 | 282.6 |
| 7 |  RxnMiner · Doubao Seed 2.1 Pro | 603 | 45.32 | 65.82 | 46.89 | 68.46 | 1.005 | 1246.9 |
| 8 |  RxnMiner · GLM 5.3 Flash | 603 | 40.61 | 52.98 | 41.10 | 53.57 | 0.148 | 2786.2 |
| 9 | Kimi K3 | 603 | 39.59 | 52.51 | 39.90 | 63.98 | 0.614 | 1085.6 |
| 10 | GPT-5.6-Sol | 603 | 35.55 | 51.29 | 35.89 | 62.45 | 0.151 | 202.1 |
| 11 | Doubao Seed 2.1 Pro | 603 | 33.99 | 48.13 | 34.47 | 61.31 | 0.238 | 1070.1 |
| 12 |  Gemini 3.1 Pro | 603 | 31.19 | 43.87 | 31.61 | 57.14 | 0.191 | 246.3 |
| 13 | GLM 5.3 Flash | 603 | 30.87 | 46.47 | 31.31 | 52.79 | 0.0355 | 1022.6 |
| 14 | ChemEAGLE | 603 | 25.05 | 49.71 | 24.96 | 47.48 | 0.550 | 546.7 |
| 15 | Qwen3.8-Flash-Next | 603 | 20.01 | 36.27 | 20.80 | 49.25 | 0.0072 | 247.3 |
| 16 | DeepSeek V4.1 Flash | 603 | 15.67 | 24.07 | 15.82 | 27.50 | 0.127 | 882.8 |
| 17 | MERMaid | 603 | 5.77 | 15.22 | 5.85 | 16.26 | 0.0914 | |
| 18 | OpenChemIE | 603 | 9.79 | 8.12 | 5.56 | 17.52 | 0 | 5.1 |
| 19 | MinerU.Chem | 603 | 7.08 | 2.61 | 2.10 | 15.12 | 0 | 211.5 |
| 20 | RxnScribe | 603 | 6.15 | 3.35 | 2.02 | 16.66 | 0 | 12.1 |
| 21 | RDE2 | 603 | 2.03 | 1.85 | 0.88 | 7.12 | 0 | 391.8 |
| 22 | RxnCaption | 603 | 0.00 | 0.00 | 0.00 | 0.00 | 0 | 84.2 |
| 23 | RxnID | 603 | 0.00 | 0.00 | 0.00 | 0.00 | 0 | 29.6 |
Scores are percentages. Cost and time are per-figure averages; missing costs display as 0, and missing times are blank. Initially sorted by reaction F1.
HOW TO READ THE RESULTS
One full benchmark. An explicit denominator.
Precision, recall, and F1 are calculated per figure, then averaged with equal figure weights. Macro-F1 averages each figure’s F1, with zero when precision and recall are both zero. Failed and empty answers stay in the denominator.
Strict scoring requires molecular connectivity and stereochemistry to agree. Connectivity-only scoring ignores stereochemistry. Reaction F1 combines predicted-reaction precision with product-occurrence recall; molecule metrics compare individual structures.
All scored lanes use the same 603 figures. Pending lanes have no published scores and are excluded from ranking.