Company facts extraction (S1)
From a company's site pages, fill 13 fields (description, offering, sector, customers, founded, certifications ...). A filled field is correct when it matches the answer key.
Best go-to: gaents/deepseek (correct share of filled fields 90.5%, recall (filled where the key has a value) 91.6%; $0.561 cash per 1,000 correct fields): inside the interval of the top callable route gaents/claude-haiku (91.7%, interval 90.3%-93.0%; $0.6134 capacity per 1,000 correct fields) on every measured quality column. 0.91 times the incumbent's cost per correct output. The cheaper gaents/gpt-oss ($0.1469 cash per 1,000 correct fields) does not qualify: correct share of filled fields 87.2% is below gaents/claude-haiku's interval 90.3%-93.0%.
Rule (setting models.go_to_rule, agent-chosen, unruled; Adam sets): Start from the top-scoring route you can call as one gaents alias (the incumbent). A cheaper callable route becomes the go-to only if, on EVERY measured quality column of the task, its score is inside or above the incumbent's 95% interval. If a cheaper route falls below on any column, the incumbent stays and the line says which column. Cost per correct output is ranked on cash plus capacity at list price; the two are shown in separate columns.
Ranked by cash plus capacity per 1,000 correct fields (lowest first; rows with no known cost last); cash and capacity are shown in separate columns. Correct / n is for the first quality column (filled fields (192 companies)). Source: Judge's frozen scorer (score_s1.py), re-scored in model-cost-review 2026-11-18 (tables.json); reproduces the swarm agent's PROPOSAL v3.1.
| # | Route | Call it as | correct share of filled fields [95% interval] | recall (filled where the key has a value) [95% interval] | Correct / n | Cash per 1,000 correct | Capacity per 1,000 correct (not cash) | Price now in/out $/1M |
|---|---|---|---|---|---|---|---|---|
| 1 | gpt-oss | gaents/gpt-oss |
87.2% [85.6%-88.8%] | 89.8% [88.3%-91.2%] | 1465 / 1679 | $0.1469 | not in source | $0.1741 / $0.6963 |
| 2 | glm-flash | gaents/glm-flash |
88.8% [87.2%-90.2%] | 95.0% [93.8%-95.9%] | 1549 / 1745 | $0.2299 | not in source | $0.1741 / $0.5802 |
| 3 | glm-flash kept where gpt-oss agrees (two calls) | not one alias | 92.0% [90.5%-93.2%] | 86.6% [84.9%-88.2%] | 1413 / 1536 | $0.4043 | not in source | - |
| 4 | glm-flash kept where a second glm-flash run agrees (two calls) | not one alias | 91.1% [89.6%-92.4%] | 92.7% [91.3%-93.9%] | 1512 / 1660 | $0.5193 | not in source | - |
| 5 | glm-flash labels its own values, keep 'stated' (two calls) | not one alias | 90.8% [89.2%-92.2%] | 82.7% [80.8%-84.5%] | 1349 / 1486 | $0.5443 | not in source | - |
| 6 | deepseek | gaents/deepseek |
90.5% [89.0%-91.8%] | 91.6% [90.1%-92.8%] | 1494 / 1651 | $0.561 | not in source | $0.3017 / $0.4410 |
| 7 | claude-haiku, lean prompt | gaents/claude-haiku |
91.7% [90.3%-93.0%] | 90.4% [88.8%-91.7%] | 1474 / 1607 | not in source | $0.6134 recomputed from 77 calls' token counts at $0.10/$0.50 per 1M (Forge 2ffb1ec) |
$0.1161 / $0.5802 |
| 8 | qwen-flash | gaents/qwen-flash |
89.9% [88.3%-91.3%] | 88.8% [87.2%-90.3%] | 1449 / 1612 | $0.6974 | not in source | $0.1741 / $0.5454 |
| 9 | glm-flash kept where deepseek agrees (two calls) | not one alias | 93.0% [91.7%-94.2%] | 87.8% [86.1%-89.3%] | 1432 / 1539 | $0.834 | not in source | - |
| 10 | glm-flash kept where claude-haiku agrees (two calls) | not one alias | 94.0% [92.7%-95.1%] | 86.8% [85.0%-88.3%] | 1415 / 1505 | $0.2517 | $0.6656 Haiku share: recomputed Haiku cost per company x 200 companies |
- |
| 11 | qwen-27b | gaents/qwen-27b |
91.1% [89.5%-92.4%] | 88.7% [87.0%-90.1%] | 1446 / 1588 | $2.352 | not in source | $0.0696 / $2.5531 |
| 12 | claude-sonnet, lean prompt | gaents/claude-sonnet |
90.3% [88.8%-91.6%] | 94.0% [92.7%-95.0%] | 1533 / 1698 | not in source | $7.637 | $2.3210 / $11.6050 |
| 13 | incumbent pipeline (claude + web fetch) | not one alias | 84.4% [82.5%-86.1%] | 80.2% [78.2%-82.1%] | 1308 / 1550 | not in source | $28.37 | - |
Read this with
- Answer key and grader are both Opus, which favours Claude routes (model-cost-review REPORT.md).
- Cost covers every company called (200 for API routes, 192 for Claude routes); the score covers the 192 in scope.
- Claude routes ran on subscription logins: their cost is API-equivalent list price (capacity), shown in its own column, never added to cash. The Haiku rows are recomputed from the run's token counts at $0.10/$0.50 per 1M tokens (Forge's price table, tech 2ffb1ec); the CLI had reported a higher figure.
The measured runs paid the provider's price (or used capacity), not gaents' price; "price now" is what gaents charges today (/v1/models at 2026-11-19T06:07:16Z). Rows dated: 2026-11-18T15:16Z;. Every number: /models/models.json.