# Head-to-head: mathtec/whisper-media-transcriber (ours) vs the Store leaders and Groq hosted Whisper, 2026-11-18

Beads: hustle-qtk, hustle-jghg. The question: can our listing honestly claim "better" against the transcription leaders, and is Groq a cheaper backend for us?

## Outcome first

- **Accuracy: ours loses (n=15 matched clips).** Ours (standard `small`) had a pooled WER of **5.40%**. On the same 15 clips steadyfetch had **3.00%**, tidytools **3.25%**, Groq `whisper-large-v3-turbo` **3.00%** and Groq `whisper-large-v3` **2.80%**. Ours lost or tied on 15 of 15 clips against steadyfetch. The biggest gap was non-English: **10.86% vs 3.71%** (de/fr/es; ours 13.3%, 13.6% and 5.3%).
- **Against memo23 (the leader by 30-day users) it is a tie on 4 clips: 10.49% ours vs 10.67% memo23 `small`.** Both lost to the turbo-based Actors on the same 4 clips (5.42% steadyfetch, 5.79% tidytools).
- **Price: ours is 9-10x the two $0.006 leaders.** This org (STARTER, BRONZE tier) paid **$0.0615 per real audio minute** for ours: $0.81 for 13.18 audio minutes, charged as 15 started minutes at $0.054. It paid $0.0063 for steadyfetch, $0.0068 for tidytools and **$0.0692 for memo23** (per-second billing, but a 60 s minimum per file plus a $0.03/GB start fee).
- **Speed: ours is the slowest file-based option apart from memo23.** Run seconds per audio minute were **31.0 ours** (4 GB), 2.8 steadyfetch, 6.1 tidytools and 88.3 memo23 (2 GB, includes start-up). Groq's API took 0.23 s per audio minute for turbo and 0.41 s for large-v3 (client wall time).
- **Failure handling: all four Actors pass, and Groq passes as an API.** Every Actor returned a row with a plain reason for the HTML page and for the 404, and none charged per minute for either. memo23 still charges its start fee for every run.
- **Two of the three leaders are hosted large-v3-turbo.** tidytools' own rows say `model: whisper-large-v3-turbo, asrBackend: workers-ai` (Cloudflare). steadyfetch's text was **byte-identical to our Groq turbo call on 15 of 16 clips**, and the long file was the exception. That strongly suggests a Groq turbo backend (an inference from the output; steadyfetch does not say so).
- **Groq as our backend: list cost $0.00067/min (turbo) vs our measured $0.006/min on CPU, about 9x cheaper, with better WER on this set.** The pricing section below gives the price this implies.

**Claims the numbers support for the listing:** none on accuracy, speed or price against steadyfetch or tidytools. Against memo23, "the same accuracy as memo23 small at a similar effective price" (n=4) holds. The supported facts for all four are "failed links return a row with the reason and are not charged" and "timestamped segments, language auto-detect (3/3 correct)".

**Claims NOT supported:** "more accurate" (it is worse, 5.40% vs 3.00%), "faster", "cheaper", and "better non-English". Our 10-23 premium claim ("fewer errors than the leading Store transcriber") was memo23 on one file. It does not carry over to the current turbo-based leaders, and premium was **not re-run today** (see Limits).

## Store leaders today (Store API, 2026-11-18 ~10:20Z)

Search terms: transcription, audio transcription, video transcriber, whisper, speech to text, transcribe audio, audio to text, podcast transcription, mp3 to text, memo23. Sorted by `stats.totalUsers30Days`. Raw data is in `h2h-transcriber/store/`.

| Actor | 30-day users | all-time users | Price (BRONZE) | Engine (own statement) | Tested |
|---|---|---|---|---|---|
| lance_api/douyin-transcripts-scraper | 825 | 2958 | per result | Douyin only | no (platform-specific) |
| tubetext/instagram-reels-transcript | 545 | 776 | per event | Reels only | no (platform-specific) |
| **memo23/video-audio-transcriber** | **299** | 853 | $0.0008/s ($0.048/min), $0.03 per GB at start (2 GB default = $0.06), 60 s minimum per transcript | faster-whisper in the Actor, tiny/base/small (default `base`) | yes, `small`, 4 clips |
| **steadyfetch/media-transcriber** | **231** | 501 | $0.006 per started minute, all tiers | not stated; text is identical to Groq turbo | yes, 16 clips |
| **tidytools/audio-transcriber** | **224** | 333 | $0.006/started min (BRONZE; $0.0048 GOLD+) | whisper-large-v3-turbo on Cloudflare Workers AI (row fields) | yes, 15 clips |
| kaz_kakyo/audio-transcriber | 84 | 369 | $0.01/min | Deepgram nova-3 | no |
| mathtec/whisper-media-transcriber (ours) | 1 | 2 | $0.054/started min standard, $0.0607 premium (FREE $0.06 / $0.0674) | faster-whisper small / large-v3-turbo on CPU | yes, 15 clips + 2 failures |

## Method

1. **Inputs.** I built 16 clips with references plus 2 failure URLs (`scripts/build_clips.py`, `inputs.json`). All audio is public domain, synthetic, or CC-BY. No private org audio went to any third party. Short clips were kept under 60 s so that ours and memo23 bill exactly one minute each.
2. **Hosting.** The clips went to a key-value store in the Hustle org (`h2h-transcriber-2026-11-18`, id `4Avq6hBLhrSsqaT0t`) behind signed public record URLs, the same route as the 10-16 test. Every Actor received the same URLs. Groq received the same mp3 bytes as a file upload, and the failure URLs through its `url` parameter.
3. **Runs.** I ran one async batch run per Actor from the Hustle org (`mathtechustle`, STARTER/BRONZE), each with `maxTotalChargeUsd`, default memory and build `latest`. Language was left on auto-detect everywhere. Ours used `model=small, outputFormat=segments`. memo23 used `model=small` (its most accurate option at the same price; its default is `base`). steadyfetch used `outputFormats=[text,segments]` and tidytools used `saveSubtitles=false`. Each Groq model got one request per clip with `response_format=verbose_json`, `timestamp_granularities=segment` and `temperature=0`. No Actor settings or prices were touched.
4. **Budget choices.** The $1.50 cap set the coverage. memo23 got 4 clips (one per category). The long clip ran only on steadyfetch and Groq: on ours it would have cost 22 × $0.054 = $1.19. Our premium model was not run.
5. **Scoring** (`scripts/score.py`, `scripts/tables.py`). English uses Whisper's EnglishTextNormalizer on both sides plus jiwer, the same scorer as `apify-transcribe/tests/evalkit.py`. Non-English uses Whisper's BasicTextNormalizer plus jiwer. Pooled WER is (S+D+I)/reference words over a matched set. $ is the settled run's `usageTotalUsd` divided by real audio minutes. Speed is the run's `stats.runTimeSecs` divided by audio minutes; it includes start-up and download, and the runs were batches.

## Inputs (16 clips + 2 failure URLs)

| id | category | s | ref words | recipe |
|---|---|---|---|---|
| clean_tts_a | clean | 54.5 | 158 | Kokoro TTS meeting script, lines 0-1 |
| clean_tts_b | clean | 53.8 | 147 | Kokoro TTS, lines 4-5 (British + American voices) |
| clean_tech_names | clean | 59.5 | 134 | Kokoro TTS, 15 invented/rare proper nouns, no prompt |
| reader_sonnet_db / gw / jtm / ks / mac | accented (human readers) | 54.8 / 58.3 / 50.5 / 51.2 / 43.0 | 123 each | LibriVox volunteer readers of Sonnet 130, cut to the poem only. Readers chosen because their poem reading fits in 60 s |
| noisy_pink_0db | noisy | 54.5 | 158 | clean_tts_a + pink noise, 0 dB SNR |
| noisy_music_10db | noisy | 53.8 | 147 | clean_tts_b + synthesised music, 10 dB |
| noisy_phone | noisy | 54.5 | 158 | clean_tts_a through a phone channel (300-3400 Hz, 8 kHz mu-law) |
| noisy_reader_pink_5db | noisy | 47.1 | 123 | LibriVox reader njm + pink noise, 5 dB |
| nonen_de / fr / es | non-English | 50.6 / 50.3 / 54.6 | 90 / 142 / 113 | Multilingual LibriSpeech test split (CC-BY 4.0), one speaker, consecutive utterances |
| long_21min | long | 1314.3 | 3380 | LibriVox "Gift of the Magi" (story only) + full Kokoro meeting + tech names + 2 sonnet readers (apc, ce), 1.5 s gaps |
| fail_html | failure | - | - | `https://example.com/` (an HTML page) |
| fail_404 | failure | - | - | `https://example.com/missing-file.mp3` (HTTP 404) |

Caveats: the TTS clips are easy (0% for almost everyone). The sonnet reference contains "damask'd", which costs each system up to 2 words. The MLS references spell numbers as words.

## Results

### WER on matched sets (pooled %, lower is better)

| System | 15 short: pooled | median clip | clean (3) | accented (5) | noisy (4) | non-English (3) | memo23's 4 clips | long 21.9 min |
|---|---|---|---|---|---|---|---|---|
| **ours small** | **5.40** | 4.03 | 7.48 | 4.35 | 1.70 | **10.86** | 10.49 | not run (cost) |
| memo23 small | - | - | - | - | - | - | **10.67** | not run |
| steadyfetch | 3.00 | 2.04 | 5.22 | 2.74 | 1.19 | 3.71 | 5.42 | 2.08 |
| tidytools | 3.25 | 2.65 | 5.44 | 3.39 | 1.19 | 3.71 | 5.79 | not run |
| Groq large-v3-turbo | 3.00 | 2.04 | 5.22 | 2.74 | 1.19 | 3.71 | 5.42 | **1.49** |
| Groq large-v3 | **2.80** | 1.77 | 6.12 | 2.42 | 0.85 | 2.57 | 6.15 | 3.48 |

On the long file Groq large-v3 **dropped a 52-word block at the start of the meeting section** (a deletion block right after the story ends), so its long-file WER is more than twice turbo's.

### Per clip (WER %)

| clip | ours | memo23 | steadyfetch | tidytools | Groq turbo | Groq v3 |
|---|---|---|---|---|---|---|
| clean_tts_a | 0.00 | - | 0.00 | 1.26 | 0.00 | 0.00 |
| clean_tts_b | 0.00 | - | 0.00 | 0.00 | 0.00 | 0.00 |
| clean_tech_names | 24.44 | 20.74 | 17.04 | 16.30 | 17.04 | 20.00 |
| reader_sonnet_db | 4.03 | 4.84 | 3.23 | 4.03 | 3.23 | 3.23 |
| reader_sonnet_gw | 3.23 | - | 1.61 | 2.42 | 1.61 | 1.61 |
| reader_sonnet_jtm | 5.65 | - | 4.84 | 4.84 | 4.84 | 4.84 |
| reader_sonnet_ks | 4.84 | - | 1.61 | 2.42 | 1.61 | 0.00 |
| reader_sonnet_mac | 4.03 | - | 2.42 | 3.23 | 2.42 | 2.42 |
| noisy_pink_0db | 1.26 | - | 0.63 | 0.00 | 0.63 | 0.00 |
| noisy_music_10db | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| noisy_phone | 0.00 | - | 0.00 | 0.63 | 0.00 | 0.00 |
| noisy_reader_pink_5db | 6.45 | - | 4.84 | 4.84 | 4.84 | 4.03 |
| nonen_de | 13.33 | - | 7.78 | 5.56 | 7.78 | 4.44 |
| nonen_fr | 13.61 | 17.01 | 2.04 | 3.40 | 2.04 | 2.04 |
| nonen_es | 5.31 | - | 2.65 | 2.65 | 2.65 | 1.77 |
| long_21min | - | - | 2.08 | - | 1.49 | 3.48 |

### Cost, speed, output

| System | $ charged (this test) | $ per real audio min | run s per audio min | memory | segments with timestamps | non-English language detected | HTML page / 404 |
|---|---|---|---|---|---|---|---|
| ours small | 0.81 (15 × $0.054) | **0.0615** | 31.0 (median item RTF ≈ 0.47) | 4096 | 15/15 | de, fr, es (3/3) | row `failed`, "URL returned a web page…" / "HTTP 404"; $0 |
| memo23 small | 0.252 (240 billed s + 2 start events) | 0.0692 | 88.3 | 2048 | 4/4 | fr (1/1) | `bad_input` / `source_unavailable` rows; start fee still charged |
| steadyfetch | 0.222 (37 started min) | 0.0063 | 2.8 | 4096 | 16/16 | German, French, Spanish | `unsupported_site` / `unreachable`, "Nothing was charged" |
| tidytools | 0.09 (15 started min) | 0.0068 | 6.1 | 1024 | 15/15 | de, fr, es | `success:false` with reason, "(not charged)" |
| Groq turbo | $0 cash (free-tier key); list $0.0234 | list 0.00067 | 0.23 (API wall) | - | 16/16 | German, French, Spanish | HTTP 400 `invalid_media_file` / `media_fetch_failed` |
| Groq large-v3 | $0 cash; list $0.0649 | list 0.00185 | 0.41 | - | 16/16 | German, French, Spanish | same |

Ours stopped at its own `maxTotalChargeUsd` ($0.82) after 15 minutes, and that is why the two failure URLs in that batch came back `not_processed` ("your maximum charge for this run is reached"). They were re-run alone (run M36KuR5hDVAp4JpeF), cost $0 and gave the rows above.

## Groq: pricing and terms (read 2026-11-18)

- **Price** (console.groq.com/docs/models and /docs/speech-to-text; the groq.com/pricing page is rendered client-side, so its raw HTML has no figures): `whisper-large-v3` **$0.111 per hour**, `whisper-large-v3-turbo` **$0.04 per hour**. "Minimum Billed Length 10 seconds. If you submit a request less than this, you will still be billed for 10 seconds." Max file size is 25 MB on the free tier and 100 MB on the dev tier; larger files go through the `url` parameter.
- **Rate limits** (docs/rate-limits, free tier): 20 RPM, 2K RPD, **7.2K audio-seconds/hour and 28.8K audio-seconds/day** (8 h/day) per Whisper model. The models page lists 200K/400K audio-seconds/hour (large-v3/turbo) for paid use. The key used (`~/.config/groq/.env`, Adam's shared key) returned `x-ratelimit-limit-requests: 2000`, which matches the free tier. **Cash spent: $0; at list price this test would have cost $0.0883.**
- **Terms** (Groq Services Agreement, "Last Modified: June 22, 2026"), quoted:
  - 3.1: "Groq grants you a non-exclusive, non-transferable, and limited right to access and use the Cloud Services … including the right to use Groq's APIs to integrate the Cloud Services and AI Model Services into your Customer Application and to make the Cloud Services and AI Model Services available to End Users through your Customer Applications."
  - Definition: "'Customer Application' means Customer's applications, products, or services in which Customer may integrate the Cloud Services and AI Model Services as permitted by this Agreement."
  - 6.3 Restrictions: "Customer will not, and will not allow its Authorized Users to, … (c) sell, resell, sublicense, transfer, or distribute any of the Cloud Services except as expressly approved by Groq; (d) access or use the Cloud Services … (iii) in a manner intended to avoid incurring Fees …"
  - How an Actor that sells transcripts made through Groq sits between 3.1 and 6.3(c) is a judgement for Adam, not something this test decides. tidytools states its model and backend in its output, and steadyfetch does not.
- Raw pages: `h2h-transcriber/groq-pages/`.

## Pricing implications

The 60% rule: price = cost / (1 − 0.2 Apify fee − 0.6 margin) = **cost / 0.2 = 5 × cost**.

| Backend | Cost per audio min | Source | Price by the 60% rule | Listed today (BRONZE / FREE) |
|---|---|---|---|---|
| ours, `small` on CPU (Actor) | $0.0060 | measured 10-16, run W75AAoeE3IbxaNamv (pricing/apify-costs.json) | **$0.030** | $0.054 / $0.060 (Adam's override) |
| ours, `large-v3-turbo` on CPU | $0.01347 | measured 10-23, run kFz2Q78HYSXhItTU0 | **$0.0674** | $0.0607 / $0.0674 |
| Groq turbo, API only | $0.00067 | list price | $0.0033 | - |
| **Groq turbo + thin Actor wrapper** | **≈ $0.0013** = $0.00067 + ≈ $0.00063 Apify compute | wrapper compute **not measured on our code**; derived from steadyfetch's run (98.9 s at 4 GB for 35.1 audio min = 0.0031 CU/min) × ≈ $0.20/CU implied by our own 10-16 run ($0.0769 for 0.396 CU) | **≈ $0.0065** | - |
| Groq large-v3 + wrapper | ≈ $0.0025 | same assumption | ≈ $0.0124 | - |

Reading:
- On this set a Groq turbo backend costs about **one fifth of our `small` cost** and about **one tenth of our premium cost**, and it was more accurate than either: Groq turbo 3.00% vs our small 5.40%. Our premium was not re-measured here.
- Our 60% rule applied to Groq turbo gives about **$0.0065/min, which is where steadyfetch and tidytools already sit ($0.006).** Our CPU `small` cost of $0.006/min alone already equals their whole price.
- Unknowns that move these numbers:
  - The wrapper's real compute (memory could be 1 GB, as tidytools uses, rather than 4 GB).
  - Groq's 10 s minimum per request.
  - Splitting files over 25/100 MB.
  - Free-tier limits: 8 audio-hours/day, so production needs the paid tier.
  - The 6.3(c) question above.

## Spend

- **Apify: $1.374 charged across 5 runs** (cap $1.50): ours $0.81, ours failures $0.00, steadyfetch $0.222, tidytools $0.09, memo23 $0.252. `/users/me/limits` `monthlyUsageUsd` went from 12.3911 to 13.7665, a delta of **$1.3754**. The extra $0.0014 over the run charges is assumed to be key-value-store storage and requests; no other org runs fell in the window.
- **Groq: $0 cash (free-tier key); list-price equivalent $0.0883** (cap $0.20).
- Local CPU: faster-whisper `small` word alignment (`scripts/align.py`), used only to find cut points and never scored.

## Limits of this test

- n = 15 short clips (13.2 audio min) plus 1 long clip, one run per system, one time of day. memo23 got only 4 clips.
- **Our premium (`large-v3-turbo`) was not run**, and **ours was not run on the long clip**. Both were left out for the cap; the long clip alone would have cost $1.19 at our price. The only long-file data for ours is the 10-16 / 10-23 runs on a 12.9-min interview.
- Most "clean" and "noisy" clips are TTS, which nearly every system transcribes perfectly. Human-reader and non-English clips separate the systems. There are no phone calls, crosstalk or real far-field recordings.
- Speed is per batch run, so it includes start-up, image pull and download, and the systems run items concurrently to different degrees.
- Our differentiators were not exercised: VAD (silence/music invention), `quality_flags` and the vocabulary prompt.

## Files

All under `/home/a/Projects/swarm/agents/hustle/reports/market/h2h-transcriber/`:
- `inputs.json`: clips, recipes and reference texts. `clips/*.mp3`: the audio. `hosting.json`: signed URLs. `align.json`: cut-point word timings.
- `store/`: Store search results, Actor objects and input schemas.
- `runs/apify/{ours_small,ours_fails,steadyfetch,tidytools,memo23_small}.json`: input, cap, settled run object, dataset items and log tail.
- `runs/groq/<model>/<clip>.json`: response, headers, wall time and list cost.
- `per-clip.json`, `summary.json`, `tables.json`: computed metrics. `limits-before.json`, `limits-after.json`: account usage.
- `groq-pages/`: raw Groq pricing, docs and terms pages.
- `scripts/`: `align.py`, `build_clips.py`, `host.py`, `apify.py`, `apify_runs.py`, `groq_run.py`, `score.py`, `tables.py`. Tokens and keys are read at run time and never written.
