# Head-to-head: mathtec/youtube-transcript (ours) vs pintostudio/youtube-transcript-scraper (Store leader), 2026-11-09

Bead: hustle-qtk. Purpose: find out whether our Store listing can honestly say "better", and if so where.

## Outcome first

- **Transcript content: equal (n=28).** On all 28 videos where both returned a transcript, the two outputs are **identical**: same text after whitespace normalisation (28/28), same segment count (28/28) and same segment start times (max difference 0.00 s). Both also match an independent fetch of the same YouTube caption track (difflib token ratio 1.000 for both, 28/28). Both Actors pass YouTube's own caption track through unchanged. Neither is more complete, and a quality claim cannot separate them.
- **Success rate: equal.** 28/30 each. Both failed only on the 2 expected private/removed videos.
- **Price: ours is lower.** Measured on the Hustle org account (STARTER plan): **$0.003** per successful transcript for ours vs **$0.010** for the leader, so ours costs 0.30x as much (n=28 each). Neither charged for the failed videos.
- **Failure handling: ours is cleaner (n=2).** Ours writes a dataset row with `status: failed` and a plain error ("This video is private" / "This video is unavailable"), and the run ends SUCCEEDED. The leader writes **no row** in either case. On the private video its run ended **FAILED** with a misleading "Temporary error (TIMEOUT) … try again in a few minutes" after 30.9 s. On the removed video the run SUCCEEDED and the reason appeared only in the run status message.
- **Speed: mixed, no "faster" claim.** Run seconds per video (single-video runs, 28 successes): median **4.3 s ours vs 5.3 s leader**. Ours has a longer tail, though: p90 16.9 s vs 8.0 s, and max 31.0 s vs 10.7 s. On human-caption videos the median was 12.7 s for ours vs 5.4 s for the leader (n=10). Our logs show the slow time sits between Actor init and exit, with no per-video log lines to explain it.
- **Batching: ours only.** The leader accepts one `videoUrl` per run. Ours took 25 English videos in one run: 82.4 s, 23 transcripts plus 2 clean failed rows, about 3.3 s per video, $0.069. The batch output was byte-identical to the single runs (28/28).
- **Metadata: ours returns more.** Ours returns title, channel, language, languageName, isGenerated (human vs auto), languageFallback, availableLanguages, durationSeconds, segmentCount and plain `text`. The leader returns only `data: [{start, dur, text}]` as strings.

**Claims the numbers support for the listing:**
- "Same transcript as the category leader at 30% of the price" (n=28, identical text and timestamps).
- "Failed videos return a row with the reason and are not charged" (n=2).
- "Many videos, channels or playlists per run" (an input-schema fact, measured with 25 in one run).
- "Tells you whether captions are human or auto-generated" (a field the leader lacks).

**Claims the numbers do NOT support:**
- "More accurate", "more complete" or "better quality": the transcripts were identical.
- "Faster": the medians favour ours, but the tail is worse, and n=28 single runs.

## Method

1. **Inputs.** I took 30 public video ids from org artefacts and probed each one on 2026-11-09 from withheld (a residential IP) to list its caption tracks. The probe reads the YouTube watch page `captionTracks`, where `kind=asr` means auto-generated and anything else is uploaded. The candidate pool was 190 ids (`h2h-youtube-transcript/candidate-pool-probe.json`), and the selection is in `inputs.json`.
2. **Reference.** For every non-failing video I fetched the caption track independently from withheld (innertube ANDROID client, then timedtext json3; `reference-tracks.json`). Human-caption videos use the uploaded track as reference, and auto-only videos use the asr track. Nothing here passed through either Actor.
3. **Runs.** Both Actors ran from the Hustle Apify org (`mathtechustle`, STARTER). This org does not own `mathtec/youtube-transcript`, so it pays the public price like any customer. Each video got one single-video run per Actor, started at the same moment (4 videos, so 8 runs, at a time) so that both met the same YouTube conditions. Every run passed `maxTotalChargeUsd` ($0.02 per single run; $0.004 × n for batch runs). Default memory, build `latest`, and no Actor settings or prices were changed.
4. **Equivalent settings.** Ours: `videos=[url]`, `languages=[lang]`, `preferManual=true` (the default), `outputFormat=segments` (text plus timestamps). Leader: `videoUrl=url`, `targetLanguage=lang`. Its schema has no other options, and it returned the human track whenever one existed.
5. **Metrics.** Success means a non-empty transcript. Words are `\w+` tokens on lowercased text. Similarity is `difflib.SequenceMatcher` ratio on token lists, pairwise and against the reference. Seconds are the run's `stats.runTimeSecs`, with client wall time also recorded. The $ figure is charged events × event price, read from the run object after it settled. Charged counts read right after a run finished were sometimes still 0, so the counts were re-read later.
6. **Batch.** One extra run of ours per language group (en 25, nl 2, es/hi/kn 1 each) measured its native multi-video mode.

## Inputs (30)

The non-English rows (21–24) are short clips whose only track is a non-English asr track. Their content may be partly music, which YouTube's language ID handles poorly. Row 25 tests choosing a specific uploaded track (Spanish) from a video with five uploaded languages.

| # | videoId | category | lang requested | length | tracks (uploaded / asr) | reference (kind, words) | why chosen | source (under /home/a/Projects/) |
|---|---|---|---|---|---|---|---|---|
| 1 | `PdE-waSx-d8` | human | en | 4h14m | en / en | up 47835 | Wolfram on Lex #376; uploaded en + asr; also a long human track | wolfram/podcasts/transcripts/lex-fridman/MANIFEST.json |
| 2 | `lXUZvyajciY` | human | en | 2h26m | en,es / en | up 26294 | Karpathy on Dwarkesh; two uploaded languages | karpathy/videos/INDEX.md (external talks) |
| 3 | `tsTeEkzO9xc` | human | en | 1h32m | en / en | up 15518 | Karpathy Berkeley hackathon keynote | karpathy/videos/INDEX.md (external talks) |
| 4 | `bZQun8Y4L2A` | human | en | 42m40s | en-US / - | up 7753 | State of GPT; only an uploaded en-US track, no asr (tests regional code) | karpathy/videos/INDEX.md (external talks) |
| 5 | `cdiD-9MMpb0` | human | en | 3h28m | en-US / en | up 38601 | Karpathy on Lex #333; en-US uploaded | karpathy/videos/INDEX.md (external talks) |
| 6 | `98aUXLpyLZ0` | human | en | 2m16s | en / en | up 194 | Craig batch1 row flagged isGenerated=false; short | craig/requests/from-hustle-2026-10-23-youtube-batch1.json |
| 7 | `zBOoEpsjWAo` | human | en | 20m16s | en / en | up 3587 | Sundar Pichai interview | alternatives/data/alternatives.json |
| 8 | `MCPzPBy_bhM` | human | en | 18m11s | en / en | up 2410 | Doodle weekly 10-26 | doodle/reports/weekly/2026-10-26.md |
| 9 | `KHcukYSw53E` | human | en | 18m09s | en / en | up 2339 | Doodle weekly 10-26 | doodle/reports/weekly/2026-10-26.md |
| 10 | `WSlfKfQ8Q5Q` | human | en | 9m50s | en-GB / en | up 1754 | uploaded en-GB (tests regional code) | history/adam-asks.json |
| 11 | `zjkBMFhNj_g` | auto | en | 59m48s | - / en | asr 12197 | Karpathy Intro to LLMs | karpathy/videos/INDEX.md |
| 12 | `g6bOwQdCJrc` | auto | en | 31m31s | - / en | asr 6805 | Karpathy CVPR'21 keynote | karpathy/videos/INDEX.md (external talks) |
| 13 | `oBklltKXtDE` | auto | en | 11m11s | - / en | asr 2422 | Karpathy PyTorch at Tesla | karpathy/videos/INDEX.md (external talks) |
| 14 | `y57wwucbXR8` | auto | en | 17m54s | - / en | asr 3975 | Karpathy Software 2.0 | karpathy/videos/INDEX.md (external talks) |
| 15 | `AKkWQ4PSpfk` | auto | en | 3m58s | - / en + 20 auto-translated | asr 551 | CoinGeek; many asr languages listed | swarm/agents/hustle/services/digest/state.json |
| 16 | `sAykCAGgzkg` | auto | en | 3m07s | - / en | asr 360 | CoinGeek | swarm/agents/hustle/services/digest/state.json |
| 17 | `RCYdu7BlkQI` | auto | en | 27m15s | - / en + 20 auto-translated | asr 4196 | Wolfram R&D | swarm/agents/hustle/services/digest/state.json |
| 18 | `-z9ZSC7Xes0` | auto | en | 11m07s | - / en | asr 1365 | Zoo text-to-CAD | alternatives/data/alternatives.json |
| 19 | `k6XujXpIXDU` | auto | en | 12m26s | - / en | asr 1649 | listed in Adam asks and in the 10-14 tweet triage | history/adam-asks.json; alternatives/research/tweet-triage-2026-10-14-scout.md |
| 20 | `afLpBvEeVa4` | auto | en | 45m05s | - / en | asr 6935 | Parliamentary Review podcast | history/adam-asks.json |
| 21 | `ndF990bG08g` | non-en | nl | 1m29s | - / nl | asr 149 | only track is nl asr | craig/requests/from-hustle-2026-10-23-youtube-batch1.json |
| 22 | `2BucdRecAdA` | non-en | nl | 0m50s | - / nl | asr 85 | only track is nl asr | history/adam-asks.json |
| 23 | `QDnwZcDHRjA` | non-en | hi | 0m50s | - / hi | asr 9 | only track is hi asr | history/adam-asks.json |
| 24 | `9F6-cKqCvKw` | non-en | kn | 0m44s | - / kn | asr 22 | only track is kn asr | history/adam-asks.json |
| 25 | `t5Y2CwCsnbA` | non-en | es | 1h00m | en,el,pl,pt,es / en | up 9356 | asks for the uploaded Spanish track out of 5 uploaded languages | craig/requests/from-hustle-2026-10-23-youtube-batch1.json |
| 26 | `s49L87BUsfo` | long | en | 3h55m | - / en | asr 32235 | Hodlonaut v Wright testimony | craig/requests/from-hustle-2026-10-23-youtube-batch1.json |
| 27 | `l8pRSuU81PU` | long | en | 4h01m | - / en | asr 43505 | Karpathy reproduce GPT-2 | karpathy/videos/INDEX.md |
| 28 | `ez773teNFYA` | long | en | 3h11m | - / en | asr 34441 | Wolfram on Lex #89 | wolfram/podcasts/transcripts/lex-fridman/MANIFEST.json |
| 29 | `DeebhvBEUqA` | private/removed | en | - | - / - | - | batch1 "This video is private"; LOGIN_REQUIRED on 11-09 | craig/requests/from-hustle-2026-10-23-youtube-batch1.json |
| 30 | `-P-zEwwOMx4` | private/removed | en | - | - / - | - | batch1 "This video is unavailable"; ERROR on 11-09 | craig/requests/from-hustle-2026-10-23-youtube-batch1.json |

Category sizes: human 10, auto 10, non-English 5, long (over 60 min, auto-only) 3, private/removed 2. Four of the human videos (rows 1, 2, 3, 5) and one non-English video (row 25) are also over 60 minutes.

## Results by category

| category | n | ok ours / leader | $ ours / leader | $ per ok ours / leader | median run s ours / leader | median words (identical) | timestamps ours / leader | median sim to reference ours / leader |
|---|---|---|---|---|---|---|---|---|
| all | 30 | 28 / 28 | 0.084 / 0.28 | 0.003 / 0.010 | 4.3 / 5.3 | 3781 | 28 / 28 | 1.000 / 1.000 |
| human | 10 | 10 / 10 | 0.03 / 0.10 | 0.003 / 0.010 | 12.7 / 5.4 | 5670 | 10 / 10 | 1.000 / 1.000 |
| auto | 10 | 10 / 10 | 0.03 / 0.10 | 0.003 / 0.010 | 4.1 / 5.0 | 3199 | 10 / 10 | 1.000 / 1.000 |
| non-en | 5 | 5 / 5 | 0.015 / 0.05 | 0.003 / 0.010 | 4.4 / 5.2 | 85 | 5 / 5 | 1.000 / 1.000 |
| long | 3 | 3 / 3 | 0.009 / 0.03 | 0.003 / 0.010 | 4.2 / 7.7 | 34441 | 3 / 3 | 1.000 / 1.000 |
| private/removed | 2 | 0 / 0 | 0 / 0 | - | 8.2 / 20.0 | - | - | - |

Timing over the 28 successes: ours median 4.3 s, p90 16.9 s, max 31.0 s; leader median 5.3 s, p90 8.0 s, max 10.7 s. Client wall time (POST to finished) had a median of 7.2 s for ours and 9.3 s for the leader.

Language handling: both returned the requested language in 5/5 non-English cases (nl, nl, hi, kn, es). For the regional-only tracks (en-US ×2, en-GB) both returned the track when asked for `en`. Ours labels the result (`language: en-US`, `languageFallback: false`); the leader has no language field.

## Per video

| videoId | cat | ours ok | leader ok | words (both) | ref words | sim to ref ours / leader | ours vs leader sim | lang returned (ours) | ours isGenerated | run s ours / leader | $ ours / leader |
|---|---|---|---|---|---|---|---|---|---|---|---|
| `PdE-waSx-d8` | human | Y | Y | 47835 | 47835 | 1.0 / 1.0 | 1.0 | en | False | 13.5 / 6.2 | 0.003 / 0.01 |
| `lXUZvyajciY` | human | Y | Y | 26294 | 26294 | 1.0 / 1.0 | 1.0 | en | False | 11.5 / 4.6 | 0.003 / 0.01 |
| `tsTeEkzO9xc` | human | Y | Y | 15518 | 15518 | 1.0 / 1.0 | 1.0 | en | False | 15.1 / 4.9 | 0.003 / 0.01 |
| `bZQun8Y4L2A` | human | Y | Y | 7753 | 7753 | 1.0 / 1.0 | 1.0 | en-US | False | 3.2 / 6.9 | 0.003 / 0.01 |
| `cdiD-9MMpb0` | human | Y | Y | 38601 | 38601 | 1.0 / 1.0 | 1.0 | en-US | False | 31.0 / 9.0 | 0.003 / 0.01 |
| `98aUXLpyLZ0` | human | Y | Y | 194 | 194 | 1.0 / 1.0 | 1.0 | en | False | 17.8 / 5.2 | 0.003 / 0.01 |
| `zBOoEpsjWAo` | human | Y | Y | 3587 | 3587 | 1.0 / 1.0 | 1.0 | en | False | 3.0 / 5.5 | 0.003 / 0.01 |
| `MCPzPBy_bhM` | human | Y | Y | 2410 | 2410 | 1.0 / 1.0 | 1.0 | en | False | 16.9 / 8.7 | 0.003 / 0.01 |
| `KHcukYSw53E` | human | Y | Y | 2339 | 2339 | 1.0 / 1.0 | 1.0 | en | False | 11.9 / 5.3 | 0.003 / 0.01 |
| `WSlfKfQ8Q5Q` | human | Y | Y | 1754 | 1754 | 1.0 / 1.0 | 1.0 | en-GB | False | 2.9 / 4.1 | 0.003 / 0.01 |
| `zjkBMFhNj_g` | auto | Y | Y | 12197 | 12197 | 1.0 / 1.0 | 1.0 | en | True | 2.3 / 4.9 | 0.003 / 0.01 |
| `g6bOwQdCJrc` | auto | Y | Y | 6805 | 6805 | 1.0 / 1.0 | 1.0 | en | True | 23.6 / 5.1 | 0.003 / 0.01 |
| `oBklltKXtDE` | auto | Y | Y | 2422 | 2422 | 1.0 / 1.0 | 1.0 | en | True | 3.3 / 3.3 | 0.003 / 0.01 |
| `y57wwucbXR8` | auto | Y | Y | 3975 | 3975 | 1.0 / 1.0 | 1.0 | en | True | 6.4 / 6.4 | 0.003 / 0.01 |
| `AKkWQ4PSpfk` | auto | Y | Y | 551 | 551 | 1.0 / 1.0 | 1.0 | en | True | 4.8 / 4.3 | 0.003 / 0.01 |
| `sAykCAGgzkg` | auto | Y | Y | 360 | 360 | 1.0 / 1.0 | 1.0 | en | True | 4.1 / 4.3 | 0.003 / 0.01 |
| `RCYdu7BlkQI` | auto | Y | Y | 4196 | 4196 | 1.0 / 1.0 | 1.0 | en | True | 4.2 / 6.6 | 0.003 / 0.01 |
| `-z9ZSC7Xes0` | auto | Y | Y | 1365 | 1365 | 1.0 / 1.0 | 1.0 | en | True | 4.2 / 10.7 | 0.003 / 0.01 |
| `k6XujXpIXDU` | auto | Y | Y | 1649 | 1649 | 1.0 / 1.0 | 1.0 | en | True | 2.8 / 4.6 | 0.003 / 0.01 |
| `afLpBvEeVa4` | auto | Y | Y | 6935 | 6935 | 1.0 / 1.0 | 1.0 | en | True | 2.9 / 6.4 | 0.003 / 0.01 |
| `ndF990bG08g` | non-en | Y | Y | 149 | 149 | 1.0 / 1.0 | 1.0 | nl | True | 5.9 / 5.1 | 0.003 / 0.01 |
| `2BucdRecAdA` | non-en | Y | Y | 85 | 85 | 1.0 / 1.0 | 1.0 | nl | True | 3.8 / 5.2 | 0.003 / 0.01 |
| `QDnwZcDHRjA` | non-en | Y | Y | 9 | 9 | 1.0 / 1.0 | 1.0 | hi | True | 5.3 / 5.0 | 0.003 / 0.01 |
| `9F6-cKqCvKw` | non-en | Y | Y | 22 | 22 | 1.0 / 1.0 | 1.0 | kn | True | 4.4 / 5.2 | 0.003 / 0.01 |
| `t5Y2CwCsnbA` | non-en | Y | Y | 9356 | 9356 | 1.0 / 1.0 | 1.0 | es | False | 2.9 / 5.3 | 0.003 / 0.01 |
| `s49L87BUsfo` | long | Y | Y | 32235 | 32235 | 1.0 / 1.0 | 1.0 | en | True | 3.0 / 7.7 | 0.003 / 0.01 |
| `l8pRSuU81PU` | long | Y | Y | 43505 | 43505 | 1.0 / 1.0 | 1.0 | en | True | 8.2 / 8.0 | 0.003 / 0.01 |
| `ez773teNFYA` | long | Y | Y | 34441 | 34441 | 1.0 / 1.0 | 1.0 | en | True | 4.2 / 6.6 | 0.003 / 0.01 |
| `DeebhvBEUqA` | private | N: row "This video is private" | N: no row; run FAILED "Temporary error (TIMEOUT)" | 0 | - | - | - | - | - | 13.9 / 30.9 | 0 / 0 |
| `-P-zEwwOMx4` | removed | N: row "This video is unavailable" | N: no row; run SUCCEEDED, statusMessage VIDEO_UNAVAILABLE | 0 | - | - | - | - | - | 2.5 / 9.1 | 0 / 0 |

## Spend

- **This test (attributed per run): $0.448.** Single runs cost $0.364 (ours 28 × $0.003 = $0.084; leader 28 × $0.010 = $0.280). The ours batch runs cost $0.084. The hard cap was $1.00.
- **`/v2/users/me/limits`:** `monthlyUsageUsd` was 5.1988 before and 6.2114 after the single runs, a **delta of $1.0126**. This delta is **not** all from this test. In the same window (08:37–08:50Z) the same org account ran 22 runs of `memo23/linkedin-company-employees-scraper` (not mine) costing $0.867, and one more was still running. The limits snapshot was taken before the batch runs, so add $0.084 for those. Per-run attribution: `runs/*/*.json` (`run.usageTotalUsd`, `run.chargedEventCounts`).
- The listing price in `our-vs-leader-2026-10-20.tsv` ($0.0033) is our FREE-tier price. This STARTER account paid the BRONZE tier price, $0.003. The leader charges a flat $0.01 on every tier.

## Limits of this test

- A single pass at one time of day. The leader ran on its default 1024 MB and ours on 512 MB, both Actor defaults.
- Neither Actor was tested on a video with **no** captions. Ours has an optional paid Whisper fallback (`transcribeIfNoCaptions`) and the leader has none, but that was not measured here.
- Both Actors read YouTube's caption tracks, so "quality" here can only mean "returns the track". Text accuracy is YouTube's and is the same for both.

## Files

All under `/home/a/Projects/swarm/agents/hustle/reports/market/h2h-youtube-transcript/`:
- `inputs.json`: the 30 inputs with reason, source and track probe.
- `candidate-pool-probe.json`: the 190-id pool probe.
- `reference-tracks.json`: the independent caption fetch.
- `runs/ours/*.json`, `runs/leader/*.json`, `runs/ours-batch/*.json`: input, cap, run object (settled, plus `run_at_capture`), dataset items, log tail and client timing.
- `per-video.json`, `summary.json`: the computed metrics.
- `limits-before.json`, `limits-after.json`: account usage snapshots.
- `scripts/`: `runner.py`, `analyse.py`, `ref2.py`, `tracks.py`, `apify.py`. The token is read from `.secrets/apify-org.env` at run time and is never written to disk.
