# Head-to-head: mathtec/youtube-comments-scraper (ours) vs streamers/youtube-comments-scraper (Store leader), 2026-11-09

Bead: hustle-qtk. Purpose: find out whether our Store listing can honestly say "better", and if so where.

Leader confirmed from the public Store API on 2026-11-09 (search "youtube comments", sorted by 30-day users): **streamers/youtube-comments-scraper, 3,992 users**. Second is apidojo/youtube-comments-scraper with 1,264 users (`h2h-google-trends/actors/store-search-2026-11-09.json`). Ours had 0 users in 30 days.

## Outcome first

- **Completeness against YouTube's displayed count: equal (n=5 videos where all comments fit under the cap).** Both returned **85 of 86** displayed comments (replies included). Each missed one **different** reply: ours missed 1 of 8 replies on `lyRu0GOO4uQ` (30/31), and the leader missed 1 of 11 replies on `UQWGWpzZ9RU` (33/34). Both got 1/1 and 20/20 on the other two videos.
- **Same comments: 238 of 245 ids shared.** Of the 7 ids each returned that the other did not, 1 per side is the missed reply above. The other 12 come from how each Actor fills a capped page (below). Likes were identical on 238/238 shared comments.
- **Text: equal except @-mention spacing (4/238 differ).** On all 4, YouTube puts the mention inside non-breaking spaces. The leader keeps them (`\xa0@Crypto_Current\xa0Rick, can I…`). Ours drops them and **glues the mention to the next word** (`@Crypto_CurrentRick, can I…`), which is a readability defect in ours. The other 234 texts are identical.
- **Capped videos: different composition, same size.** On the 4 videos with more than 40 comments, both returned exactly 40 rows. Ours puts replies inline, right after their parent, and counts them towards the cap. The leader returns all top-level comments first and replies last (seen on `zT6lUZVSSl4`: 15 comments, then 5 replies), so a capped leader run had 0 replies. On `bRQEWPA832A` that gave ours 35 comments + 5 replies and the leader 40 comments. Neither is wrong; the user gets a different 40.
- **Comments off / zero comments: ours clearer and cheaper.** Ours writes a row `status: empty` with **"comments are turned off for this video (or it is private or members-only)"** on the comments-off video, and **"this video has no comments"** on the video with comments on and 0 comments. Each costs only its $0.0007 start fee. The leader writes the **same** row on both, `error: NO_RESULTS` with the note *"if you want to collect videos, set limits above 0"*, which is misleading because limits were 40. It charged **2 result events ($0.003)** for each.
- **Price: ours is lower.** On this STARTER (BRONZE) account: ours **$0.0315 for 245 comments = $0.000129 per comment** ($0.0001 per comment + $0.0007 start per run); leader **$0.3735 = $0.001524 per comment** ($0.0015 per comment, no start fee, plus the 4 charged events on the 2 empty videos). So ours costs **0.085x as much** (about 1/12). Per 40-comment video: ours $0.0047 vs leader $0.06.
- **Speed: ours faster.** Run seconds per single-video run: median **3.8 s ours vs 12.3 s leader**, p90 6.3 vs 15.3, max 19.3 vs 15.4 (n=10 each). Ours' 19.3 s was on `bRQEWPA832A`.
- **Fields: each has some the other lacks.** **Leader only:** `commentsCount`, the video's exact total comment count on every row (2,513 and 2,461,578 where YouTube displays "2.5K" and "2.4M"); `hasCreatorHeart` (ours has `isHearted`). **Ours only:** `authorChannelId`, author channel URL and avatar, `isPinned`, `isEdited`, `hasImage`, `likesIsApprox`/`likesText`, `rank`, `commentUrl`, video title and channel, and a per-row `status`. Both have reply count, parent id and author-is-owner.

**Claims the numbers support for the listing:**
- "Same comments as the category leader at about 1/12 of the price" (85/86 vs 85/86 of displayed counts; 238/245 shared ids; likes identical).
- "Replies come right under their comment" (an ordering fact: every reply directly follows its parent on all 5 videos where ours returned replies).
- "Comments-off and no-comment videos return a row saying which, and only the run start is charged" (n=1 each; the leader returned the same error for both and charged 2 events each).
- "Author channel id and pinned flag on every row" (fields the leader lacks).

**Claims the numbers do NOT support:**
- "More complete": 85/86 each.
- "Exact text": ours drops the spacing around @-mentions on 4/238 comments, gluing the handle to the next word.
- "Includes the video's total comment count": the leader has it and ours does not.

**What the test says to fix (facts for whoever owns the Actor):** mention runs lose their surrounding non-breaking spaces, so `@handle` is joined to the next word (4 cases on 2 videos). One reply out of 8 on `lyRu0GOO4uQ` was not returned (`UgwiUkoLA2wlfnrMKnV4AaABAg.ANiyygGDP7nAP892dCkb-3`, whose parent was returned).

## Method

1. **Inputs (10 videos).** Small CryptoCurrent videos (20, 31 and 34 displayed comments), a Wolfram R&D video with 1, and a CoinGeek video with comments on and 0 comments, all from the digest channels (`services/digest/channels.json`). Also a CoComelon (made-for-kids) video whose page says "Comments are turned off.", a medium 3Blue1Brown video (265), a Veritasium Short (580, given as a `/shorts/` link), Karpathy's "How I use LLMs" (2.5K) and Rick Astley (2.4M, ours' prefill example). The list and reasons are in `inputs.json`.
2. **Reference.** From withheld (residential IP), before and after the runs (`scripts/ccprobe.py` → `ref/probe-pre.json`, `ref/probe-post.json`), I read each watch page's `ytInitialData`: the comments panel's displayed count, the comment section and its "Comments are turned off" message. I then made the first comments continuation request (innertube WEB `/next`) to confirm whether a comment box and threads exist. The CoinGeek video showed a comment box and 0 threads, which means comments on and zero comments. Pre and post counts were the same on every video. The one exception is `bRQEWPA832A`, where the pre-run fetch got a page variant without the comments panel; it showed 265 both in the selection probe and post-run. YouTube's displayed count includes replies.
3. **Runs.** All runs came from the Hustle org (`mathtechustle`, STARTER/BRONZE). This org does not own `mathtec/youtube-comments-scraper`, so it pays the public price. Each video got one single-video run per Actor, started at the same moment. Every run was async with `maxTotalChargeUsd` (ours $0.02; leader $0.50, its Store minimum, so real spend was bounded by `maxComments` and by a client guard that aborts above 50 rows). Default memory (ours 256 MB, leader 1024 MB) and build; no Actor settings or prices changed. A pilot on `zT6lUZVSSl4` confirmed that the leader returns replies (5/20), so ours ran with replies on.
4. **Equivalent settings** (exact inputs are in each `runs/*/<videoId>.json`):

   | Actor | input |
   |---|---|
   | ours | `videos: [url]`, `maxCommentsPerVideo: 40`, `sort: "newest"`, `includeReplies: true`, `includeVideoTitle: true` |
   | leader | `startUrls: [{url}]`, `maxComments: 40`, `sortCommentsBy: "NEWEST_FIRST"` |

5. **Metrics.** Completeness = unique comment ids / min(displayed, 40), on the 5 videos whose exact displayed count is ≤ 40. Shared = the same comment id from both. $ is `usageTotalUsd` per run after the runs settled; on every run it equals charged events × event price. Seconds are `stats.runTimeSecs`.

## Results

| Actor | comment rows | full-count videos: got / displayed | capped videos at 40 | $ total | $ per comment | $ per video | median s | p90 s | max s |
|---|---|---|---|---|---|---|---|---|---|
| ours | 245 | 85 / 86 | 4/4 | 0.0315 | 0.000129 | 0.00315 | 3.8 | 6.3 | 19.3 |
| leader | 245 | 85 / 86 | 4/4 | 0.3735 | 0.001524 | 0.03735 | 12.3 | 15.3 | 15.4 |

Shared ids 238; only ours 7; only leader 7; text equal on 234/238 shared; likes equal on 238/238.

## Per video

| videoId | category | displayed (pre / post) | expected rows (min(displayed, 40)) | ours rows (replies) | leader rows (replies) | shared ids | only ours / only leader | text equal on shared | $ ours / leader | run s ours / leader |
|---|---|---|---|---|---|---|---|---|---|---|
| `oosKQjEhANI` | tiny (1 displayed) | 1 / 1 | 1 | 1 (0) | 1 (0) | 1 | 0 / 0 | 1/1 | 0.0008 / 0.0015 | 3.1 / 11.5 |
| `zT6lUZVSSl4` | small (20) | 20 / 20 | 20 | 20 (5) | 20 (5) | 20 | 0 / 0 | 20/20 | 0.0027 / 0.03 | 4.0 / 13.0 |
| `lyRu0GOO4uQ` | small (31) | 31 / 31 | 31 | 30 (7) | 31 (8) | 30 | 0 / 1 | 29/30 | 0.0037 / 0.0465 | 3.2 / 15.3 |
| `UQWGWpzZ9RU` | small (34) | 34 / 34 | 34 | 34 (11) | 33 (10) | 33 | 1 / 0 | 30/33 | 0.0041 / 0.0495 | 3.6 / 15.4 |
| `lge1uHCO2_0` | zero (comments on, 0 comments) | - / - (comment box, 0 threads) | 0 | 0: `empty` "this video has no comments" | 0: "NO_RESULTS" ("…set limits above 0") | 0 | 0 / 0 | - | 0.0007 / 0.003 | 6.1 / 14.7 |
| `Fe5UG6MSy_8` | comments off | - / - ("Comments are turned off.") | 0 | 0: `empty` "comments are turned off for this video (or it is private or members-only)" | 0: "NO_RESULTS" ("…set limits above 0") | 0 | 0 / 0 | - | 0.0007 / 0.003 | 5.9 / 5.0 |
| `bRQEWPA832A` | medium (265) | variant page / 265 | 40 | 40 (5) | 40 (0) | 35 | 5 / 5 | 35/35 | 0.0047 / 0.06 | 19.3 / 8.1 |
| `rL0qEmTRI-I` | Short (580) | 580 / 580 | 40 | 40 (1) | 40 (0) | 39 | 1 / 1 | 39/39 | 0.0047 / 0.06 | 6.3 / 15.3 |
| `EWvNQjAaOHw` | big (2.5K) | 2.5K / 2.5K | 40 | 40 (0) | 40 (0) | 40 | 0 / 0 | 40/40 | 0.0047 / 0.06 | 2.2 / 8.9 |
| `dQw4w9WgXcQ` | huge (2.4M) | 2.4M / 2.4M | 40 | 40 (0) | 40 (0) | 40 | 0 / 0 | 40/40 | 0.0047 / 0.06 | 2.7 / 8.9 |

The only-ours / only-leader ids on `bRQEWPA832A` and `rL0qEmTRI-I` are ours' inline replies against the leader's next top-level comments. Their parents are in both outputs.

## Spend

- **This test (attributed per run): $0.405.** Ours $0.0315 + leader $0.3735. The three h2h tests in this batch came to $0.8734 together, against a hard cap of $1.50 shared across all three.
- **`/v2/users/me/limits`:** one before/after pair covers the whole three-test batch (`h2h-google-trends/limits-before.json`, `limits-after.json`): a delta of $0.8765, against $0.8734 summed over the 104 run ids. No other org runs started in the window, so $0.003 is unattributed platform usage.

## Limits of this test

- Full-count completeness was measured on only 5 small videos (86 comments). Deep pagination on large threads (thousands of comments, long reply chains) was not tested, and that is where scrapers usually lose rows.
- One sort order (newest) and one cap (40). "Top" sort was not compared.
- Single-video runs. Ours' $0.0007 start fee is amortised in multi-video runs and was not measured that way here. The leader has no start fee.
- One comments-off video and one zero-comment video.

## Files

All under `/home/a/Projects/swarm/agents/hustle/reports/market/h2h-youtube-comments/`:
- `inputs.json`: the 10 videos, categories, reasons, sources and the pre-run probe.
- `ref/cand-probe.json`, `ref/kids-ids.txt`, `ref/cand-ids.txt` (selection), `ref/probe-pre.json`, `ref/probe-post.json`: the independent watch-page and comments-continuation fetches from withheld.
- `actors/*.json`: each Actor's metadata, pricing and latest-build input schema at test time.
- `runs/{ours,leader}/<videoId>.json`: input, cap, settled run object (`run`, plus `run_at_capture`), dataset items, log and client timing.
- `per-video.json`, `summary.json`: the computed metrics.
- `scripts/`: `apify.py`, `runlib.py`, `ccprobe.py`, `runner.py`, `analyse.py`. The token is read from `.secrets/apify-org.env` at run time and is never written to disk.
