mikkeli/brfc-stream-pirate
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Cut the README to what you need to run this

Browse Source 
It had grown into a changelog: how each fact was discovered, which earlier
claim it overturned, what the sample size was at the time. Useful while
learning the system, noise once you just want to record a match. Rewritten as
a reference, 295 lines to ~195, losing the archaeology and keeping the facts.

Conclusions that were buried in narrative are now rules. Sample-size markers
survive only where they change whether a claim can be leaned on.

Documents --feed, including that it is the setting that matters and that
getting it wrong fails silently.

Two corrections while here. Of five observed endings, three self-terminated
and two ran forever, across both feed types with nothing predicting which --
so the cap stays, but "the stream never ends" was too strong. And preflight
no longer claims to mint a JWT: /oauth/browser-token hands back the session's
existing token, so the expiry it prints is not a freshness signal. The check
is unaffected, since it rests on the 403 a dead session returns.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fr1fLMnVGNAKH5L2BmhMxn
This commit is contained in:
Mikkeli Matlock
2026-09-26 22:33:49 +09:00
parent 847fe1e13f
commit 9c22a02fab
2 changed files with 161 additions and 254 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+150 -250
View File
@@ -11,285 +11,185 @@ makes, using your own paid session.
| File | Job |
|---|---|
| `record_live.py` | Unattended recorder: finds the live entry itself, mints a master, records |
| `preflight.py` | Pre-match check: does the session still mint a JWT, can this host record |
| `watchdog.py` | Backup starter: if nothing is *progressing*, kill the stale recorder and start one |
| `tampermonkey/brfc-stream-sniffer.user.js` | Hooks fetch/XHR in the page, captures the manifest URL verbatim, POSTs it to the catcher |
| `catcher.py` | Listens on 127.0.0.1:8787, hands the URL to yt-dlp, records to `recordings/` |
| `discover.py` | Drives the player with Playwright to capture the token-minting traffic |
| `record_live.py` | Unattended recorder: finds the live entry, mints a master, records |
| `preflight.py` | Pre-match check: is the session live, can this host record |
| `watchdog.py` | Backup starter: if nothing is *progressing*, start a recorder |
| `catcher.py` | Listens on 127.0.0.1:8787, hands a sniffed URL to yt-dlp |
| `tampermonkey/brfc-stream-sniffer.user.js` | Hooks fetch/XHR, POSTs the manifest URL to the catcher |
| `discover.py` | Drives the player with Playwright to capture token-minting traffic |
| `probe.py` | Byte-exact manifest fetch + concurrency test |
## Recording a match
python preflight.py # exit 0 only if the session is live
Mandatory, hours before kickoff. A dead session is invisible locally and the
only fix needs a human at a browser. Then schedule the recorder for ~30 min
before kickoff:
python record_live.py --max-wait 75 --duration 180 --feed video
**Pick the feed, not just the rendition.** The live page can expose two entries
for one match -- an audio feed and a video feed -- and which manifest loads
first is a race that has come out differently on the same night. `--feed`
chooses between entries and is the setting that matters:
| `--feed` | Behaviour |
|---|---|
| `video` (default) | record a video feed; fall back to audio if none is offered |
| `audio` | the reverse |
| `any` | whichever was discovered first -- the old behaviour |
| `both` | record every feed in parallel |
| `--entry <id>` | skip discovery entirely and pin one |
Feeds are classified by the codecs in their master, never by entry id. A
non-preferred feed stays on the list as a fallback, because a feed you hold no
pass for resolves a master fine and only fails later on the segments; the
recorder falls through to the next candidate when a download dies immediately.
`--prefer` is the *other* choice -- which rendition inside one master -- and
`best` is almost always right. A wrong `--feed` cannot be rescued by `--prefer`,
and it fails quietly: a complete, playable, healthy recording of the wrong
thing. `--feed both` costs ~210 MB extra and removes the question.
**Warm the scheduled task.** After registering it, launch it once by hand and
stop it again. A freshly registered task has been seen to hang at interpreter
startup -- alive, zero CPU, nothing logged, frozen before any of our code runs.
Relaunching fixed it; root cause unknown. Stopping the test run matters as much:
`MultipleInstances IgnoreNew` means a test left running swallows the real
trigger silently.
Optionally schedule `watchdog.py` ~20 min later as a backup. It checks whether
recording is *progressing* (a growing `.ts`, or a freshly written log), not
whether a process exists -- a hung recorder passes the process check.
### Manual alternative
python catcher.py # leave running
Install the userscript, open the match page, press play. The sniffer catches the
manifest, the catcher starts yt-dlp. Config via environment:
| Variable | Default | Effect |
|---|---|---|
| `BRFC_BROWSER` | `firefox` | which browser's cookie jar yt-dlp reads |
| `BRFC_DURATION_MIN` | `150` | catcher's recording cap, minutes |
Check concurrency against a live token with `probe.py --url "<manifest>"
--parallel 3`.
## The token
Manifest URLs carry an Akamai path token:
hdntl=exp=<unix>~acl=/*~id=<n>~data=hdntl~hmac=<sig>
- `exp` -- ~24h validity
- `acl=/*` -- wildcard over the whole host, not just your match
- `hmac` -- signature over the literal path
It is a **bearer credential** with `acl=/*` over the whole host, valid to `exp`.
Anyone holding a live URL can pull the stream. Keep it out of pastebins and
issue trackers; `stream-url.txt` and `cookies.txt` are gitignored.
It is a bearer credential. Anyone holding a live URL can pull the stream until
`exp`. Keep it out of pastebins and issue trackers. `stream-url.txt` and
`cookies.txt` are gitignored.
Two tokens, not one:
**Do not re-encode the path.** curl normalises `%2f` to `%2F`, which
invalidates the HMAC. `probe.py` sends the request target verbatim.
## Use Firefox for watching
yt-dlp lifts session cookies from the browser. Chrome 127+ on Windows uses
app-bound cookie encryption and extraction fails. Firefox works with no fuss.
set BRFC_BROWSER=firefox # default
## Workflow
pip install yt-dlp # done
python catcher.py # leave running
Install the userscript in Tampermonkey, open the match page, press play. The
sniffer catches the manifest, the catcher starts yt-dlp. Recording is MPEG-TS
(`--hls-use-mpegts`) so a killed process still leaves a playable file.
For a 3am kickoff: Task Scheduler opens the match page a few minutes before,
the browser autoplays, the sniffer fires, the catcher records. Nobody is awake.
### Before every match
python preflight.py # exit 0 only if the server minted a JWT
Not optional and not a formality. The session can be dead while every local
signal looks fine (see "The cookie expiry is not the session lifetime"), and the
only fix -- signing in by hand in Firefox -- needs a human and some lead time.
Run it hours before kickoff.
Then register the task, launch it once by hand, and stop it again before the
real trigger (see "a freshly registered task can hang on its first launch").
## Concurrency check
An Akamai edge token is a stateless HMAC -- the edge validates signature and
expiry and serves; it holds no session table, so it cannot count viewers. Any
concurrent-stream limit lives in the entitlement API that *issues* tokens,
upstream of the CDN. Confirm rather than assume, while the browser is playing:
python probe.py --url "<manifest from the sniffer>" --parallel 3
3/3 alongside the playing browser means four concurrent readers of one token
and the edge does not care.
---
## Findings (live matches, 2026-08-29 onward)
Verified against live streams, starting with a 23:00 JST kickoff on 2026-08-29
that recorded 1080p end to end. Later matches are dated where they corrected
something.
### Two token types, not one
The player uses a **two-stage token chain**:
| Token | Where | TTL | Purpose |
| Token | Where | TTL | Fetches |
|---|---|---|---|
| `hdnts=` | query string, `acl=/*` unencoded | **~25 seconds** | fetch the master playlist |
| `hdntl=` | URL path, `acl=%2f*` encoded | **~24 hours** | media playlists + segments |
| `hdnts=` | query string, `acl=/*` unencoded | ~25 seconds | master playlist |
| `hdntl=` | URL path, `acl=%2f*` encoded | ~24 hours | media playlists, segments |
The master response rewrites each variant URL with a 24h `hdntl` token. So the
`hdntl` token is the durable one worth handing to a downloader. Chasing a
`hdnts` URL by hand is pointless -- it dies before you can paste it.
The master response rewrites each variant URL with a 24h `hdntl`. Chasing a
`hdnts` URL by hand is pointless -- it expires before you can paste it.
### Confirmed by experiment
**Do not re-encode the path.** curl normalises `%2f` to `%2F`, invalidating the
HMAC. `probe.py` sends the request target verbatim.
- **Concurrency is not limited.** Two independent ffmpeg downloads ran at full
bitrate *while Firefox played the same match*. Three concurrent readers of
one token, no complaint from the edge. Akamai token auth is a stateless HMAC
check; it has no viewer counter.
- **No CDN cookies exist.** All 13 relevant cookies are on `bristolrovers.co.uk`;
nothing is scoped to `akamaized.net`. The token in the URL is the whole auth.
- **Stream is 1920x1080 H.264 + AAC ~5.2 Mbit/s** -> roughly 4.7 GB per match.
- **Output is MPEG-TS** even though yt-dlp names it `.mp4`. That is what
`--hls-use-mpegts` buys: a killed process still leaves a playable file.
- Player is **Bitmovin**, embedded on the club site. `0_xm5vrrw6` is a Kaltura
entry id. It does **not** identify the match -- see below.
## Rules that are not preferences
### The entry id names the feed, not the match
- **Hand the downloader a master URL, not a media one.** Python gets 403 on
path-token media playlists; ffmpeg gets 200. yt-dlp fetches playlists with
Python before ffmpeg runs, so it must be given a `hdnts` master.
`record_live.py` mints one immediately before recording.
- **Force `--downloader ffmpeg`.** yt-dlp's native HLS downloader dies on the
path-token URLs for the same reason.
- **Keep the duration cap.** A feed can keep serving after full time -- a static
slate on video, digital silence on audio -- at full bitrate, with no
`#EXT-X-ENDLIST` and no stall to detect. Of five observed endings three
self-terminated and two ran forever, on both feed types, with no pattern that
predicts which. Treat a stream that stops by itself as luck.
- **Use Firefox as the session holder.** Chrome/Edge lock their cookie DB while
running (yt-dlp issue 7271) and Chrome 127+ on Windows adds app-bound cookie
encryption; an unattended job cannot close a browser to read them.
- **Do not judge a token by `probe.py`.** It inherits the path-token 403, so a
live token looks dead through it.
Disproved 2026-09-19. Four observations, two ids:
## Facts
| Entry id | Seen | Feed |
|---|---|---|
| `0_xm5vrrw6` | 2026-08-29, 2026-09-19 | video |
| `0_z236v0e7` | 2026-09-05, 2026-09-12 | audio |
- 1920x1080 H.264 + AAC ~5.2 Mbit/s. Audio-only feeds are ~140 kbps AAC.
- Output is MPEG-TS even when named `.mp4` -- that is what `--hls-use-mpegts`
buys: a killed process still leaves a playable file.
- Budget **~36 MB/min video** (~6.5 GB for a 180 min cap), **~1.2 MB/min audio**
(~210 MB).
- **Concurrency is not limited.** An Akamai edge token is a stateless HMAC with
no viewer counter. Three readers of one token ran at full bitrate alongside a
playing browser.
- **No CDN cookies exist.** All relevant cookies are on `bristolrovers.co.uk`,
none on `akamaized.net`. The token in the URL is the whole auth.
- Player is Bitmovin. The Kaltura entry id names the **feed, not the match**:
`0_xm5vrrw6` (video) and `0_z236v0e7` (audio) each recurred weeks apart.
Nothing may use it to answer "have I recorded this game" -- use the date.
`catcher.py` is unaffected; it dedupes on liveness, not identity.
Three weeks apart, the same id came back for the video feed. So the id is a
channel: it distinguishes the video feed from the audio one and nothing more.
## The session
`catcher.py` is not broken by this, because it dedupes on *liveness* -- "is a
downloader for this key still alive" -- and the key clears when the downloader
hits its cap. But do not build anything that treats the id as a match
identifier: "have I already recorded this game" cannot be answered from it, and
neither can "is this a new match or a reload". Use the kickoff date for that.
The `SSESS` cookie is the whole login. Everything downstream -- JWT, entry API,
master playlist -- rides it, in plain Python, with no browser or fingerprint
forging.
### Gotcha: which URL form a Python client can fetch
**Cookie expiry is not session lifetime.** The server can drop a session while
the browser holds a cookie that looks perfectly healthy:
Earlier notes here blamed a TLS handshake difference. That was wrong. The 403
splits by **token form**, not by client:
SSESS... 76 days left on the clock
GET /oauth/browser-token -> 403 {"error":"Unauthenticated"}
Set-Cookie: SSESS...=deleted; expires=Thu, 01 Jan 1970
| URL form | Python | ffmpeg |
|---|---|---|
| `index.m3u8?hdnts=...` (master, query token) | 200 | 200 |
| `hdntl=.../index_N.m3u8` (media, path token) | 403 | 200 |
One session died after six days; its replacement survived seven and counting.
The lifetime is unpredictable and only the server knows it, so a local cookie
check proves nothing and `preflight.py` is not a formality.
Consequences:
Other session behaviour:
- Mint a **master** URL and hand that to the downloader. `record_live.py` does
this: it calls the entry API for a fresh master immediately before recording.
- Force `--downloader ffmpeg`. yt-dlp's native HLS downloader fetches variant
playlists with Python and dies on the path-token URLs. This is required, not
a preference.
- `probe.py` is diagnostic scaffolding and inherits the path-token 403 -- do
not use it to conclude a token is dead.
- **Sessions are concurrent.** Logging in via a second browser does not
invalidate the first -- Drupal issues a session per browser. So a second
device can hold its own session without disturbing the recorder's. *One
observation, checked immediately after the second login.*
- `/oauth/browser-token` returns the session's **existing** JWT rather than
minting a fresh one per call, so the token's expiry is not a freshness
signal. A dead session returns 403, which is what the check relies on.
- The cookie *value* changed once between weeks with no re-login recorded.
Unexplained. Treat a copied cookie as a snapshot that can go stale on its own.
- The session cookie is **httpOnly**, so `document.cookie` cannot read it. Any
provisioning scheme must use a platform cookie store, or it will silently
collect every cookie except the one that matters.
### Firefox, not Chrome
### Decision: do not automate login
yt-dlp reads cookies from a running Firefox. Chrome/Edge lock their cookie DB
while running (yt-dlp issue 7271), and an unattended job cannot close the
browser to read them.
Automating it is the riskiest step in the chain and the only one needing a
stored password, against an unknown `captcha_token` and a lockout that would
take out the working session too. Instead assume the session can be dead before
any given match and make `preflight.py` routine.
### The stream never ends
After full time the feed switches to a static league slate and keeps serving
indefinitely at full CBR bitrate (~4.7 Mbit/s to encode a still image). There
is no `#EXT-X-ENDLIST`, so ffmpeg never finishes, and there is no stall to
detect either -- from the downloader's point of view it is a healthy live
stream forever.
**Keep the duration cap.** `catcher.py` and `record_live.py` pass
`ffmpeg_o:-t $DURATION_MIN*60`. Without it, a feed that behaves like the video
one above fills the disk with a logo.
This holds for audio too, which earlier notes here doubted. On 2026-09-12 the
audio feed for a 20:30 kickoff went to **digital silence** -- `volumedetect`
reports -91.0 dB on both mean and max, the noise floor, not a quiet passage --
at 02:35:20 into the file, and then kept serving that silence at a steady
156 kbps until the 180 min cap fired at `rc=0`. Nothing about the stream said
it was over.
So of three observations, two feeds ran forever and one did not:
| Date | Feed | Ended how |
|---|---|---|
| 2026-08-29 | video | never; stopped manually at 116 min |
| 2026-09-02 | audio | self-terminated, `rc=0` at 165 min |
| 2026-09-12 | audio | never; silence to the 180 min cap |
The 2026-09-02 self-termination is now the outlier, not the rule, and it is the
only one of the three. **The cap is load-bearing on both feed types.** Treat a
stream that stops by itself as luck.
Budget ~36 MB/min for video (measured 2026-09-19: 1080p, 60 s -> 36 MB), so
~6.5 GB for a 180 min cap. Audio is ~1.2 MB/min, ~210 MB for the same cap.
---
Cookie *provisioning* to a second host remains the open problem.
## Status
Unattended recording works. A scheduled task started `record_live.py` at 03:15
for an 03:45 kickoff, found the stream in 16 seconds, and recorded 165 min of
audio to a playable MPEG-TS with nobody awake.
Unattended recording works end to end, scheduled, with nobody awake. Five
matches recorded as of 2026-09-20, including one night capturing both feeds in
parallel.
Sample sizes are small but no longer one: four recorded matches as of
2026-09-19, two video and two audio. Two earlier claims here have since been
disproved outright -- the entry id does not identify the match, and the audio
feed does not reliably end itself. Both are corrected above. Assume the next
claim in this file is the one about to fall over.
**Verified:** the token chain in plain Python; recording both feed types via
ffmpeg; Task Scheduler launching the job; concurrent readers of one token; a
dead session being detectable before kickoff; the duration cap firing at `rc=0`
on a feed that would not stop.
### Verified
- Token chain end to end in plain Python: session cookie -> JWT -> entry API
-> master playlist. No browser, no fingerprint forging.
- Recording via ffmpeg, video (1080p) and audio (~122 kbps AAC).
- Scheduled unattended run, including that Task Scheduler launches it correctly.
- Concurrency: several readers of one token, alongside a playing browser.
- A dead session is detectable before kickoff, cheaply and read-only
(`preflight.py`), and distinguishable from a blocked request by the
`{"error":"Unauthenticated"}` body and the cookie-deleting `Set-Cookie`.
- The duration cap firing cleanly at `rc=0` on a feed that would not stop.
### Gotcha: a freshly registered task can hang on its first launch
Seen 2026-09-12. The first launch of a newly registered scheduled task hung
dead at interpreter startup -- process alive, 0.015 s of CPU and 21 modules
loaded after three minutes, nothing written to any log, a `conhost` child
allocated. Not a crash and not a loop: frozen before Python ran any of our
code. Killing it and launching the identical task again worked immediately and
it has not recurred. Root cause unknown.
Unattended, that is a silently lost match. So after registering a task, **launch
it once by hand and stop it again**, which makes the real firing a warm start.
Stopping matters as much as starting: `MultipleInstances IgnoreNew` means a test
run left alive will swallow the scheduled trigger without a word.
### Not verified
- **Login.** Everything above rides an `SSESS` cookie lifted from a desktop
Firefox profile. Obtaining a session *from credentials* has never been run.
`/user/login` is a standard Drupal form (`name`, `pass`, `form_build_id`,
`form_id`, `captcha_token`) with no reCAPTCHA script and no 2FA mention, but
`captcha_token` is an unknown and testing it risks a lockout that would take
out the working session too.
### The cookie expiry is not the session lifetime
Disproved 2026-09-12, and this one bites. The `SSESS` cookie carries a ~90 day
expiry, and earlier notes here read that as the session lifetime. It is not.
The server can drop the session while the browser still holds a cookie that
looks perfectly healthy:
SSESSaa90be30f47... 76 days left on the clock
GET /oauth/browser-token -> 403 {"error":"Unauthenticated"}
Set-Cookie: SSESSaa90be30f47...=deleted; expires=Thu, 01 Jan 1970
That session had last worked six days earlier, at the end of the 2026-09-06
recording. Nothing local showed it: right expiry, right domain, right profile.
Only asking the server revealed it.
Then the *replacement* session survived the next seven days fine. So the
lifetime is not a week either -- it is unpredictable, and somewhere between six
days and the 90 the cookie advertises.
Two consequences:
- **A local cookie check proves nothing.** Presence and expiry are necessary,
not sufficient. `preflight.py` exists for this: it spends one request asking
the server to mint a JWT, which is the only answer that counts. Run it hours
before kickoff, not minutes.
- The cookie *value* also changed between 2026-09-12 and 2026-09-19 with no
re-login recorded (48 -> 50 chars). Unexplained. Treat a copied cookie as a
snapshot that can go stale on its own, which matters for provisioning one to
another host.
### Decision: still do not automate login, but detect it early
The old reasoning for this was arithmetic, and the arithmetic was wrong: it
assumed a ~87 day cookie meant about four manual logins a year. A session that
can die in six days means the refresh could be needed weekly, which is a real
cost rather than a rounding error.
The decision survives the correction anyway, for the reason that was always the
strong one: automating login is the riskiest step in the chain and the only one
needing a stored password, against an unknown `captcha_token` and a lockout that
would take out the working session. What changes is the mitigation. Do not rely
on remembering to refresh it quarterly -- assume it can be dead before any given
match, and make the check routine:
python preflight.py # exit 0 only if the server minted a JWT
Cookie *provisioning* remains the open problem for a second host, and the
rotating value above makes it harder, not easier.
Use Firefox as the session holder. Two independent reasons: its cookie DB stays
readable while running (Chrome/Edge lock theirs), and it holds the login
durably where Chrome has been observed dropping it within a day.
**Not verified:** obtaining a session *from credentials*. `/user/login` is a
standard Drupal form (`name`, `pass`, `form_build_id`, `form_id`,
`captcha_token`) with no reCAPTCHA script and no 2FA mention, but
`captcha_token` is unknown and probing it risks a lockout.