Mikkeli Matlock
84c29cc0b7
The TLS-handshake explanation for the 403 was wrong. The split is by token form, not by client: Python fetches the master (query token) fine and fails only on the media playlist (path token). That is why record_live.py mints a master and why --downloader ffmpeg is required. "The stream never ends" was one video observation stated as a rule, and an audio stream has since contradicted it by terminating cleanly well inside the cap. Both are now recorded as single observations rather than conclusions. Also records the state of play: unattended recording works, login has never been tested, and the decision not to automate it -- the cookie outlives a quarter, so automating the riskiest step to save four manual logins a year is a bad trade. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kyt7QHQvLJtQRuiY4wAqKB
186 lines
7.8 KiB
Markdown
186 lines
7.8 KiB
Markdown
# brfc-stream-pirate
|
|
|
|
Personal time-shifting archive for Bristol Rovers video pass streams.
|
|
Kickoffs land at unfriendly hours in JST; this records them unattended.
|
|
|
|
Stream is **plain HLS, no DRM** (StreamAMG behind Akamai). Nothing here breaks
|
|
a protection measure -- it replays the same authenticated request the player
|
|
makes, using your own paid session.
|
|
|
|
## Pieces
|
|
|
|
| File | Job |
|
|
|---|---|
|
|
| `tampermonkey/brfc-stream-sniffer.user.js` | Hooks fetch/XHR in the page, captures the manifest URL verbatim, POSTs it to the catcher |
|
|
| `catcher.py` | Listens on 127.0.0.1:8787, hands the URL to yt-dlp, records to `recordings/` |
|
|
| `probe.py` | Byte-exact manifest fetch + concurrency test |
|
|
|
|
## The token
|
|
|
|
Manifest URLs carry an Akamai path token:
|
|
|
|
hdntl=exp=<unix>~acl=/*~id=<n>~data=hdntl~hmac=<sig>
|
|
|
|
- `exp` -- ~24h validity
|
|
- `acl=/*` -- wildcard over the whole host, not just your match
|
|
- `hmac` -- signature over the literal path
|
|
|
|
It is a bearer credential. Anyone holding a live URL can pull the stream until
|
|
`exp`. Keep it out of pastebins and issue trackers. `stream-url.txt` and
|
|
`cookies.txt` are gitignored.
|
|
|
|
**Do not re-encode the path.** curl normalises `%2f` to `%2F`, which
|
|
invalidates the HMAC. `probe.py` sends the request target verbatim.
|
|
|
|
## Use Firefox for watching
|
|
|
|
yt-dlp lifts session cookies from the browser. Chrome 127+ on Windows uses
|
|
app-bound cookie encryption and extraction fails. Firefox works with no fuss.
|
|
|
|
set BRFC_BROWSER=firefox # default
|
|
|
|
## Workflow
|
|
|
|
pip install yt-dlp # done
|
|
python catcher.py # leave running
|
|
|
|
Install the userscript in Tampermonkey, open the match page, press play. The
|
|
sniffer catches the manifest, the catcher starts yt-dlp. Recording is MPEG-TS
|
|
(`--hls-use-mpegts`) so a killed process still leaves a playable file.
|
|
|
|
For a 3am kickoff: Task Scheduler opens the match page a few minutes before,
|
|
the browser autoplays, the sniffer fires, the catcher records. Nobody is awake.
|
|
|
|
## Concurrency check
|
|
|
|
An Akamai edge token is a stateless HMAC -- the edge validates signature and
|
|
expiry and serves; it holds no session table, so it cannot count viewers. Any
|
|
concurrent-stream limit lives in the entitlement API that *issues* tokens,
|
|
upstream of the CDN. Confirm rather than assume, while the browser is playing:
|
|
|
|
python probe.py --url "<manifest from the sniffer>" --parallel 3
|
|
|
|
3/3 alongside the playing browser means four concurrent readers of one token
|
|
and the edge does not care.
|
|
|
|
---
|
|
|
|
## Findings (2026-08-29, live 2300 JST match)
|
|
|
|
Verified against a live stream. Recorded 1080p successfully end to end.
|
|
|
|
### Two token types, not one
|
|
|
|
The player uses a **two-stage token chain**:
|
|
|
|
| Token | Where | TTL | Purpose |
|
|
|---|---|---|---|
|
|
| `hdnts=` | query string, `acl=/*` unencoded | **~25 seconds** | fetch the master playlist |
|
|
| `hdntl=` | URL path, `acl=%2f*` encoded | **~24 hours** | media playlists + segments |
|
|
|
|
The master response rewrites each variant URL with a 24h `hdntl` token. So the
|
|
`hdntl` token is the durable one worth handing to a downloader. Chasing a
|
|
`hdnts` URL by hand is pointless -- it dies before you can paste it.
|
|
|
|
### Confirmed by experiment
|
|
|
|
- **Concurrency is not limited.** Two independent ffmpeg downloads ran at full
|
|
bitrate *while Firefox played the same match*. Three concurrent readers of
|
|
one token, no complaint from the edge. Akamai token auth is a stateless HMAC
|
|
check; it has no viewer counter.
|
|
- **No CDN cookies exist.** All 13 relevant cookies are on `bristolrovers.co.uk`;
|
|
nothing is scoped to `akamaized.net`. The token in the URL is the whole auth.
|
|
- **Stream is 1920x1080 H.264 + AAC ~5.2 Mbit/s** -> roughly 4.7 GB per match.
|
|
- **Output is MPEG-TS** even though yt-dlp names it `.mp4`. That is what
|
|
`--hls-use-mpegts` buys: a killed process still leaves a playable file.
|
|
- Player is **Bitmovin**, embedded on the club site. `0_xm5vrrw6` is a Kaltura
|
|
entry id and identifies the match -- the catcher dedupes on it.
|
|
|
|
### Gotcha: which URL form a Python client can fetch
|
|
|
|
Earlier notes here blamed a TLS handshake difference. That was wrong. The 403
|
|
splits by **token form**, not by client:
|
|
|
|
| URL form | Python | ffmpeg |
|
|
|---|---|---|
|
|
| `index.m3u8?hdnts=...` (master, query token) | 200 | 200 |
|
|
| `hdntl=.../index_N.m3u8` (media, path token) | 403 | 200 |
|
|
|
|
Consequences:
|
|
|
|
- Mint a **master** URL and hand that to the downloader. `record_live.py` does
|
|
this: it calls the entry API for a fresh master immediately before recording.
|
|
- Force `--downloader ffmpeg`. yt-dlp's native HLS downloader fetches variant
|
|
playlists with Python and dies on the path-token URLs. This is required, not
|
|
a preference.
|
|
- `probe.py` is diagnostic scaffolding and inherits the path-token 403 -- do
|
|
not use it to conclude a token is dead.
|
|
|
|
### Firefox, not Chrome
|
|
|
|
yt-dlp reads cookies from a running Firefox. Chrome/Edge lock their cookie DB
|
|
while running (yt-dlp issue 7271), and an unattended job cannot close the
|
|
browser to read them.
|
|
|
|
### The stream never ends
|
|
|
|
After full time the feed switches to a static league slate and keeps serving
|
|
indefinitely at full CBR bitrate (~4.7 Mbit/s to encode a still image). There
|
|
is no `#EXT-X-ENDLIST`, so ffmpeg never finishes, and there is no stall to
|
|
detect either -- from the downloader's point of view it is a healthy live
|
|
stream forever.
|
|
|
|
**Keep the duration cap.** `catcher.py` and `record_live.py` pass
|
|
`ffmpeg_o:-t $DURATION_MIN*60`. Without it, a feed that behaves like the video
|
|
one above fills the disk with a logo.
|
|
|
|
Note the sample size. This is **one video observation**, and it is contradicted
|
|
by **one audio observation**: on 2026-09-02 an audio stream ended cleanly by
|
|
itself (`rc=0` at 165 min, against a 180 min cap that never fired). So "the
|
|
stream never ends" is a property of that video feed, not an established rule.
|
|
Treat the cap as a safety net rather than as the only stop condition.
|
|
|
|
Observed: video, 116 min recorded for a 23:00 kickoff, still running at 01:04
|
|
and stopped manually. Audio, 165 min, self-terminated. Budget ~3.5-4 GB per copy per match at 40 MB/min.
|
|
|
|
---
|
|
|
|
## Status
|
|
|
|
Unattended recording works. A scheduled task started `record_live.py` at 03:15
|
|
for an 03:45 kickoff, found the stream in 16 seconds, and recorded 165 min of
|
|
audio to a playable MPEG-TS with nobody awake.
|
|
|
|
Sample sizes are one. One video capture, one audio capture. Where the two
|
|
disagree (stream termination, whether an entry id persists between matches),
|
|
neither is established -- do not build on either without more matches.
|
|
|
|
### Verified
|
|
|
|
- Token chain end to end in plain Python: session cookie -> JWT -> entry API
|
|
-> master playlist. No browser, no fingerprint forging.
|
|
- Recording via ffmpeg, video (1080p) and audio (~122 kbps AAC).
|
|
- Scheduled unattended run, including that Task Scheduler launches it correctly.
|
|
- Concurrency: several readers of one token, alongside a playing browser.
|
|
|
|
### Not verified
|
|
|
|
- **Login.** Everything above rides an `SSESS` cookie lifted from a desktop
|
|
Firefox profile. Obtaining a session *from credentials* has never been run.
|
|
`/user/login` is a standard Drupal form (`name`, `pass`, `form_build_id`,
|
|
`form_id`, `captcha_token`) with no reCAPTCHA script and no 2FA mention, but
|
|
`captcha_token` is an unknown and testing it risks a lockout that would take
|
|
out the working session too.
|
|
|
|
### Decision: do not automate login
|
|
|
|
The cookie lasts ~87 days and matches are weekly, so automating login saves
|
|
about four manual actions a year -- while being the riskiest step and the only
|
|
one needing a stored password. Refresh the cookie by logging in manually on the
|
|
desktop each quarter and provisioning it to the recording host instead. That
|
|
makes cookie *provisioning* the open problem, not the captcha.
|
|
|
|
Use Firefox as the session holder. Two independent reasons: its cookie DB stays
|
|
readable while running (Chrome/Edge lock theirs), and it holds the login
|
|
durably where Chrome has been observed dropping it within a day.
|