diff --git a/README.md b/README.md index 7eaa7f4..4a58cae 100644 --- a/README.md +++ b/README.md @@ -11,285 +11,185 @@ makes, using your own paid session. | File | Job | |---|---| -| `record_live.py` | Unattended recorder: finds the live entry itself, mints a master, records | -| `preflight.py` | Pre-match check: does the session still mint a JWT, can this host record | -| `watchdog.py` | Backup starter: if nothing is *progressing*, kill the stale recorder and start one | -| `tampermonkey/brfc-stream-sniffer.user.js` | Hooks fetch/XHR in the page, captures the manifest URL verbatim, POSTs it to the catcher | -| `catcher.py` | Listens on 127.0.0.1:8787, hands the URL to yt-dlp, records to `recordings/` | -| `discover.py` | Drives the player with Playwright to capture the token-minting traffic | +| `record_live.py` | Unattended recorder: finds the live entry, mints a master, records | +| `preflight.py` | Pre-match check: is the session live, can this host record | +| `watchdog.py` | Backup starter: if nothing is *progressing*, start a recorder | +| `catcher.py` | Listens on 127.0.0.1:8787, hands a sniffed URL to yt-dlp | +| `tampermonkey/brfc-stream-sniffer.user.js` | Hooks fetch/XHR, POSTs the manifest URL to the catcher | +| `discover.py` | Drives the player with Playwright to capture token-minting traffic | | `probe.py` | Byte-exact manifest fetch + concurrency test | +## Recording a match + + python preflight.py # exit 0 only if the session is live + +Mandatory, hours before kickoff. A dead session is invisible locally and the +only fix needs a human at a browser. Then schedule the recorder for ~30 min +before kickoff: + + python record_live.py --max-wait 75 --duration 180 --feed video + +**Pick the feed, not just the rendition.** The live page can expose two entries +for one match -- an audio feed and a video feed -- and which manifest loads +first is a race that has come out differently on the same night. `--feed` +chooses between entries and is the setting that matters: + +| `--feed` | Behaviour | +|---|---| +| `video` (default) | record a video feed; fall back to audio if none is offered | +| `audio` | the reverse | +| `any` | whichever was discovered first -- the old behaviour | +| `both` | record every feed in parallel | +| `--entry ` | skip discovery entirely and pin one | + +Feeds are classified by the codecs in their master, never by entry id. A +non-preferred feed stays on the list as a fallback, because a feed you hold no +pass for resolves a master fine and only fails later on the segments; the +recorder falls through to the next candidate when a download dies immediately. + +`--prefer` is the *other* choice -- which rendition inside one master -- and +`best` is almost always right. A wrong `--feed` cannot be rescued by `--prefer`, +and it fails quietly: a complete, playable, healthy recording of the wrong +thing. `--feed both` costs ~210 MB extra and removes the question. + +**Warm the scheduled task.** After registering it, launch it once by hand and +stop it again. A freshly registered task has been seen to hang at interpreter +startup -- alive, zero CPU, nothing logged, frozen before any of our code runs. +Relaunching fixed it; root cause unknown. Stopping the test run matters as much: +`MultipleInstances IgnoreNew` means a test left running swallows the real +trigger silently. + +Optionally schedule `watchdog.py` ~20 min later as a backup. It checks whether +recording is *progressing* (a growing `.ts`, or a freshly written log), not +whether a process exists -- a hung recorder passes the process check. + +### Manual alternative + + python catcher.py # leave running + +Install the userscript, open the match page, press play. The sniffer catches the +manifest, the catcher starts yt-dlp. Config via environment: + +| Variable | Default | Effect | +|---|---|---| +| `BRFC_BROWSER` | `firefox` | which browser's cookie jar yt-dlp reads | +| `BRFC_DURATION_MIN` | `150` | catcher's recording cap, minutes | + +Check concurrency against a live token with `probe.py --url "" +--parallel 3`. + ## The token Manifest URLs carry an Akamai path token: hdntl=exp=~acl=/*~id=~data=hdntl~hmac= -- `exp` -- ~24h validity -- `acl=/*` -- wildcard over the whole host, not just your match -- `hmac` -- signature over the literal path +It is a **bearer credential** with `acl=/*` over the whole host, valid to `exp`. +Anyone holding a live URL can pull the stream. Keep it out of pastebins and +issue trackers; `stream-url.txt` and `cookies.txt` are gitignored. -It is a bearer credential. Anyone holding a live URL can pull the stream until -`exp`. Keep it out of pastebins and issue trackers. `stream-url.txt` and -`cookies.txt` are gitignored. +Two tokens, not one: -**Do not re-encode the path.** curl normalises `%2f` to `%2F`, which -invalidates the HMAC. `probe.py` sends the request target verbatim. - -## Use Firefox for watching - -yt-dlp lifts session cookies from the browser. Chrome 127+ on Windows uses -app-bound cookie encryption and extraction fails. Firefox works with no fuss. - - set BRFC_BROWSER=firefox # default - -## Workflow - - pip install yt-dlp # done - python catcher.py # leave running - -Install the userscript in Tampermonkey, open the match page, press play. The -sniffer catches the manifest, the catcher starts yt-dlp. Recording is MPEG-TS -(`--hls-use-mpegts`) so a killed process still leaves a playable file. - -For a 3am kickoff: Task Scheduler opens the match page a few minutes before, -the browser autoplays, the sniffer fires, the catcher records. Nobody is awake. - -### Before every match - - python preflight.py # exit 0 only if the server minted a JWT - -Not optional and not a formality. The session can be dead while every local -signal looks fine (see "The cookie expiry is not the session lifetime"), and the -only fix -- signing in by hand in Firefox -- needs a human and some lead time. -Run it hours before kickoff. - -Then register the task, launch it once by hand, and stop it again before the -real trigger (see "a freshly registered task can hang on its first launch"). - -## Concurrency check - -An Akamai edge token is a stateless HMAC -- the edge validates signature and -expiry and serves; it holds no session table, so it cannot count viewers. Any -concurrent-stream limit lives in the entitlement API that *issues* tokens, -upstream of the CDN. Confirm rather than assume, while the browser is playing: - - python probe.py --url "" --parallel 3 - -3/3 alongside the playing browser means four concurrent readers of one token -and the edge does not care. - ---- - -## Findings (live matches, 2026-08-29 onward) - -Verified against live streams, starting with a 23:00 JST kickoff on 2026-08-29 -that recorded 1080p end to end. Later matches are dated where they corrected -something. - -### Two token types, not one - -The player uses a **two-stage token chain**: - -| Token | Where | TTL | Purpose | +| Token | Where | TTL | Fetches | |---|---|---|---| -| `hdnts=` | query string, `acl=/*` unencoded | **~25 seconds** | fetch the master playlist | -| `hdntl=` | URL path, `acl=%2f*` encoded | **~24 hours** | media playlists + segments | +| `hdnts=` | query string, `acl=/*` unencoded | ~25 seconds | master playlist | +| `hdntl=` | URL path, `acl=%2f*` encoded | ~24 hours | media playlists, segments | -The master response rewrites each variant URL with a 24h `hdntl` token. So the -`hdntl` token is the durable one worth handing to a downloader. Chasing a -`hdnts` URL by hand is pointless -- it dies before you can paste it. +The master response rewrites each variant URL with a 24h `hdntl`. Chasing a +`hdnts` URL by hand is pointless -- it expires before you can paste it. -### Confirmed by experiment +**Do not re-encode the path.** curl normalises `%2f` to `%2F`, invalidating the +HMAC. `probe.py` sends the request target verbatim. -- **Concurrency is not limited.** Two independent ffmpeg downloads ran at full - bitrate *while Firefox played the same match*. Three concurrent readers of - one token, no complaint from the edge. Akamai token auth is a stateless HMAC - check; it has no viewer counter. -- **No CDN cookies exist.** All 13 relevant cookies are on `bristolrovers.co.uk`; - nothing is scoped to `akamaized.net`. The token in the URL is the whole auth. -- **Stream is 1920x1080 H.264 + AAC ~5.2 Mbit/s** -> roughly 4.7 GB per match. -- **Output is MPEG-TS** even though yt-dlp names it `.mp4`. That is what - `--hls-use-mpegts` buys: a killed process still leaves a playable file. -- Player is **Bitmovin**, embedded on the club site. `0_xm5vrrw6` is a Kaltura - entry id. It does **not** identify the match -- see below. +## Rules that are not preferences -### The entry id names the feed, not the match +- **Hand the downloader a master URL, not a media one.** Python gets 403 on + path-token media playlists; ffmpeg gets 200. yt-dlp fetches playlists with + Python before ffmpeg runs, so it must be given a `hdnts` master. + `record_live.py` mints one immediately before recording. +- **Force `--downloader ffmpeg`.** yt-dlp's native HLS downloader dies on the + path-token URLs for the same reason. +- **Keep the duration cap.** A feed can keep serving after full time -- a static + slate on video, digital silence on audio -- at full bitrate, with no + `#EXT-X-ENDLIST` and no stall to detect. Of five observed endings three + self-terminated and two ran forever, on both feed types, with no pattern that + predicts which. Treat a stream that stops by itself as luck. +- **Use Firefox as the session holder.** Chrome/Edge lock their cookie DB while + running (yt-dlp issue 7271) and Chrome 127+ on Windows adds app-bound cookie + encryption; an unattended job cannot close a browser to read them. +- **Do not judge a token by `probe.py`.** It inherits the path-token 403, so a + live token looks dead through it. -Disproved 2026-09-19. Four observations, two ids: +## Facts -| Entry id | Seen | Feed | -|---|---|---| -| `0_xm5vrrw6` | 2026-08-29, 2026-09-19 | video | -| `0_z236v0e7` | 2026-09-05, 2026-09-12 | audio | +- 1920x1080 H.264 + AAC ~5.2 Mbit/s. Audio-only feeds are ~140 kbps AAC. +- Output is MPEG-TS even when named `.mp4` -- that is what `--hls-use-mpegts` + buys: a killed process still leaves a playable file. +- Budget **~36 MB/min video** (~6.5 GB for a 180 min cap), **~1.2 MB/min audio** + (~210 MB). +- **Concurrency is not limited.** An Akamai edge token is a stateless HMAC with + no viewer counter. Three readers of one token ran at full bitrate alongside a + playing browser. +- **No CDN cookies exist.** All relevant cookies are on `bristolrovers.co.uk`, + none on `akamaized.net`. The token in the URL is the whole auth. +- Player is Bitmovin. The Kaltura entry id names the **feed, not the match**: + `0_xm5vrrw6` (video) and `0_z236v0e7` (audio) each recurred weeks apart. + Nothing may use it to answer "have I recorded this game" -- use the date. + `catcher.py` is unaffected; it dedupes on liveness, not identity. -Three weeks apart, the same id came back for the video feed. So the id is a -channel: it distinguishes the video feed from the audio one and nothing more. +## The session -`catcher.py` is not broken by this, because it dedupes on *liveness* -- "is a -downloader for this key still alive" -- and the key clears when the downloader -hits its cap. But do not build anything that treats the id as a match -identifier: "have I already recorded this game" cannot be answered from it, and -neither can "is this a new match or a reload". Use the kickoff date for that. +The `SSESS` cookie is the whole login. Everything downstream -- JWT, entry API, +master playlist -- rides it, in plain Python, with no browser or fingerprint +forging. -### Gotcha: which URL form a Python client can fetch +**Cookie expiry is not session lifetime.** The server can drop a session while +the browser holds a cookie that looks perfectly healthy: -Earlier notes here blamed a TLS handshake difference. That was wrong. The 403 -splits by **token form**, not by client: + SSESS... 76 days left on the clock + GET /oauth/browser-token -> 403 {"error":"Unauthenticated"} + Set-Cookie: SSESS...=deleted; expires=Thu, 01 Jan 1970 -| URL form | Python | ffmpeg | -|---|---|---| -| `index.m3u8?hdnts=...` (master, query token) | 200 | 200 | -| `hdntl=.../index_N.m3u8` (media, path token) | 403 | 200 | +One session died after six days; its replacement survived seven and counting. +The lifetime is unpredictable and only the server knows it, so a local cookie +check proves nothing and `preflight.py` is not a formality. -Consequences: +Other session behaviour: -- Mint a **master** URL and hand that to the downloader. `record_live.py` does - this: it calls the entry API for a fresh master immediately before recording. -- Force `--downloader ffmpeg`. yt-dlp's native HLS downloader fetches variant - playlists with Python and dies on the path-token URLs. This is required, not - a preference. -- `probe.py` is diagnostic scaffolding and inherits the path-token 403 -- do - not use it to conclude a token is dead. +- **Sessions are concurrent.** Logging in via a second browser does not + invalidate the first -- Drupal issues a session per browser. So a second + device can hold its own session without disturbing the recorder's. *One + observation, checked immediately after the second login.* +- `/oauth/browser-token` returns the session's **existing** JWT rather than + minting a fresh one per call, so the token's expiry is not a freshness + signal. A dead session returns 403, which is what the check relies on. +- The cookie *value* changed once between weeks with no re-login recorded. + Unexplained. Treat a copied cookie as a snapshot that can go stale on its own. +- The session cookie is **httpOnly**, so `document.cookie` cannot read it. Any + provisioning scheme must use a platform cookie store, or it will silently + collect every cookie except the one that matters. -### Firefox, not Chrome +### Decision: do not automate login -yt-dlp reads cookies from a running Firefox. Chrome/Edge lock their cookie DB -while running (yt-dlp issue 7271), and an unattended job cannot close the -browser to read them. +Automating it is the riskiest step in the chain and the only one needing a +stored password, against an unknown `captcha_token` and a lockout that would +take out the working session too. Instead assume the session can be dead before +any given match and make `preflight.py` routine. -### The stream never ends - -After full time the feed switches to a static league slate and keeps serving -indefinitely at full CBR bitrate (~4.7 Mbit/s to encode a still image). There -is no `#EXT-X-ENDLIST`, so ffmpeg never finishes, and there is no stall to -detect either -- from the downloader's point of view it is a healthy live -stream forever. - -**Keep the duration cap.** `catcher.py` and `record_live.py` pass -`ffmpeg_o:-t $DURATION_MIN*60`. Without it, a feed that behaves like the video -one above fills the disk with a logo. - -This holds for audio too, which earlier notes here doubted. On 2026-09-12 the -audio feed for a 20:30 kickoff went to **digital silence** -- `volumedetect` -reports -91.0 dB on both mean and max, the noise floor, not a quiet passage -- -at 02:35:20 into the file, and then kept serving that silence at a steady -156 kbps until the 180 min cap fired at `rc=0`. Nothing about the stream said -it was over. - -So of three observations, two feeds ran forever and one did not: - -| Date | Feed | Ended how | -|---|---|---| -| 2026-08-29 | video | never; stopped manually at 116 min | -| 2026-09-02 | audio | self-terminated, `rc=0` at 165 min | -| 2026-09-12 | audio | never; silence to the 180 min cap | - -The 2026-09-02 self-termination is now the outlier, not the rule, and it is the -only one of the three. **The cap is load-bearing on both feed types.** Treat a -stream that stops by itself as luck. - -Budget ~36 MB/min for video (measured 2026-09-19: 1080p, 60 s -> 36 MB), so -~6.5 GB for a 180 min cap. Audio is ~1.2 MB/min, ~210 MB for the same cap. - ---- +Cookie *provisioning* to a second host remains the open problem. ## Status -Unattended recording works. A scheduled task started `record_live.py` at 03:15 -for an 03:45 kickoff, found the stream in 16 seconds, and recorded 165 min of -audio to a playable MPEG-TS with nobody awake. +Unattended recording works end to end, scheduled, with nobody awake. Five +matches recorded as of 2026-09-20, including one night capturing both feeds in +parallel. -Sample sizes are small but no longer one: four recorded matches as of -2026-09-19, two video and two audio. Two earlier claims here have since been -disproved outright -- the entry id does not identify the match, and the audio -feed does not reliably end itself. Both are corrected above. Assume the next -claim in this file is the one about to fall over. +**Verified:** the token chain in plain Python; recording both feed types via +ffmpeg; Task Scheduler launching the job; concurrent readers of one token; a +dead session being detectable before kickoff; the duration cap firing at `rc=0` +on a feed that would not stop. -### Verified - -- Token chain end to end in plain Python: session cookie -> JWT -> entry API - -> master playlist. No browser, no fingerprint forging. -- Recording via ffmpeg, video (1080p) and audio (~122 kbps AAC). -- Scheduled unattended run, including that Task Scheduler launches it correctly. -- Concurrency: several readers of one token, alongside a playing browser. -- A dead session is detectable before kickoff, cheaply and read-only - (`preflight.py`), and distinguishable from a blocked request by the - `{"error":"Unauthenticated"}` body and the cookie-deleting `Set-Cookie`. -- The duration cap firing cleanly at `rc=0` on a feed that would not stop. - -### Gotcha: a freshly registered task can hang on its first launch - -Seen 2026-09-12. The first launch of a newly registered scheduled task hung -dead at interpreter startup -- process alive, 0.015 s of CPU and 21 modules -loaded after three minutes, nothing written to any log, a `conhost` child -allocated. Not a crash and not a loop: frozen before Python ran any of our -code. Killing it and launching the identical task again worked immediately and -it has not recurred. Root cause unknown. - -Unattended, that is a silently lost match. So after registering a task, **launch -it once by hand and stop it again**, which makes the real firing a warm start. -Stopping matters as much as starting: `MultipleInstances IgnoreNew` means a test -run left alive will swallow the scheduled trigger without a word. - -### Not verified - -- **Login.** Everything above rides an `SSESS` cookie lifted from a desktop - Firefox profile. Obtaining a session *from credentials* has never been run. - `/user/login` is a standard Drupal form (`name`, `pass`, `form_build_id`, - `form_id`, `captcha_token`) with no reCAPTCHA script and no 2FA mention, but - `captcha_token` is an unknown and testing it risks a lockout that would take - out the working session too. - -### The cookie expiry is not the session lifetime - -Disproved 2026-09-12, and this one bites. The `SSESS` cookie carries a ~90 day -expiry, and earlier notes here read that as the session lifetime. It is not. -The server can drop the session while the browser still holds a cookie that -looks perfectly healthy: - - SSESSaa90be30f47... 76 days left on the clock - GET /oauth/browser-token -> 403 {"error":"Unauthenticated"} - Set-Cookie: SSESSaa90be30f47...=deleted; expires=Thu, 01 Jan 1970 - -That session had last worked six days earlier, at the end of the 2026-09-06 -recording. Nothing local showed it: right expiry, right domain, right profile. -Only asking the server revealed it. - -Then the *replacement* session survived the next seven days fine. So the -lifetime is not a week either -- it is unpredictable, and somewhere between six -days and the 90 the cookie advertises. - -Two consequences: - -- **A local cookie check proves nothing.** Presence and expiry are necessary, - not sufficient. `preflight.py` exists for this: it spends one request asking - the server to mint a JWT, which is the only answer that counts. Run it hours - before kickoff, not minutes. -- The cookie *value* also changed between 2026-09-12 and 2026-09-19 with no - re-login recorded (48 -> 50 chars). Unexplained. Treat a copied cookie as a - snapshot that can go stale on its own, which matters for provisioning one to - another host. - -### Decision: still do not automate login, but detect it early - -The old reasoning for this was arithmetic, and the arithmetic was wrong: it -assumed a ~87 day cookie meant about four manual logins a year. A session that -can die in six days means the refresh could be needed weekly, which is a real -cost rather than a rounding error. - -The decision survives the correction anyway, for the reason that was always the -strong one: automating login is the riskiest step in the chain and the only one -needing a stored password, against an unknown `captcha_token` and a lockout that -would take out the working session. What changes is the mitigation. Do not rely -on remembering to refresh it quarterly -- assume it can be dead before any given -match, and make the check routine: - - python preflight.py # exit 0 only if the server minted a JWT - -Cookie *provisioning* remains the open problem for a second host, and the -rotating value above makes it harder, not easier. - -Use Firefox as the session holder. Two independent reasons: its cookie DB stays -readable while running (Chrome/Edge lock theirs), and it holds the login -durably where Chrome has been observed dropping it within a day. +**Not verified:** obtaining a session *from credentials*. `/user/login` is a +standard Drupal form (`name`, `pass`, `form_build_id`, `form_id`, +`captcha_token`) with no reCAPTCHA script and no 2FA mention, but +`captcha_token` is unknown and probing it risks a lockout. diff --git a/preflight.py b/preflight.py index f2cdb73..7ec5744 100644 --- a/preflight.py +++ b/preflight.py @@ -9,8 +9,9 @@ automate login"). python preflight.py # session + host checks python preflight.py --entry 0_z236v0e7 # also exercise the entry API -Exit code is 0 only if the session minted a JWT. Anything else means tonight's -scheduled run would fail, and it would fail at kickoff with nobody watching. +Exit code is 0 only if the server returned a JWT for this session. Anything +else means tonight's scheduled run would fail, and it would fail at kickoff +with nobody watching. Prints no secrets. Cookie values and tokens are reported by length only. """ @@ -103,7 +104,13 @@ def check_cookies(jar): def check_session(brfc): - """The real login test: does the site still mint a JWT off this cookie?""" + """The real login test: will the server hand back a JWT for this cookie? + + It returns the session's existing token rather than minting a fresh one per + call, so the expiry printed below is the token's remaining life and not a + freshness signal. That does not weaken the check: a dead session answers + 403, which is the thing we are looking for. + """ cj = http.cookiejar.CookieJar() for c in brfc: cj.set_cookie(c) @@ -135,7 +142,7 @@ def check_session(brfc): exp = claims.get("exp") when = (f", expires {datetime.fromtimestamp(exp, timezone.utc):%H:%M:%S}Z " f"({(exp - time.time()) / 60:+.0f} min)" if exp else "") - say(OK, f"session mints a JWT ({len(jwt)} chars{when})") + say(OK, f"session is live, server returned a JWT ({len(jwt)} chars{when})") return jwt