From c39de9a017df56ff250ca66b9d47b987201e3484 Mon Sep 17 00:00:00 2001 From: Mikkeli Matlock Date: Sat, 19 Sep 2026 21:48:10 +0900 Subject: [PATCH] Correct three claims that three weeks of matches disproved The cookie expiry is not the session lifetime. A session died six days into a cookie with 76 days left on it, returning 403 {"error":"Unauthenticated"} and a Set-Cookie that deleted the thing. Nothing local showed it. The replacement then survived seven days, so the lifetime is not a week either -- it is unpredictable, and only the server can answer. The old reasoning for not automating login was arithmetic built on the 90 day figure; the decision survives on its real merits, but the mitigation changes from refreshing quarterly to checking before every match. The entry id names the feed, not the match. 0_xm5vrrw6 came back three weeks apart for video, 0_z236v0e7 twice for audio. catcher.py is unaffected -- it dedupes on liveness, not identity -- but nothing should ask it "have I recorded this game". The audio feed does not reliably end itself. On 2026-09-12 it went to digital silence at -91 dB and served that silence to the 180 min cap. The one audio stream that self-terminated is now the outlier of three. The cap is load-bearing on both feed types. Also records the scheduled-task startup hang and the warm-start habit it forces, and the per-minute sizes measured for both feeds. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Fr1fLMnVGNAKH5L2BmhMxn --- README.md | 148 ++++++++++++++++++++++++++++++++++++++++++++++------- catcher.py | 6 ++- 2 files changed, 133 insertions(+), 21 deletions(-) diff --git a/README.md b/README.md index 1be37ac..7eaa7f4 100644 --- a/README.md +++ b/README.md @@ -11,8 +11,12 @@ makes, using your own paid session. | File | Job | |---|---| +| `record_live.py` | Unattended recorder: finds the live entry itself, mints a master, records | +| `preflight.py` | Pre-match check: does the session still mint a JWT, can this host record | +| `watchdog.py` | Backup starter: if nothing is *progressing*, kill the stale recorder and start one | | `tampermonkey/brfc-stream-sniffer.user.js` | Hooks fetch/XHR in the page, captures the manifest URL verbatim, POSTs it to the catcher | | `catcher.py` | Listens on 127.0.0.1:8787, hands the URL to yt-dlp, records to `recordings/` | +| `discover.py` | Drives the player with Playwright to capture the token-minting traffic | | `probe.py` | Byte-exact manifest fetch + concurrency test | ## The token @@ -51,6 +55,18 @@ sniffer catches the manifest, the catcher starts yt-dlp. Recording is MPEG-TS For a 3am kickoff: Task Scheduler opens the match page a few minutes before, the browser autoplays, the sniffer fires, the catcher records. Nobody is awake. +### Before every match + + python preflight.py # exit 0 only if the server minted a JWT + +Not optional and not a formality. The session can be dead while every local +signal looks fine (see "The cookie expiry is not the session lifetime"), and the +only fix -- signing in by hand in Firefox -- needs a human and some lead time. +Run it hours before kickoff. + +Then register the task, launch it once by hand, and stop it again before the +real trigger (see "a freshly registered task can hang on its first launch"). + ## Concurrency check An Akamai edge token is a stateless HMAC -- the edge validates signature and @@ -65,9 +81,11 @@ and the edge does not care. --- -## Findings (2026-08-29, live 2300 JST match) +## Findings (live matches, 2026-08-29 onward) -Verified against a live stream. Recorded 1080p successfully end to end. +Verified against live streams, starting with a 23:00 JST kickoff on 2026-08-29 +that recorded 1080p end to end. Later matches are dated where they corrected +something. ### Two token types, not one @@ -94,7 +112,25 @@ The master response rewrites each variant URL with a 24h `hdntl` token. So the - **Output is MPEG-TS** even though yt-dlp names it `.mp4`. That is what `--hls-use-mpegts` buys: a killed process still leaves a playable file. - Player is **Bitmovin**, embedded on the club site. `0_xm5vrrw6` is a Kaltura - entry id and identifies the match -- the catcher dedupes on it. + entry id. It does **not** identify the match -- see below. + +### The entry id names the feed, not the match + +Disproved 2026-09-19. Four observations, two ids: + +| Entry id | Seen | Feed | +|---|---|---| +| `0_xm5vrrw6` | 2026-08-29, 2026-09-19 | video | +| `0_z236v0e7` | 2026-09-05, 2026-09-12 | audio | + +Three weeks apart, the same id came back for the video feed. So the id is a +channel: it distinguishes the video feed from the audio one and nothing more. + +`catcher.py` is not broken by this, because it dedupes on *liveness* -- "is a +downloader for this key still alive" -- and the key clears when the downloader +hits its cap. But do not build anything that treats the id as a match +identifier: "have I already recorded this game" cannot be answered from it, and +neither can "is this a new match or a reload". Use the kickoff date for that. ### Gotcha: which URL form a Python client can fetch @@ -134,14 +170,27 @@ stream forever. `ffmpeg_o:-t $DURATION_MIN*60`. Without it, a feed that behaves like the video one above fills the disk with a logo. -Note the sample size. This is **one video observation**, and it is contradicted -by **one audio observation**: on 2026-09-02 an audio stream ended cleanly by -itself (`rc=0` at 165 min, against a 180 min cap that never fired). So "the -stream never ends" is a property of that video feed, not an established rule. -Treat the cap as a safety net rather than as the only stop condition. +This holds for audio too, which earlier notes here doubted. On 2026-09-12 the +audio feed for a 20:30 kickoff went to **digital silence** -- `volumedetect` +reports -91.0 dB on both mean and max, the noise floor, not a quiet passage -- +at 02:35:20 into the file, and then kept serving that silence at a steady +156 kbps until the 180 min cap fired at `rc=0`. Nothing about the stream said +it was over. -Observed: video, 116 min recorded for a 23:00 kickoff, still running at 01:04 -and stopped manually. Audio, 165 min, self-terminated. Budget ~3.5-4 GB per copy per match at 40 MB/min. +So of three observations, two feeds ran forever and one did not: + +| Date | Feed | Ended how | +|---|---|---| +| 2026-08-29 | video | never; stopped manually at 116 min | +| 2026-09-02 | audio | self-terminated, `rc=0` at 165 min | +| 2026-09-12 | audio | never; silence to the 180 min cap | + +The 2026-09-02 self-termination is now the outlier, not the rule, and it is the +only one of the three. **The cap is load-bearing on both feed types.** Treat a +stream that stops by itself as luck. + +Budget ~36 MB/min for video (measured 2026-09-19: 1080p, 60 s -> 36 MB), so +~6.5 GB for a 180 min cap. Audio is ~1.2 MB/min, ~210 MB for the same cap. --- @@ -151,9 +200,11 @@ Unattended recording works. A scheduled task started `record_live.py` at 03:15 for an 03:45 kickoff, found the stream in 16 seconds, and recorded 165 min of audio to a playable MPEG-TS with nobody awake. -Sample sizes are one. One video capture, one audio capture. Where the two -disagree (stream termination, whether an entry id persists between matches), -neither is established -- do not build on either without more matches. +Sample sizes are small but no longer one: four recorded matches as of +2026-09-19, two video and two audio. Two earlier claims here have since been +disproved outright -- the entry id does not identify the match, and the audio +feed does not reliably end itself. Both are corrected above. Assume the next +claim in this file is the one about to fall over. ### Verified @@ -162,6 +213,24 @@ neither is established -- do not build on either without more matches. - Recording via ffmpeg, video (1080p) and audio (~122 kbps AAC). - Scheduled unattended run, including that Task Scheduler launches it correctly. - Concurrency: several readers of one token, alongside a playing browser. +- A dead session is detectable before kickoff, cheaply and read-only + (`preflight.py`), and distinguishable from a blocked request by the + `{"error":"Unauthenticated"}` body and the cookie-deleting `Set-Cookie`. +- The duration cap firing cleanly at `rc=0` on a feed that would not stop. + +### Gotcha: a freshly registered task can hang on its first launch + +Seen 2026-09-12. The first launch of a newly registered scheduled task hung +dead at interpreter startup -- process alive, 0.015 s of CPU and 21 modules +loaded after three minutes, nothing written to any log, a `conhost` child +allocated. Not a crash and not a loop: frozen before Python ran any of our +code. Killing it and launching the identical task again worked immediately and +it has not recurred. Root cause unknown. + +Unattended, that is a silently lost match. So after registering a task, **launch +it once by hand and stop it again**, which makes the real firing a warm start. +Stopping matters as much as starting: `MultipleInstances IgnoreNew` means a test +run left alive will swallow the scheduled trigger without a word. ### Not verified @@ -172,13 +241,54 @@ neither is established -- do not build on either without more matches. `captcha_token` is an unknown and testing it risks a lockout that would take out the working session too. -### Decision: do not automate login +### The cookie expiry is not the session lifetime -The cookie lasts ~87 days and matches are weekly, so automating login saves -about four manual actions a year -- while being the riskiest step and the only -one needing a stored password. Refresh the cookie by logging in manually on the -desktop each quarter and provisioning it to the recording host instead. That -makes cookie *provisioning* the open problem, not the captcha. +Disproved 2026-09-12, and this one bites. The `SSESS` cookie carries a ~90 day +expiry, and earlier notes here read that as the session lifetime. It is not. +The server can drop the session while the browser still holds a cookie that +looks perfectly healthy: + + SSESSaa90be30f47... 76 days left on the clock + GET /oauth/browser-token -> 403 {"error":"Unauthenticated"} + Set-Cookie: SSESSaa90be30f47...=deleted; expires=Thu, 01 Jan 1970 + +That session had last worked six days earlier, at the end of the 2026-09-06 +recording. Nothing local showed it: right expiry, right domain, right profile. +Only asking the server revealed it. + +Then the *replacement* session survived the next seven days fine. So the +lifetime is not a week either -- it is unpredictable, and somewhere between six +days and the 90 the cookie advertises. + +Two consequences: + +- **A local cookie check proves nothing.** Presence and expiry are necessary, + not sufficient. `preflight.py` exists for this: it spends one request asking + the server to mint a JWT, which is the only answer that counts. Run it hours + before kickoff, not minutes. +- The cookie *value* also changed between 2026-09-12 and 2026-09-19 with no + re-login recorded (48 -> 50 chars). Unexplained. Treat a copied cookie as a + snapshot that can go stale on its own, which matters for provisioning one to + another host. + +### Decision: still do not automate login, but detect it early + +The old reasoning for this was arithmetic, and the arithmetic was wrong: it +assumed a ~87 day cookie meant about four manual logins a year. A session that +can die in six days means the refresh could be needed weekly, which is a real +cost rather than a rounding error. + +The decision survives the correction anyway, for the reason that was always the +strong one: automating login is the riskiest step in the chain and the only one +needing a stored password, against an unknown `captcha_token` and a lockout that +would take out the working session. What changes is the mitigation. Do not rely +on remembering to refresh it quarterly -- assume it can be dead before any given +match, and make the check routine: + + python preflight.py # exit 0 only if the server minted a JWT + +Cookie *provisioning* remains the open problem for a second host, and the +rotating value above makes it harder, not easier. Use Firefox as the session holder. Two independent reasons: its cookie DB stays readable while running (Chrome/Edge lock theirs), and it holds the login diff --git a/catcher.py b/catcher.py index 66b6c63..4d0c1d9 100644 --- a/catcher.py +++ b/catcher.py @@ -32,8 +32,10 @@ BROWSER = os.environ.get("BRFC_BROWSER", "firefox") # ends cleanly. Pad past 90 min for half-time, stoppage and post-match. DURATION_MIN = int(os.environ.get("BRFC_DURATION_MIN", "150")) -# Kaltura entry id (e.g. 0_xm5vrrw6) identifies the match. Every manifest for -# one game -- master and each variant, each with its own token -- shares it. +# Kaltura entry id (e.g. 0_xm5vrrw6). Every manifest for one game -- master and +# each variant, each with its own token -- shares it. It names the *feed*, not +# the match: the same id came back three weeks apart (see README). That is fine +# here, because the dedupe below is on liveness, not on "seen this before". ASSET_RE = re.compile(r"/(\d+_[a-z0-9]+)/", re.I) _procs = {} # asset key -> Popen of the running downloader