Correct three claims that three weeks of matches disproved
The cookie expiry is not the session lifetime. A session died six days into a
cookie with 76 days left on it, returning 403 {"error":"Unauthenticated"} and
a Set-Cookie that deleted the thing. Nothing local showed it. The replacement
then survived seven days, so the lifetime is not a week either -- it is
unpredictable, and only the server can answer. The old reasoning for not
automating login was arithmetic built on the 90 day figure; the decision
survives on its real merits, but the mitigation changes from refreshing
quarterly to checking before every match.
The entry id names the feed, not the match. 0_xm5vrrw6 came back three weeks
apart for video, 0_z236v0e7 twice for audio. catcher.py is unaffected -- it
dedupes on liveness, not identity -- but nothing should ask it "have I
recorded this game".
The audio feed does not reliably end itself. On 2026-09-12 it went to digital
silence at -91 dB and served that silence to the 180 min cap. The one audio
stream that self-terminated is now the outlier of three. The cap is
load-bearing on both feed types.
Also records the scheduled-task startup hang and the warm-start habit it
forces, and the per-minute sizes measured for both feeds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fr1fLMnVGNAKH5L2BmhMxn
This commit is contained in:
@@ -11,8 +11,12 @@ makes, using your own paid session.
|
||||
|
||||
| File | Job |
|
||||
|---|---|
|
||||
| `record_live.py` | Unattended recorder: finds the live entry itself, mints a master, records |
|
||||
| `preflight.py` | Pre-match check: does the session still mint a JWT, can this host record |
|
||||
| `watchdog.py` | Backup starter: if nothing is *progressing*, kill the stale recorder and start one |
|
||||
| `tampermonkey/brfc-stream-sniffer.user.js` | Hooks fetch/XHR in the page, captures the manifest URL verbatim, POSTs it to the catcher |
|
||||
| `catcher.py` | Listens on 127.0.0.1:8787, hands the URL to yt-dlp, records to `recordings/` |
|
||||
| `discover.py` | Drives the player with Playwright to capture the token-minting traffic |
|
||||
| `probe.py` | Byte-exact manifest fetch + concurrency test |
|
||||
|
||||
## The token
|
||||
@@ -51,6 +55,18 @@ sniffer catches the manifest, the catcher starts yt-dlp. Recording is MPEG-TS
|
||||
For a 3am kickoff: Task Scheduler opens the match page a few minutes before,
|
||||
the browser autoplays, the sniffer fires, the catcher records. Nobody is awake.
|
||||
|
||||
### Before every match
|
||||
|
||||
python preflight.py # exit 0 only if the server minted a JWT
|
||||
|
||||
Not optional and not a formality. The session can be dead while every local
|
||||
signal looks fine (see "The cookie expiry is not the session lifetime"), and the
|
||||
only fix -- signing in by hand in Firefox -- needs a human and some lead time.
|
||||
Run it hours before kickoff.
|
||||
|
||||
Then register the task, launch it once by hand, and stop it again before the
|
||||
real trigger (see "a freshly registered task can hang on its first launch").
|
||||
|
||||
## Concurrency check
|
||||
|
||||
An Akamai edge token is a stateless HMAC -- the edge validates signature and
|
||||
@@ -65,9 +81,11 @@ and the edge does not care.
|
||||
|
||||
---
|
||||
|
||||
## Findings (2026-08-29, live 2300 JST match)
|
||||
## Findings (live matches, 2026-08-29 onward)
|
||||
|
||||
Verified against a live stream. Recorded 1080p successfully end to end.
|
||||
Verified against live streams, starting with a 23:00 JST kickoff on 2026-08-29
|
||||
that recorded 1080p end to end. Later matches are dated where they corrected
|
||||
something.
|
||||
|
||||
### Two token types, not one
|
||||
|
||||
@@ -94,7 +112,25 @@ The master response rewrites each variant URL with a 24h `hdntl` token. So the
|
||||
- **Output is MPEG-TS** even though yt-dlp names it `.mp4`. That is what
|
||||
`--hls-use-mpegts` buys: a killed process still leaves a playable file.
|
||||
- Player is **Bitmovin**, embedded on the club site. `0_xm5vrrw6` is a Kaltura
|
||||
entry id and identifies the match -- the catcher dedupes on it.
|
||||
entry id. It does **not** identify the match -- see below.
|
||||
|
||||
### The entry id names the feed, not the match
|
||||
|
||||
Disproved 2026-09-19. Four observations, two ids:
|
||||
|
||||
| Entry id | Seen | Feed |
|
||||
|---|---|---|
|
||||
| `0_xm5vrrw6` | 2026-08-29, 2026-09-19 | video |
|
||||
| `0_z236v0e7` | 2026-09-05, 2026-09-12 | audio |
|
||||
|
||||
Three weeks apart, the same id came back for the video feed. So the id is a
|
||||
channel: it distinguishes the video feed from the audio one and nothing more.
|
||||
|
||||
`catcher.py` is not broken by this, because it dedupes on *liveness* -- "is a
|
||||
downloader for this key still alive" -- and the key clears when the downloader
|
||||
hits its cap. But do not build anything that treats the id as a match
|
||||
identifier: "have I already recorded this game" cannot be answered from it, and
|
||||
neither can "is this a new match or a reload". Use the kickoff date for that.
|
||||
|
||||
### Gotcha: which URL form a Python client can fetch
|
||||
|
||||
@@ -134,14 +170,27 @@ stream forever.
|
||||
`ffmpeg_o:-t $DURATION_MIN*60`. Without it, a feed that behaves like the video
|
||||
one above fills the disk with a logo.
|
||||
|
||||
Note the sample size. This is **one video observation**, and it is contradicted
|
||||
by **one audio observation**: on 2026-09-02 an audio stream ended cleanly by
|
||||
itself (`rc=0` at 165 min, against a 180 min cap that never fired). So "the
|
||||
stream never ends" is a property of that video feed, not an established rule.
|
||||
Treat the cap as a safety net rather than as the only stop condition.
|
||||
This holds for audio too, which earlier notes here doubted. On 2026-09-12 the
|
||||
audio feed for a 20:30 kickoff went to **digital silence** -- `volumedetect`
|
||||
reports -91.0 dB on both mean and max, the noise floor, not a quiet passage --
|
||||
at 02:35:20 into the file, and then kept serving that silence at a steady
|
||||
156 kbps until the 180 min cap fired at `rc=0`. Nothing about the stream said
|
||||
it was over.
|
||||
|
||||
Observed: video, 116 min recorded for a 23:00 kickoff, still running at 01:04
|
||||
and stopped manually. Audio, 165 min, self-terminated. Budget ~3.5-4 GB per copy per match at 40 MB/min.
|
||||
So of three observations, two feeds ran forever and one did not:
|
||||
|
||||
| Date | Feed | Ended how |
|
||||
|---|---|---|
|
||||
| 2026-08-29 | video | never; stopped manually at 116 min |
|
||||
| 2026-09-02 | audio | self-terminated, `rc=0` at 165 min |
|
||||
| 2026-09-12 | audio | never; silence to the 180 min cap |
|
||||
|
||||
The 2026-09-02 self-termination is now the outlier, not the rule, and it is the
|
||||
only one of the three. **The cap is load-bearing on both feed types.** Treat a
|
||||
stream that stops by itself as luck.
|
||||
|
||||
Budget ~36 MB/min for video (measured 2026-09-19: 1080p, 60 s -> 36 MB), so
|
||||
~6.5 GB for a 180 min cap. Audio is ~1.2 MB/min, ~210 MB for the same cap.
|
||||
|
||||
---
|
||||
|
||||
@@ -151,9 +200,11 @@ Unattended recording works. A scheduled task started `record_live.py` at 03:15
|
||||
for an 03:45 kickoff, found the stream in 16 seconds, and recorded 165 min of
|
||||
audio to a playable MPEG-TS with nobody awake.
|
||||
|
||||
Sample sizes are one. One video capture, one audio capture. Where the two
|
||||
disagree (stream termination, whether an entry id persists between matches),
|
||||
neither is established -- do not build on either without more matches.
|
||||
Sample sizes are small but no longer one: four recorded matches as of
|
||||
2026-09-19, two video and two audio. Two earlier claims here have since been
|
||||
disproved outright -- the entry id does not identify the match, and the audio
|
||||
feed does not reliably end itself. Both are corrected above. Assume the next
|
||||
claim in this file is the one about to fall over.
|
||||
|
||||
### Verified
|
||||
|
||||
@@ -162,6 +213,24 @@ neither is established -- do not build on either without more matches.
|
||||
- Recording via ffmpeg, video (1080p) and audio (~122 kbps AAC).
|
||||
- Scheduled unattended run, including that Task Scheduler launches it correctly.
|
||||
- Concurrency: several readers of one token, alongside a playing browser.
|
||||
- A dead session is detectable before kickoff, cheaply and read-only
|
||||
(`preflight.py`), and distinguishable from a blocked request by the
|
||||
`{"error":"Unauthenticated"}` body and the cookie-deleting `Set-Cookie`.
|
||||
- The duration cap firing cleanly at `rc=0` on a feed that would not stop.
|
||||
|
||||
### Gotcha: a freshly registered task can hang on its first launch
|
||||
|
||||
Seen 2026-09-12. The first launch of a newly registered scheduled task hung
|
||||
dead at interpreter startup -- process alive, 0.015 s of CPU and 21 modules
|
||||
loaded after three minutes, nothing written to any log, a `conhost` child
|
||||
allocated. Not a crash and not a loop: frozen before Python ran any of our
|
||||
code. Killing it and launching the identical task again worked immediately and
|
||||
it has not recurred. Root cause unknown.
|
||||
|
||||
Unattended, that is a silently lost match. So after registering a task, **launch
|
||||
it once by hand and stop it again**, which makes the real firing a warm start.
|
||||
Stopping matters as much as starting: `MultipleInstances IgnoreNew` means a test
|
||||
run left alive will swallow the scheduled trigger without a word.
|
||||
|
||||
### Not verified
|
||||
|
||||
@@ -172,13 +241,54 @@ neither is established -- do not build on either without more matches.
|
||||
`captcha_token` is an unknown and testing it risks a lockout that would take
|
||||
out the working session too.
|
||||
|
||||
### Decision: do not automate login
|
||||
### The cookie expiry is not the session lifetime
|
||||
|
||||
The cookie lasts ~87 days and matches are weekly, so automating login saves
|
||||
about four manual actions a year -- while being the riskiest step and the only
|
||||
one needing a stored password. Refresh the cookie by logging in manually on the
|
||||
desktop each quarter and provisioning it to the recording host instead. That
|
||||
makes cookie *provisioning* the open problem, not the captcha.
|
||||
Disproved 2026-09-12, and this one bites. The `SSESS` cookie carries a ~90 day
|
||||
expiry, and earlier notes here read that as the session lifetime. It is not.
|
||||
The server can drop the session while the browser still holds a cookie that
|
||||
looks perfectly healthy:
|
||||
|
||||
SSESSaa90be30f47... 76 days left on the clock
|
||||
GET /oauth/browser-token -> 403 {"error":"Unauthenticated"}
|
||||
Set-Cookie: SSESSaa90be30f47...=deleted; expires=Thu, 01 Jan 1970
|
||||
|
||||
That session had last worked six days earlier, at the end of the 2026-09-06
|
||||
recording. Nothing local showed it: right expiry, right domain, right profile.
|
||||
Only asking the server revealed it.
|
||||
|
||||
Then the *replacement* session survived the next seven days fine. So the
|
||||
lifetime is not a week either -- it is unpredictable, and somewhere between six
|
||||
days and the 90 the cookie advertises.
|
||||
|
||||
Two consequences:
|
||||
|
||||
- **A local cookie check proves nothing.** Presence and expiry are necessary,
|
||||
not sufficient. `preflight.py` exists for this: it spends one request asking
|
||||
the server to mint a JWT, which is the only answer that counts. Run it hours
|
||||
before kickoff, not minutes.
|
||||
- The cookie *value* also changed between 2026-09-12 and 2026-09-19 with no
|
||||
re-login recorded (48 -> 50 chars). Unexplained. Treat a copied cookie as a
|
||||
snapshot that can go stale on its own, which matters for provisioning one to
|
||||
another host.
|
||||
|
||||
### Decision: still do not automate login, but detect it early
|
||||
|
||||
The old reasoning for this was arithmetic, and the arithmetic was wrong: it
|
||||
assumed a ~87 day cookie meant about four manual logins a year. A session that
|
||||
can die in six days means the refresh could be needed weekly, which is a real
|
||||
cost rather than a rounding error.
|
||||
|
||||
The decision survives the correction anyway, for the reason that was always the
|
||||
strong one: automating login is the riskiest step in the chain and the only one
|
||||
needing a stored password, against an unknown `captcha_token` and a lockout that
|
||||
would take out the working session. What changes is the mitigation. Do not rely
|
||||
on remembering to refresh it quarterly -- assume it can be dead before any given
|
||||
match, and make the check routine:
|
||||
|
||||
python preflight.py # exit 0 only if the server minted a JWT
|
||||
|
||||
Cookie *provisioning* remains the open problem for a second host, and the
|
||||
rotating value above makes it harder, not easier.
|
||||
|
||||
Use Firefox as the session holder. Two independent reasons: its cookie DB stays
|
||||
readable while running (Chrome/Edge lock theirs), and it holds the login
|
||||
|
||||
+4
-2
@@ -32,8 +32,10 @@ BROWSER = os.environ.get("BRFC_BROWSER", "firefox")
|
||||
# ends cleanly. Pad past 90 min for half-time, stoppage and post-match.
|
||||
DURATION_MIN = int(os.environ.get("BRFC_DURATION_MIN", "150"))
|
||||
|
||||
# Kaltura entry id (e.g. 0_xm5vrrw6) identifies the match. Every manifest for
|
||||
# one game -- master and each variant, each with its own token -- shares it.
|
||||
# Kaltura entry id (e.g. 0_xm5vrrw6). Every manifest for one game -- master and
|
||||
# each variant, each with its own token -- shares it. It names the *feed*, not
|
||||
# the match: the same id came back three weeks apart (see README). That is fine
|
||||
# here, because the dedupe below is on liveness, not on "seen this before".
|
||||
ASSET_RE = re.compile(r"/(\d+_[a-z0-9]+)/", re.I)
|
||||
|
||||
_procs = {} # asset key -> Popen of the running downloader
|
||||
|
||||
Reference in New Issue
Block a user