Correct two disproved claims and record what is actually verified
The TLS-handshake explanation for the 403 was wrong. The split is by token form, not by client: Python fetches the master (query token) fine and fails only on the media playlist (path token). That is why record_live.py mints a master and why --downloader ffmpeg is required. "The stream never ends" was one video observation stated as a rule, and an audio stream has since contradicted it by terminating cleanly well inside the cap. Both are now recorded as single observations rather than conclusions. Also records the state of play: unattended recording works, login has never been tested, and the decision not to automate it -- the cookie outlives a quarter, so automating the riskiest step to save four manual logins a year is a bad trade. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kyt7QHQvLJtQRuiY4wAqKB
This commit is contained in:
@@ -96,13 +96,25 @@ The master response rewrites each variant URL with a 24h `hdntl` token. So the
|
|||||||
- Player is **Bitmovin**, embedded on the club site. `0_xm5vrrw6` is a Kaltura
|
- Player is **Bitmovin**, embedded on the club site. `0_xm5vrrw6` is a Kaltura
|
||||||
entry id and identifies the match -- the catcher dedupes on it.
|
entry id and identifies the match -- the catcher dedupes on it.
|
||||||
|
|
||||||
### Gotcha: use ffmpeg/yt-dlp, not a hand-rolled HTTP client
|
### Gotcha: which URL form a Python client can fetch
|
||||||
|
|
||||||
Python's `http.client` gets a **403 on every request**, with any header set,
|
Earlier notes here blamed a TLS handshake difference. That was wrong. The 403
|
||||||
including ffmpeg's exact headers, on an unexpired token from the same IP. The
|
splits by **token form**, not by client:
|
||||||
difference is below HTTP (TLS handshake). ffmpeg fetches the same URL fine, so
|
|
||||||
just use yt-dlp. `probe.py` is diagnostic scaffolding and inherits this 403 --
|
| URL form | Python | ffmpeg |
|
||||||
do not use it to conclude a token is dead.
|
|---|---|---|
|
||||||
|
| `index.m3u8?hdnts=...` (master, query token) | 200 | 200 |
|
||||||
|
| `hdntl=.../index_N.m3u8` (media, path token) | 403 | 200 |
|
||||||
|
|
||||||
|
Consequences:
|
||||||
|
|
||||||
|
- Mint a **master** URL and hand that to the downloader. `record_live.py` does
|
||||||
|
this: it calls the entry API for a fresh master immediately before recording.
|
||||||
|
- Force `--downloader ffmpeg`. yt-dlp's native HLS downloader fetches variant
|
||||||
|
playlists with Python and dies on the path-token URLs. This is required, not
|
||||||
|
a preference.
|
||||||
|
- `probe.py` is diagnostic scaffolding and inherits the path-token 403 -- do
|
||||||
|
not use it to conclude a token is dead.
|
||||||
|
|
||||||
### Firefox, not Chrome
|
### Firefox, not Chrome
|
||||||
|
|
||||||
@@ -118,9 +130,56 @@ is no `#EXT-X-ENDLIST`, so ffmpeg never finishes, and there is no stall to
|
|||||||
detect either -- from the downloader's point of view it is a healthy live
|
detect either -- from the downloader's point of view it is a healthy live
|
||||||
stream forever.
|
stream forever.
|
||||||
|
|
||||||
**A hard duration cap is therefore the only stop condition.** `catcher.py`
|
**Keep the duration cap.** `catcher.py` and `record_live.py` pass
|
||||||
passes `ffmpeg_o:-t $DURATION_MIN*60`. Do not remove it; without it an
|
`ffmpeg_o:-t $DURATION_MIN*60`. Without it, a feed that behaves like the video
|
||||||
unattended run fills the disk with a logo.
|
one above fills the disk with a logo.
|
||||||
|
|
||||||
Observed: 116 min recorded for a 23:00 kickoff, still going at 01:04 and only
|
Note the sample size. This is **one video observation**, and it is contradicted
|
||||||
stopped manually. Budget ~3.5-4 GB per copy per match at 40 MB/min.
|
by **one audio observation**: on 2026-09-02 an audio stream ended cleanly by
|
||||||
|
itself (`rc=0` at 165 min, against a 180 min cap that never fired). So "the
|
||||||
|
stream never ends" is a property of that video feed, not an established rule.
|
||||||
|
Treat the cap as a safety net rather than as the only stop condition.
|
||||||
|
|
||||||
|
Observed: video, 116 min recorded for a 23:00 kickoff, still running at 01:04
|
||||||
|
and stopped manually. Audio, 165 min, self-terminated. Budget ~3.5-4 GB per copy per match at 40 MB/min.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Unattended recording works. A scheduled task started `record_live.py` at 03:15
|
||||||
|
for an 03:45 kickoff, found the stream in 16 seconds, and recorded 165 min of
|
||||||
|
audio to a playable MPEG-TS with nobody awake.
|
||||||
|
|
||||||
|
Sample sizes are one. One video capture, one audio capture. Where the two
|
||||||
|
disagree (stream termination, whether an entry id persists between matches),
|
||||||
|
neither is established -- do not build on either without more matches.
|
||||||
|
|
||||||
|
### Verified
|
||||||
|
|
||||||
|
- Token chain end to end in plain Python: session cookie -> JWT -> entry API
|
||||||
|
-> master playlist. No browser, no fingerprint forging.
|
||||||
|
- Recording via ffmpeg, video (1080p) and audio (~122 kbps AAC).
|
||||||
|
- Scheduled unattended run, including that Task Scheduler launches it correctly.
|
||||||
|
- Concurrency: several readers of one token, alongside a playing browser.
|
||||||
|
|
||||||
|
### Not verified
|
||||||
|
|
||||||
|
- **Login.** Everything above rides an `SSESS` cookie lifted from a desktop
|
||||||
|
Firefox profile. Obtaining a session *from credentials* has never been run.
|
||||||
|
`/user/login` is a standard Drupal form (`name`, `pass`, `form_build_id`,
|
||||||
|
`form_id`, `captcha_token`) with no reCAPTCHA script and no 2FA mention, but
|
||||||
|
`captcha_token` is an unknown and testing it risks a lockout that would take
|
||||||
|
out the working session too.
|
||||||
|
|
||||||
|
### Decision: do not automate login
|
||||||
|
|
||||||
|
The cookie lasts ~87 days and matches are weekly, so automating login saves
|
||||||
|
about four manual actions a year -- while being the riskiest step and the only
|
||||||
|
one needing a stored password. Refresh the cookie by logging in manually on the
|
||||||
|
desktop each quarter and provisioning it to the recording host instead. That
|
||||||
|
makes cookie *provisioning* the open problem, not the captcha.
|
||||||
|
|
||||||
|
Use Firefox as the session holder. Two independent reasons: its cookie DB stays
|
||||||
|
readable while running (Chrome/Edge lock theirs), and it holds the login
|
||||||
|
durably where Chrome has been observed dropping it within a day.
|
||||||
|
|||||||
Reference in New Issue
Block a user