Compare commits

..

8 Commits

Author SHA1 Message Date
Mikkeli Matlock 9c22a02fab Cut the README to what you need to run this
It had grown into a changelog: how each fact was discovered, which earlier
claim it overturned, what the sample size was at the time. Useful while
learning the system, noise once you just want to record a match. Rewritten as
a reference, 295 lines to ~195, losing the archaeology and keeping the facts.

Conclusions that were buried in narrative are now rules. Sample-size markers
survive only where they change whether a claim can be leaned on.

Documents --feed, including that it is the setting that matters and that
getting it wrong fails silently.

Two corrections while here. Of five observed endings, three self-terminated
and two ran forever, across both feed types with nothing predicting which --
so the cap stays, but "the stream never ends" was too strong. And preflight
no longer claims to mint a JWT: /oauth/browser-token hands back the session's
existing token, so the expiry it prints is not a freshness signal. The check
is unaffected, since it rests on the 403 a dead session returns.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fr1fLMnVGNAKH5L2BmhMxn
2026-09-26 22:33:49 +09:00
Mikkeli Matlock 847fe1e13f Choose the feed, not just the rendition
The live page can expose two entries for one match, an audio feed and a video
feed, and discovery returned whichever manifest loaded first. That race came
out differently twice on the same night: at 21:11 the page served the video
entry, at 22:30 the audio one. The scheduled run recorded audio against a video
pass, and nothing looked wrong -- it fails by producing a complete, playable,
healthy recording of the wrong thing.

--prefer cannot rescue that; it picks a rendition inside one master, after the
entry is already chosen. So add --feed, which chooses between entries:

  video (default)  prefer a video feed, fall back to audio
  audio            the reverse
  any              first discovered, the old behaviour
  both             record every feed in parallel
  --entry <id>     skip discovery and pin one

find_manifest now returns every entry it sees and keeps watching 15s past the
first hit, rather than returning on it. Each candidate is classified by the
codecs in its master, never by entry id -- ids name a feed, but relying on that
would bake in an observation that has already been wrong once.

A preference is not a filter. Non-preferred feeds stay on the list, because a
feed you hold no pass for resolves a master fine and only 403s later on the
segments; the recorder falls through to the next candidate when a download dies
immediately, so a wrong guess costs seconds rather than the match.

Output files are tagged -video / -audio, which matters once --feed both writes
two at once.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fr1fLMnVGNAKH5L2BmhMxn
2026-09-26 22:33:34 +09:00
Mikkeli Matlock c39de9a017 Correct three claims that three weeks of matches disproved
The cookie expiry is not the session lifetime. A session died six days into a
cookie with 76 days left on it, returning 403 {"error":"Unauthenticated"} and
a Set-Cookie that deleted the thing. Nothing local showed it. The replacement
then survived seven days, so the lifetime is not a week either -- it is
unpredictable, and only the server can answer. The old reasoning for not
automating login was arithmetic built on the 90 day figure; the decision
survives on its real merits, but the mitigation changes from refreshing
quarterly to checking before every match.

The entry id names the feed, not the match. 0_xm5vrrw6 came back three weeks
apart for video, 0_z236v0e7 twice for audio. catcher.py is unaffected -- it
dedupes on liveness, not identity -- but nothing should ask it "have I
recorded this game".

The audio feed does not reliably end itself. On 2026-09-12 it went to digital
silence at -91 dB and served that silence to the 180 min cap. The one audio
stream that self-terminated is now the outlier of three. The cap is
load-bearing on both feed types.

Also records the scheduled-task startup hang and the warm-start habit it
forces, and the per-minute sizes measured for both feeds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fr1fLMnVGNAKH5L2BmhMxn
2026-09-19 21:48:10 +09:00
Mikkeli Matlock 14ee5ec26c Add a pre-match session check and a recovery watchdog
Two failure modes cost a match silently, and neither was detectable from
inside the recorder.

preflight.py: the session can be dead while every local signal looks fine --
right cookie, right expiry, right profile. Only the server knows. This spends
one request asking it to mint a JWT and exits non-zero if it will not, so a
dead login surfaces hours before kickoff instead of at it. Reports cookies and
tokens by length; it never prints a value.

watchdog.py: a scheduled launch has been seen to hang at interpreter startup,
alive but frozen before any of our code runs. So "is a recorder process
running" is the wrong question -- the hung process answers yes. This asks
whether recording is *progressing*, via a growing .ts or a freshly written
log, and only intervenes when both signals are stale. A healthy capture is
never interrupted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fr1fLMnVGNAKH5L2BmhMxn
2026-09-19 21:47:19 +09:00
Mikkeli Matlock fcf509fda5 Log what the master offers and let the caller pin audio or video
The recorder passed no -f, so yt-dlp used its default, which prefers video.
That was never a decision: on 2 Sep the master held exactly one rendition, so
there was nothing to choose between. If a master ever carries a video variant
the account is not entitled to, the default would take it and 403 on segments.

--prefer audio|video|best makes it explicit. best keeps the old behaviour.

Every run now logs the renditions the master advertises, parsed from
EXT-X-STREAM-INF, so we build evidence about whether video and audio ever
appear together instead of assuming entitlement shapes the master.

Observed so far, both audio nights: one rendition, mp4a.40.2 at 140 kbps.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kyt7QHQvLJtQRuiY4wAqKB
2026-09-05 19:01:18 +09:00
Mikkeli Matlock 84c29cc0b7 Correct two disproved claims and record what is actually verified
The TLS-handshake explanation for the 403 was wrong. The split is by token
form, not by client: Python fetches the master (query token) fine and fails
only on the media playlist (path token). That is why record_live.py mints a
master and why --downloader ffmpeg is required.

"The stream never ends" was one video observation stated as a rule, and an
audio stream has since contradicted it by terminating cleanly well inside the
cap. Both are now recorded as single observations rather than conclusions.

Also records the state of play: unattended recording works, login has never
been tested, and the decision not to automate it -- the cookie outlives a
quarter, so automating the riskiest step to save four manual logins a year is
a bad trade.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kyt7QHQvLJtQRuiY4wAqKB
2026-09-05 18:49:00 +09:00
Mikkeli Matlock f476e85209 Add unattended recorder that finds the live stream itself
record_live.py runs the whole chain alone: headless browser only to learn the
entry id (it appears at runtime; static HTML has none when nothing is live),
then plain Python mints a fresh master playlist, then ffmpeg records.

No password needed. Firefox session cookies run to late November, well past
any kickoff, so the job authenticates from the stored session.

Three failures found by testing against a live feed rather than at kickoff:

- yt-dlp's native HLS downloader fetches variant playlists with Python and
  gets 403/404 on path-token URLs. Forced --downloader ffmpeg; this is
  required, not a preference.
- ffmpeg 4.3.1 has no -reconnect_on_network_error. Removed here and in
  catcher.py, which had the same unsupported flag.
- A master playlist outlives its stream: after a match ends the master is
  still served while its variants 404. So a found master does not mean a live
  stream, and a downloader that dies immediately means "not up yet" rather
  than "failed". It now retries until the wait window expires.

Untested: the actual segment download for an audio-only stream, because
nothing is live right now. Video was verified working on 29 Aug.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kyt7QHQvLJtQRuiY4wAqKB
2026-09-01 21:00:08 +09:00
Mikkeli Matlock ce92e55255 Map the full token chain and prove it needs no browser
Playwright discovery against the live feed found every link, then the whole
chain was replayed in plain urllib with no browser at all:

  SSESS cookie -> GET /oauth/browser-token          -> JWT (90 min)
  JWT + x-api-key -> GET /v1/entry/{id}             -> hdnts master URL (~25s)
  master -> hdntl token (24h) -> ffmpeg

Both auth hops live off the Akamai CDN, so plain Python reaches them; ffmpeg
handles the CDN, which is the only host Python gets 403s from. Nothing needs
a fingerprint forged and Camoufox is now a fallback rather than the plan.

Two assumptions this disproved:
- The play click is not part of auth. Manifests arrive before any click, so
  a headless run needs no interaction to mint a token.
- The 90 minute JWT does not constrain a 2 hour recording. It is spent before
  ffmpeg starts; the 24h hdntl token carries the match.

discover.py drives the page with Playwright and identifies the mint and token
endpoints by what they RETURN rather than by guessing endpoint names, so it
survives the endpoints being renamed. It rides existing Firefox cookies and
needs no password. Verified working with --headless.

Sniffer v0.5 adds muted autoplay: the browser only has to trigger the manifest
fetch, ffmpeg records real audio separately, and muted autoplay is permitted
where audible is not. It stands down once playback starts, because the control
is a toggle and clicking twice would pause the recording.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kyt7QHQvLJtQRuiY4wAqKB
2026-09-01 11:45:04 +09:00
8 changed files with 1247 additions and 94 deletions
+1
View File
@@ -5,3 +5,4 @@ cookies.txt
*.mkv *.mkv
*.mp4 *.mp4
*.ts *.ts
__pycache__/
+160 -91
View File
@@ -11,116 +11,185 @@ makes, using your own paid session.
| File | Job | | File | Job |
|---|---| |---|---|
| `tampermonkey/brfc-stream-sniffer.user.js` | Hooks fetch/XHR in the page, captures the manifest URL verbatim, POSTs it to the catcher | | `record_live.py` | Unattended recorder: finds the live entry, mints a master, records |
| `catcher.py` | Listens on 127.0.0.1:8787, hands the URL to yt-dlp, records to `recordings/` | | `preflight.py` | Pre-match check: is the session live, can this host record |
| `watchdog.py` | Backup starter: if nothing is *progressing*, start a recorder |
| `catcher.py` | Listens on 127.0.0.1:8787, hands a sniffed URL to yt-dlp |
| `tampermonkey/brfc-stream-sniffer.user.js` | Hooks fetch/XHR, POSTs the manifest URL to the catcher |
| `discover.py` | Drives the player with Playwright to capture token-minting traffic |
| `probe.py` | Byte-exact manifest fetch + concurrency test | | `probe.py` | Byte-exact manifest fetch + concurrency test |
## Recording a match
python preflight.py # exit 0 only if the session is live
Mandatory, hours before kickoff. A dead session is invisible locally and the
only fix needs a human at a browser. Then schedule the recorder for ~30 min
before kickoff:
python record_live.py --max-wait 75 --duration 180 --feed video
**Pick the feed, not just the rendition.** The live page can expose two entries
for one match -- an audio feed and a video feed -- and which manifest loads
first is a race that has come out differently on the same night. `--feed`
chooses between entries and is the setting that matters:
| `--feed` | Behaviour |
|---|---|
| `video` (default) | record a video feed; fall back to audio if none is offered |
| `audio` | the reverse |
| `any` | whichever was discovered first -- the old behaviour |
| `both` | record every feed in parallel |
| `--entry <id>` | skip discovery entirely and pin one |
Feeds are classified by the codecs in their master, never by entry id. A
non-preferred feed stays on the list as a fallback, because a feed you hold no
pass for resolves a master fine and only fails later on the segments; the
recorder falls through to the next candidate when a download dies immediately.
`--prefer` is the *other* choice -- which rendition inside one master -- and
`best` is almost always right. A wrong `--feed` cannot be rescued by `--prefer`,
and it fails quietly: a complete, playable, healthy recording of the wrong
thing. `--feed both` costs ~210 MB extra and removes the question.
**Warm the scheduled task.** After registering it, launch it once by hand and
stop it again. A freshly registered task has been seen to hang at interpreter
startup -- alive, zero CPU, nothing logged, frozen before any of our code runs.
Relaunching fixed it; root cause unknown. Stopping the test run matters as much:
`MultipleInstances IgnoreNew` means a test left running swallows the real
trigger silently.
Optionally schedule `watchdog.py` ~20 min later as a backup. It checks whether
recording is *progressing* (a growing `.ts`, or a freshly written log), not
whether a process exists -- a hung recorder passes the process check.
### Manual alternative
python catcher.py # leave running
Install the userscript, open the match page, press play. The sniffer catches the
manifest, the catcher starts yt-dlp. Config via environment:
| Variable | Default | Effect |
|---|---|---|
| `BRFC_BROWSER` | `firefox` | which browser's cookie jar yt-dlp reads |
| `BRFC_DURATION_MIN` | `150` | catcher's recording cap, minutes |
Check concurrency against a live token with `probe.py --url "<manifest>"
--parallel 3`.
## The token ## The token
Manifest URLs carry an Akamai path token: Manifest URLs carry an Akamai path token:
hdntl=exp=<unix>~acl=/*~id=<n>~data=hdntl~hmac=<sig> hdntl=exp=<unix>~acl=/*~id=<n>~data=hdntl~hmac=<sig>
- `exp` -- ~24h validity It is a **bearer credential** with `acl=/*` over the whole host, valid to `exp`.
- `acl=/*` -- wildcard over the whole host, not just your match Anyone holding a live URL can pull the stream. Keep it out of pastebins and
- `hmac` -- signature over the literal path issue trackers; `stream-url.txt` and `cookies.txt` are gitignored.
It is a bearer credential. Anyone holding a live URL can pull the stream until Two tokens, not one:
`exp`. Keep it out of pastebins and issue trackers. `stream-url.txt` and
`cookies.txt` are gitignored.
**Do not re-encode the path.** curl normalises `%2f` to `%2F`, which | Token | Where | TTL | Fetches |
invalidates the HMAC. `probe.py` sends the request target verbatim.
## Use Firefox for watching
yt-dlp lifts session cookies from the browser. Chrome 127+ on Windows uses
app-bound cookie encryption and extraction fails. Firefox works with no fuss.
set BRFC_BROWSER=firefox # default
## Workflow
pip install yt-dlp # done
python catcher.py # leave running
Install the userscript in Tampermonkey, open the match page, press play. The
sniffer catches the manifest, the catcher starts yt-dlp. Recording is MPEG-TS
(`--hls-use-mpegts`) so a killed process still leaves a playable file.
For a 3am kickoff: Task Scheduler opens the match page a few minutes before,
the browser autoplays, the sniffer fires, the catcher records. Nobody is awake.
## Concurrency check
An Akamai edge token is a stateless HMAC -- the edge validates signature and
expiry and serves; it holds no session table, so it cannot count viewers. Any
concurrent-stream limit lives in the entitlement API that *issues* tokens,
upstream of the CDN. Confirm rather than assume, while the browser is playing:
python probe.py --url "<manifest from the sniffer>" --parallel 3
3/3 alongside the playing browser means four concurrent readers of one token
and the edge does not care.
---
## Findings (2026-08-29, live 2300 JST match)
Verified against a live stream. Recorded 1080p successfully end to end.
### Two token types, not one
The player uses a **two-stage token chain**:
| Token | Where | TTL | Purpose |
|---|---|---|---| |---|---|---|---|
| `hdnts=` | query string, `acl=/*` unencoded | **~25 seconds** | fetch the master playlist | | `hdnts=` | query string, `acl=/*` unencoded | ~25 seconds | master playlist |
| `hdntl=` | URL path, `acl=%2f*` encoded | **~24 hours** | media playlists + segments | | `hdntl=` | URL path, `acl=%2f*` encoded | ~24 hours | media playlists, segments |
The master response rewrites each variant URL with a 24h `hdntl` token. So the The master response rewrites each variant URL with a 24h `hdntl`. Chasing a
`hdntl` token is the durable one worth handing to a downloader. Chasing a `hdnts` URL by hand is pointless -- it expires before you can paste it.
`hdnts` URL by hand is pointless -- it dies before you can paste it.
### Confirmed by experiment **Do not re-encode the path.** curl normalises `%2f` to `%2F`, invalidating the
HMAC. `probe.py` sends the request target verbatim.
- **Concurrency is not limited.** Two independent ffmpeg downloads ran at full ## Rules that are not preferences
bitrate *while Firefox played the same match*. Three concurrent readers of
one token, no complaint from the edge. Akamai token auth is a stateless HMAC
check; it has no viewer counter.
- **No CDN cookies exist.** All 13 relevant cookies are on `bristolrovers.co.uk`;
nothing is scoped to `akamaized.net`. The token in the URL is the whole auth.
- **Stream is 1920x1080 H.264 + AAC ~5.2 Mbit/s** -> roughly 4.7 GB per match.
- **Output is MPEG-TS** even though yt-dlp names it `.mp4`. That is what
`--hls-use-mpegts` buys: a killed process still leaves a playable file.
- Player is **Bitmovin**, embedded on the club site. `0_xm5vrrw6` is a Kaltura
entry id and identifies the match -- the catcher dedupes on it.
### Gotcha: use ffmpeg/yt-dlp, not a hand-rolled HTTP client - **Hand the downloader a master URL, not a media one.** Python gets 403 on
path-token media playlists; ffmpeg gets 200. yt-dlp fetches playlists with
Python before ffmpeg runs, so it must be given a `hdnts` master.
`record_live.py` mints one immediately before recording.
- **Force `--downloader ffmpeg`.** yt-dlp's native HLS downloader dies on the
path-token URLs for the same reason.
- **Keep the duration cap.** A feed can keep serving after full time -- a static
slate on video, digital silence on audio -- at full bitrate, with no
`#EXT-X-ENDLIST` and no stall to detect. Of five observed endings three
self-terminated and two ran forever, on both feed types, with no pattern that
predicts which. Treat a stream that stops by itself as luck.
- **Use Firefox as the session holder.** Chrome/Edge lock their cookie DB while
running (yt-dlp issue 7271) and Chrome 127+ on Windows adds app-bound cookie
encryption; an unattended job cannot close a browser to read them.
- **Do not judge a token by `probe.py`.** It inherits the path-token 403, so a
live token looks dead through it.
Python's `http.client` gets a **403 on every request**, with any header set, ## Facts
including ffmpeg's exact headers, on an unexpired token from the same IP. The
difference is below HTTP (TLS handshake). ffmpeg fetches the same URL fine, so
just use yt-dlp. `probe.py` is diagnostic scaffolding and inherits this 403 --
do not use it to conclude a token is dead.
### Firefox, not Chrome - 1920x1080 H.264 + AAC ~5.2 Mbit/s. Audio-only feeds are ~140 kbps AAC.
- Output is MPEG-TS even when named `.mp4` -- that is what `--hls-use-mpegts`
buys: a killed process still leaves a playable file.
- Budget **~36 MB/min video** (~6.5 GB for a 180 min cap), **~1.2 MB/min audio**
(~210 MB).
- **Concurrency is not limited.** An Akamai edge token is a stateless HMAC with
no viewer counter. Three readers of one token ran at full bitrate alongside a
playing browser.
- **No CDN cookies exist.** All relevant cookies are on `bristolrovers.co.uk`,
none on `akamaized.net`. The token in the URL is the whole auth.
- Player is Bitmovin. The Kaltura entry id names the **feed, not the match**:
`0_xm5vrrw6` (video) and `0_z236v0e7` (audio) each recurred weeks apart.
Nothing may use it to answer "have I recorded this game" -- use the date.
`catcher.py` is unaffected; it dedupes on liveness, not identity.
yt-dlp reads cookies from a running Firefox. Chrome/Edge lock their cookie DB ## The session
while running (yt-dlp issue 7271), and an unattended job cannot close the
browser to read them.
### The stream never ends The `SSESS` cookie is the whole login. Everything downstream -- JWT, entry API,
master playlist -- rides it, in plain Python, with no browser or fingerprint
forging.
After full time the feed switches to a static league slate and keeps serving **Cookie expiry is not session lifetime.** The server can drop a session while
indefinitely at full CBR bitrate (~4.7 Mbit/s to encode a still image). There the browser holds a cookie that looks perfectly healthy:
is no `#EXT-X-ENDLIST`, so ffmpeg never finishes, and there is no stall to
detect either -- from the downloader's point of view it is a healthy live
stream forever.
**A hard duration cap is therefore the only stop condition.** `catcher.py` SSESS... 76 days left on the clock
passes `ffmpeg_o:-t $DURATION_MIN*60`. Do not remove it; without it an GET /oauth/browser-token -> 403 {"error":"Unauthenticated"}
unattended run fills the disk with a logo. Set-Cookie: SSESS...=deleted; expires=Thu, 01 Jan 1970
Observed: 116 min recorded for a 23:00 kickoff, still going at 01:04 and only One session died after six days; its replacement survived seven and counting.
stopped manually. Budget ~3.5-4 GB per copy per match at 40 MB/min. The lifetime is unpredictable and only the server knows it, so a local cookie
check proves nothing and `preflight.py` is not a formality.
Other session behaviour:
- **Sessions are concurrent.** Logging in via a second browser does not
invalidate the first -- Drupal issues a session per browser. So a second
device can hold its own session without disturbing the recorder's. *One
observation, checked immediately after the second login.*
- `/oauth/browser-token` returns the session's **existing** JWT rather than
minting a fresh one per call, so the token's expiry is not a freshness
signal. A dead session returns 403, which is what the check relies on.
- The cookie *value* changed once between weeks with no re-login recorded.
Unexplained. Treat a copied cookie as a snapshot that can go stale on its own.
- The session cookie is **httpOnly**, so `document.cookie` cannot read it. Any
provisioning scheme must use a platform cookie store, or it will silently
collect every cookie except the one that matters.
### Decision: do not automate login
Automating it is the riskiest step in the chain and the only one needing a
stored password, against an unknown `captcha_token` and a lockout that would
take out the working session too. Instead assume the session can be dead before
any given match and make `preflight.py` routine.
Cookie *provisioning* to a second host remains the open problem.
## Status
Unattended recording works end to end, scheduled, with nobody awake. Five
matches recorded as of 2026-09-20, including one night capturing both feeds in
parallel.
**Verified:** the token chain in plain Python; recording both feed types via
ffmpeg; Task Scheduler launching the job; concurrent readers of one token; a
dead session being detectable before kickoff; the duration cap firing at `rc=0`
on a feed that would not stop.
**Not verified:** obtaining a session *from credentials*. `/user/login` is a
standard Drupal form (`name`, `pass`, `form_build_id`, `form_id`,
`captcha_token`) with no reCAPTCHA script and no 2FA mention, but
`captcha_token` is unknown and probing it risks a lockout.
+6 -3
View File
@@ -32,8 +32,10 @@ BROWSER = os.environ.get("BRFC_BROWSER", "firefox")
# ends cleanly. Pad past 90 min for half-time, stoppage and post-match. # ends cleanly. Pad past 90 min for half-time, stoppage and post-match.
DURATION_MIN = int(os.environ.get("BRFC_DURATION_MIN", "150")) DURATION_MIN = int(os.environ.get("BRFC_DURATION_MIN", "150"))
# Kaltura entry id (e.g. 0_xm5vrrw6) identifies the match. Every manifest for # Kaltura entry id (e.g. 0_xm5vrrw6). Every manifest for one game -- master and
# one game -- master and each variant, each with its own token -- shares it. # each variant, each with its own token -- shares it. It names the *feed*, not
# the match: the same id came back three weeks apart (see README). That is fine
# here, because the dedupe below is on liveness, not on "seen this before".
ASSET_RE = re.compile(r"/(\d+_[a-z0-9]+)/", re.I) ASSET_RE = re.compile(r"/(\d+_[a-z0-9]+)/", re.I)
_procs = {} # asset key -> Popen of the running downloader _procs = {} # asset key -> Popen of the running downloader
@@ -68,8 +70,9 @@ def start_recording(key, url, user_agent, page):
# yt-dlp's own retry flags do NOT reach the ffmpeg downloader, which is # yt-dlp's own retry flags do NOT reach the ffmpeg downloader, which is
# what actually pulls live HLS. These are ffmpeg's own reconnect knobs. # what actually pulls live HLS. These are ffmpeg's own reconnect knobs.
"--downloader-args", "--downloader-args",
# ffmpeg 4.3.1 has no -reconnect_on_network_error.
"ffmpeg_i:-reconnect 1 -reconnect_streamed 1 " "ffmpeg_i:-reconnect 1 -reconnect_streamed 1 "
"-reconnect_on_network_error 1 -reconnect_delay_max 30", "-reconnect_delay_max 30",
"--downloader-args", f"ffmpeg_o:-t {DURATION_MIN * 60}", "--downloader-args", f"ffmpeg_o:-t {DURATION_MIN * 60}",
"-o", out, "-o", out,
url, url,
+203
View File
@@ -0,0 +1,203 @@
#!/usr/bin/env python3
"""Drive the player with Playwright and capture the traffic that mints a token.
Purpose: find which request produces a stream URL, so the mint can eventually
run without a GUI. Playwright sees request AND response bodies natively, so
this replaces the userscript's hooks with something far less fragile.
No password is needed. Your existing Firefox session cookies are injected into
a fresh browser context, so this rides the login you already have.
python discover.py --url "<match page url>"
python discover.py --url "..." --headless
Ports to Camoufox on halogen by swapping the launcher: Camoufox speaks the
Playwright protocol, so the flow below is unchanged.
"""
import argparse
import json
import os
import re
import sys
from datetime import datetime
ROOT = os.path.dirname(os.path.abspath(__file__))
LOGS = os.path.join(ROOT, "logs")
TRACE = os.path.join(LOGS, "pw-trace.jsonl")
# Only these domains' cookies are handed to the browser. Everything else in
# the Firefox jar is irrelevant here and should not be copied around.
COOKIE_DOMAINS = ("bristolrovers", "streamamg", "kaltura", "akamaized")
# Bodies we never want to read: media segments would be dumped into the log.
SKIP_BODY = (".ts", ".mp4", ".m4s", ".aac", ".jpg", ".jpeg", ".png", ".gif",
".webp", ".svg", ".woff", ".woff2", ".css", ".ico")
MAX_BODY = 20000
def firefox_cookies():
"""Lift the live session from Firefox. Requires no password."""
from yt_dlp.cookies import extract_cookies_from_browser
import io, contextlib
buf = io.StringIO()
with contextlib.redirect_stderr(buf):
jar = extract_cookies_from_browser("firefox")
out = []
for c in jar:
dom = (c.domain or "")
if not any(k in dom.lower() for k in COOKIE_DOMAINS):
continue
out.append({
"name": c.name,
"value": c.value or "",
"domain": dom,
"path": c.path or "/",
"expires": float(c.expires) if c.expires else -1,
"httpOnly": False,
"secure": bool(c.secure),
"sameSite": "Lax",
})
return out
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--url", required=True, help="match page URL")
ap.add_argument("--headless", action="store_true",
help="run without a window (what halogen will do)")
ap.add_argument("--seconds", type=int, default=90,
help="how long to watch after load")
args = ap.parse_args()
try:
from playwright.sync_api import sync_playwright
except ImportError:
sys.exit("playwright not installed: pip install playwright "
"&& python -m playwright install firefox")
os.makedirs(LOGS, exist_ok=True)
cookies = firefox_cookies()
print(f"injecting {len(cookies)} session cookies")
manifests, mints, jwts, seen = [], [], [], set()
fh = open(TRACE, "a", encoding="utf-8")
def write(rec):
rec["ts"] = datetime.now().isoformat()
fh.write(json.dumps(rec, ensure_ascii=False) + chr(10))
fh.flush()
with sync_playwright() as p:
browser = p.firefox.launch(headless=args.headless)
ctx = browser.new_context(
user_agent=("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) "
"Gecko/20100101 Firefox/131.0"),
viewport={"width": 1440, "height": 900},
)
if cookies:
try:
ctx.add_cookies(cookies)
except Exception as exc:
print(f"cookie injection failed: {exc}")
page = ctx.new_page()
def on_response(resp):
url = resp.url
if url in seen:
return
low = url.split("?")[0].lower()
if ".m3u8" in low:
seen.add(url)
manifests.append(url)
print(f" manifest: {url[:120]}")
write({"kind": "manifest", "url": url, "status": resp.status})
return
if any(low.endswith(e) for e in SKIP_BODY):
return
try:
body = resp.text()
except Exception:
return
if not body:
return
# The JWT that authorises the mint is issued somewhere on the club
# site off the session cookie. Detect it the same way: by output.
# A JWT is three base64 segments starting with a b64 '{"alg"'.
if re.search(r"eyJ[A-Za-z0-9_-]{8,}\.eyJ[A-Za-z0-9_-]{8,}\.", body):
seen.add(url)
req = resp.request
rec = {
"kind": "JWT-ISSUER",
"method": req.method,
"url": url,
"status": resp.status,
"reqHeaders": dict(req.headers),
"reqBody": (req.post_data or None),
"body": body[:MAX_BODY],
}
jwts.append(rec)
write(rec)
print(f" *** JWT ISSUER *** {req.method} {url[:100]}")
return
# The mint call is whatever RETURNS a manifest URL, whatever it is
# named. Detect by output, not by guessing endpoint names.
if "m3u8" in body.lower():
seen.add(url)
req = resp.request
rec = {
"kind": "MINT",
"method": req.method,
"url": url,
"status": resp.status,
"reqHeaders": dict(req.headers),
"reqBody": (req.post_data or None),
"body": body[:MAX_BODY],
}
mints.append(rec)
write(rec)
print(f" *** MINT *** {req.method} {url[:110]}")
page.on("response", on_response)
print(f"loading {args.url}")
page.goto(args.url, wait_until="domcontentloaded", timeout=60000)
# The player needs a click. Muted playback is enough to trigger the
# manifest fetch, and muted autoplay is not blocked.
page.wait_for_timeout(5000)
for sel in (".bmpui-ui-hugeplaybacktogglebutton",
".bmpui-ui-playbacktogglebutton",
"button[aria-label*='play' i]"):
try:
el = page.query_selector(sel)
if el and el.is_visible():
el.click()
print(f"clicked {sel}")
break
except Exception:
pass
try:
page.evaluate("() => { const v = document.querySelector('video');"
" if (v) { v.muted = true; v.play(); } }")
except Exception:
pass
print(f"watching for {args.seconds}s ...")
page.wait_for_timeout(args.seconds * 1000)
browser.close()
fh.close()
print(f"\nmanifests seen : {len(manifests)}")
print(f"mint calls : {len(mints)}")
for m in mints:
print(f"\n {m['method']} {m['url']}")
print(f" status {m['status']} reqBody={str(m['reqBody'])[:200]}")
print(f"\nfull trace -> {TRACE}")
if __name__ == "__main__":
main()
+231
View File
@@ -0,0 +1,231 @@
#!/usr/bin/env python3
"""Pre-match check: is the session still good, and can this host record?
Run it a few hours before kickoff, not five minutes before. The one thing it
cannot fix for you is a dead login: refreshing that means opening Firefox and
signing in by hand, which takes a human (see README, "Decision: do not
automate login").
python preflight.py # session + host checks
python preflight.py --entry 0_z236v0e7 # also exercise the entry API
Exit code is 0 only if the server returned a JWT for this session. Anything
else means tonight's scheduled run would fail, and it would fail at kickoff
with nobody watching.
Prints no secrets. Cookie values and tokens are reported by length only.
"""
import argparse
import base64
import contextlib
import http.cookiejar
import io
import json
import os
import re
import shutil
import sys
import time
import urllib.error
import urllib.request
from datetime import datetime, timezone
ROOT = os.path.dirname(os.path.abspath(__file__))
RECORDINGS = os.path.join(ROOT, "recordings")
UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) "
"Gecko/20100101 Firefox/131.0")
TOKEN_URL = ("https://www.bristolrovers.co.uk/oauth/browser-token"
"?client_id=streamamg&token_type=urn%3Aietf%3Aparams%3Aoauth%3Atoken-type%3Aid_token")
# Public client key, shipped to every browser -- not a secret.
API_KEY = "50ivbQ3IXpF2mwT4tPAJ2V6cVm4CGRT2YKmDK5v6"
OK, BAD, WARN = "ok ", "FAIL", "warn"
def say(status, msg):
print(f"[{status}] {msg}", flush=True)
def check_deps():
"""Everything record_live.py reaches for, checked before kickoff."""
good = True
for mod, hint in (("yt_dlp", "pip install yt-dlp"),
("playwright", "pip install playwright && "
"python -m playwright install firefox")):
try:
__import__(mod)
say(OK, f"{mod} importable")
except ImportError:
say(BAD, f"{mod} missing -- {hint}")
good = False
if shutil.which("ffmpeg"):
say(OK, "ffmpeg on PATH")
else:
say(BAD, "ffmpeg not on PATH -- the downloader is forced to ffmpeg")
good = False
return good
def check_disk(need_gb=8):
"""A video match is ~4.7 GB. Audio is ~200 MB. Budget for video."""
os.makedirs(RECORDINGS, exist_ok=True)
free_gb = shutil.disk_usage(RECORDINGS).free / 2**30
if free_gb >= need_gb:
say(OK, f"{free_gb:.0f} GB free on the recordings volume")
return True
say(BAD, f"only {free_gb:.1f} GB free, want >= {need_gb} GB for a video match")
return False
def firefox_jar():
from yt_dlp.cookies import extract_cookies_from_browser
with contextlib.redirect_stderr(io.StringIO()):
return extract_cookies_from_browser("firefox")
def check_cookies(jar):
"""Cookie *present and unexpired* is necessary, not sufficient. The server
can invalidate a session while the browser still holds the cookie, which is
exactly what check_session() is for."""
brfc = [c for c in jar if "bristolrovers" in (c.domain or "").lower()]
say(OK if brfc else BAD, f"{len(brfc)} bristolrovers cookies in the Firefox jar")
sess = [c for c in brfc if c.name.startswith(("SSESS", "SESS"))]
if not sess:
say(BAD, "no SSESS/SESS cookie -- Firefox is not logged in")
return brfc, False
for c in sess:
if not c.expires:
say(WARN, f"{c.name[:16]}... is a session cookie (dies with Firefox)")
continue
days = (c.expires - time.time()) / 86400
say(OK if days > 1 else BAD,
f"{c.name[:16]}... len={len(c.value or '')} expires in {days:.0f}d")
return brfc, True
def check_session(brfc):
"""The real login test: will the server hand back a JWT for this cookie?
It returns the session's existing token rather than minting a fresh one per
call, so the expiry printed below is the token's remaining life and not a
freshness signal. That does not weaken the check: a dead session answers
403, which is the thing we are looking for.
"""
cj = http.cookiejar.CookieJar()
for c in brfc:
cj.set_cookie(c)
op = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
hdrs = {"User-Agent": UA, "Accept": "application/json, text/plain, */*"}
try:
resp = op.open(urllib.request.Request(TOKEN_URL, headers=hdrs), timeout=20)
body = resp.read().decode("utf-8", "replace")
except urllib.error.HTTPError as exc:
detail = exc.read().decode("utf-8", "replace")[:200]
say(BAD, f"browser-token returned HTTP {exc.code}: {detail}")
if exc.code in (401, 403):
say(BAD, "session is dead server-side -- sign in again in Firefox")
return None
except Exception as exc:
say(BAD, f"browser-token unreachable: {type(exc).__name__}: {exc}")
return None
m = re.search(r"(eyJ[\w-]+\.eyJ[\w-]+\.[\w-]+)", body)
if not m:
say(BAD, "no JWT in the browser-token response -- session is stale")
return None
jwt = m.group(1)
seg = jwt.split(".")[1]
try:
claims = json.loads(base64.urlsafe_b64decode(seg + "=" * (-len(seg) % 4)))
except Exception:
claims = {}
exp = claims.get("exp")
when = (f", expires {datetime.fromtimestamp(exp, timezone.utc):%H:%M:%S}Z "
f"({(exp - time.time()) / 60:+.0f} min)" if exp else "")
say(OK, f"session is live, server returned a JWT ({len(jwt)} chars{when})")
return jwt
def check_entry(jwt, entry_id):
"""Optional: exercise the rest of the chain against a known entry id.
A master playlist outlives its stream, so a 200 here does not prove the
match is live -- it proves the entitlement still resolves.
"""
req = urllib.request.Request(
f"https://api.playback.streamamg.com/v1/entry/{entry_id}",
headers={"User-Agent": UA, "Authorization": "Bearer " + jwt,
"x-api-key": API_KEY,
"Origin": "https://www.bristolrovers.co.uk",
"Referer": "https://www.bristolrovers.co.uk/"})
try:
data = json.loads(urllib.request.urlopen(req, timeout=20)
.read().decode("utf-8", "replace"))
except urllib.error.HTTPError as exc:
say(WARN, f"entry {entry_id}: HTTP {exc.code} "
f"(stale entry id, or not entitled)")
return
except Exception as exc:
say(WARN, f"entry {entry_id}: {type(exc).__name__}: {exc}")
return
hls = (data.get("media") or {}).get("hls")
if not hls:
say(WARN, f"entry {entry_id} resolved but carries no HLS master")
return
say(OK, f"entry {entry_id} resolves to a master playlist")
try:
body = urllib.request.urlopen(urllib.request.Request(
hls, headers={"User-Agent": UA,
"Referer": "https://www.bristolrovers.co.uk/"}),
timeout=15).read().decode("utf-8", "replace")
except Exception as exc:
say(WARN, f"could not read the master: {type(exc).__name__}: {exc}")
return
PREFIX = "#EXT-X-STREAM-INF:"
streams = [ln[len(PREFIX):] for ln in body.splitlines()
if ln.startswith(PREFIX)]
say(OK, f"master offers {len(streams)} rendition(s):")
for st in streams:
bw = re.search(r"BANDWIDTH=(\d+)", st)
res = re.search(r"RESOLUTION=([\dx]+)", st)
cod = re.search(r'CODECS="([^"]+)"', st)
codecs = cod.group(1) if cod else "?"
kind = "audio-only" if ("avc" not in codecs and "hvc" not in codecs) else "video"
print(f" {kind:11s} {int(bw.group(1)) // 1000 if bw else '?':>6} kbps "
f"{res.group(1) if res else '-':>10} {codecs}")
def main():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--entry", help="entry id to exercise the entry API against")
args = ap.parse_args()
print(f"preflight {datetime.now():%Y-%m-%d %H:%M:%S}\n")
host_ok = check_deps() & check_disk()
try:
jar = firefox_jar()
except Exception as exc:
say(BAD, f"cannot read Firefox cookies: {type(exc).__name__}: {exc}")
return 1
brfc, have_cookie = check_cookies(jar)
jwt = check_session(brfc) if have_cookie else None
if jwt and args.entry:
check_entry(jwt, args.entry)
print()
if not jwt:
say(BAD, "NOT READY -- sign in at https://www.bristolrovers.co.uk/user/login "
"in Firefox, then run this again")
return 1
if not host_ok:
say(BAD, "session is good but this host cannot record -- fix the above")
return 1
say(OK, "READY")
return 0
if __name__ == "__main__":
sys.exit(main())
+448
View File
@@ -0,0 +1,448 @@
#!/usr/bin/env python3
"""Unattended recorder: find the live stream, then record it.
Why this needs a headless browser at all: auth and the token mint are plain
HTTP (see discover.py), but the *entry id* for a given match only appears in
the page at runtime -- the static HTML carries no id when nothing is live. So
we render the page headlessly just long enough to learn the manifest URL, then
hand it to ffmpeg and close the browser.
Nothing here needs your password. It rides the Firefox session cookies, which
outlive kickoff by months.
# start now, keep looking for up to 45 min, then record 150 min
python record_live.py --max-wait 45 --duration 150
# take both feeds at once, or pin one when you know which you hold
python record_live.py --feed both
python record_live.py --entry 0_xm5vrrw6
Two different choices, often confused. **--feed** picks which *entry* to record
when the page exposes more than one for the same match, typically an audio feed
and a video feed. **--prefer** picks a rendition *inside* one entry's master.
Getting --feed wrong cannot be rescued by --prefer, and it fails quietly: you
get a complete, playable, perfectly healthy recording of the wrong thing.
# or wait until a wall-clock time first (03:15 before an 03:45 kickoff)
python record_live.py --at "2026-09-02 03:15" --max-wait 60
Recording stops on the duration cap. That is deliberate and load-bearing: the
feed does not emit #EXT-X-ENDLIST -- after full time it switches to a static
slate and streams forever.
"""
import argparse
import contextlib
import io
import os
import re
import subprocess
import sys
import time
from datetime import datetime
ROOT = os.path.dirname(os.path.abspath(__file__))
RECORDINGS = os.path.join(ROOT, "recordings")
LOGS = os.path.join(ROOT, "logs")
LIVE_URL = "https://www.bristolrovers.co.uk/video/live"
UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) "
"Gecko/20100101 Firefox/131.0")
COOKIE_DOMAINS = ("bristolrovers", "streamamg", "kaltura", "akamaized")
def log(msg):
line = f"[{datetime.now():%Y-%m-%d %H:%M:%S}] {msg}"
print(line, flush=True)
os.makedirs(LOGS, exist_ok=True)
with open(os.path.join(LOGS, "record_live.log"), "a", encoding="utf-8") as fh:
fh.write(line + chr(10))
def firefox_cookies():
from yt_dlp.cookies import extract_cookies_from_browser
with contextlib.redirect_stderr(io.StringIO()):
jar = extract_cookies_from_browser("firefox")
out = []
for c in jar:
dom = c.domain or ""
if not any(k in dom.lower() for k in COOKIE_DOMAINS):
continue
out.append({"name": c.name, "value": c.value or "", "domain": dom,
"path": c.path or "/",
"expires": float(c.expires) if c.expires else -1,
"httpOnly": False, "secure": bool(c.secure),
"sameSite": "Lax"})
return out
TOKEN_URL = ("https://www.bristolrovers.co.uk/oauth/browser-token"
"?client_id=streamamg&token_type=urn%3Aietf%3Aparams%3Aoauth%3Atoken-type%3Aid_token")
# Public client key, shipped to every browser -- not a secret.
API_KEY = "50ivbQ3IXpF2mwT4tPAJ2V6cVm4CGRT2YKmDK5v6"
def mint_master(entry_id, quiet=False):
"""Mint a fresh master playlist URL in plain Python. No browser.
Returns (master_url, rendition_kinds). The master token lives ~25 seconds,
so mint immediately before you use the URL -- classify with one call, then
mint again for the recording.
Handing yt-dlp the *media* playlist (hdntl, path token) fails: Python
clients get 403 on that URL form, and yt-dlp fetches the playlist with
urllib before ffmpeg ever runs. The *master* (hdnts, query token) is fetched
fine by Python, so we mint one here and pass that instead. It lives only
~25 seconds, which is why it is minted immediately before recording.
"""
import http.cookiejar, json, re, urllib.request
from yt_dlp.cookies import extract_cookies_from_browser
with contextlib.redirect_stderr(io.StringIO()):
jar = extract_cookies_from_browser("firefox")
cj = http.cookiejar.CookieJar()
for c in jar:
if "bristolrovers" in (c.domain or "").lower():
cj.set_cookie(c)
op = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
hdrs = {"User-Agent": UA, "Accept": "application/json, text/plain, */*"}
body = op.open(urllib.request.Request(TOKEN_URL, headers=hdrs),
timeout=20).read().decode("utf-8", "replace")
m = re.search(r"(eyJ[\w-]+\.eyJ[\w-]+\.[\w-]+)", body)
if not m:
log(" mint: no JWT in browser-token response (session expired?)")
return None, []
jwt = m.group(1)
req = urllib.request.Request(
f"https://api.playback.streamamg.com/v1/entry/{entry_id}",
headers={**hdrs, "Authorization": "Bearer " + jwt, "x-api-key": API_KEY,
"Origin": "https://www.bristolrovers.co.uk",
"Referer": "https://www.bristolrovers.co.uk/"})
data = json.loads(urllib.request.urlopen(req, timeout=20)
.read().decode("utf-8", "replace"))
hls = (data.get("media") or {}).get("hls") or None
kinds = log_renditions(hls, quiet=quiet) if hls else []
return hls, kinds
def log_renditions(master_url, quiet=False):
"""Log what the master offers, and return the kind of each rendition.
The return value is what feed_kind() classifies an entry on. Codecs are the
only trustworthy signal: entry ids look alike, and the same id comes back
week after week for the same feed.
"""
import urllib.request
try:
req = urllib.request.Request(master_url, headers={"User-Agent": UA,
"Referer": "https://www.bristolrovers.co.uk/"})
body = urllib.request.urlopen(req, timeout=15).read().decode("utf-8", "replace")
except Exception as exc:
log(f" (could not read master for rendition list: {exc})")
return []
PREFIX = "#EXT-X-STREAM-INF:"
streams = [ln[len(PREFIX):] for ln in body.splitlines()
if ln.startswith(PREFIX)]
kinds = []
if not quiet:
log(f" master offers {len(streams)} rendition(s):")
for st in streams:
bw = re.search(r"BANDWIDTH=(\d+)", st)
res = re.search(r"RESOLUTION=([\dx]+)", st)
cod = re.search(r'CODECS="([^"]+)"', st)
codecs = cod.group(1) if cod else "?"
kind = "audio-only" if ("avc" not in codecs and "hvc" not in codecs) else "video"
kinds.append(kind)
if not quiet:
log(f" {kind:11s} {int(bw.group(1))//1000 if bw else '?':>6} kbps "
f"{res.group(1) if res else '-':>10} {codecs}")
return kinds
def feed_kind(kinds):
"""'video' if the master carries any video rendition, else 'audio'."""
if not kinds:
return "unknown"
return "video" if any(k == "video" for k in kinds) else "audio"
def find_manifest(timeout_s=90, settle_s=15):
"""Load the live page headlessly and return every entry id it exposes.
The browser is needed only for this: the entry id appears at runtime, and
the static HTML carries none when nothing is live. Everything after this
is plain HTTP.
Returns a list, not one id, because the page can expose more than one feed
for the same match -- an audio entry and a video entry. Which manifest
loads first is a race, and it has been observed to come out differently on
the same night. Returning the first one silently records the wrong feed.
So keep watching for `settle_s` after the first hit, and let the caller
choose.
"""
from playwright.sync_api import sync_playwright
entries = [] # discovery order, de-duplicated
first_seen = [None]
with sync_playwright() as p:
browser = p.firefox.launch(headless=True)
ctx = browser.new_context(user_agent=UA,
viewport={"width": 1440, "height": 900})
ctx.add_cookies(firefox_cookies())
page = ctx.new_page()
def on_response(resp):
u = resp.url
if ".m3u8" not in u.split("?")[0].lower():
return
if "hdntl=" not in u and "hdnts=" not in u:
return
m = re.search(r"/(\d+_[a-z0-9]+)/", u, re.I)
if not m or m.group(1) in entries:
return
entries.append(m.group(1))
if first_seen[0] is None:
first_seen[0] = time.time()
page.on("response", on_response)
try:
page.goto(LIVE_URL, wait_until="domcontentloaded", timeout=60000)
except Exception as exc:
log(f" page load failed: {exc}")
browser.close()
return None
# The manifest is fetched on load, before any click. We still click,
# because a stream that has only just gone live may need the nudge.
deadline = time.time() + timeout_s
clicked = False
while time.time() < deadline:
# Stop once the page has settled: a feed the player never requests
# will not appear no matter how long we wait.
if first_seen[0] and time.time() - first_seen[0] > settle_s:
break
if not clicked:
for sel in (".bmpui-ui-hugeplaybacktogglebutton",
".bmpui-ui-playbacktogglebutton",
"button[aria-label*='play' i]"):
try:
el = page.query_selector(sel)
if el and el.is_visible():
el.click()
clicked = True
break
except Exception:
pass
with contextlib.suppress(Exception):
page.evaluate("() => { const v=document.querySelector('video');"
" if (v) { v.muted = true; v.play(); } }")
page.wait_for_timeout(2000)
browser.close()
return entries
def record(url, duration_min, prefer='best', tag=None):
stamp = datetime.now().strftime("%Y%m%d-%H%M%S")
name = f"brfc-{stamp}-{tag}.ts" if tag else f"brfc-{stamp}.ts"
out = os.path.join(RECORDINGS, name)
os.makedirs(RECORDINGS, exist_ok=True)
logfile = os.path.join(LOGS, f"yt-dlp-{stamp}.log")
cmd = [
sys.executable, "-m", "yt_dlp",
# Force ffmpeg. yt-dlp's native HLS downloader fetches variant
# playlists with Python and gets 403/404 on the path-token URLs;
# ffmpeg fetches them fine. This is not a preference, it is required.
"--downloader", "ffmpeg",
"--hls-use-mpegts", # a killed process still leaves a playable file
"--no-part",
"--cookies-from-browser", "firefox",
"--user-agent", UA,
"--referer", "https://www.bristolrovers.co.uk/",
"--downloader-args",
# ffmpeg 4.3.1 has no -reconnect_on_network_error.
"ffmpeg_i:-reconnect 1 -reconnect_streamed 1 "
"-reconnect_delay_max 30",
"--downloader-args", f"ffmpeg_o:-t {duration_min * 60}",
"-o", out,
]
if prefer == "audio":
# Pin audio when the entitlement is audio-only: if a video rendition
# ever appears in the master, taking it would 403 on the segments.
cmd += ["-f", "bestaudio/best"]
elif prefer == "video":
cmd += ["-f", "bestvideo*+bestaudio/best"]
cmd.append(url)
log(f"recording -> {os.path.basename(out)} (cap {duration_min} min)")
log(f" yt-dlp log -> {os.path.basename(logfile)}")
with open(logfile, "w", encoding="utf-8") as fh:
return subprocess.Popen(cmd, stdout=fh, stderr=subprocess.STDOUT, cwd=ROOT)
def classify_entries(entries):
"""Label each discovered entry video or audio, by minting its master once.
Returns [(entry_id, kind)] in discovery order, skipping anything that will
not mint. The master minted here is thrown away -- its token lives ~25
seconds, so the recording mints its own.
"""
out = []
for entry in entries:
try:
url, kinds = mint_master(entry)
except Exception as exc:
log(f" {entry}: mint failed: {type(exc).__name__}: {exc}")
continue
if not url:
continue
kind = feed_kind(kinds)
log(f" {entry} is a {kind} feed")
out.append((entry, kind))
return out
def choose_feeds(chosen, want):
"""Order classified entries by preference. 'both' and 'any' take them all.
A preference is not a filter: the non-preferred feeds stay on the list as
fallbacks, because a feed you hold no pass for still resolves a master and
only fails later, on the segments.
"""
if want in ("both", "any"):
return chosen
first = [c for c in chosen if c[1] == want]
rest = [c for c in chosen if c[1] != want]
if not first and rest:
log(f" no {want} feed on offer -- falling back to "
f"{', '.join(k for _, k in rest)}")
return first + rest
def run_recorders(targets, args):
"""Record, and report whether a recording actually ran.
'both' starts every feed at once: concurrency is not limited and a second
copy is cheap insurance. Otherwise take the first choice and fall through
to the next only when the downloader dies immediately -- which is what an
unentitled feed looks like, the master resolving and the segments 403ing.
"""
def start(entry, kind):
try:
url, _ = mint_master(entry, quiet=True)
except Exception as exc:
log(f" {entry}: mint failed: {type(exc).__name__}: {exc}")
return None
if not url:
log(f" {entry}: could not re-mint a master; skipping")
return None
proc = record(url, args.duration, args.prefer, tag=kind)
log(f" {entry} ({kind}): downloader pid {proc.pid}")
return proc, time.time()
def finish(entry, kind, proc, t0):
proc.wait()
ran = time.time() - t0
log(f" {entry} ({kind}): rc={proc.returncode} after {ran:.0f}s")
# A master outlives its stream, so minting one proves nothing about
# whether the match is up. Only a download that actually ran does.
if proc.returncode == 0 or ran > 120:
return True
log(" died immediately -- not live yet, or no pass for this feed")
return False
if args.feed == "both":
started = []
for entry, kind in targets:
s = start(entry, kind)
if s is not None:
started.append((entry, kind) + s)
# Every one must be waited on, so collect results before reducing --
# any() would short-circuit and orphan a running downloader.
results = [finish(*row) for row in started]
return any(results)
# One feed at a time, falling through on immediate death so an unentitled
# first choice does not cost the match.
for entry, kind in targets:
s = start(entry, kind)
if s is None:
continue
if finish(entry, kind, *s):
return True
return False
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--at", help='wall-clock start, e.g. "2026-09-02 03:15"')
ap.add_argument("--max-wait", type=int, default=45,
help="minutes to keep looking for a live stream")
ap.add_argument("--duration", type=int, default=150,
help="recording cap in minutes")
ap.add_argument("--poll", type=int, default=60, help="seconds between attempts")
ap.add_argument("--prefer", choices=("best", "audio", "video"), default="best",
help="which rendition to take when the master offers both; "
"'best' is yt-dlp's default and prefers video")
ap.add_argument("--entry", help="skip discovery and record this entry id, "
"bypassing --feed entirely")
ap.add_argument("--feed", choices=("video", "audio", "any", "both"),
default="video",
help="which FEED to record when the page exposes more than "
"one entry. 'video' prefers a video feed and falls "
"back to audio; 'both' records every feed in parallel. "
"Distinct from --prefer, which picks a rendition "
"inside one feed and cannot rescue a wrong feed.")
args = ap.parse_args()
if args.at:
target = datetime.strptime(args.at, "%Y-%m-%d %H:%M")
wait = (target - datetime.now()).total_seconds()
if wait > 0:
log(f"sleeping {wait/3600:.2f}h until {target:%Y-%m-%d %H:%M}")
time.sleep(wait)
deadline = time.time() + args.max_wait * 60
attempt = 0
while time.time() < deadline:
attempt += 1
if args.entry:
entries = [args.entry]
log(f"attempt {attempt}: pinned entry {args.entry}")
else:
log(f"attempt {attempt}: looking for a live stream")
try:
entries = find_manifest()
except Exception as exc:
log(f" error: {type(exc).__name__}: {exc}")
entries = []
if not entries:
log(f" nothing live yet; retrying in {args.poll}s")
time.sleep(args.poll)
continue
log(f" live entries: {', '.join(entries)}")
chosen = classify_entries(entries)
if not chosen:
log(" could not mint a manifest for any entry; retrying")
time.sleep(args.poll)
continue
targets = choose_feeds(chosen, args.feed)
if not targets:
log(f" no {args.feed} feed among {[k for _, k in chosen]}; retrying")
time.sleep(args.poll)
continue
if run_recorders(targets, args) or time.time() >= deadline:
return 0
time.sleep(args.poll)
log("gave up: no live stream appeared within the wait window")
return 1
if __name__ == "__main__":
sys.exit(main())
+70
View File
@@ -27,6 +27,13 @@
// once the endpoint is known -- it writes a lot. // once the endpoint is known -- it writes a lot.
const TRACE = true; const TRACE = true;
// Start playback without a human. The player needs a click, but the browser
// only has to trigger the manifest fetch -- ffmpeg records the real audio
// independently, so playing MUTED costs nothing and muted autoplay is
// permitted by Firefox policy where audible autoplay is not.
const AUTOPLAY = true;
const AUTOPLAY_TIMEOUT_MS = 180000; // pre-match margin is long; keep trying
const IDLE_OPACITY = 0.063; // 16/255 const IDLE_OPACITY = 0.063; // 16/255
const MAX_BODY = 20000; // never log a whole media segment const MAX_BODY = 20000; // never log a whole media segment
@@ -188,4 +195,67 @@
panel.appendChild(row); panel.appendChild(row);
} }
} }
// --- autoplay -------------------------------------------------------------
// Runs in whichever frame owns the <video>, which may be the player iframe.
function findVideo() {
const v = document.querySelector('video');
return (v && v.isConnected) ? v : null;
}
function clickPlayButton() {
const sels = [
'.bmpui-ui-hugeplaybacktogglebutton', // Bitmovin centre button
'.bmpui-ui-playbacktogglebutton', // Bitmovin control bar
'button[aria-label*="play" i]',
'[class*="play-button"]', '[class*="playButton"]'
];
for (const s of sels) {
const el = document.querySelector(s);
if (el && el.offsetParent !== null) { // must be visible to be real
el.click();
console.log('[brfc-sniffer] autoplay: clicked', s);
return true;
}
}
return false;
}
function startAutoplay() {
if (!AUTOPLAY) return;
const deadline = Date.now() + AUTOPLAY_TIMEOUT_MS;
const timer = setInterval(() => {
const v = findVideo();
// Done once something is actually playing. Critically, we must stop
// here: the play control is a TOGGLE, so clicking it again would pause
// playback and silently end the recording.
if (v && !v.paused && !v.ended && v.readyState > 2) {
console.log('[brfc-sniffer] autoplay: playing, standing down');
clearInterval(timer);
return;
}
if (Date.now() > deadline) {
console.log('[brfc-sniffer] autoplay: gave up after',
AUTOPLAY_TIMEOUT_MS / 1000, 's');
clearInterval(timer);
return;
}
if (v) {
v.muted = true; // audible autoplay is blocked; muted is not
const p = v.play();
if (p && p.catch) p.catch(() => { clickPlayButton(); });
} else {
clickPlayButton(); // player chrome may exist before <video>
}
}, 2000);
}
if (document.readyState === 'loading') {
document.addEventListener('DOMContentLoaded', startAutoplay);
} else {
startAutoplay();
}
})(); })();
+128
View File
@@ -0,0 +1,128 @@
#!/usr/bin/env python3
"""Backup starter: confirm something is actually recording, else start it.
Schedule this a few minutes before kickoff, behind the primary recording task.
It exists because a scheduled launch of record_live.py has been seen to hang
dead at interpreter startup -- process alive, zero CPU, nothing logged, forever.
Unattended, that loses the match silently.
So "is a recorder process running" is the wrong question: the hung process
answers yes. This asks whether recording is *progressing*:
1. a .ts in recordings/ grew in the last --quiet-secs -> healthy, leave it
2. record_live.log was written in the last --quiet-secs -> mid search loop,
leave it
3. neither -> stale. Kill any
recorder process
and start a fresh
one.
Killing only ever targets a recorder that has gone quiet on both signals, so a
healthy capture is never interrupted.
python watchdog.py --duration 170 --max-wait 55
"""
import argparse
import glob
import os
import subprocess
import sys
import time
from datetime import datetime
ROOT = os.path.dirname(os.path.abspath(__file__))
RECORDINGS = os.path.join(ROOT, "recordings")
LOGS = os.path.join(ROOT, "logs")
RECORDER = os.path.join(ROOT, "record_live.py")
def log(msg):
line = f"[{datetime.now():%Y-%m-%d %H:%M:%S}] {msg}"
print(line, flush=True)
os.makedirs(LOGS, exist_ok=True)
with open(os.path.join(LOGS, "watchdog.log"), "a", encoding="utf-8") as fh:
fh.write(line + chr(10))
def powershell(cmd):
try:
out = subprocess.run(["powershell", "-NoProfile", "-Command", cmd],
capture_output=True, text=True, timeout=60)
return out.stdout.strip()
except Exception as exc:
log(f" powershell failed: {type(exc).__name__}: {exc}")
return ""
def recorder_pids():
"""PIDs of any running record_live.py, hung or healthy."""
out = powershell(
"Get-CimInstance Win32_Process -Filter \"Name='python.exe'\" | "
"Where-Object { $_.CommandLine -match 'record_live' } | "
"ForEach-Object { $_.ProcessId }")
return [int(x) for x in out.split() if x.strip().isdigit()]
def newest_mtime(pattern):
files = glob.glob(pattern)
return max((os.path.getmtime(f) for f in files), default=0)
def main():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--duration", type=int, default=170,
help="recording cap in minutes for a run we start")
ap.add_argument("--max-wait", type=int, default=55,
help="minutes to keep looking for a live stream")
ap.add_argument("--prefer", choices=("best", "audio", "video"), default="best")
ap.add_argument("--quiet-secs", type=int, default=180,
help="how stale both signals must be before we intervene")
args = ap.parse_args()
log("watchdog: checking whether anything is recording")
now = time.time()
ts_age = now - newest_mtime(os.path.join(RECORDINGS, "*.ts"))
log_age = now - newest_mtime(os.path.join(LOGS, "record_live.log"))
pids = recorder_pids()
log(f" recorder pids {pids or 'none'}; newest .ts {ts_age:.0f}s old; "
f"record_live.log {log_age:.0f}s old")
if ts_age < args.quiet_secs:
log(" a recording is growing -- healthy, nothing to do")
return 0
if pids and log_age < args.quiet_secs:
log(" recorder is logging -- mid search loop, nothing to do")
return 0
if pids:
log(f" recorder {pids} is alive but both signals are stale -- killing it")
for pid in pids:
powershell(f"Stop-Process -Id {pid} -Force -ErrorAction SilentlyContinue")
time.sleep(3)
still = recorder_pids()
if still:
log(f" WARNING: {still} survived the kill; starting anyway")
else:
log(" nothing is recording")
cmd = [sys.executable, RECORDER,
"--max-wait", str(args.max_wait),
"--duration", str(args.duration),
"--prefer", args.prefer]
log(f" starting: {' '.join(cmd[1:])}")
# Detached: the watchdog task exits immediately, the recorder outlives it.
flags = 0
if os.name == "nt":
flags = subprocess.CREATE_NEW_PROCESS_GROUP | getattr(
subprocess, "DETACHED_PROCESS", 0)
proc = subprocess.Popen(cmd, cwd=ROOT, creationflags=flags,
stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL)
log(f" started recorder pid {proc.pid}")
return 0
if __name__ == "__main__":
sys.exit(main())