brfc-stream-pirate/README.md

# brfc-stream-pirate

Personal time-shifting archive for Bristol Rovers video pass streams.
Kickoffs land at unfriendly hours in JST; this records them unattended.

Stream is **plain HLS, no DRM** (StreamAMG behind Akamai). Nothing here breaks
a protection measure -- it replays the same authenticated request the player
makes, using your own paid session.

## Pieces

| File | Job |
|---|---|
| `record_live.py` | Unattended recorder: finds the live entry, mints a master, records |
| `preflight.py` | Pre-match check: is the session live, can this host record |
| `watchdog.py` | Backup starter: if nothing is *progressing*, start a recorder |
| `catcher.py` | Listens on 127.0.0.1:8787, hands a sniffed URL to yt-dlp |
| `tampermonkey/brfc-stream-sniffer.user.js` | Hooks fetch/XHR, POSTs the manifest URL to the catcher |
| `discover.py` | Drives the player with Playwright to capture token-minting traffic |
| `probe.py` | Byte-exact manifest fetch + concurrency test |

## Recording a match

    python preflight.py                   # exit 0 only if the session is live

Mandatory, hours before kickoff. A dead session is invisible locally and the
only fix needs a human at a browser. Then schedule the recorder for ~30 min
before kickoff:

    python record_live.py --max-wait 75 --duration 180 --feed video

**Pick the feed, not just the rendition.** The live page can expose two entries
for one match -- an audio feed and a video feed -- and which manifest loads
first is a race that has come out differently on the same night. `--feed`
chooses between entries and is the setting that matters:

| `--feed` | Behaviour |
|---|---|
| `video` (default) | record a video feed; fall back to audio if none is offered |
| `audio` | the reverse |
| `any` | whichever was discovered first -- the old behaviour |
| `both` | record every feed in parallel |
| `--entry <id>` | skip discovery entirely and pin one |

Feeds are classified by the codecs in their master, never by entry id. A
non-preferred feed stays on the list as a fallback, because a feed you hold no
pass for resolves a master fine and only fails later on the segments; the
recorder falls through to the next candidate when a download dies immediately.

`--prefer` is the *other* choice -- which rendition inside one master -- and
`best` is almost always right. A wrong `--feed` cannot be rescued by `--prefer`,
and it fails quietly: a complete, playable, healthy recording of the wrong
thing. `--feed both` costs ~210 MB extra and removes the question.

**Warm the scheduled task.** After registering it, launch it once by hand and
stop it again. A freshly registered task has been seen to hang at interpreter
startup -- alive, zero CPU, nothing logged, frozen before any of our code runs.
Relaunching fixed it; root cause unknown. Stopping the test run matters as much:
`MultipleInstances IgnoreNew` means a test left running swallows the real
trigger silently.

Optionally schedule `watchdog.py` ~20 min later as a backup. It checks whether
recording is *progressing* (a growing `.ts`, or a freshly written log), not
whether a process exists -- a hung recorder passes the process check.

### Manual alternative

    python catcher.py                     # leave running

Install the userscript, open the match page, press play. The sniffer catches the
manifest, the catcher starts yt-dlp. Config via environment:

| Variable | Default | Effect |
|---|---|---|
| `BRFC_BROWSER` | `firefox` | which browser's cookie jar yt-dlp reads |
| `BRFC_DURATION_MIN` | `150` | catcher's recording cap, minutes |

Check concurrency against a live token with `probe.py --url "<manifest>"
--parallel 3`.

## The token

Manifest URLs carry an Akamai path token:

    hdntl=exp=<unix>~acl=/*~id=<n>~data=hdntl~hmac=<sig>

It is a **bearer credential** with `acl=/*` over the whole host, valid to `exp`.
Anyone holding a live URL can pull the stream. Keep it out of pastebins and
issue trackers; `stream-url.txt` and `cookies.txt` are gitignored.

Two tokens, not one:

| Token | Where | TTL | Fetches |
|---|---|---|---|
| `hdnts=` | query string, `acl=/*` unencoded | ~25 seconds | master playlist |
| `hdntl=` | URL path, `acl=%2f*` encoded | ~24 hours | media playlists, segments |

The master response rewrites each variant URL with a 24h `hdntl`. Chasing a
`hdnts` URL by hand is pointless -- it expires before you can paste it.

**Do not re-encode the path.** curl normalises `%2f` to `%2F`, invalidating the
HMAC. `probe.py` sends the request target verbatim.

## Rules that are not preferences

- **Hand the downloader a master URL, not a media one.** Python gets 403 on
  path-token media playlists; ffmpeg gets 200. yt-dlp fetches playlists with
  Python before ffmpeg runs, so it must be given a `hdnts` master.
  `record_live.py` mints one immediately before recording.
- **Force `--downloader ffmpeg`.** yt-dlp's native HLS downloader dies on the
  path-token URLs for the same reason.
- **Keep the duration cap.** A feed can keep serving after full time -- a static
  slate on video, digital silence on audio -- at full bitrate, with no
  `#EXT-X-ENDLIST` and no stall to detect. Of five observed endings three
  self-terminated and two ran forever, on both feed types, with no pattern that
  predicts which. Treat a stream that stops by itself as luck.
- **Use Firefox as the session holder.** Chrome/Edge lock their cookie DB while
  running (yt-dlp issue 7271) and Chrome 127+ on Windows adds app-bound cookie
  encryption; an unattended job cannot close a browser to read them.
- **Do not judge a token by `probe.py`.** It inherits the path-token 403, so a
  live token looks dead through it.

## Facts

- 1920x1080 H.264 + AAC ~5.2 Mbit/s. Audio-only feeds are ~140 kbps AAC.
- Output is MPEG-TS even when named `.mp4` -- that is what `--hls-use-mpegts`
  buys: a killed process still leaves a playable file.
- Budget **~36 MB/min video** (~6.5 GB for a 180 min cap), **~1.2 MB/min audio**
  (~210 MB).
- **Concurrency is not limited.** An Akamai edge token is a stateless HMAC with
  no viewer counter. Three readers of one token ran at full bitrate alongside a
  playing browser.
- **No CDN cookies exist.** All relevant cookies are on `bristolrovers.co.uk`,
  none on `akamaized.net`. The token in the URL is the whole auth.
- Player is Bitmovin. The Kaltura entry id names the **feed, not the match**:
  `0_xm5vrrw6` (video) and `0_z236v0e7` (audio) each recurred weeks apart.
  Nothing may use it to answer "have I recorded this game" -- use the date.
  `catcher.py` is unaffected; it dedupes on liveness, not identity.

## The session

The `SSESS` cookie is the whole login. Everything downstream -- JWT, entry API,
master playlist -- rides it, in plain Python, with no browser or fingerprint
forging.

**Cookie expiry is not session lifetime.** The server can drop a session while
the browser holds a cookie that looks perfectly healthy:

    SSESS...  76 days left on the clock
    GET /oauth/browser-token  ->  403 {"error":"Unauthenticated"}
    Set-Cookie: SSESS...=deleted; expires=Thu, 01 Jan 1970

One session died after six days; its replacement survived seven and counting.
The lifetime is unpredictable and only the server knows it, so a local cookie
check proves nothing and `preflight.py` is not a formality.

Other session behaviour:

- **Sessions are concurrent.** Logging in via a second browser does not
  invalidate the first -- Drupal issues a session per browser. So a second
  device can hold its own session without disturbing the recorder's. *One
  observation, checked immediately after the second login.*
- `/oauth/browser-token` returns the session's **existing** JWT rather than
  minting a fresh one per call, so the token's expiry is not a freshness
  signal. A dead session returns 403, which is what the check relies on.
- The cookie *value* changed once between weeks with no re-login recorded.
  Unexplained. Treat a copied cookie as a snapshot that can go stale on its own.
- The session cookie is **httpOnly**, so `document.cookie` cannot read it. Any
  provisioning scheme must use a platform cookie store, or it will silently
  collect every cookie except the one that matters.

### Decision: do not automate login

Automating it is the riskiest step in the chain and the only one needing a
stored password, against an unknown `captcha_token` and a lockout that would
take out the working session too. Instead assume the session can be dead before
any given match and make `preflight.py` routine.

Cookie *provisioning* to a second host remains the open problem.

## Status

Unattended recording works end to end, scheduled, with nobody awake. Five
matches recorded as of 2026-09-20, including one night capturing both feeds in
parallel.

**Verified:** the token chain in plain Python; recording both feed types via
ffmpeg; Task Scheduler launching the job; concurrent readers of one token; a
dead session being detectable before kickoff; the duration cap firing at `rc=0`
on a feed that would not stop.

**Not verified:** obtaining a session *from credentials*. `/user/login` is a
standard Drupal form (`name`, `pass`, `form_build_id`, `form_id`,
`captcha_token`) with no reCAPTCHA script and no 2FA mention, but
`captcha_token` is unknown and probing it risks a lockout.