Capture Bristol Rovers video pass streams for personal time-shifting
Verified end to end against a live match: 1080p H.264/AAC MPEG-TS at ~5.2 Mbit/s, recorded while the browser played the same stream. Components: - tampermonkey sniffer hooks fetch/XHR at document-start, captures the manifest URL verbatim and POSTs it to the local catcher. v0.4 also traces the API chain to find which call mints the token. - catcher receives the URL and launches yt-dlp. Dedupes on the Kaltura asset id and checks downloader liveness, so a mid-match page reload is both safe and the recovery path for a dead downloader. - creds stores secrets in the OS vault rather than a gitignored file, and refuses to run on a backend that does not encrypt. Protocol notes are in the README: the stream uses a two-stage token chain (a ~25s hdnts token for the master playlist, a ~24h hdntl token for media), concurrency is unrestricted, and the live window is ~30s with no DVR. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kyt7QHQvLJtQRuiY4wAqKB
This commit is contained in:
@@ -0,0 +1,7 @@
|
|||||||
|
recordings/
|
||||||
|
logs/
|
||||||
|
stream-url.txt
|
||||||
|
cookies.txt
|
||||||
|
*.mkv
|
||||||
|
*.mp4
|
||||||
|
*.ts
|
||||||
@@ -0,0 +1,111 @@
|
|||||||
|
# brfc-stream-pirate
|
||||||
|
|
||||||
|
Personal time-shifting archive for Bristol Rovers video pass streams.
|
||||||
|
Kickoffs land at unfriendly hours in JST; this records them unattended.
|
||||||
|
|
||||||
|
Stream is **plain HLS, no DRM** (StreamAMG behind Akamai). Nothing here breaks
|
||||||
|
a protection measure -- it replays the same authenticated request the player
|
||||||
|
makes, using your own paid session.
|
||||||
|
|
||||||
|
## Pieces
|
||||||
|
|
||||||
|
| File | Job |
|
||||||
|
|---|---|
|
||||||
|
| `tampermonkey/brfc-stream-sniffer.user.js` | Hooks fetch/XHR in the page, captures the manifest URL verbatim, POSTs it to the catcher |
|
||||||
|
| `catcher.py` | Listens on 127.0.0.1:8787, hands the URL to yt-dlp, records to `recordings/` |
|
||||||
|
| `probe.py` | Byte-exact manifest fetch + concurrency test |
|
||||||
|
|
||||||
|
## The token
|
||||||
|
|
||||||
|
Manifest URLs carry an Akamai path token:
|
||||||
|
|
||||||
|
hdntl=exp=<unix>~acl=/*~id=<n>~data=hdntl~hmac=<sig>
|
||||||
|
|
||||||
|
- `exp` -- ~24h validity
|
||||||
|
- `acl=/*` -- wildcard over the whole host, not just your match
|
||||||
|
- `hmac` -- signature over the literal path
|
||||||
|
|
||||||
|
It is a bearer credential. Anyone holding a live URL can pull the stream until
|
||||||
|
`exp`. Keep it out of pastebins and issue trackers. `stream-url.txt` and
|
||||||
|
`cookies.txt` are gitignored.
|
||||||
|
|
||||||
|
**Do not re-encode the path.** curl normalises `%2f` to `%2F`, which
|
||||||
|
invalidates the HMAC. `probe.py` sends the request target verbatim.
|
||||||
|
|
||||||
|
## Use Firefox for watching
|
||||||
|
|
||||||
|
yt-dlp lifts session cookies from the browser. Chrome 127+ on Windows uses
|
||||||
|
app-bound cookie encryption and extraction fails. Firefox works with no fuss.
|
||||||
|
|
||||||
|
set BRFC_BROWSER=firefox # default
|
||||||
|
|
||||||
|
## Workflow
|
||||||
|
|
||||||
|
pip install yt-dlp # done
|
||||||
|
python catcher.py # leave running
|
||||||
|
|
||||||
|
Install the userscript in Tampermonkey, open the match page, press play. The
|
||||||
|
sniffer catches the manifest, the catcher starts yt-dlp. Recording is MPEG-TS
|
||||||
|
(`--hls-use-mpegts`) so a killed process still leaves a playable file.
|
||||||
|
|
||||||
|
For a 3am kickoff: Task Scheduler opens the match page a few minutes before,
|
||||||
|
the browser autoplays, the sniffer fires, the catcher records. Nobody is awake.
|
||||||
|
|
||||||
|
## Concurrency check
|
||||||
|
|
||||||
|
An Akamai edge token is a stateless HMAC -- the edge validates signature and
|
||||||
|
expiry and serves; it holds no session table, so it cannot count viewers. Any
|
||||||
|
concurrent-stream limit lives in the entitlement API that *issues* tokens,
|
||||||
|
upstream of the CDN. Confirm rather than assume, while the browser is playing:
|
||||||
|
|
||||||
|
python probe.py --url "<manifest from the sniffer>" --parallel 3
|
||||||
|
|
||||||
|
3/3 alongside the playing browser means four concurrent readers of one token
|
||||||
|
and the edge does not care.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Findings (2026-08-29, live 2300 JST match)
|
||||||
|
|
||||||
|
Verified against a live stream. Recorded 1080p successfully end to end.
|
||||||
|
|
||||||
|
### Two token types, not one
|
||||||
|
|
||||||
|
The player uses a **two-stage token chain**:
|
||||||
|
|
||||||
|
| Token | Where | TTL | Purpose |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `hdnts=` | query string, `acl=/*` unencoded | **~25 seconds** | fetch the master playlist |
|
||||||
|
| `hdntl=` | URL path, `acl=%2f*` encoded | **~24 hours** | media playlists + segments |
|
||||||
|
|
||||||
|
The master response rewrites each variant URL with a 24h `hdntl` token. So the
|
||||||
|
`hdntl` token is the durable one worth handing to a downloader. Chasing a
|
||||||
|
`hdnts` URL by hand is pointless -- it dies before you can paste it.
|
||||||
|
|
||||||
|
### Confirmed by experiment
|
||||||
|
|
||||||
|
- **Concurrency is not limited.** Two independent ffmpeg downloads ran at full
|
||||||
|
bitrate *while Firefox played the same match*. Three concurrent readers of
|
||||||
|
one token, no complaint from the edge. Akamai token auth is a stateless HMAC
|
||||||
|
check; it has no viewer counter.
|
||||||
|
- **No CDN cookies exist.** All 13 relevant cookies are on `bristolrovers.co.uk`;
|
||||||
|
nothing is scoped to `akamaized.net`. The token in the URL is the whole auth.
|
||||||
|
- **Stream is 1920x1080 H.264 + AAC ~5.2 Mbit/s** -> roughly 4.7 GB per match.
|
||||||
|
- **Output is MPEG-TS** even though yt-dlp names it `.mp4`. That is what
|
||||||
|
`--hls-use-mpegts` buys: a killed process still leaves a playable file.
|
||||||
|
- Player is **Bitmovin**, embedded on the club site. `0_xm5vrrw6` is a Kaltura
|
||||||
|
entry id and identifies the match -- the catcher dedupes on it.
|
||||||
|
|
||||||
|
### Gotcha: use ffmpeg/yt-dlp, not a hand-rolled HTTP client
|
||||||
|
|
||||||
|
Python's `http.client` gets a **403 on every request**, with any header set,
|
||||||
|
including ffmpeg's exact headers, on an unexpired token from the same IP. The
|
||||||
|
difference is below HTTP (TLS handshake). ffmpeg fetches the same URL fine, so
|
||||||
|
just use yt-dlp. `probe.py` is diagnostic scaffolding and inherits this 403 --
|
||||||
|
do not use it to conclude a token is dead.
|
||||||
|
|
||||||
|
### Firefox, not Chrome
|
||||||
|
|
||||||
|
yt-dlp reads cookies from a running Firefox. Chrome/Edge lock their cookie DB
|
||||||
|
while running (yt-dlp issue 7271), and an unattended job cannot close the
|
||||||
|
browser to read them.
|
||||||
+151
@@ -0,0 +1,151 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Receive a manifest URL from the Tampermonkey sniffer and record the match.
|
||||||
|
|
||||||
|
Runs on localhost only. The userscript POSTs whatever manifest URL the player
|
||||||
|
actually requested; we hand it to yt-dlp.
|
||||||
|
|
||||||
|
Reload resilience is the whole point of the liveness check below. A page
|
||||||
|
reload mid-match is expected (network blips, tab crashes), and it must be
|
||||||
|
*safe* -- it must not start a duplicate recording, and it must restart a
|
||||||
|
recording that has died. So we dedupe on "is a downloader for this match
|
||||||
|
still alive", not "have I seen this match before".
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import threading
|
||||||
|
from datetime import datetime
|
||||||
|
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||||
|
|
||||||
|
HOST, PORT = "127.0.0.1", 8787
|
||||||
|
ROOT = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
RECORDINGS = os.path.join(ROOT, "recordings")
|
||||||
|
LOGS = os.path.join(ROOT, "logs")
|
||||||
|
|
||||||
|
# yt-dlp lifts session cookies from here. Firefox is required on Windows:
|
||||||
|
# Chrome/Edge lock their cookie DB while running (yt-dlp issue 7271).
|
||||||
|
BROWSER = os.environ.get("BRFC_BROWSER", "firefox")
|
||||||
|
|
||||||
|
# Hard stop, so an unattended run cannot record forever if the stream never
|
||||||
|
# ends cleanly. Pad past 90 min for half-time, stoppage and post-match.
|
||||||
|
DURATION_MIN = int(os.environ.get("BRFC_DURATION_MIN", "150"))
|
||||||
|
|
||||||
|
# Kaltura entry id (e.g. 0_xm5vrrw6) identifies the match. Every manifest for
|
||||||
|
# one game -- master and each variant, each with its own token -- shares it.
|
||||||
|
ASSET_RE = re.compile(r"/(\d+_[a-z0-9]+)/", re.I)
|
||||||
|
|
||||||
|
_procs = {} # asset key -> Popen of the running downloader
|
||||||
|
_lock = threading.Lock()
|
||||||
|
|
||||||
|
|
||||||
|
def log(msg):
|
||||||
|
line = f"[{datetime.now():%Y-%m-%d %H:%M:%S}] {msg}"
|
||||||
|
print(line, flush=True)
|
||||||
|
os.makedirs(LOGS, exist_ok=True)
|
||||||
|
with open(os.path.join(LOGS, "catcher.log"), "a", encoding="utf-8") as fh:
|
||||||
|
fh.write(line + "\n")
|
||||||
|
|
||||||
|
|
||||||
|
def asset_key(url):
|
||||||
|
m = ASSET_RE.search(url)
|
||||||
|
return m.group(1) if m else url
|
||||||
|
|
||||||
|
|
||||||
|
def start_recording(key, url, user_agent, page):
|
||||||
|
stamp = datetime.now().strftime("%Y%m%d-%H%M%S")
|
||||||
|
out = os.path.join(RECORDINGS, f"brfc-{key}-{stamp}.ts")
|
||||||
|
os.makedirs(RECORDINGS, exist_ok=True)
|
||||||
|
|
||||||
|
cmd = [
|
||||||
|
sys.executable, "-m", "yt_dlp",
|
||||||
|
"--hls-use-mpegts", # truncated file still plays
|
||||||
|
"--no-part",
|
||||||
|
"--cookies-from-browser", BROWSER,
|
||||||
|
"--user-agent", user_agent or "Mozilla/5.0",
|
||||||
|
"--referer", page or "https://www.bristolrovers.co.uk/",
|
||||||
|
# yt-dlp's own retry flags do NOT reach the ffmpeg downloader, which is
|
||||||
|
# what actually pulls live HLS. These are ffmpeg's own reconnect knobs.
|
||||||
|
"--downloader-args",
|
||||||
|
"ffmpeg_i:-reconnect 1 -reconnect_streamed 1 "
|
||||||
|
"-reconnect_on_network_error 1 -reconnect_delay_max 30",
|
||||||
|
"--downloader-args", f"ffmpeg_o:-t {DURATION_MIN * 60}",
|
||||||
|
"-o", out,
|
||||||
|
url,
|
||||||
|
]
|
||||||
|
logfile = os.path.join(LOGS, f"yt-dlp-{stamp}.log")
|
||||||
|
fh = open(logfile, "w", encoding="utf-8")
|
||||||
|
proc = subprocess.Popen(cmd, stdout=fh, stderr=subprocess.STDOUT, cwd=ROOT)
|
||||||
|
log(f"START {key} -> {os.path.basename(out)} (pid {proc.pid}, cap {DURATION_MIN}min)")
|
||||||
|
return proc
|
||||||
|
|
||||||
|
|
||||||
|
class Handler(BaseHTTPRequestHandler):
|
||||||
|
def do_POST(self):
|
||||||
|
if self.path not in ("/stream", "/trace"):
|
||||||
|
self.send_error(404)
|
||||||
|
return
|
||||||
|
n = int(self.headers.get("Content-Length", 0))
|
||||||
|
try:
|
||||||
|
body = json.loads(self.rfile.read(n) or b"{}")
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
self.send_error(400, "bad json")
|
||||||
|
return
|
||||||
|
|
||||||
|
if self.path == "/trace":
|
||||||
|
self.do_TRACE_post(body)
|
||||||
|
self.send_response(200)
|
||||||
|
self.send_header("Access-Control-Allow-Origin", "*")
|
||||||
|
self.end_headers()
|
||||||
|
self.wfile.write(b'{"ok":true}')
|
||||||
|
return
|
||||||
|
|
||||||
|
url = body.get("url", "")
|
||||||
|
if not url:
|
||||||
|
self.send_error(400, "no url")
|
||||||
|
return
|
||||||
|
|
||||||
|
key = asset_key(url)
|
||||||
|
action = "ignored"
|
||||||
|
with _lock:
|
||||||
|
proc = _procs.get(key)
|
||||||
|
if proc is not None and proc.poll() is None:
|
||||||
|
action = "already recording" # healthy: reload is a no-op
|
||||||
|
else:
|
||||||
|
if proc is not None:
|
||||||
|
log(f"RESTART {key}: downloader had exited (rc={proc.returncode})")
|
||||||
|
_procs[key] = start_recording(key, url, body.get("userAgent"),
|
||||||
|
body.get("page"))
|
||||||
|
action = "started"
|
||||||
|
|
||||||
|
# Always 200 the browser; the player re-requests manifests constantly.
|
||||||
|
self.send_response(200)
|
||||||
|
self.send_header("Content-Type", "application/json")
|
||||||
|
self.send_header("Access-Control-Allow-Origin", "*")
|
||||||
|
self.end_headers()
|
||||||
|
self.wfile.write(json.dumps({"ok": True, "action": action}).encode())
|
||||||
|
|
||||||
|
def do_TRACE_post(self, body):
|
||||||
|
"""Discovery mode: append the request/response the page made.
|
||||||
|
|
||||||
|
Used to find which API call mints the stream token, so the chain can
|
||||||
|
run without a GUI browser. Written as JSONL for easy grepping.
|
||||||
|
"""
|
||||||
|
os.makedirs(LOGS, exist_ok=True)
|
||||||
|
with open(os.path.join(LOGS, "trace.jsonl"), "a", encoding="utf-8") as fh:
|
||||||
|
fh.write(json.dumps(body, ensure_ascii=False) + chr(10))
|
||||||
|
|
||||||
|
def do_OPTIONS(self):
|
||||||
|
self.send_response(204)
|
||||||
|
self.send_header("Access-Control-Allow-Origin", "*")
|
||||||
|
self.send_header("Access-Control-Allow-Headers", "Content-Type")
|
||||||
|
self.end_headers()
|
||||||
|
|
||||||
|
def log_message(self, *args):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
log(f"catcher listening on http://{HOST}:{PORT}/stream (cookies: {BROWSER})")
|
||||||
|
HTTPServer((HOST, PORT), Handler).serve_forever()
|
||||||
@@ -0,0 +1,156 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Credential storage for unattended recording.
|
||||||
|
|
||||||
|
Secrets live in the Windows Credential Manager (DPAPI, keyed to your Windows
|
||||||
|
account), not in a file. Nothing here can be committed by accident because
|
||||||
|
nothing here is written to the repo.
|
||||||
|
|
||||||
|
python creds.py --set-login store email + password (prompts)
|
||||||
|
python creds.py --show show what is stored (never the password)
|
||||||
|
python creds.py --clear remove everything
|
||||||
|
|
||||||
|
What this does and does not protect against:
|
||||||
|
|
||||||
|
DOES -- plaintext at rest, and the classic "gitignored file gets committed
|
||||||
|
after a git add -A" accident.
|
||||||
|
DOES -- readable by another Windows user on this machine.
|
||||||
|
NOT -- a process already running as you. Credential Manager unlocks with
|
||||||
|
your login session, so anything running as you can read it. That is
|
||||||
|
inherent to unattended operation: a job that logs in without you
|
||||||
|
present must be able to reach the secret without you present.
|
||||||
|
|
||||||
|
Caveat for Task Scheduler: "Run whether user is logged on or not" runs in a
|
||||||
|
session where DPAPI may not be able to decrypt these entries. Prefer "Run only
|
||||||
|
when user is logged on" for any task that needs credentials.
|
||||||
|
|
||||||
|
PLATFORM NOTE -- this module is only safe where a real vault backend exists:
|
||||||
|
|
||||||
|
Windows WinVaultKeyring (Credential Manager, DPAPI). Good.
|
||||||
|
Linux+GUI SecretService (GNOME Keyring / KWallet). Good.
|
||||||
|
Headless No D-Bus session, no unlocked keyring. `keyring` NOT SAFE.
|
||||||
|
Linux falls back to a fail or plaintext backend.
|
||||||
|
|
||||||
|
On a headless Pi or server, do NOT use this module. Use systemd-creds, which
|
||||||
|
encrypts to the host/TPM key and is decrypted only for the service:
|
||||||
|
|
||||||
|
systemd-creds encrypt --name=brfc-pass - /etc/credstore.encrypted/brfc-pass
|
||||||
|
# then in the unit: LoadCredential=brfc-pass
|
||||||
|
# and read it from: $CREDENTIALS_DIRECTORY/brfc-pass
|
||||||
|
|
||||||
|
Run `python creds.py --check` on each host before trusting it with a secret.
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import getpass
|
||||||
|
import sys
|
||||||
|
|
||||||
|
try:
|
||||||
|
import keyring
|
||||||
|
except ImportError:
|
||||||
|
sys.exit("keyring not installed. Run: pip install keyring")
|
||||||
|
|
||||||
|
SERVICE = "brfc-stream-pirate"
|
||||||
|
|
||||||
|
# Keys held in the vault.
|
||||||
|
K_USER = "username"
|
||||||
|
K_PASS = "password"
|
||||||
|
K_COOKIE = "session-cookie" # cached login session, refreshed periodically
|
||||||
|
|
||||||
|
|
||||||
|
def get(key):
|
||||||
|
return keyring.get_password(SERVICE, key)
|
||||||
|
|
||||||
|
|
||||||
|
def put(key, value):
|
||||||
|
keyring.set_password(SERVICE, key, value)
|
||||||
|
|
||||||
|
|
||||||
|
def drop(key):
|
||||||
|
try:
|
||||||
|
keyring.delete_password(SERVICE, key)
|
||||||
|
except keyring.errors.PasswordDeleteError:
|
||||||
|
pass # already absent
|
||||||
|
|
||||||
|
|
||||||
|
def load_login():
|
||||||
|
"""Return (username, password), or None if not configured yet."""
|
||||||
|
user, pw = get(K_USER), get(K_PASS)
|
||||||
|
return (user, pw) if user and pw else None
|
||||||
|
|
||||||
|
|
||||||
|
def load_cookie():
|
||||||
|
"""Cached session cookie, or None. Callers must handle it being stale."""
|
||||||
|
return get(K_COOKIE)
|
||||||
|
|
||||||
|
|
||||||
|
def save_cookie(value):
|
||||||
|
put(K_COOKIE, value)
|
||||||
|
|
||||||
|
|
||||||
|
# Backends that do not actually protect anything. `keyring` selects these
|
||||||
|
# silently on a headless box, so a secret stored there is effectively plaintext.
|
||||||
|
UNSAFE_BACKENDS = {"PlaintextKeyring", "Keyring", "fail.Keyring", "EnvironCredential"}
|
||||||
|
|
||||||
|
|
||||||
|
def backend():
|
||||||
|
return keyring.get_keyring().__class__.__name__
|
||||||
|
|
||||||
|
|
||||||
|
def backend_is_safe():
|
||||||
|
name = backend()
|
||||||
|
return name not in UNSAFE_BACKENDS and "fail" not in name.lower()
|
||||||
|
|
||||||
|
|
||||||
|
def _mask(s):
|
||||||
|
if not s:
|
||||||
|
return "(not set)"
|
||||||
|
return s[:2] + "*" * max(0, len(s) - 4) + s[-2:] if len(s) > 4 else "*" * len(s)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||||
|
g = ap.add_mutually_exclusive_group(required=True)
|
||||||
|
g.add_argument("--set-login", action="store_true", help="store email + password")
|
||||||
|
g.add_argument("--show", action="store_true", help="show stored username")
|
||||||
|
g.add_argument("--clear", action="store_true", help="remove all stored secrets")
|
||||||
|
g.add_argument("--check", action="store_true",
|
||||||
|
help="report whether this host has a real vault backend")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
if args.check:
|
||||||
|
name = backend()
|
||||||
|
safe = backend_is_safe()
|
||||||
|
print(f"backend : {name}")
|
||||||
|
print(f"verdict : {'SAFE' if safe else 'NOT SAFE -- do not store secrets here'}")
|
||||||
|
if not safe:
|
||||||
|
print(" use systemd-creds on this host (see module docstring)")
|
||||||
|
sys.exit(0 if safe else 1)
|
||||||
|
|
||||||
|
if not backend_is_safe():
|
||||||
|
sys.exit(f"refusing to use backend {backend()!r}: it does not encrypt. "
|
||||||
|
"Run 'python creds.py --check' for guidance.")
|
||||||
|
|
||||||
|
if args.set_login:
|
||||||
|
user = input("email: ").strip()
|
||||||
|
if not user:
|
||||||
|
sys.exit("aborted: empty email")
|
||||||
|
pw = getpass.getpass("password: ")
|
||||||
|
if not pw:
|
||||||
|
sys.exit("aborted: empty password")
|
||||||
|
put(K_USER, user)
|
||||||
|
put(K_PASS, pw)
|
||||||
|
print(f"stored in Windows Credential Manager under '{SERVICE}'")
|
||||||
|
|
||||||
|
elif args.show:
|
||||||
|
print(f"service : {SERVICE}")
|
||||||
|
print(f"username: {get(K_USER) or '(not set)'}")
|
||||||
|
print(f"password: {'(set)' if get(K_PASS) else '(not set)'}")
|
||||||
|
print(f"cookie : {_mask(get(K_COOKIE))}")
|
||||||
|
|
||||||
|
elif args.clear:
|
||||||
|
for k in (K_USER, K_PASS, K_COOKIE):
|
||||||
|
drop(k)
|
||||||
|
print("cleared")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,95 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Byte-exact manifest fetcher + concurrency test.
|
||||||
|
|
||||||
|
curl normalises percent-escapes (%2f -> %2F), which breaks an Akamai path
|
||||||
|
token because the HMAC covers the literal path. This sends the request target
|
||||||
|
verbatim via http.client, and can fire N of them at once to see whether the
|
||||||
|
edge objects to concurrent readers of one token.
|
||||||
|
|
||||||
|
python probe.py --url "<manifest url>" # single fetch
|
||||||
|
python probe.py --url "<...>" --parallel 3 # concurrency test
|
||||||
|
python probe.py --url "<...>" --cookie "name=val" # with a CDN cookie
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import http.client
|
||||||
|
import sys
|
||||||
|
import threading
|
||||||
|
from urllib.parse import urlsplit
|
||||||
|
|
||||||
|
UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||||||
|
"(KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36")
|
||||||
|
|
||||||
|
|
||||||
|
def fetch(url, cookie=None, referer="https://www.bristolrovers.co.uk/", tag=""):
|
||||||
|
parts = urlsplit(url)
|
||||||
|
# Rebuild the target from the raw split so nothing is re-encoded.
|
||||||
|
target = parts.path + (("?" + parts.query) if parts.query else "")
|
||||||
|
conn = http.client.HTTPSConnection(parts.netloc, timeout=15)
|
||||||
|
try:
|
||||||
|
conn.putrequest("GET", target, skip_accept_encoding=True)
|
||||||
|
conn.putheader("User-Agent", UA)
|
||||||
|
conn.putheader("Accept", "*/*")
|
||||||
|
conn.putheader("Referer", referer)
|
||||||
|
conn.putheader("Origin", referer.rstrip("/"))
|
||||||
|
if cookie:
|
||||||
|
conn.putheader("Cookie", cookie)
|
||||||
|
conn.endheaders()
|
||||||
|
resp = conn.getresponse()
|
||||||
|
body = resp.read()
|
||||||
|
print(f"{tag}HTTP {resp.status} {resp.reason} ({len(body)} bytes)")
|
||||||
|
if resp.status == 200:
|
||||||
|
text = body.decode("utf-8", "replace")
|
||||||
|
variants = [l for l in text.splitlines()
|
||||||
|
if l.startswith("#EXT-X-STREAM-INF") or l.endswith(".m3u8")]
|
||||||
|
segs = [l for l in text.splitlines() if l and not l.startswith("#")]
|
||||||
|
if variants:
|
||||||
|
print(f"{tag} master playlist, {len(variants)//2 or len(variants)} rendition(s):")
|
||||||
|
for v in variants[:12]:
|
||||||
|
print(f"{tag} {v}")
|
||||||
|
else:
|
||||||
|
print(f"{tag} media playlist, {len(segs)} segment(s)")
|
||||||
|
for s in segs[:2]:
|
||||||
|
print(f"{tag} {s[:110]}")
|
||||||
|
else:
|
||||||
|
print(f"{tag} {body.decode('utf-8', 'replace')[:200].strip()}")
|
||||||
|
return resp.status
|
||||||
|
except Exception as exc:
|
||||||
|
print(f"{tag}ERROR {type(exc).__name__}: {exc}")
|
||||||
|
return -1
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--url", required=True)
|
||||||
|
ap.add_argument("--cookie", default=None)
|
||||||
|
ap.add_argument("--parallel", type=int, default=1,
|
||||||
|
help="fire N simultaneous fetches to test concurrency")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
if args.parallel <= 1:
|
||||||
|
sys.exit(0 if fetch(args.url, args.cookie) == 200 else 1)
|
||||||
|
|
||||||
|
print(f"firing {args.parallel} simultaneous requests "
|
||||||
|
f"(keep the browser playing while this runs)\n")
|
||||||
|
results = {}
|
||||||
|
start = threading.Barrier(args.parallel)
|
||||||
|
|
||||||
|
def worker(i):
|
||||||
|
start.wait() # release them together
|
||||||
|
results[i] = fetch(args.url, args.cookie, tag=f"[{i}] ")
|
||||||
|
|
||||||
|
threads = [threading.Thread(target=worker, args=(i,)) for i in range(args.parallel)]
|
||||||
|
for t in threads:
|
||||||
|
t.start()
|
||||||
|
for t in threads:
|
||||||
|
t.join()
|
||||||
|
|
||||||
|
ok = sum(1 for v in results.values() if v == 200)
|
||||||
|
print(f"\n{ok}/{args.parallel} succeeded concurrently"
|
||||||
|
f"{' -- plus the browser itself' if ok else ''}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,191 @@
|
|||||||
|
// ==UserScript==
|
||||||
|
// @name BRFC stream sniffer
|
||||||
|
// @namespace local.brfc.archive
|
||||||
|
// @version 0.4
|
||||||
|
// @description Capture the manifest URL the player requests, and trace the API call that mints its token.
|
||||||
|
// @match *://*.bristolrovers.co.uk/*
|
||||||
|
// @match *://*.streamamg.com/*
|
||||||
|
// @match *://*.streamamg.co.uk/*
|
||||||
|
// @match *://*.kaltura.com/*
|
||||||
|
// @match *://*.kaltura.co.uk/*
|
||||||
|
// @match *://*.akamaized.net/*
|
||||||
|
// @match *://*.akamaihd.net/*
|
||||||
|
// @noframes false
|
||||||
|
// @run-at document-start
|
||||||
|
// @grant GM_xmlhttpRequest
|
||||||
|
// @connect 127.0.0.1
|
||||||
|
// ==/UserScript==
|
||||||
|
|
||||||
|
(function () {
|
||||||
|
'use strict';
|
||||||
|
|
||||||
|
const CATCHER = 'http://127.0.0.1:8787/stream';
|
||||||
|
const TRACER = 'http://127.0.0.1:8787/trace';
|
||||||
|
|
||||||
|
// Discovery mode: record the request/response chain that produces a stream
|
||||||
|
// URL, so the token mint can be replayed without a GUI browser. Set false
|
||||||
|
// once the endpoint is known -- it writes a lot.
|
||||||
|
const TRACE = true;
|
||||||
|
|
||||||
|
const IDLE_OPACITY = 0.063; // 16/255
|
||||||
|
const MAX_BODY = 20000; // never log a whole media segment
|
||||||
|
|
||||||
|
console.log('[brfc-sniffer] v0.4 active in frame:', location.href);
|
||||||
|
|
||||||
|
const seen = new Map();
|
||||||
|
|
||||||
|
const isManifest = u => typeof u === 'string' && /\.m3u8(\?|$)/i.test(u);
|
||||||
|
// Media and assets are noise. Everything else is a mint-call candidate.
|
||||||
|
const isNoise = u => /\.(ts|mp4|m4s|aac|jpe?g|png|gif|webp|svg|woff2?|css)(\?|$)/i.test(u);
|
||||||
|
|
||||||
|
function post(endpoint, payload) {
|
||||||
|
try {
|
||||||
|
GM_xmlhttpRequest({
|
||||||
|
method: 'POST',
|
||||||
|
url: endpoint,
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
data: JSON.stringify(payload),
|
||||||
|
onerror: () => {}
|
||||||
|
});
|
||||||
|
} catch (e) { /* catcher not running; never break the page */ }
|
||||||
|
}
|
||||||
|
|
||||||
|
function record(url, via) {
|
||||||
|
let abs = url;
|
||||||
|
if (!/^https?:/i.test(url)) {
|
||||||
|
try { abs = new URL(url, location.href).href; } catch (e) { /* keep raw */ }
|
||||||
|
}
|
||||||
|
if (seen.has(abs)) return;
|
||||||
|
seen.set(abs, { url: abs, via });
|
||||||
|
console.log('[brfc-sniffer]', via, abs);
|
||||||
|
render();
|
||||||
|
post(CATCHER, {
|
||||||
|
url: abs, via, page: location.href,
|
||||||
|
userAgent: navigator.userAgent, ts: new Date().toISOString()
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// The mint call is identified by what it RETURNS, not by its name. Any
|
||||||
|
// response mentioning m3u8 is a candidate, whatever the endpoint is called.
|
||||||
|
function trace(entry, bodyText) {
|
||||||
|
if (!TRACE) return;
|
||||||
|
const mints = !!bodyText && /m3u8/i.test(bodyText);
|
||||||
|
if (!mints && !/api|token|entitle|session|auth|manifest|playback|licen/i.test(entry.url)) return;
|
||||||
|
entry.mintsManifest = mints;
|
||||||
|
entry.body = bodyText ? bodyText.slice(0, MAX_BODY) : null;
|
||||||
|
entry.page = location.href;
|
||||||
|
entry.ts = new Date().toISOString();
|
||||||
|
if (mints) console.log('[brfc-sniffer] *** MINT CALL ***', entry.method, entry.url);
|
||||||
|
post(TRACER, entry);
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- fetch ----------------------------------------------------------------
|
||||||
|
const _fetch = window.fetch;
|
||||||
|
if (_fetch) {
|
||||||
|
window.fetch = function (input, init) {
|
||||||
|
let url = null, method = 'GET';
|
||||||
|
try {
|
||||||
|
url = (typeof input === 'string') ? input : (input && input.url) || null;
|
||||||
|
method = (init && init.method) || (input && input.method) || 'GET';
|
||||||
|
if (isManifest(url)) record(url, 'fetch');
|
||||||
|
} catch (e) { /* ignore */ }
|
||||||
|
|
||||||
|
const p = _fetch.apply(this, arguments);
|
||||||
|
if (TRACE && url && !isNoise(url)) {
|
||||||
|
p.then(resp => {
|
||||||
|
try {
|
||||||
|
const reqBody = (init && typeof init.body === 'string')
|
||||||
|
? init.body.slice(0, MAX_BODY) : null;
|
||||||
|
resp.clone().text().then(
|
||||||
|
t => trace({ via: 'fetch', method, url, status: resp.status, reqBody }, t),
|
||||||
|
() => {}
|
||||||
|
);
|
||||||
|
} catch (e) { /* ignore */ }
|
||||||
|
}, () => {});
|
||||||
|
}
|
||||||
|
return p;
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- XHR ------------------------------------------------------------------
|
||||||
|
const _open = XMLHttpRequest.prototype.open;
|
||||||
|
const _send = XMLHttpRequest.prototype.send;
|
||||||
|
|
||||||
|
XMLHttpRequest.prototype.open = function (method, url) {
|
||||||
|
try {
|
||||||
|
this.__brfc = { method: method, url: url };
|
||||||
|
if (isManifest(url)) record(url, 'xhr');
|
||||||
|
} catch (e) { /* ignore */ }
|
||||||
|
return _open.apply(this, arguments);
|
||||||
|
};
|
||||||
|
|
||||||
|
XMLHttpRequest.prototype.send = function (body) {
|
||||||
|
try {
|
||||||
|
const info = this.__brfc;
|
||||||
|
if (TRACE && info && info.url && !isNoise(info.url)) {
|
||||||
|
const reqBody = (typeof body === 'string') ? body.slice(0, MAX_BODY) : null;
|
||||||
|
this.addEventListener('load', () => {
|
||||||
|
let text = null;
|
||||||
|
try {
|
||||||
|
if (this.responseType === '' || this.responseType === 'text') {
|
||||||
|
text = this.responseText;
|
||||||
|
}
|
||||||
|
} catch (e) { /* binary response */ }
|
||||||
|
trace({ via: 'xhr', method: info.method, url: info.url,
|
||||||
|
status: this.status, reqBody: reqBody }, text);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
} catch (e) { /* ignore */ }
|
||||||
|
return _send.apply(this, arguments);
|
||||||
|
};
|
||||||
|
|
||||||
|
// --- <video src> ----------------------------------------------------------
|
||||||
|
const _setAttr = Element.prototype.setAttribute;
|
||||||
|
Element.prototype.setAttribute = function (name, value) {
|
||||||
|
try {
|
||||||
|
if (/^src$/i.test(name) && isManifest(value)) record(value, 'attr');
|
||||||
|
} catch (e) { /* ignore */ }
|
||||||
|
return _setAttr.apply(this, arguments);
|
||||||
|
};
|
||||||
|
|
||||||
|
// --- panel ----------------------------------------------------------------
|
||||||
|
let panel;
|
||||||
|
function render() {
|
||||||
|
if (window.top !== window.self) return;
|
||||||
|
if (!panel) {
|
||||||
|
panel = document.createElement('div');
|
||||||
|
panel.style.cssText = [
|
||||||
|
'position:fixed', 'right:12px', 'bottom:12px', 'z-index:2147483647',
|
||||||
|
'max-width:min(560px,90vw)', 'max-height:40vh', 'overflow:auto',
|
||||||
|
'background:#101418', 'color:#e6edf3',
|
||||||
|
'font:12px/1.45 ui-monospace,Menlo,Consolas,monospace',
|
||||||
|
'border:1px solid #2b333c', 'border-radius:8px', 'padding:10px 12px',
|
||||||
|
'box-shadow:0 8px 24px rgba(0,0,0,.45)',
|
||||||
|
'opacity:' + IDLE_OPACITY,
|
||||||
|
'transition:opacity .18s ease'
|
||||||
|
].join(';');
|
||||||
|
panel.addEventListener('mouseenter', () => { panel.style.opacity = '1'; });
|
||||||
|
panel.addEventListener('mouseleave', () => { panel.style.opacity = String(IDLE_OPACITY); });
|
||||||
|
(document.body || document.documentElement).appendChild(panel);
|
||||||
|
}
|
||||||
|
panel.innerHTML = '<b style="color:#5ac8fa">BRFC sniffer v0.4</b> '
|
||||||
|
+ seen.size + ' manifest(s)'
|
||||||
|
+ (TRACE ? ' <span style="color:#e3b341">[tracing]</span>' : '') + '<br>';
|
||||||
|
for (const rec of seen.values()) {
|
||||||
|
const row = document.createElement('div');
|
||||||
|
row.style.cssText = 'margin-top:8px;word-break:break-all;'
|
||||||
|
+ 'border-top:1px solid #2b333c;padding-top:6px';
|
||||||
|
const btn = document.createElement('button');
|
||||||
|
btn.textContent = 'copy';
|
||||||
|
btn.style.cssText = 'margin-right:8px;cursor:pointer;background:#1f6feb;'
|
||||||
|
+ 'color:#fff;border:0;border-radius:4px;padding:2px 8px';
|
||||||
|
btn.onclick = () => navigator.clipboard.writeText(rec.url).then(() => {
|
||||||
|
btn.textContent = 'copied';
|
||||||
|
setTimeout(() => { btn.textContent = 'copy'; }, 1200);
|
||||||
|
});
|
||||||
|
row.appendChild(btn);
|
||||||
|
row.appendChild(document.createTextNode('[' + rec.via + '] ' + rec.url));
|
||||||
|
panel.appendChild(row);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})();
|
||||||
Reference in New Issue
Block a user