Compare commits
2 Commits
fcf509fda5
...
c39de9a017
| Author | SHA1 | Date | |
|---|---|---|---|
 Mikkeli Matlock
|
c39de9a017 | ||
|
Mikkeli Matlock
|
14ee5ec26c |
@@ -11,8 +11,12 @@ makes, using your own paid session.
|
||||
|
||||
| File | Job |
|
||||
|---|---|
|
||||
| `record_live.py` | Unattended recorder: finds the live entry itself, mints a master, records |
|
||||
| `preflight.py` | Pre-match check: does the session still mint a JWT, can this host record |
|
||||
| `watchdog.py` | Backup starter: if nothing is *progressing*, kill the stale recorder and start one |
|
||||
| `tampermonkey/brfc-stream-sniffer.user.js` | Hooks fetch/XHR in the page, captures the manifest URL verbatim, POSTs it to the catcher |
|
||||
| `catcher.py` | Listens on 127.0.0.1:8787, hands the URL to yt-dlp, records to `recordings/` |
|
||||
| `discover.py` | Drives the player with Playwright to capture the token-minting traffic |
|
||||
| `probe.py` | Byte-exact manifest fetch + concurrency test |
|
||||
|
||||
## The token
|
||||
@@ -51,6 +55,18 @@ sniffer catches the manifest, the catcher starts yt-dlp. Recording is MPEG-TS
|
||||
For a 3am kickoff: Task Scheduler opens the match page a few minutes before,
|
||||
the browser autoplays, the sniffer fires, the catcher records. Nobody is awake.
|
||||
|
||||
### Before every match
|
||||
|
||||
python preflight.py # exit 0 only if the server minted a JWT
|
||||
|
||||
Not optional and not a formality. The session can be dead while every local
|
||||
signal looks fine (see "The cookie expiry is not the session lifetime"), and the
|
||||
only fix -- signing in by hand in Firefox -- needs a human and some lead time.
|
||||
Run it hours before kickoff.
|
||||
|
||||
Then register the task, launch it once by hand, and stop it again before the
|
||||
real trigger (see "a freshly registered task can hang on its first launch").
|
||||
|
||||
## Concurrency check
|
||||
|
||||
An Akamai edge token is a stateless HMAC -- the edge validates signature and
|
||||
@@ -65,9 +81,11 @@ and the edge does not care.
|
||||
|
||||
---
|
||||
|
||||
## Findings (2026-08-29, live 2300 JST match)
|
||||
## Findings (live matches, 2026-08-29 onward)
|
||||
|
||||
Verified against a live stream. Recorded 1080p successfully end to end.
|
||||
Verified against live streams, starting with a 23:00 JST kickoff on 2026-08-29
|
||||
that recorded 1080p end to end. Later matches are dated where they corrected
|
||||
something.
|
||||
|
||||
### Two token types, not one
|
||||
|
||||
@@ -94,7 +112,25 @@ The master response rewrites each variant URL with a 24h `hdntl` token. So the
|
||||
- **Output is MPEG-TS** even though yt-dlp names it `.mp4`. That is what
|
||||
`--hls-use-mpegts` buys: a killed process still leaves a playable file.
|
||||
- Player is **Bitmovin**, embedded on the club site. `0_xm5vrrw6` is a Kaltura
|
||||
entry id and identifies the match -- the catcher dedupes on it.
|
||||
entry id. It does **not** identify the match -- see below.
|
||||
|
||||
### The entry id names the feed, not the match
|
||||
|
||||
Disproved 2026-09-19. Four observations, two ids:
|
||||
|
||||
| Entry id | Seen | Feed |
|
||||
|---|---|---|
|
||||
| `0_xm5vrrw6` | 2026-08-29, 2026-09-19 | video |
|
||||
| `0_z236v0e7` | 2026-09-05, 2026-09-12 | audio |
|
||||
|
||||
Three weeks apart, the same id came back for the video feed. So the id is a
|
||||
channel: it distinguishes the video feed from the audio one and nothing more.
|
||||
|
||||
`catcher.py` is not broken by this, because it dedupes on *liveness* -- "is a
|
||||
downloader for this key still alive" -- and the key clears when the downloader
|
||||
hits its cap. But do not build anything that treats the id as a match
|
||||
identifier: "have I already recorded this game" cannot be answered from it, and
|
||||
neither can "is this a new match or a reload". Use the kickoff date for that.
|
||||
|
||||
### Gotcha: which URL form a Python client can fetch
|
||||
|
||||
@@ -134,14 +170,27 @@ stream forever.
|
||||
`ffmpeg_o:-t $DURATION_MIN*60`. Without it, a feed that behaves like the video
|
||||
one above fills the disk with a logo.
|
||||
|
||||
Note the sample size. This is **one video observation**, and it is contradicted
|
||||
by **one audio observation**: on 2026-09-02 an audio stream ended cleanly by
|
||||
itself (`rc=0` at 165 min, against a 180 min cap that never fired). So "the
|
||||
stream never ends" is a property of that video feed, not an established rule.
|
||||
Treat the cap as a safety net rather than as the only stop condition.
|
||||
This holds for audio too, which earlier notes here doubted. On 2026-09-12 the
|
||||
audio feed for a 20:30 kickoff went to **digital silence** -- `volumedetect`
|
||||
reports -91.0 dB on both mean and max, the noise floor, not a quiet passage --
|
||||
at 02:35:20 into the file, and then kept serving that silence at a steady
|
||||
156 kbps until the 180 min cap fired at `rc=0`. Nothing about the stream said
|
||||
it was over.
|
||||
|
||||
Observed: video, 116 min recorded for a 23:00 kickoff, still running at 01:04
|
||||
and stopped manually. Audio, 165 min, self-terminated. Budget ~3.5-4 GB per copy per match at 40 MB/min.
|
||||
So of three observations, two feeds ran forever and one did not:
|
||||
|
||||
| Date | Feed | Ended how |
|
||||
|---|---|---|
|
||||
| 2026-08-29 | video | never; stopped manually at 116 min |
|
||||
| 2026-09-02 | audio | self-terminated, `rc=0` at 165 min |
|
||||
| 2026-09-12 | audio | never; silence to the 180 min cap |
|
||||
|
||||
The 2026-09-02 self-termination is now the outlier, not the rule, and it is the
|
||||
only one of the three. **The cap is load-bearing on both feed types.** Treat a
|
||||
stream that stops by itself as luck.
|
||||
|
||||
Budget ~36 MB/min for video (measured 2026-09-19: 1080p, 60 s -> 36 MB), so
|
||||
~6.5 GB for a 180 min cap. Audio is ~1.2 MB/min, ~210 MB for the same cap.
|
||||
|
||||
---
|
||||
|
||||
@@ -151,9 +200,11 @@ Unattended recording works. A scheduled task started `record_live.py` at 03:15
|
||||
for an 03:45 kickoff, found the stream in 16 seconds, and recorded 165 min of
|
||||
audio to a playable MPEG-TS with nobody awake.
|
||||
|
||||
Sample sizes are one. One video capture, one audio capture. Where the two
|
||||
disagree (stream termination, whether an entry id persists between matches),
|
||||
neither is established -- do not build on either without more matches.
|
||||
Sample sizes are small but no longer one: four recorded matches as of
|
||||
2026-09-19, two video and two audio. Two earlier claims here have since been
|
||||
disproved outright -- the entry id does not identify the match, and the audio
|
||||
feed does not reliably end itself. Both are corrected above. Assume the next
|
||||
claim in this file is the one about to fall over.
|
||||
|
||||
### Verified
|
||||
|
||||
@@ -162,6 +213,24 @@ neither is established -- do not build on either without more matches.
|
||||
- Recording via ffmpeg, video (1080p) and audio (~122 kbps AAC).
|
||||
- Scheduled unattended run, including that Task Scheduler launches it correctly.
|
||||
- Concurrency: several readers of one token, alongside a playing browser.
|
||||
- A dead session is detectable before kickoff, cheaply and read-only
|
||||
(`preflight.py`), and distinguishable from a blocked request by the
|
||||
`{"error":"Unauthenticated"}` body and the cookie-deleting `Set-Cookie`.
|
||||
- The duration cap firing cleanly at `rc=0` on a feed that would not stop.
|
||||
|
||||
### Gotcha: a freshly registered task can hang on its first launch
|
||||
|
||||
Seen 2026-09-12. The first launch of a newly registered scheduled task hung
|
||||
dead at interpreter startup -- process alive, 0.015 s of CPU and 21 modules
|
||||
loaded after three minutes, nothing written to any log, a `conhost` child
|
||||
allocated. Not a crash and not a loop: frozen before Python ran any of our
|
||||
code. Killing it and launching the identical task again worked immediately and
|
||||
it has not recurred. Root cause unknown.
|
||||
|
||||
Unattended, that is a silently lost match. So after registering a task, **launch
|
||||
it once by hand and stop it again**, which makes the real firing a warm start.
|
||||
Stopping matters as much as starting: `MultipleInstances IgnoreNew` means a test
|
||||
run left alive will swallow the scheduled trigger without a word.
|
||||
|
||||
### Not verified
|
||||
|
||||
@@ -172,13 +241,54 @@ neither is established -- do not build on either without more matches.
|
||||
`captcha_token` is an unknown and testing it risks a lockout that would take
|
||||
out the working session too.
|
||||
|
||||
### Decision: do not automate login
|
||||
### The cookie expiry is not the session lifetime
|
||||
|
||||
The cookie lasts ~87 days and matches are weekly, so automating login saves
|
||||
about four manual actions a year -- while being the riskiest step and the only
|
||||
one needing a stored password. Refresh the cookie by logging in manually on the
|
||||
desktop each quarter and provisioning it to the recording host instead. That
|
||||
makes cookie *provisioning* the open problem, not the captcha.
|
||||
Disproved 2026-09-12, and this one bites. The `SSESS` cookie carries a ~90 day
|
||||
expiry, and earlier notes here read that as the session lifetime. It is not.
|
||||
The server can drop the session while the browser still holds a cookie that
|
||||
looks perfectly healthy:
|
||||
|
||||
SSESSaa90be30f47... 76 days left on the clock
|
||||
GET /oauth/browser-token -> 403 {"error":"Unauthenticated"}
|
||||
Set-Cookie: SSESSaa90be30f47...=deleted; expires=Thu, 01 Jan 1970
|
||||
|
||||
That session had last worked six days earlier, at the end of the 2026-09-06
|
||||
recording. Nothing local showed it: right expiry, right domain, right profile.
|
||||
Only asking the server revealed it.
|
||||
|
||||
Then the *replacement* session survived the next seven days fine. So the
|
||||
lifetime is not a week either -- it is unpredictable, and somewhere between six
|
||||
days and the 90 the cookie advertises.
|
||||
|
||||
Two consequences:
|
||||
|
||||
- **A local cookie check proves nothing.** Presence and expiry are necessary,
|
||||
not sufficient. `preflight.py` exists for this: it spends one request asking
|
||||
the server to mint a JWT, which is the only answer that counts. Run it hours
|
||||
before kickoff, not minutes.
|
||||
- The cookie *value* also changed between 2026-09-12 and 2026-09-19 with no
|
||||
re-login recorded (48 -> 50 chars). Unexplained. Treat a copied cookie as a
|
||||
snapshot that can go stale on its own, which matters for provisioning one to
|
||||
another host.
|
||||
|
||||
### Decision: still do not automate login, but detect it early
|
||||
|
||||
The old reasoning for this was arithmetic, and the arithmetic was wrong: it
|
||||
assumed a ~87 day cookie meant about four manual logins a year. A session that
|
||||
can die in six days means the refresh could be needed weekly, which is a real
|
||||
cost rather than a rounding error.
|
||||
|
||||
The decision survives the correction anyway, for the reason that was always the
|
||||
strong one: automating login is the riskiest step in the chain and the only one
|
||||
needing a stored password, against an unknown `captcha_token` and a lockout that
|
||||
would take out the working session. What changes is the mitigation. Do not rely
|
||||
on remembering to refresh it quarterly -- assume it can be dead before any given
|
||||
match, and make the check routine:
|
||||
|
||||
python preflight.py # exit 0 only if the server minted a JWT
|
||||
|
||||
Cookie *provisioning* remains the open problem for a second host, and the
|
||||
rotating value above makes it harder, not easier.
|
||||
|
||||
Use Firefox as the session holder. Two independent reasons: its cookie DB stays
|
||||
readable while running (Chrome/Edge lock theirs), and it holds the login
|
||||
|
||||
+4
-2
@@ -32,8 +32,10 @@ BROWSER = os.environ.get("BRFC_BROWSER", "firefox")
|
||||
# ends cleanly. Pad past 90 min for half-time, stoppage and post-match.
|
||||
DURATION_MIN = int(os.environ.get("BRFC_DURATION_MIN", "150"))
|
||||
|
||||
# Kaltura entry id (e.g. 0_xm5vrrw6) identifies the match. Every manifest for
|
||||
# one game -- master and each variant, each with its own token -- shares it.
|
||||
# Kaltura entry id (e.g. 0_xm5vrrw6). Every manifest for one game -- master and
|
||||
# each variant, each with its own token -- shares it. It names the *feed*, not
|
||||
# the match: the same id came back three weeks apart (see README). That is fine
|
||||
# here, because the dedupe below is on liveness, not on "seen this before".
|
||||
ASSET_RE = re.compile(r"/(\d+_[a-z0-9]+)/", re.I)
|
||||
|
||||
_procs = {} # asset key -> Popen of the running downloader
|
||||
|
||||
+224
@@ -0,0 +1,224 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Pre-match check: is the session still good, and can this host record?
|
||||
|
||||
Run it a few hours before kickoff, not five minutes before. The one thing it
|
||||
cannot fix for you is a dead login: refreshing that means opening Firefox and
|
||||
signing in by hand, which takes a human (see README, "Decision: do not
|
||||
automate login").
|
||||
|
||||
python preflight.py # session + host checks
|
||||
python preflight.py --entry 0_z236v0e7 # also exercise the entry API
|
||||
|
||||
Exit code is 0 only if the session minted a JWT. Anything else means tonight's
|
||||
scheduled run would fail, and it would fail at kickoff with nobody watching.
|
||||
|
||||
Prints no secrets. Cookie values and tokens are reported by length only.
|
||||
"""
|
||||
import argparse
|
||||
import base64
|
||||
import contextlib
|
||||
import http.cookiejar
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from datetime import datetime, timezone
|
||||
|
||||
ROOT = os.path.dirname(os.path.abspath(__file__))
|
||||
RECORDINGS = os.path.join(ROOT, "recordings")
|
||||
UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) "
|
||||
"Gecko/20100101 Firefox/131.0")
|
||||
TOKEN_URL = ("https://www.bristolrovers.co.uk/oauth/browser-token"
|
||||
"?client_id=streamamg&token_type=urn%3Aietf%3Aparams%3Aoauth%3Atoken-type%3Aid_token")
|
||||
# Public client key, shipped to every browser -- not a secret.
|
||||
API_KEY = "50ivbQ3IXpF2mwT4tPAJ2V6cVm4CGRT2YKmDK5v6"
|
||||
|
||||
OK, BAD, WARN = "ok ", "FAIL", "warn"
|
||||
|
||||
|
||||
def say(status, msg):
|
||||
print(f"[{status}] {msg}", flush=True)
|
||||
|
||||
|
||||
def check_deps():
|
||||
"""Everything record_live.py reaches for, checked before kickoff."""
|
||||
good = True
|
||||
for mod, hint in (("yt_dlp", "pip install yt-dlp"),
|
||||
("playwright", "pip install playwright && "
|
||||
"python -m playwright install firefox")):
|
||||
try:
|
||||
__import__(mod)
|
||||
say(OK, f"{mod} importable")
|
||||
except ImportError:
|
||||
say(BAD, f"{mod} missing -- {hint}")
|
||||
good = False
|
||||
if shutil.which("ffmpeg"):
|
||||
say(OK, "ffmpeg on PATH")
|
||||
else:
|
||||
say(BAD, "ffmpeg not on PATH -- the downloader is forced to ffmpeg")
|
||||
good = False
|
||||
return good
|
||||
|
||||
|
||||
def check_disk(need_gb=8):
|
||||
"""A video match is ~4.7 GB. Audio is ~200 MB. Budget for video."""
|
||||
os.makedirs(RECORDINGS, exist_ok=True)
|
||||
free_gb = shutil.disk_usage(RECORDINGS).free / 2**30
|
||||
if free_gb >= need_gb:
|
||||
say(OK, f"{free_gb:.0f} GB free on the recordings volume")
|
||||
return True
|
||||
say(BAD, f"only {free_gb:.1f} GB free, want >= {need_gb} GB for a video match")
|
||||
return False
|
||||
|
||||
|
||||
def firefox_jar():
|
||||
from yt_dlp.cookies import extract_cookies_from_browser
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
return extract_cookies_from_browser("firefox")
|
||||
|
||||
|
||||
def check_cookies(jar):
|
||||
"""Cookie *present and unexpired* is necessary, not sufficient. The server
|
||||
can invalidate a session while the browser still holds the cookie, which is
|
||||
exactly what check_session() is for."""
|
||||
brfc = [c for c in jar if "bristolrovers" in (c.domain or "").lower()]
|
||||
say(OK if brfc else BAD, f"{len(brfc)} bristolrovers cookies in the Firefox jar")
|
||||
sess = [c for c in brfc if c.name.startswith(("SSESS", "SESS"))]
|
||||
if not sess:
|
||||
say(BAD, "no SSESS/SESS cookie -- Firefox is not logged in")
|
||||
return brfc, False
|
||||
for c in sess:
|
||||
if not c.expires:
|
||||
say(WARN, f"{c.name[:16]}... is a session cookie (dies with Firefox)")
|
||||
continue
|
||||
days = (c.expires - time.time()) / 86400
|
||||
say(OK if days > 1 else BAD,
|
||||
f"{c.name[:16]}... len={len(c.value or '')} expires in {days:.0f}d")
|
||||
return brfc, True
|
||||
|
||||
|
||||
def check_session(brfc):
|
||||
"""The real login test: does the site still mint a JWT off this cookie?"""
|
||||
cj = http.cookiejar.CookieJar()
|
||||
for c in brfc:
|
||||
cj.set_cookie(c)
|
||||
op = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
|
||||
hdrs = {"User-Agent": UA, "Accept": "application/json, text/plain, */*"}
|
||||
try:
|
||||
resp = op.open(urllib.request.Request(TOKEN_URL, headers=hdrs), timeout=20)
|
||||
body = resp.read().decode("utf-8", "replace")
|
||||
except urllib.error.HTTPError as exc:
|
||||
detail = exc.read().decode("utf-8", "replace")[:200]
|
||||
say(BAD, f"browser-token returned HTTP {exc.code}: {detail}")
|
||||
if exc.code in (401, 403):
|
||||
say(BAD, "session is dead server-side -- sign in again in Firefox")
|
||||
return None
|
||||
except Exception as exc:
|
||||
say(BAD, f"browser-token unreachable: {type(exc).__name__}: {exc}")
|
||||
return None
|
||||
|
||||
m = re.search(r"(eyJ[\w-]+\.eyJ[\w-]+\.[\w-]+)", body)
|
||||
if not m:
|
||||
say(BAD, "no JWT in the browser-token response -- session is stale")
|
||||
return None
|
||||
jwt = m.group(1)
|
||||
seg = jwt.split(".")[1]
|
||||
try:
|
||||
claims = json.loads(base64.urlsafe_b64decode(seg + "=" * (-len(seg) % 4)))
|
||||
except Exception:
|
||||
claims = {}
|
||||
exp = claims.get("exp")
|
||||
when = (f", expires {datetime.fromtimestamp(exp, timezone.utc):%H:%M:%S}Z "
|
||||
f"({(exp - time.time()) / 60:+.0f} min)" if exp else "")
|
||||
say(OK, f"session mints a JWT ({len(jwt)} chars{when})")
|
||||
return jwt
|
||||
|
||||
|
||||
def check_entry(jwt, entry_id):
|
||||
"""Optional: exercise the rest of the chain against a known entry id.
|
||||
|
||||
A master playlist outlives its stream, so a 200 here does not prove the
|
||||
match is live -- it proves the entitlement still resolves.
|
||||
"""
|
||||
req = urllib.request.Request(
|
||||
f"https://api.playback.streamamg.com/v1/entry/{entry_id}",
|
||||
headers={"User-Agent": UA, "Authorization": "Bearer " + jwt,
|
||||
"x-api-key": API_KEY,
|
||||
"Origin": "https://www.bristolrovers.co.uk",
|
||||
"Referer": "https://www.bristolrovers.co.uk/"})
|
||||
try:
|
||||
data = json.loads(urllib.request.urlopen(req, timeout=20)
|
||||
.read().decode("utf-8", "replace"))
|
||||
except urllib.error.HTTPError as exc:
|
||||
say(WARN, f"entry {entry_id}: HTTP {exc.code} "
|
||||
f"(stale entry id, or not entitled)")
|
||||
return
|
||||
except Exception as exc:
|
||||
say(WARN, f"entry {entry_id}: {type(exc).__name__}: {exc}")
|
||||
return
|
||||
|
||||
hls = (data.get("media") or {}).get("hls")
|
||||
if not hls:
|
||||
say(WARN, f"entry {entry_id} resolved but carries no HLS master")
|
||||
return
|
||||
say(OK, f"entry {entry_id} resolves to a master playlist")
|
||||
try:
|
||||
body = urllib.request.urlopen(urllib.request.Request(
|
||||
hls, headers={"User-Agent": UA,
|
||||
"Referer": "https://www.bristolrovers.co.uk/"}),
|
||||
timeout=15).read().decode("utf-8", "replace")
|
||||
except Exception as exc:
|
||||
say(WARN, f"could not read the master: {type(exc).__name__}: {exc}")
|
||||
return
|
||||
PREFIX = "#EXT-X-STREAM-INF:"
|
||||
streams = [ln[len(PREFIX):] for ln in body.splitlines()
|
||||
if ln.startswith(PREFIX)]
|
||||
say(OK, f"master offers {len(streams)} rendition(s):")
|
||||
for st in streams:
|
||||
bw = re.search(r"BANDWIDTH=(\d+)", st)
|
||||
res = re.search(r"RESOLUTION=([\dx]+)", st)
|
||||
cod = re.search(r'CODECS="([^"]+)"', st)
|
||||
codecs = cod.group(1) if cod else "?"
|
||||
kind = "audio-only" if ("avc" not in codecs and "hvc" not in codecs) else "video"
|
||||
print(f" {kind:11s} {int(bw.group(1)) // 1000 if bw else '?':>6} kbps "
|
||||
f"{res.group(1) if res else '-':>10} {codecs}")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--entry", help="entry id to exercise the entry API against")
|
||||
args = ap.parse_args()
|
||||
|
||||
print(f"preflight {datetime.now():%Y-%m-%d %H:%M:%S}\n")
|
||||
host_ok = check_deps() & check_disk()
|
||||
|
||||
try:
|
||||
jar = firefox_jar()
|
||||
except Exception as exc:
|
||||
say(BAD, f"cannot read Firefox cookies: {type(exc).__name__}: {exc}")
|
||||
return 1
|
||||
brfc, have_cookie = check_cookies(jar)
|
||||
|
||||
jwt = check_session(brfc) if have_cookie else None
|
||||
if jwt and args.entry:
|
||||
check_entry(jwt, args.entry)
|
||||
|
||||
print()
|
||||
if not jwt:
|
||||
say(BAD, "NOT READY -- sign in at https://www.bristolrovers.co.uk/user/login "
|
||||
"in Firefox, then run this again")
|
||||
return 1
|
||||
if not host_ok:
|
||||
say(BAD, "session is good but this host cannot record -- fix the above")
|
||||
return 1
|
||||
say(OK, "READY")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+128
@@ -0,0 +1,128 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Backup starter: confirm something is actually recording, else start it.
|
||||
|
||||
Schedule this a few minutes before kickoff, behind the primary recording task.
|
||||
It exists because a scheduled launch of record_live.py has been seen to hang
|
||||
dead at interpreter startup -- process alive, zero CPU, nothing logged, forever.
|
||||
Unattended, that loses the match silently.
|
||||
|
||||
So "is a recorder process running" is the wrong question: the hung process
|
||||
answers yes. This asks whether recording is *progressing*:
|
||||
|
||||
1. a .ts in recordings/ grew in the last --quiet-secs -> healthy, leave it
|
||||
2. record_live.log was written in the last --quiet-secs -> mid search loop,
|
||||
leave it
|
||||
3. neither -> stale. Kill any
|
||||
recorder process
|
||||
and start a fresh
|
||||
one.
|
||||
|
||||
Killing only ever targets a recorder that has gone quiet on both signals, so a
|
||||
healthy capture is never interrupted.
|
||||
|
||||
python watchdog.py --duration 170 --max-wait 55
|
||||
"""
|
||||
import argparse
|
||||
import glob
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime
|
||||
|
||||
ROOT = os.path.dirname(os.path.abspath(__file__))
|
||||
RECORDINGS = os.path.join(ROOT, "recordings")
|
||||
LOGS = os.path.join(ROOT, "logs")
|
||||
RECORDER = os.path.join(ROOT, "record_live.py")
|
||||
|
||||
|
||||
def log(msg):
|
||||
line = f"[{datetime.now():%Y-%m-%d %H:%M:%S}] {msg}"
|
||||
print(line, flush=True)
|
||||
os.makedirs(LOGS, exist_ok=True)
|
||||
with open(os.path.join(LOGS, "watchdog.log"), "a", encoding="utf-8") as fh:
|
||||
fh.write(line + chr(10))
|
||||
|
||||
|
||||
def powershell(cmd):
|
||||
try:
|
||||
out = subprocess.run(["powershell", "-NoProfile", "-Command", cmd],
|
||||
capture_output=True, text=True, timeout=60)
|
||||
return out.stdout.strip()
|
||||
except Exception as exc:
|
||||
log(f" powershell failed: {type(exc).__name__}: {exc}")
|
||||
return ""
|
||||
|
||||
|
||||
def recorder_pids():
|
||||
"""PIDs of any running record_live.py, hung or healthy."""
|
||||
out = powershell(
|
||||
"Get-CimInstance Win32_Process -Filter \"Name='python.exe'\" | "
|
||||
"Where-Object { $_.CommandLine -match 'record_live' } | "
|
||||
"ForEach-Object { $_.ProcessId }")
|
||||
return [int(x) for x in out.split() if x.strip().isdigit()]
|
||||
|
||||
|
||||
def newest_mtime(pattern):
|
||||
files = glob.glob(pattern)
|
||||
return max((os.path.getmtime(f) for f in files), default=0)
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--duration", type=int, default=170,
|
||||
help="recording cap in minutes for a run we start")
|
||||
ap.add_argument("--max-wait", type=int, default=55,
|
||||
help="minutes to keep looking for a live stream")
|
||||
ap.add_argument("--prefer", choices=("best", "audio", "video"), default="best")
|
||||
ap.add_argument("--quiet-secs", type=int, default=180,
|
||||
help="how stale both signals must be before we intervene")
|
||||
args = ap.parse_args()
|
||||
|
||||
log("watchdog: checking whether anything is recording")
|
||||
now = time.time()
|
||||
|
||||
ts_age = now - newest_mtime(os.path.join(RECORDINGS, "*.ts"))
|
||||
log_age = now - newest_mtime(os.path.join(LOGS, "record_live.log"))
|
||||
pids = recorder_pids()
|
||||
log(f" recorder pids {pids or 'none'}; newest .ts {ts_age:.0f}s old; "
|
||||
f"record_live.log {log_age:.0f}s old")
|
||||
|
||||
if ts_age < args.quiet_secs:
|
||||
log(" a recording is growing -- healthy, nothing to do")
|
||||
return 0
|
||||
if pids and log_age < args.quiet_secs:
|
||||
log(" recorder is logging -- mid search loop, nothing to do")
|
||||
return 0
|
||||
|
||||
if pids:
|
||||
log(f" recorder {pids} is alive but both signals are stale -- killing it")
|
||||
for pid in pids:
|
||||
powershell(f"Stop-Process -Id {pid} -Force -ErrorAction SilentlyContinue")
|
||||
time.sleep(3)
|
||||
still = recorder_pids()
|
||||
if still:
|
||||
log(f" WARNING: {still} survived the kill; starting anyway")
|
||||
else:
|
||||
log(" nothing is recording")
|
||||
|
||||
cmd = [sys.executable, RECORDER,
|
||||
"--max-wait", str(args.max_wait),
|
||||
"--duration", str(args.duration),
|
||||
"--prefer", args.prefer]
|
||||
log(f" starting: {' '.join(cmd[1:])}")
|
||||
# Detached: the watchdog task exits immediately, the recorder outlives it.
|
||||
flags = 0
|
||||
if os.name == "nt":
|
||||
flags = subprocess.CREATE_NEW_PROCESS_GROUP | getattr(
|
||||
subprocess, "DETACHED_PROCESS", 0)
|
||||
proc = subprocess.Popen(cmd, cwd=ROOT, creationflags=flags,
|
||||
stdin=subprocess.DEVNULL,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL)
|
||||
log(f" started recorder pid {proc.pid}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user