diff --git a/CHANGELOG.md b/CHANGELOG.md index 8f642b9..3a57a01 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,42 @@ # Changelog +## v2.5 — 2026-07-10 + +### Added + +#### GPS Triage + +viofosync can now fetch a clip's **GPS track before downloading the recording itself** — a skeleton trace pulled in seconds over Wi-Fi, rather than minutes of transferring the full MP4. With triage on, every queued clip is scouted for its route first, so the archive's journey maps, stops, and place names appear ahead of the footage arriving, and the download queue can be organised around where you actually drove. Off by default — enable **GPS triage** in Settings. + +#### Sync Filtering + +GPS-driven controls over what actually syncs — more sync policies to come: + +- **Read-only-only sync** — restrict syncing to locked (event) clips, so only footage the camera has protected is pulled. +- **Location skip** — define **named locations** (Home by default) with a dwell radius; with GPS triage on, clips that dwell at a location flagged *exclude recordings* are auto-skipped, so parking-mode footage recorded on your own driveway never downloads. Location names also label journey and stop endpoints on the archive maps. + +#### Clip Locking & Delete-from-Camera + +You can now manually **mark a clip read-only** to retain it indefinitely — locked clips are protected from retention pruning and manual delete. + +#### Journeys & Archive UX + +- A **per-journey completion pie** shows at a glance how much of each detected trip has finished downloading. +- Journey grouping is **padded past the GPS stop radius** so a trip takes in its pull-away and pull-in clips, with the trace, start, and end markers all unified to that same padded window. +- Unified **clip actions dropdown** (queue / export / delete) replaces the scattered per-clip buttons. +- Archive navigation now **preserves tab state, restores map views, and adds FLIP animations**, so the view no longer resets as clips index in. + +### Changed + +- Single-channel captures get **compact filenames** — no redundant camera suffix when only one lens is present — and a downloaded clip's sidecar GPS is used as a fallback source for its track. Thanks to @nittanygeek (#24). +- Redundant **Extract GPS** button was removed now that triage handles extraction. + +### Fixed + +- The dashcam lists and serves the clip it is *still recording* and reports a stale size for it. An **active-recording guard** now holds every lens of the newest capture group out of triage and the download drain until the camera's `cmd=3014` record flag reports stopped, freeing superseded captures as new ones arrive and deferring any clip that keeps growing mid-download. (#26) +- The sync worker now **reports its state on start**, so the status badge no longer shows a false "paused" before the first cycle. +- `RangeReader` now raises `TruncatedRead` on a short range response instead of silently accepting a truncated GPS read. + ## v2.4 — 2026-06-18 ### Added diff --git a/README.md b/README.md index f5b7343..e5b3348 100644 --- a/README.md +++ b/README.md @@ -10,8 +10,11 @@ Self-hosted web app for syncing, browsing, and exporting recordings from a Viofo - **Automatic Wi-Fi sync** - clips copy from the dashcam in your car when it joins your home wi-fi. - **Download control** - skip clips you don't want, retry failed ones, and prioritise recent footage, all from the download queue. +- **GPS triage** - fetches each clip's GPS trace in seconds without downloading the full recording, so journeys, stops, and place names appear before the footage does — and the download queue can be organised around where you actually drove. +- **Sync filtering** - restrict syncing to locked (event) clips only, or auto-skip anything recorded while parked at a named location such as home. GPS-driven, and the groundwork for richer download/delete ordering to come. - **Archive browser** - clips grouped by day, played in your browser; hover a clip to scrub a quick preview. Nothing to install on your phone or laptop. - **Journey maps** - automatic journey detection with each trip shown on a map with stops detected and place names looked up automatically. +- **Clip retention control** - mark any clip read-only to keep it indefinitely (protected from pruning and delete), or delete a clip straight off the camera's SD card once it's safely saved. - **Video editor** - trim clips, cut between cameras, and add a picture-in-picture inset to any segment, then export a single video. - **Flexible exports** - original clips, joined front/rear, picture-in-picture, or edited cuts; hardware-accelerated where your system supports it. - **Storage management** - set a size or age limit and the oldest footage is pruned to fit; optional auto-delete clears the camera's SD card once a clip is safely saved. @@ -48,10 +51,10 @@ Self-hosted web app for syncing, browsing, and exporting recordings from a Viofo viofosync targets any Viofo Wi-Fi dashcam that uses the standard `…F` / `…R` / `…T` / `…I` recording filenames, so models beyond those listed below should work — reports of other cameras are welcome. -| Camera | Channels | Tested by | -| --- | --- | --- | -| Viofo A229 Pro | Front + Rear | maintainer | -| Viofo A329 | Front + Rear + Telephoto | [@jusii](https://github.com/jusii) | +| Camera | +| -------------- | +| Viofo A229 Pro | +| Viofo A329 | Three-camera models are supported with either a telephoto (`T`) or interior/cabin (`I`) third lens. The telephoto channel was validated live on the A329; the interior channel was validated against real cabin-cam footage contributed alongside that work. @@ -235,6 +238,8 @@ Camera control — reading and safely adjusting dashcam settings over Wi-Fi — Three-camera support (telephoto and interior lenses), the single-source camera registry and other improvements were contributed by [@jusii](https://github.com/jusii) (#17, #18, #20). +Single channel camera work by [@nittanygeek](https://github.com/nittanygeek) (#24). + The GPX extraction logic uses the method described at [https://sergei.nz/extracting-gps-data-from-viofo-a119-and-other-novatek-powered-cameras/](https://sergei.nz/extracting-gps-data-from-viofo-a119-and-other-novatek-powered-cameras/). This software is unaffiliated with Viofo or any other vendor. diff --git a/tests/test_archive_completion.py b/tests/test_archive_completion.py new file mode 100644 index 0000000..70444e6 --- /dev/null +++ b/tests/test_archive_completion.py @@ -0,0 +1,163 @@ +"""_apply_completion attaches per-journey data-completeness (downloaded +duration / total, front clips only, skipped excluded).""" +from __future__ import annotations + +from web.db import Database +from web.routers import archive + +DATE = "2026-06-18" + + +def _payload(*windows): + return {"journeys": [ + {"start_ts": gs, "end_ts": ge, + "group_start_ts": gs, "group_end_ts": ge} for gs, ge in windows + ]} + + +def _clip(db, ts, cam="F", *, dur=60.0, seq=1): + with db.write() as c: + c.execute( + "INSERT INTO clip_index (path, basename, group_name, timestamp, " + "camera, sequence, event_type, duration_s, scanned_at) " + "VALUES (?,?,?,?,?,?,?,?,?)", + (f"/rec/{ts}{cam}.MP4", f"{ts}{cam}.MP4", DATE, ts, cam, seq, + "normal", dur, ts), + ) + + +def _queued(db, ts, cam="F", *, state="pending", triaged=1, skip=None): + fn = f"2026_0618_{ts:06d}_0001{cam}.MP4" + with db.write() as c: + c.execute( + "INSERT INTO download_queue (filename, source_dir, camera, " + "event_type, state, enqueued_at, recorded_at, triaged_at, " + "gps_points, skip_reason) VALUES (?,?,?,?,?,?,?,?,?,?)", + (fn, "/DCIM", cam, "normal", state, ts, ts, + (ts if triaged else None), (5 if triaged else None), skip), + ) + + +def test_all_downloaded_is_complete(tmp_path): + db = Database(str(tmp_path / "v.db")) + _clip(db, 1000, dur=60.0) + _clip(db, 1060, dur=60.0) + p = _payload((1000, 1200)) + archive._apply_completion(db, DATE, p) + assert p["journeys"][0]["completion"] == 1.0 + assert p["journeys"][0]["completion_detail"]["downloaded_clips"] == 2 + + +def test_skipped_clip_excluded_from_denominator(tmp_path): + # An intentionally-skipped (e.g. geofenced) clip is absent from the map + # and must not drag completion down: a downloaded clip + a skipped one + # reads 100%. + db = Database(str(tmp_path / "v.db")) + _clip(db, 1000, dur=60.0) + _queued(db, 1060, state="skipped", skip="geofence") + p = _payload((1000, 1200)) + archive._apply_completion(db, DATE, p) + assert p["journeys"][0]["completion"] == 1.0 + assert p["journeys"][0]["completion_detail"]["total_clips"] == 1 + + +def test_none_downloaded_is_zero(tmp_path): + db = Database(str(tmp_path / "v.db")) + _queued(db, 1000) + _queued(db, 1060) + p = _payload((1000, 1200)) + archive._apply_completion(db, DATE, p) + assert p["journeys"][0]["completion"] == 0.0 + assert p["journeys"][0]["completion_detail"]["total_clips"] == 2 + + +def test_mixed_is_fraction_by_duration(tmp_path): + db = Database(str(tmp_path / "v.db")) + _clip(db, 1000, dur=60.0) + _queued(db, 1060) + p = _payload((1000, 1200)) + archive._apply_completion(db, DATE, p) + assert p["journeys"][0]["completion"] == 0.5 + + +def test_rear_clip_ignored(tmp_path): + db = Database(str(tmp_path / "v.db")) + _clip(db, 1000, cam="F", dur=60.0) + _clip(db, 1000, cam="R", dur=60.0, seq=2) + p = _payload((1000, 1200)) + archive._apply_completion(db, DATE, p) + assert p["journeys"][0]["completion_detail"]["total_clips"] == 1 + + +def test_prefixed_front_counts(tmp_path): + db = Database(str(tmp_path / "v.db")) + _clip(db, 1000, cam="PF", dur=60.0) + p = _payload((1000, 1200)) + archive._apply_completion(db, DATE, p) + assert p["journeys"][0]["completion_detail"]["downloaded_clips"] == 1 + + +def test_duration_s_null_uses_gap(tmp_path): + db = Database(str(tmp_path / "v.db")) + _clip(db, 1000, dur=None) + _clip(db, 1090, dur=None) + p = _payload((1000, 1200)) + archive._apply_completion(db, DATE, p) + assert p["journeys"][0]["completion"] == 1.0 + assert p["journeys"][0]["completion_detail"]["total_s"] == 90 + 60 + + +def test_only_clips_in_window_count(tmp_path): + db = Database(str(tmp_path / "v.db")) + _clip(db, 1000, dur=60.0) + _clip(db, 5000, dur=60.0) + p = _payload((900, 1200)) + archive._apply_completion(db, DATE, p) + assert p["journeys"][0]["completion_detail"]["total_clips"] == 1 + + +def test_empty_journey_reads_complete(tmp_path): + db = Database(str(tmp_path / "v.db")) + p = _payload((1000, 1200)) + archive._apply_completion(db, DATE, p) + # Empty window (no clips at all) -> no pie, not a misleading 100%. + assert p["journeys"][0]["completion"] is None + assert p["journeys"][0]["completion_detail"]["total_clips"] == 0 + + +def test_untriaged_pending_excluded(tmp_path): + db = Database(str(tmp_path / "v.db")) + _queued(db, 1000, triaged=0) # triaged_at NULL -> not on the map + p = _payload((900, 1200)) + archive._apply_completion(db, DATE, p) + assert p["journeys"][0]["completion_detail"]["total_clips"] == 0 + # Empty window (nothing counted) -> no pie, not a misleading 100%. + assert p["journeys"][0]["completion"] is None + + +def test_geofenced_only_window_has_no_pie(tmp_path): + # A journey recovered purely from geofence-skipped skeletons has no + # downloadable clips in its window: the pie is omitted (None), not a + # misleading 100%. + db = Database(str(tmp_path / "v.db")) + _queued(db, 1060, state="skipped", skip="geofence") + p = _payload((1000, 1200)) + archive._apply_completion(db, DATE, p) + assert p["journeys"][0]["completion"] is None + + +def test_pending_without_gps_lock_excluded(tmp_path): + # triaged but gps_points=0 (no lock) -> absent from the journey map, + # so excluded from the denominator. + db = Database(str(tmp_path / "v.db")) + with db.write() as c: + c.execute( + "INSERT INTO download_queue (filename, source_dir, camera, " + "event_type, state, enqueued_at, recorded_at, triaged_at, " + "gps_points) VALUES (?,?,?,?,?,?,?,?,?)", + ("2026_0618_001000_0001F.MP4", "/DCIM", "F", "normal", + "pending", 1000, 1000, 1000, 0), + ) + p = _payload((900, 1200)) + archive._apply_completion(db, DATE, p) + assert p["journeys"][0]["completion_detail"]["total_clips"] == 0 diff --git a/tests/test_archive_geofenced_filter.py b/tests/test_archive_geofenced_filter.py new file mode 100644 index 0000000..9991dfa --- /dev/null +++ b/tests/test_archive_geofenced_filter.py @@ -0,0 +1,183 @@ +"""The archive's opt-in "GPS-excluded" view: geofence-skipped clips as +tiles (remote_day_clips) — user-skipped clips stay invisible.""" +from __future__ import annotations + +import datetime as _dt +import time +import types + +from web.db import Database +from web.routers import archive + + +def _req(db, *, gps_triage): + """Minimal stand-in for a FastAPI Request: the archive endpoints only + read request.app.state.db and .settings_provider.get().""" + snap = types.SimpleNamespace(gps_triage=gps_triage) + provider = types.SimpleNamespace(get=lambda: snap) + state = types.SimpleNamespace(db=db, settings_provider=provider) + return types.SimpleNamespace(app=types.SimpleNamespace(state=state)) + + +def _seed_queue(db, filename, *, camera, state="pending", skip_reason=None, + recorded_at=None, triaged=True, gps_points=1): + triaged_at = int(time.time()) if triaged else None + with db.write() as c: + c.execute( + "INSERT INTO download_queue " + "(filename, source_dir, camera, event_type, state, skip_reason, " + " enqueued_at, recorded_at, triaged_at, gps_points) " + "VALUES (?,?,?,?,?,?,?,?,?,?)", + (filename, "/DCIM/Movie", camera, "normal", state, skip_reason, + int(time.time()), recorded_at, triaged_at, gps_points), + ) + + +def test_remote_day_clips_geofenced_only_with_flag(tmp_path): + db = Database(str(tmp_path / "v.db")) + _seed_queue(db, "2026_0618_080000_0001F.MP4", camera="F", + state="skipped", skip_reason="geofence", recorded_at=1_000) + _seed_queue(db, "2026_0618_090000_0002F.MP4", camera="F", + state="skipped", skip_reason="user", recorded_at=2_000) + + # Default: no skipped rows at all (pins today's behaviour). + assert archive.remote_day_clips(db, "2026-06-18") == [] + + clips = archive.remote_day_clips(db, "2026-06-18", include_geofenced=True) + assert [c["front"]["filename"] for c in clips] == [ + "2026_0618_080000_0001F.MP4" + ] + assert clips[0]["front"]["state"] == "skipped" + assert clips[0]["front"]["skip_reason"] == "geofence" + + +def test_remote_day_clips_flag_keeps_pending(tmp_path): + # The flag widens the set; it must not replace it. + db = Database(str(tmp_path / "v.db")) + _seed_queue(db, "2026_0618_080000_0001F.MP4", camera="F", + recorded_at=1_000) + _seed_queue(db, "2026_0618_090000_0002F.MP4", camera="F", + state="skipped", skip_reason="geofence", recorded_at=2_000) + clips = archive.remote_day_clips(db, "2026-06-18", include_geofenced=True) + assert len(clips) == 2 + + +def _moving_gpx(lat0, lon0, n, start, step_lon=0.0008, dt_s=60): + pts = [] + for i in range(n): + t = start + _dt.timedelta(seconds=i * dt_s) + iso = t.strftime("%Y-%m-%dT%H:%M:%SZ") + pts.append( + f'' + f"1390" + ) + return ( + '' + "" + "".join(pts) + "" + ) + + +def _seed_geofenced_skeleton(db, rec, fn, start): + (rec / ".triage").mkdir(parents=True, exist_ok=True) + (rec / ".triage" / (fn + ".gpx")).write_text( + _moving_gpx(53.0, -2.0, 6, start) + ) + ts = int(start.replace(tzinfo=_dt.UTC).timestamp()) + _seed_queue(db, fn, camera="F", state="skipped", + skip_reason="geofence", recorded_at=ts) + return ts + + +def test_route_flag_recovers_geofenced_journey(tmp_path): + db = Database(str(tmp_path / "v.db")) + rec = tmp_path / "rec" + fn = "2026_0618_080000_0001F.MP4" + _seed_geofenced_skeleton(db, rec, fn, + _dt.datetime(2026, 6, 18, 8, 0, 0)) + + off = archive.build_route_payload(db, str(rec), "2026-06-18", None) + assert off["journeys"] == [] + assert "geofenced_tracks" not in off + + on = archive.build_route_payload( + db, str(rec), "2026-06-18", None, include_geofenced=True, + ) + assert len(on["journeys"]) == 1 + assert [t["filename"] for t in on["geofenced_tracks"]] == [fn] + track = on["geofenced_tracks"][0] + coords = track["geojson"]["geometry"]["coordinates"] + assert len(coords) == 6 + # Each track carries per-point timestamps so the client can scope it to + # the owning journey's window (otherwise every journey map would draw + # every track). One timestamp per coordinate, ascending. + assert len(track["times"]) == len(coords) + assert track["times"] == sorted(track["times"]) + # No downloadable clip in the window -> no pie (completion is None). + assert on["journeys"][0]["completion"] is None + + +def test_geofenced_tracks_scoped_to_separate_journeys(tmp_path): + # Two journeys hours apart, each with its own geofence-skipped clip. Each + # track's times must fall within exactly one journey's window, so the + # client can draw each recovered stretch only on the map it belongs to. + db = Database(str(tmp_path / "v.db")) + rec = tmp_path / "rec" + fn_a = "2026_0618_080000_0001F.MP4" + fn_b = "2026_0618_140000_0002F.MP4" + _seed_geofenced_skeleton(db, rec, fn_a, _dt.datetime(2026, 6, 18, 8, 0, 0)) + _seed_geofenced_skeleton(db, rec, fn_b, _dt.datetime(2026, 6, 18, 14, 0, 0)) + + on = archive.build_route_payload( + db, str(rec), "2026-06-18", None, include_geofenced=True, + ) + assert len(on["journeys"]) == 2 + tracks = {t["filename"]: t["times"] for t in on["geofenced_tracks"]} + windows = [(j["start_ts"], j["end_ts"]) for j in on["journeys"]] + # Every track sits wholly inside exactly one journey window. + for _fn, times in tracks.items(): + in_window = [ + (lo, hi) for lo, hi in windows + if all(lo <= t <= hi for t in times) + ] + assert len(in_window) == 1, f"{_fn} not scoped to one journey" + + +def _route_req(db, rec): + snap = types.SimpleNamespace( + gps_triage=True, recordings=str(rec), locations=(), + ) + provider = types.SimpleNamespace(get=lambda: snap) + state = types.SimpleNamespace(db=db, settings_provider=provider) + return types.SimpleNamespace(app=types.SimpleNamespace(state=state)) + + +def test_get_day_show_geofenced_param(tmp_path): + db = Database(str(tmp_path / "v.db")) + _seed_queue(db, "2026_0618_080000_0001F.MP4", camera="F", + state="skipped", skip_reason="geofence", recorded_at=1_000) + req = _req(db, gps_triage=True) + + off = archive.get_day(req, "2026-06-18", None, None, True, True, True, + False) + assert off["clips"] == [] + + on = archive.get_day(req, "2026-06-18", None, None, True, True, True, + True) + assert len(on["clips"]) == 1 + assert on["clips"][0]["front"]["skip_reason"] == "geofence" + + +def test_get_route_show_geofenced_param(tmp_path): + db = Database(str(tmp_path / "v.db")) + rec = tmp_path / "rec" + _seed_geofenced_skeleton(db, rec, "2026_0618_080000_0001F.MP4", + _dt.datetime(2026, 6, 18, 8, 0, 0)) + req = _route_req(db, rec) + + off = archive.get_route(req, "2026-06-18", False) + assert "geofenced_tracks" not in off + + on = archive.get_route(req, "2026-06-18", True) + assert len(on["geofenced_tracks"]) == 1 + assert len(on["journeys"]) == 1 diff --git a/tests/test_archive_group_window.py b/tests/test_archive_group_window.py new file mode 100644 index 0000000..bf16097 --- /dev/null +++ b/tests/test_archive_group_window.py @@ -0,0 +1,275 @@ +"""The archive day-grid groups clips into a journey using a PADDED window, so +pull-away / pull-in clips that sit just outside the raw GPS journey (the GPS +stop boundary lands ~STOP_RADIUS_M inside the real drive) attach to the journey +card instead of the neighbouring stop. The route payload carries the padded +window (``group_start_ts``/``group_end_ts``), reusing the same +``_expand_journey_window`` the timeline editor uses.""" +from __future__ import annotations + +import datetime as _dt + +from web.db import Database +from web.routers import archive + + +def _moving_gpx(lat0, lon0, n, start, step_lon=0.0008, dt_s=60): + """A GPX track of ``n`` points moving steadily east — continuous movement, + >200 m, no 5-min dwell — so aggregate_day yields exactly one journey.""" + pts = [] + for i in range(n): + t = start + _dt.timedelta(seconds=i * dt_s) + iso = t.strftime("%Y-%m-%dT%H:%M:%SZ") + pts.append( + f'' + f"1390" + ) + return ( + '' + "" + "".join(pts) + "" + ) + + +def _seed_journey(db, rec, day, fn, start): + daydir = rec / day + daydir.mkdir(parents=True, exist_ok=True) + path = daydir / fn + path.write_bytes(b"x") + (daydir / (fn + ".gpx")).write_text(_moving_gpx(53.0, -2.0, 6, start)) + ts = int(start.replace(tzinfo=_dt.UTC).timestamp()) + with db.write() as c: + c.execute( + "INSERT INTO clip_index " + "(path, basename, group_name, timestamp, camera, sequence, " + " event_type, size_bytes, has_gpx, gps_examined, scanned_at) " + "VALUES (?,?,?,?,?,?,?,?,1,1,?)", + (str(path), fn, day, ts, "F", 1, "normal", 1, ts), + ) + + +def test_journey_carries_padded_group_window(tmp_path): + db = Database(str(tmp_path / "v.db")) + rec = tmp_path / "rec" + day = "2026-06-18" + _seed_journey(db, rec, day, "2026_0618_080000_0001F.MP4", + _dt.datetime(2026, 6, 18, 8, 0, 0)) + + payload = archive.build_route_payload(db, str(rec), day, None) + assert payload["journeys"], "expected one journey from the moving track" + j = payload["journeys"][0] + # No parking clips on the day → padded by the full cap on each edge. + assert j["group_start_ts"] == j["start_ts"] - archive.MAX_JOURNEY_BUFFER_S + assert j["group_end_ts"] == j["end_ts"] + archive.MAX_JOURNEY_BUFFER_S + + +def test_group_window_bounded_by_parking_clip(tmp_path): + db = Database(str(tmp_path / "v.db")) + rec = tmp_path / "rec" + day = "2026-06-18" + _seed_journey(db, rec, day, "2026_0618_080000_0001F.MP4", + _dt.datetime(2026, 6, 18, 8, 0, 0)) + + # Drop a parking clip 30 s after the journey end (inside the 120 s pad) so + # the trailing edge clamps to it instead of running to the full cap. + end_ts = archive.build_route_payload(db, str(rec), day, None)["journeys"][0]["end_ts"] + with db.write() as c: + c.execute( + "INSERT INTO clip_index " + "(path, basename, group_name, timestamp, camera, sequence, " + " event_type, size_bytes, has_gpx, gps_examined, scanned_at) " + "VALUES (?,?,?,?,?,?,?,?,0,1,?)", + (str(rec / day / "park.MP4"), "2026_0618_080600_0009PF.MP4", day, + int(end_ts) + 30, "PF", 9, "parking", 1, int(end_ts)), + ) + + j = archive.build_route_payload(db, str(rec), day, None)["journeys"][0] + assert j["group_end_ts"] == j["end_ts"] + 30 + assert j["group_end_ts"] < j["end_ts"] + archive.MAX_JOURNEY_BUFFER_S + + +def test_group_window_bounded_by_queued_parking_clip(tmp_path): + """A parking clip that is still queued (not downloaded) must also bound the + pad — under GPS triage the day's parking clips live only in download_queue, + so a clip_index-only bound would never engage.""" + db = Database(str(tmp_path / "v.db")) + rec = tmp_path / "rec" + day = "2026-06-18" + _seed_journey(db, rec, day, "2026_0618_080000_0001F.MP4", + _dt.datetime(2026, 6, 18, 8, 0, 0)) + end_ts = archive.build_route_payload(db, str(rec), day, None)["journeys"][0]["end_ts"] + with db.write() as c: + c.execute( + "INSERT INTO download_queue (filename, source_dir, state, " + " event_type, recorded_at, enqueued_at) VALUES (?,?,?,?,?,0)", + ("2026_0618_080530_0009PF.MP4", "/DCIM/Movie", "pending", + "parking", int(end_ts) + 30), + ) + + j = archive.build_route_payload(db, str(rec), day, None)["journeys"][0] + assert j["group_end_ts"] == j["end_ts"] + 30 + assert j["group_end_ts"] < j["end_ts"] + archive.MAX_JOURNEY_BUFFER_S + + +def test_assemble_route_persists_merged_points(): + """The cached payload carries the day's merged points (lon, lat, ts), + sorted ascending by ts, so the on-read reframe can slice them.""" + import datetime as _dt + + from web.routers import archive + from web.services.gps import Point + + base = _dt.datetime(2026, 6, 18, 8, 0, 0, tzinfo=_dt.UTC) + pts = [ + Point(base + _dt.timedelta(seconds=i * 60), 53.0, -2.0 + i * 0.0008, 13, 90) + for i in range(6) + ] + payload = archive._assemble_route("2026-06-18", pts, [], []) + + assert "points" in payload + assert len(payload["points"]) == 6 + assert payload["points"][0] == [pts[0].lon, pts[0].lat, pts[0].t.timestamp()] + ts_col = [p[2] for p in payload["points"]] + assert ts_col == sorted(ts_col) + + +def _dwell_then_drive_gpx(lat0, lon0, start, dwell_n=7, drive_n=6, + step_lon=0.0008, dt_s=60): + """``dwell_n`` stationary fixes (a >5-min stop) followed by ``drive_n`` + fixes moving east. aggregate_day yields one stop + one journey; the last + stationary fixes sit just before the raw journey start, inside the pad.""" + import datetime as _dt + rows = [] + t = start + for _ in range(dwell_n): + iso = t.strftime("%Y-%m-%dT%H:%M:%SZ") + rows.append(f'' + f"00") + t += _dt.timedelta(seconds=dt_s) + for i in range(drive_n): + iso = t.strftime("%Y-%m-%dT%H:%M:%SZ") + rows.append( + f'' + f"1390") + t += _dt.timedelta(seconds=dt_s) + return ('' + + "".join(rows) + "") + + +def test_reframe_moves_start_back_into_swallowed_dwell(tmp_path): + """A drive out of a dwell: the stop detector trims the first ~min of + movement, but the reframe pulls the journey start back to the fixes inside + the padded window.""" + import datetime as _dt + db = Database(str(tmp_path / "v.db")) + rec = tmp_path / "rec" + day = "2026-06-18" + daydir = rec / day + daydir.mkdir(parents=True, exist_ok=True) + fn = "2026_0618_080000_0001F.MP4" + (daydir / fn).write_bytes(b"x") + (daydir / (fn + ".gpx")).write_text( + _dwell_then_drive_gpx(53.0, -2.0, _dt.datetime(2026, 6, 18, 8, 0, 0))) + ts = int(_dt.datetime(2026, 6, 18, 8, 0, 0, tzinfo=_dt.UTC).timestamp()) + with db.write() as c: + c.execute( + "INSERT INTO clip_index " + "(path, basename, group_name, timestamp, camera, sequence, " + " event_type, size_bytes, has_gpx, gps_examined, scanned_at) " + "VALUES (?,?,?,?,?,?,?,?,1,1,?)", + (str(daydir / fn), fn, day, ts, "F", 1, "normal", 1, ts)) + + payload = archive.build_route_payload(db, str(rec), day, None) + j = payload["journeys"][0] + base_ts = _dt.datetime(2026, 6, 18, 8, 0, 0, tzinfo=_dt.UTC).timestamp() + # The raw GPS journey starts at the stop boundary (last in-radius fix, +360s); + # reframing pulls the start back onto the padded window edge (+240s), proving + # the swallowed dwell maneuvering is now included. + assert j["start_ts"] == j["group_start_ts"] == base_ts + 240 + assert j["start_ts"] < base_ts + 360 # earlier than the raw (un-reframed) start + # Reframed start sits at/after the group window start, and strictly before + # the journey end (the swallowed dwell fixes were pulled in). + assert j["start_ts"] >= j["group_start_ts"] + assert j["start_ts"] < j["end_ts"] + # geojson, times and the start marker all describe the same first point. + coords = j["geojson"]["geometry"]["coordinates"] + assert j["times"][0] == j["start_ts"] + assert [j["start_lon"], j["start_lat"]] == coords[0] + # start_time is the ISO form of the reframed start_ts. + assert j["start_time"] == archive._iso_utc(j["start_ts"]) + # The merged-points scratch field is not shipped to the client. + assert "points" not in payload + + +def test_reframe_leaves_cold_start_journey_untouched(tmp_path): + """No fixes exist before the raw start (continuous movement from the first + fix), so the reframe cannot move the start earlier.""" + import datetime as _dt + db = Database(str(tmp_path / "v.db")) + rec = tmp_path / "rec" + day = "2026-06-18" + _seed_journey(db, rec, day, "2026_0618_080000_0001F.MP4", + _dt.datetime(2026, 6, 18, 8, 0, 0)) + + payload = archive.build_route_payload(db, str(rec), day, None) + j = payload["journeys"][0] + # First fix is the raw start; nothing earlier to pull in. + assert j["start_ts"] == j["times"][0] + assert j["start_ts"] > j["group_start_ts"] # window padded past the data + + +def _drive_dwell_drive_gpx(lat0, lon0, start, dt_s=60): + """drive east (6) -> dwell in place (8 = 7 min) -> drive east (6). One + session; aggregate_day yields two journeys split by one confirmed stop.""" + import datetime as _dt + rows = [] + t = start + lon = lon0 + def emit(lat, ln, sp, crs): + iso = t.strftime("%Y-%m-%dT%H:%M:%SZ") + rows.append(f'' + f"{sp}{crs}") + for _ in range(6): + lon += 0.0008 + emit(lat0, lon, 13, 90) + t += _dt.timedelta(seconds=dt_s) + for _ in range(8): # dwell in place, >5 min + emit(lat0, lon, 0, 0) + t += _dt.timedelta(seconds=dt_s) + for _ in range(6): + lon += 0.0008 + emit(lat0, lon, 13, 90) + t += _dt.timedelta(seconds=dt_s) + return ('' + + "".join(rows) + "") + + +def test_adjacent_journey_windows_never_overlap(tmp_path): + """Two drives separated by a >=5-min stop: padding is <=120 s per edge, and + a confirmed stop is >=300 s, so the journeys' padded windows can't meet. + Locks the invariant that lets the reframe slice points without a clamp.""" + import datetime as _dt + db = Database(str(tmp_path / "v.db")) + rec = tmp_path / "rec" + day = "2026-06-18" + daydir = rec / day + daydir.mkdir(parents=True, exist_ok=True) + fn = "2026_0618_080000_0001F.MP4" + (daydir / fn).write_bytes(b"x") + (daydir / (fn + ".gpx")).write_text( + _drive_dwell_drive_gpx(53.0, -2.0, _dt.datetime(2026, 6, 18, 8, 0, 0))) + ts = int(_dt.datetime(2026, 6, 18, 8, 0, 0, tzinfo=_dt.UTC).timestamp()) + with db.write() as c: + c.execute( + "INSERT INTO clip_index " + "(path, basename, group_name, timestamp, camera, sequence, " + " event_type, size_bytes, has_gpx, gps_examined, scanned_at) " + "VALUES (?,?,?,?,?,?,?,?,1,1,?)", + (str(daydir / fn), fn, day, ts, "F", 1, "normal", 1, ts)) + + payload = archive.build_route_payload(db, str(rec), day, None) + js = payload["journeys"] + assert len(js) == 2, "expected two drives split by the dwell" + a, b = sorted(js, key=lambda j: j["group_start_ts"]) + assert a["group_end_ts"] <= b["group_start_ts"], "padded windows overlap" diff --git a/tests/test_archive_home_labels.py b/tests/test_archive_home_labels.py new file mode 100644 index 0000000..7ca5b3b --- /dev/null +++ b/tests/test_archive_home_labels.py @@ -0,0 +1,103 @@ +"""Named-location label override in _apply_labels + the /geocode route.""" +from __future__ import annotations + +from types import SimpleNamespace + +from web.routers import archive + + +def _place(name, lat, lon, r, is_home=False): + return SimpleNamespace(name=name, lat=lat, lon=lon, radius_m=r, + exclude_recordings=False, is_home=is_home) + + +class _Geo: + def cache_lookup(self, lat, lon): + return "Somewhere" + + +def test_apply_labels_home_override() -> None: + payload = { + "journeys": [{"start_lat": 53.1, "start_lon": -2.0, + "end_lat": 50.0, "end_lon": 0.0}], + "stops": [{"lat": 53.1, "lon": -2.0}], + } + archive._apply_labels(payload, _Geo(), (_place("Home", 53.1, -2.0, 30, is_home=True),)) + j = payload["journeys"][0] + assert j["start_label"] == "Home" and j["start_home"] is True + assert j["start_named"] is True + assert j["end_label"] == "Somewhere" and j["end_home"] is False + assert j["end_named"] is False + s = payload["stops"][0] + assert s["label"] == "Home" and s["home"] is True + + +def test_apply_labels_no_places_falls_back() -> None: + payload = {"journeys": [], "stops": [{"lat": 53.1, "lon": -2.0}]} + archive._apply_labels(payload, _Geo(), ()) + assert payload["stops"][0]["label"] == "Somewhere" + assert payload["stops"][0]["home"] is False + + +def test_apply_labels_home_without_geocoder() -> None: + payload = {"journeys": [], "stops": [{"lat": 53.1, "lon": -2.0}]} + archive._apply_labels(payload, None, (_place("Home", 53.1, -2.0, 30, is_home=True),)) + assert payload["stops"][0]["label"] == "Home" + assert payload["stops"][0]["home"] is True + + +async def test_geocode_route_non_home_shape() -> None: + req = SimpleNamespace(app=SimpleNamespace(state=SimpleNamespace( + settings_provider=SimpleNamespace(get=lambda: SimpleNamespace(locations=())), + geocode=None, + ))) + out = await archive.geocode(req, 10.0, 10.0) + assert out["label"] is None + assert out["home"] is False + + +async def test_geocode_route_returns_home() -> None: + req = SimpleNamespace(app=SimpleNamespace(state=SimpleNamespace( + settings_provider=SimpleNamespace( + get=lambda: SimpleNamespace(locations=(_place("Home", 53.1, -2.0, 30, is_home=True),)) + ), + geocode=None, + ))) + out = await archive.geocode(req, 53.1, -2.0) + assert out["label"] == "Home" + assert out["home"] is True + assert out["named"] is True + + +def test_apply_labels_named_non_home() -> None: + payload = {"journeys": [], "stops": [{"lat": 53.1, "lon": -2.0}]} + archive._apply_labels(payload, _Geo(), (_place("Office", 53.1, -2.0, 30),)) + s = payload["stops"][0] + assert s["label"] == "Office" + assert s["home"] is False + assert s["named"] is True + + +def test_apply_labels_home_is_also_named() -> None: + payload = {"journeys": [], "stops": [{"lat": 53.1, "lon": -2.0}]} + archive._apply_labels(payload, _Geo(), (_place("Home", 53.1, -2.0, 30, is_home=True),)) + s = payload["stops"][0] + assert s["home"] is True and s["named"] is True + + +def test_apply_labels_geocode_not_named() -> None: + payload = {"journeys": [], "stops": [{"lat": 53.1, "lon": -2.0}]} + archive._apply_labels(payload, _Geo(), ()) + s = payload["stops"][0] + assert s["label"] == "Somewhere" and s["home"] is False and s["named"] is False + + +async def test_geocode_route_named_non_home() -> None: + req = SimpleNamespace(app=SimpleNamespace(state=SimpleNamespace( + settings_provider=SimpleNamespace( + get=lambda: SimpleNamespace(locations=(_place("Office", 53.1, -2.0, 30),)) + ), + geocode=None, + ))) + out = await archive.geocode(req, 53.1, -2.0) + assert out["label"] == "Office" and out["home"] is False and out["named"] is True diff --git a/tests/test_archive_stop_trim.py b/tests/test_archive_stop_trim.py new file mode 100644 index 0000000..b006ace --- /dev/null +++ b/tests/test_archive_stop_trim.py @@ -0,0 +1,85 @@ +"""_trim_stops_to_journeys shrinks each parking stop's window (and its shown +duration) so it no longer overlaps an adjacent journey's PADDED window — the +padded journey claims the pull-away/pull-in clips, so the stop should not also +advertise that time. A stop entirely absorbed by journey padding is dropped.""" +from __future__ import annotations + +import datetime as _dt + +from web.routers.archive import _trim_stops_to_journeys + + +def _iso(ts: float) -> str: + return _dt.datetime.fromtimestamp(ts, tz=_dt.UTC).isoformat() + + +def _journey(gs: float, ge: float) -> dict: + # Only the padded window fields matter to the trim. + return {"start_ts": gs, "end_ts": ge, + "group_start_ts": gs, "group_end_ts": ge} + + +def _stop(ss: float, se: float) -> dict: + return { + "start_ts": ss, "end_ts": se, + "start_time": _iso(ss), "end_time": _iso(se), + "duration_s": int(se - ss), + "lat": 1.0, "lon": 2.0, "label": None, + } + + +def test_front_overlap_trims_stop_start(): + # Journey padded end (260) reaches 60 s into the following stop [200, 500]. + payload = {"journeys": [_journey(100, 260)], "stops": [_stop(200, 500)]} + _trim_stops_to_journeys(payload) + s = payload["stops"][0] + assert s["start_ts"] == 260 + assert s["end_ts"] == 500 + assert s["duration_s"] == 240 + assert s["start_time"] == _iso(260) + + +def test_back_overlap_trims_stop_end(): + # Following journey's padded start (540) reaches back into stop [200, 560]. + payload = {"journeys": [_journey(540, 800)], "stops": [_stop(200, 560)]} + _trim_stops_to_journeys(payload) + s = payload["stops"][0] + assert s["start_ts"] == 200 + assert s["end_ts"] == 540 + assert s["duration_s"] == 340 + assert s["end_time"] == _iso(540) + + +def test_stop_between_two_journeys_trimmed_both_ends(): + payload = { + "journeys": [_journey(50, 260), _journey(540, 800)], + "stops": [_stop(200, 560)], + } + _trim_stops_to_journeys(payload) + s = payload["stops"][0] + assert (s["start_ts"], s["end_ts"]) == (260, 540) + assert s["duration_s"] == 280 + + +def test_fully_absorbed_stop_is_dropped(): + # Short stop [200, 260]; the two flanking paddings meet inside it. + payload = { + "journeys": [_journey(50, 258), _journey(255, 400)], + "stops": [_stop(200, 260)], + } + _trim_stops_to_journeys(payload) + assert payload["stops"] == [] + + +def test_isolated_stop_is_untouched(): + payload = {"journeys": [_journey(50, 260)], "stops": [_stop(1000, 1300)]} + before = dict(payload["stops"][0]) + _trim_stops_to_journeys(payload) + assert payload["stops"][0] == before + + +def test_no_journeys_leaves_stops_alone(): + payload = {"journeys": [], "stops": [_stop(200, 500)]} + before = dict(payload["stops"][0]) + _trim_stops_to_journeys(payload) + assert payload["stops"][0] == before diff --git a/tests/test_cameras_gps.py b/tests/test_cameras_gps.py new file mode 100644 index 0000000..3979f06 --- /dev/null +++ b/tests/test_cameras_gps.py @@ -0,0 +1,27 @@ +"""The camera registry designates exactly one GPS-bearing lens (front).""" +from __future__ import annotations + +from viofosync_lib.cameras import ( + CAMERAS, + GPS_CAMERA_LETTER, + is_gps_camera, +) + + +def test_exactly_one_gps_camera_and_it_is_front(): + gps_cams = [c for c in CAMERAS if c.gps] + assert len(gps_cams) == 1 + assert gps_cams[0].channel == "front" + assert GPS_CAMERA_LETTER == gps_cams[0].letter + + +def test_is_gps_camera_matches_front_only(): + assert is_gps_camera("F") is True + assert is_gps_camera("f") is True # case-insensitive + assert is_gps_camera("PF") is True # parking prefix, last letter F + assert is_gps_camera("EF") is True # event prefix + assert is_gps_camera("R") is False + assert is_gps_camera("T") is False + assert is_gps_camera("PI") is False # parking interior + assert is_gps_camera("") is False + assert is_gps_camera(None) is False diff --git a/tests/test_compact_filenames.py b/tests/test_compact_filenames.py new file mode 100644 index 0000000..0c9fd61 --- /dev/null +++ b/tests/test_compact_filenames.py @@ -0,0 +1,160 @@ +# tests/test_compact_filenames.py +"""Compact single-channel Viofo filenames: ``YYYYMMDDHHMMSS_NNNNNN.MP4``. + +Some units list recordings without datetime separator underscores or a +camera suffix. Those files are single-channel: the sole lens is the +GPS-bearing one, so suffix-less names default to the registry's GPS +camera letter and 'normal' event metadata, and every filename-derived +SQL expression (day key, camera letter, GPS-lens test) must understand +both layouts. +""" +from __future__ import annotations + +import datetime as _dt +import time + +import viofosync_lib as vfs +from viofosync_lib.cameras import GPS_CAMERA_LETTER +from web.db import Database +from web.routers import archive +from web.services import queue as q +from web.services import scanner, triage + +COMPACT = "20260618203643_000123.MP4" + + +# --- parsing layer ------------------------------------------------------- + + +def test_regex_parses_compact(): + m = vfs.downloaded_filename_re.match(COMPACT) + assert m is not None + assert m.group("year") == "2026" + assert m.group("month") == "06" + assert m.group("day") == "18" + assert m.group("hour") == "20" + assert m.group("minute") == "36" + assert m.group("second") == "43" + assert m.group("sequence") == "000123" + assert m.group("camera") == "" + + +def test_regex_still_parses_standard(): + m = vfs.downloaded_filename_re.match("2026_0618_203643_0001PF.MP4") + assert m is not None + assert m.group("sequence") == "0001" + assert m.group("camera") == "PF" + + +def test_glob_finds_compact_on_disk(tmp_path): + d = tmp_path / "2026-06-18" + d.mkdir() + (d / COMPACT).write_bytes(b"x") + got = vfs.get_downloaded_recordings(str(tmp_path), "daily") + assert (COMPACT, _dt.date(2026, 6, 18)) in got + + +def test_camera_and_event_default_to_gps_lens(): + assert q._camera_from_filename(COMPACT) == GPS_CAMERA_LETTER + assert q._event_from_filename(COMPACT) == "normal" + + +def test_scanner_meta_defaults_front(tmp_path): + d = tmp_path / "2026-06-18" + d.mkdir() + (d / COMPACT).write_bytes(b"x") + meta = scanner._clip_meta_for(str(tmp_path), "daily", COMPACT, "") + assert meta is not None + assert meta.camera == GPS_CAMERA_LETTER + assert meta.event_type == "normal" + assert meta.group_name == "2026-06-18" + assert meta.sequence == 123 + + +def test_importer_scan_item_defaults_front(): + from web.services import importer + m = vfs.downloaded_filename_re.match(COMPACT) + item = importer.scan_item_from_match( + m, COMPACT, source_rel_path=COMPACT, size=1, src_path="", + ) + assert item.camera == GPS_CAMERA_LETTER + assert item.event_type == "normal" + assert item.sequence == 123 + + +# --- SQL layer ----------------------------------------------------------- + + +def _seed(db, filename, *, state="pending", recorded_at=None, + triaged_at=None, gps_points=None, triage_attempts=0, + remote_complete=1): + now = int(time.time()) + with db.write() as c: + c.execute( + "INSERT INTO download_queue " + "(filename, source_dir, camera, event_type, state, enqueued_at, " + " recorded_at, triaged_at, gps_points, triage_attempts, " + " remote_complete) " + "VALUES (?,?,?,?,?,?,?,?,?,?,?)", + (filename, "/DCIM/Movie", q._camera_from_filename(filename), + q._event_from_filename(filename), state, now, + recorded_at, triaged_at, gps_points, triage_attempts, + remote_complete), + ) + + +def test_queue_day_key_not_malformed(tmp_path): + db = Database(str(tmp_path / "v.db")) + ts = int(_dt.datetime(2026, 6, 18, 20, 36, 43).timestamp()) + _seed(db, COMPACT, recorded_at=ts) + days = q.list_days(db) + assert [d["day"] for d in days] == ["2026-06-18"] + items = q.list_day_items(db, day="2026-06-18") + assert [it["filename"] for it in items] == [COMPACT] + assert items[0]["kind_camera"] == GPS_CAMERA_LETTER + assert items[0]["kind_event"] == "normal" + + +def test_select_targets_includes_compact(tmp_path): + db = Database(str(tmp_path / "v.db")) + old = int(time.time()) - 3600 # long past the settle window + _seed(db, COMPACT, recorded_at=old) + targets = triage.select_targets(db) + assert [t["filename"] for t in targets] == [COMPACT] + + +def test_gate_holds_compact_until_triaged(tmp_path): + # A compact clip IS the GPS-bearing lens: like a front, it must gate + # itself until triage reaches it, then be released. + db = Database(str(tmp_path / "v.db")) + _seed(db, COMPACT) + assert q.next_pending(db, triage_gate=True) is None + with db.write() as c: + c.execute( + "UPDATE download_queue SET triaged_at=?, gps_points=5 " + "WHERE filename=?", + (int(time.time()), COMPACT), + ) + item = q.next_pending(db, triage_gate=True) + assert item is not None and item.filename == COMPACT + + +def test_remote_day_clips_shows_compact_as_front(tmp_path): + db = Database(str(tmp_path / "v.db")) + ts = int(_dt.datetime(2026, 6, 18, 20, 36, 43).timestamp()) + _seed(db, COMPACT, recorded_at=ts, + triaged_at=int(time.time()), gps_points=5) + clips = archive.remote_day_clips(db, "2026-06-18") + assert len(clips) == 1 + assert clips[0]["front"] is not None + assert clips[0]["front"]["basename"] == COMPACT + assert clips[0]["event_type"] == "normal" + + +def test_gps_state_derived_from_own_columns(tmp_path): + db = Database(str(tmp_path / "v.db")) + ts = int(_dt.datetime(2026, 6, 18, 20, 36, 43).timestamp()) + _seed(db, COMPACT, recorded_at=ts, + triaged_at=int(time.time()), gps_points=5) + items = q.list_page(db, per_page=10)["items"] + assert items[0]["gps_state"] == "ok" diff --git a/tests/test_control_record_state.py b/tests/test_control_record_state.py new file mode 100644 index 0000000..8ab3263 --- /dev/null +++ b/tests/test_control_record_state.py @@ -0,0 +1,39 @@ +"""record_state: resilient camera record-flag read for the active-recording +guard. 1=recording, 0=stopped, None=unknown (unsupported command, unreachable, +absent pair, or unparseable) — callers hold the newest capture on None.""" +from __future__ import annotations + +from viofosync_lib import _control as control + + +def test_record_state_returns_recording(monkeypatch): + monkeypatch.setattr(control, "read_status_pairs", + lambda addr: [(2001, 1), (2002, 15)]) + assert control.record_state("1.2.3.4") == 1 + + +def test_record_state_returns_stopped(monkeypatch): + monkeypatch.setattr(control, "read_status_pairs", + lambda addr: [(2001, 0)]) + assert control.record_state("1.2.3.4") == 0 + + +def test_record_state_none_when_pair_absent(monkeypatch): + # A model whose cmd=3014 dump has no 2001 record pair. + monkeypatch.setattr(control, "read_status_pairs", + lambda addr: [(2002, 15), (2003, 1)]) + assert control.record_state("1.2.3.4") is None + + +def test_record_state_none_when_unreachable(monkeypatch): + def _boom(addr): + raise control.CameraUnreachable("gone") + monkeypatch.setattr(control, "read_status_pairs", _boom) + assert control.record_state("1.2.3.4") is None + + +def test_record_state_none_on_unexpected_error(monkeypatch): + def _boom(addr): + raise ValueError("garbage xml") + monkeypatch.setattr(control, "read_status_pairs", _boom) + assert control.record_state("1.2.3.4") is None diff --git a/tests/test_day_tracks.py b/tests/test_day_tracks.py new file mode 100644 index 0000000..e495ebb --- /dev/null +++ b/tests/test_day_tracks.py @@ -0,0 +1,137 @@ +"""day_gpx_paths: the shared GPX path set for a day (sidecars + skeletons).""" +from __future__ import annotations + +import os +from pathlib import Path + +import pytest + +from web.db import Database +from web.services import day_tracks + + +@pytest.fixture +def db(tmp_path: Path) -> Database: + return Database(str(tmp_path / "v.db")) + + +def _index_downloaded(db: Database, path: str, day: str, ts: int) -> None: + with db.write() as c: + c.execute( + "INSERT INTO clip_index " + "(path, basename, group_name, timestamp, camera, sequence, " + " has_gpx, scanned_at) VALUES (?,?,?,?,?,?,?,0)", + (path, os.path.basename(path), day, ts, "F", 1, 1), + ) + + +def _queue_triaged( + db: Database, filename: str, *, state="pending", skip_reason=None, +) -> None: + with db.write() as c: + c.execute( + "INSERT INTO download_queue " + "(filename, source_dir, state, skip_reason, triaged_at, " + " gps_points, enqueued_at) " + "VALUES (?,?,?,?,?,?,0)", + (filename, "/DCIM/Movie", state, skip_reason, 1, 5), + ) + + +def test_includes_sidecars_and_skeletons(db: Database, tmp_path: Path) -> None: + rec = tmp_path / "rec" + (rec / ".triage").mkdir(parents=True) + # A downloaded clip's sidecar. + _index_downloaded(db, str(rec / "2026_0618_200000_0001F.MP4"), + "2026-06-18", 1_000) + # A queued, triaged clip's skeleton. + sk = rec / ".triage" / "2026_0618_200100_0002F.MP4.gpx" + sk.write_text("") + _queue_triaged(db, "2026_0618_200100_0002F.MP4") + + paths = day_tracks.day_gpx_paths(db, str(rec), "2026-06-18") + assert str(rec / "2026_0618_200000_0001F.MP4.gpx") in paths + assert str(sk) in paths + + +def test_skipped_skeletons_only_with_broader_states(db: Database, tmp_path: Path) -> None: + rec = tmp_path / "rec" + (rec / ".triage").mkdir(parents=True) + sk = rec / ".triage" / "2026_0618_200100_0002F.MP4.gpx" + sk.write_text("") + _queue_triaged(db, "2026_0618_200100_0002F.MP4", state="skipped") + + # Default (pending/failed) excludes a skipped clip's skeleton… + assert str(sk) not in day_tracks.day_gpx_paths(db, str(rec), "2026-06-18") + # …but the geofence state set includes it. + paths = day_tracks.day_gpx_paths( + db, str(rec), "2026-06-18", + queue_states=("pending", "failed", "skipped", "downloading"), + ) + assert str(sk) in paths + + +def test_geofence_skipped_gpx_paths_filters_by_reason( + db: Database, tmp_path: Path, +) -> None: + rec = tmp_path / "rec" + (rec / ".triage").mkdir(parents=True) + sk_geo = rec / ".triage" / "2026_0618_200100_0002F.MP4.gpx" + sk_geo.write_text("") + _queue_triaged(db, "2026_0618_200100_0002F.MP4", + state="skipped", skip_reason="geofence") + sk_user = rec / ".triage" / "2026_0618_200200_0003F.MP4.gpx" + sk_user.write_text("") + _queue_triaged(db, "2026_0618_200200_0003F.MP4", + state="skipped", skip_reason="user") + sk_pend = rec / ".triage" / "2026_0618_200300_0004F.MP4.gpx" + sk_pend.write_text("") + _queue_triaged(db, "2026_0618_200300_0004F.MP4") # pending + + paths = day_tracks.geofence_skipped_gpx_paths(db, str(rec), "2026-06-18") + assert paths == [str(sk_geo)] + # And the default day set still excludes ALL skipped rows. + day = day_tracks.day_gpx_paths(db, str(rec), "2026-06-18") + assert str(sk_geo) not in day and str(sk_user) not in day + + +def test_day_parking_spans_unions_clip_index_and_queue(db: Database) -> None: + """Parking spans come from BOTH downloaded clips (clip_index, with real + duration) and queued-not-yet-downloaded clips (download_queue, a point at + recorded_at) — so the journey-window parking bound works in triage too.""" + day = "2026-06-18" + with db.write() as c: + # Downloaded parking clip: span = (ts, ts + duration_s) + c.execute( + "INSERT INTO clip_index (path, basename, group_name, timestamp, " + " camera, sequence, event_type, duration_s, has_gpx, scanned_at) " + "VALUES (?,?,?,?,?,?,?,?,0,0)", + ("/x/2026_0618_010000_0001PF.MP4", "2026_0618_010000_0001PF.MP4", + day, 1000, "PF", 1, "parking", 1800.0), + ) + # Queued (not downloaded) parking clip: point span at recorded_at + c.execute( + "INSERT INTO download_queue (filename, source_dir, state, " + " event_type, recorded_at, enqueued_at) VALUES (?,?,?,?,?,0)", + ("2026_0618_020000_0009PF.MP4", "/DCIM/Movie", "pending", + "parking", 5000), + ) + # Non-parking rows in each table must be excluded. + c.execute( + "INSERT INTO clip_index (path, basename, group_name, timestamp, " + " camera, sequence, event_type, has_gpx, scanned_at) " + "VALUES (?,?,?,?,?,?,?,0,0)", + ("/x/2026_0618_030000_0003F.MP4", "2026_0618_030000_0003F.MP4", + day, 9000, "F", 3, "normal"), + ) + c.execute( + "INSERT INTO download_queue (filename, source_dir, state, " + " event_type, recorded_at, enqueued_at) VALUES (?,?,?,?,?,0)", + ("2026_0618_040000_0004F.MP4", "/DCIM/Movie", "pending", + "normal", 12000), + ) + + spans = day_tracks.day_parking_spans(db, day) + assert (1000, 2800.0) in spans # clip_index: ts .. ts+duration + assert (5000, 5000) in spans # queue: point at recorded_at + assert len(spans) == 2 # only the two parking clips diff --git a/tests/test_db_migration.py b/tests/test_db_migration.py index 2ee9471..9310a89 100644 --- a/tests/test_db_migration.py +++ b/tests/test_db_migration.py @@ -163,3 +163,12 @@ def test_database_init_does_not_migrate( "legacy file contents must be untouched" assert not (rec / ".viofosync.db.migrated").exists(), \ "legacy file must NOT have been renamed by Database()" + + +def test_remote_complete_column_exists(tmp_path): + from web.db import Database + + db = Database(str(tmp_path / "v.db")) + with db.conn() as c: + cols = {r[1] for r in c.execute("PRAGMA table_info(download_queue)")} + assert "remote_complete" in cols diff --git a/tests/test_delete_from_camera.py b/tests/test_delete_from_camera.py new file mode 100644 index 0000000..91749fa --- /dev/null +++ b/tests/test_delete_from_camera.py @@ -0,0 +1,120 @@ +"""queue.delete_from_camera: dashcam SD-card delete, skipping locked/RO.""" +from __future__ import annotations + +from pathlib import Path +from unittest.mock import patch + +import pytest +from fastapi.testclient import TestClient + + +def _env(tmp_path: Path): + from web.db import Database + rec = tmp_path / "rec" + rec.mkdir() + return rec, Database(str(rec / "v.db")) + + +def _q(db, filename, source_dir, *, locked=0, state="pending"): + with db.write() as c: + c.execute( + "INSERT INTO download_queue (filename, source_dir, state, locked, " + "enqueued_at) VALUES (?,?,?,?,1)", (filename, source_dir, state, locked), + ) + + +def test_delete_from_camera_deletes_and_marks_gone(tmp_path): + from web.services.queue import delete_from_camera + rec, db = _env(tmp_path) + _q(db, "A.MP4", "/DCIM/Movie") + with patch("web.services.queue.vfs.delete_dashcam_file", return_value=True) as m: + res = delete_from_camera(db, ["A.MP4"], "http://cam") + m.assert_called_once_with("http://cam", "/DCIM/Movie", "A.MP4", timeout=10.0) + assert res == {"deleted": 1, "skipped": 0, "errors": 0} + with db.conn() as c: + assert c.execute("SELECT state FROM download_queue WHERE filename='A.MP4'").fetchone()["state"] == "gone" + + +def test_delete_from_camera_skips_locked_and_ro(tmp_path): + from web.services.queue import delete_from_camera + rec, db = _env(tmp_path) + _q(db, "L.MP4", "/DCIM/Movie", locked=1) + _q(db, "R.MP4", "/DCIM/Movie/RO") + with patch("web.services.queue.vfs.delete_dashcam_file", return_value=True) as m: + res = delete_from_camera(db, ["L.MP4", "R.MP4"], "http://cam") + m.assert_not_called() + assert res == {"deleted": 0, "skipped": 2, "errors": 0} + + +def test_delete_from_camera_counts_errors(tmp_path): + from web.services.queue import delete_from_camera + rec, db = _env(tmp_path) + _q(db, "A.MP4", "/DCIM/Movie") + with patch("web.services.queue.vfs.delete_dashcam_file", return_value=False): + res = delete_from_camera(db, ["A.MP4"], "http://cam") + assert res == {"deleted": 0, "skipped": 0, "errors": 1} + with db.conn() as c: + assert c.execute("SELECT state FROM download_queue WHERE filename='A.MP4'").fetchone()["state"] == "pending" + + +def test_delete_from_camera_empty(tmp_path): + from web.services.queue import delete_from_camera + rec, db = _env(tmp_path) + assert delete_from_camera(db, [], "http://cam") == {"deleted": 0, "skipped": 0, "errors": 0} + + +# ── endpoint tests ────────────────────────────────────────────────────────── + +@pytest.fixture +def authed_client(tmp_config_dir: Path, tmp_recordings_dir: Path, monkeypatch): + from web import app as app_mod + from web import settings as settings_mod + monkeypatch.setenv("VIOFOSYNC_RESTART_DISABLED", "1") + settings_mod.reset_for_tests() + application = app_mod.create_app() + with TestClient(application) as c: + c.post("/setup", data={ + "address": "192.168.1.230", + "password": "twelve-chars-min!", + "confirm": "twelve-chars-min!", + }) + csrf = c.get("/api/auth/csrf").json()["csrf"] + c.headers.update({"x-csrf-token": csrf}) + yield c + + +def test_delete_from_camera_endpoint_no_address(authed_client, monkeypatch): + """Returns ok:False/error when no dashcam address is configured.""" + import dataclasses + snap = authed_client.app.state.settings_provider.get() + no_addr_snap = dataclasses.replace(snap, address=None) + monkeypatch.setattr( + authed_client.app.state.settings_provider, "get", lambda: no_addr_snap + ) + r = authed_client.post( + "/api/queue/delete-from-camera", json={"filenames": ["A.MP4"]} + ) + assert r.status_code == 200 + data = r.json() + assert data["ok"] is False + assert "error" in data + + +def test_delete_from_camera_endpoint_success(authed_client): + """With address configured + mocked vfs, returns ok:True deleted:1.""" + db = authed_client.app.state.db + with db.write() as c: + c.execute( + "INSERT INTO download_queue (filename, source_dir, state, locked, enqueued_at) " + "VALUES ('A.MP4', '/DCIM/Movie', 'pending', 0, 1)" + ) + with patch("web.services.queue.vfs.delete_dashcam_file", return_value=True): + r = authed_client.post( + "/api/queue/delete-from-camera", json={"filenames": ["A.MP4"]} + ) + assert r.status_code == 200 + data = r.json() + assert data["ok"] is True + assert data["deleted"] == 1 + assert data["skipped"] == 0 + assert data["errors"] == 0 diff --git a/tests/test_download_deferral_refund.py b/tests/test_download_deferral_refund.py new file mode 100644 index 0000000..1abe4aa --- /dev/null +++ b/tests/test_download_deferral_refund.py @@ -0,0 +1,82 @@ +"""The active-recording overage deferral must NOT burn a download attempt. + +When download_file detects the file is still growing (streamed bytes exceed the +expected size) it raises DownloadDeferred; the sync worker must treat that like +a cancellation — return the item to pending with its attempt refunded — so a +clip the completeness guard missed can never exhaust max_attempts and get stuck +permanently 'failed'.""" +from __future__ import annotations + +import time + +import viofosync_lib as vfs +from web.db import Database +from web.services import queue as q +from web.services.hub import Hub +from web.services.sync_worker import SyncWorker + + +class _Snap: + def __init__(self, recordings): + self.recordings = recordings + self.grouping = "daily" + self.gps_extract = False + self.gps_triage = False + self.delete_after_download = False + self.download_attempts = 3 + self.max_attempts = 3 + self.timeout = 5 + self.sync_ro_only = False + + +class _Provider: + def __init__(self, snap): + self._snap = snap + + def get(self): + return self._snap + + +def _seed(db, filename): + now = int(time.time()) + with db.write() as c: + c.execute( + "INSERT INTO download_queue " + "(filename, source_dir, camera, event_type, state, enqueued_at, " + " remote_complete) VALUES (?,?,?,?,?,?,?)", + (filename, "/DCIM", filename[-5], "normal", "pending", now, 1), + ) + + +def _row(db, filename): + with db.conn() as c: + return dict(c.execute( + "SELECT * FROM download_queue WHERE filename=?", (filename,) + ).fetchone()) + + +async def test_deferral_refunds_attempt(tmp_path, monkeypatch): + db = Database(str(tmp_path / "v.db")) + rec = tmp_path / "rec" + rec.mkdir() + snap = _Snap(str(rec)) + sw = SyncWorker(db, _Provider(snap), Hub()) + sw._active_address = "1.2.3.4" + + _seed(db, "2026_0628_133416_0002F.MP4") + item = q.next_pending(db) + assert item is not None + + # Simulate download_file detecting the still-growing active file. + def _deferred(*a, **k): + raise vfs.DownloadDeferred("file still growing: streamed 3145728 > 60") + + monkeypatch.setattr(vfs, "download_file_with", _deferred) + + ok = await sw._download_one(item) + assert ok is False + + after = _row(db, "2026_0628_133416_0002F.MP4") + # Refunded: back to pending, attempt NOT burned (mirrors cancellation). + assert after["state"] == "pending" + assert after["attempts"] == 0 diff --git a/tests/test_geofence.py b/tests/test_geofence.py new file mode 100644 index 0000000..d4fa960 --- /dev/null +++ b/tests/test_geofence.py @@ -0,0 +1,252 @@ +"""Geofence detection: home_stops, evaluate_day, sweep_all.""" +from __future__ import annotations + +import datetime as _dt +from pathlib import Path + +import pytest + +from web.db import Database +from web.services import geofence +from web.services.gps import Stop +from web.settings import Place + + +@pytest.fixture +def db(tmp_path: Path) -> Database: + return Database(str(tmp_path / "v.db")) + + +def _stop(lat, lon, start, end): + return Stop( + start_time=_dt.datetime.fromtimestamp(start, _dt.UTC), + end_time=_dt.datetime.fromtimestamp(end, _dt.UTC), + center_lat=lat, center_lon=lon, point_count=10, + ) + + +def test_home_stops_filters_by_radius() -> None: + home = _stop(53.1000, -2.0000, 0, 600) # at home + away = _stop(53.2000, -2.0000, 0, 600) # ~11 km north + zones = (Place("Home", 53.1000, -2.0000, 30, True),) + out = geofence.home_stops([home, away], zones) + assert out == [home] + + +def test_home_stops_empty_zones() -> None: + assert geofence.home_stops([_stop(0, 0, 0, 1)], ()) == [] + + +# ---- evaluate_day fixtures ------------------------------------------------- + +HOME_LAT, HOME_LON = 53.1000, -2.0000 + + +def _gpx(points) -> str: + pts = "".join( + f'' + f"00" + for t, lat, lon in points + ) + return ( + '' + f"{pts}" + ) + + +def _utc(h, m, s=0): + return _dt.datetime(2026, 6, 18, h, m, s, tzinfo=_dt.UTC) + + +def _seed(db, filename, *, state="pending", source_dir="/DCIM/Movie", + recorded_at=None, triaged=True, released=None) -> None: + with db.write() as c: + c.execute( + "INSERT INTO download_queue " + "(filename, source_dir, state, recorded_at, triaged_at, " + " gps_points, geofence_released_at, enqueued_at) VALUES (?,?,?,?,?,?,?,0)", + (filename, source_dir, state, recorded_at, + 1 if triaged else None, 5 if triaged else None, released), + ) + + +def _write_skeleton(rec: Path, filename: str, points) -> None: + (rec / ".triage").mkdir(parents=True, exist_ok=True) + (rec / ".triage" / (filename + ".gpx")).write_text(_gpx(points)) + + +def test_evaluate_day_skips_in_home_dwell(db: Database, tmp_path: Path) -> None: + rec = tmp_path / "rec" + # A 7-minute dwell at home, split across the queued clip's skeleton. + dwell = [ + (_utc(20, 0).strftime("%Y-%m-%dT%H:%M:%SZ"), HOME_LAT, HOME_LON), + (_utc(20, 1).strftime("%Y-%m-%dT%H:%M:%SZ"), HOME_LAT, HOME_LON), + (_utc(20, 3).strftime("%Y-%m-%dT%H:%M:%SZ"), HOME_LAT, HOME_LON), + (_utc(20, 5).strftime("%Y-%m-%dT%H:%M:%SZ"), HOME_LAT, HOME_LON), + (_utc(20, 7).strftime("%Y-%m-%dT%H:%M:%SZ"), HOME_LAT, HOME_LON), + ] + fn_f = "2026_0618_200200_0001F.MP4" + fn_r = "2026_0618_200200_0002R.MP4" + _write_skeleton(rec, fn_f, dwell) + _seed(db, fn_f, recorded_at=int(_utc(20, 2).timestamp())) + # Rear clip shares the time window but has no skeleton of its own. + _seed(db, fn_r, recorded_at=int(_utc(20, 2).timestamp()), triaged=False) + + zones = (Place("Home", HOME_LAT, HOME_LON, 30, True),) + skipped = geofence.evaluate_day(db, str(rec), "2026-06-18", zones) + assert set(skipped) == {fn_f, fn_r} + with db.conn() as c: + states = { + r["filename"]: (r["state"], r["skip_reason"]) + for r in c.execute("SELECT filename, state, skip_reason FROM download_queue") + } + assert states[fn_f] == ("skipped", "geofence") + assert states[fn_r] == ("skipped", "geofence") + + +def test_evaluate_day_keeps_before_window_and_guarded(db: Database, tmp_path: Path) -> None: + rec = tmp_path / "rec" + dwell = [ + (_utc(20, 0).strftime("%Y-%m-%dT%H:%M:%SZ"), HOME_LAT, HOME_LON), + (_utc(20, 2).strftime("%Y-%m-%dT%H:%M:%SZ"), HOME_LAT, HOME_LON), + (_utc(20, 4).strftime("%Y-%m-%dT%H:%M:%SZ"), HOME_LAT, HOME_LON), + (_utc(20, 6).strftime("%Y-%m-%dT%H:%M:%SZ"), HOME_LAT, HOME_LON), + ] + fn_in = "2026_0618_200300_0001F.MP4" + fn_before = "2026_0618_195000_0002F.MP4" # before the dwell starts + fn_ro = "2026_0618_200300_0003F.MP4" # in window but RO + fn_rel = "2026_0618_200300_0004F.MP4" # in window but released + _write_skeleton(rec, fn_in, dwell) + _seed(db, fn_in, recorded_at=int(_utc(20, 3).timestamp())) + _seed(db, fn_before, recorded_at=int(_utc(19, 50).timestamp())) + _seed(db, fn_ro, source_dir="/DCIM/Movie/RO", + recorded_at=int(_utc(20, 3).timestamp())) + _seed(db, fn_rel, recorded_at=int(_utc(20, 3).timestamp()), released=1) + + zones = (Place("Home", HOME_LAT, HOME_LON, 30, True),) + skipped = geofence.evaluate_day(db, str(rec), "2026-06-18", zones) + assert set(skipped) == {fn_in} + + +def test_sweep_all_covers_pending_days(db: Database, tmp_path: Path) -> None: + rec = tmp_path / "rec" + for day, hh in (("2026_0618", 20), ("2026_0619", 8)): + dwell = [ + (_dt.datetime(2026, int(day[5:7]), int(day[7:9]), hh, m, + tzinfo=_dt.UTC).strftime("%Y-%m-%dT%H:%M:%SZ"), + HOME_LAT, HOME_LON) + for m in (0, 2, 4, 6) + ] + fn = f"{day}_{hh:02d}0300_0001F.MP4" + _write_skeleton(rec, fn, dwell) + _seed(db, fn, recorded_at=int( + _dt.datetime(2026, int(day[5:7]), int(day[7:9]), hh, 3, + tzinfo=_dt.UTC).timestamp())) + zones = (Place("Home", HOME_LAT, HOME_LON, 30, True),) + assert geofence.sweep_all(db, str(rec), zones) == 2 + + +def test_sweep_all_no_zones_is_noop(db: Database, tmp_path: Path) -> None: + assert geofence.sweep_all(db, str(tmp_path), ()) == 0 + + +def test_detect_states_match_skeleton_keep_states() -> None: + # The states the detector reads MUST equal the states whose skeletons the + # orphan sweep keeps, or skipped dwells get deleted out from under it. + from web.services import triage + assert set(geofence._DETECT_STATES) == set(triage.SKELETON_KEEP_STATES) + + +def test_evaluate_day_skips_leading_clip_before_first_fix(db: Database, tmp_path: Path) -> None: + rec = tmp_path / "rec" + # Dwell skeleton: first GPS fix at 20:00 (this defines the stop's start). + dwell = [ + (_utc(20, m).strftime("%Y-%m-%dT%H:%M:%SZ"), HOME_LAT, HOME_LON) + for m in (0, 2, 4, 6) + ] + fn_dwell = "2026_0618_200000_0001F.MP4" + _write_skeleton(rec, fn_dwell, dwell) + _seed(db, fn_dwell, recorded_at=int(_utc(20, 0).timestamp())) + + # Leading clip: recorded 30 s BEFORE the first fix (GPS-lock lag) -> must + # now be skipped (its footage is the start of the same home dwell). + fn_lead = "2026_0618_195930_0009F.MP4" + _seed(db, fn_lead, recorded_at=int(_utc(19, 59, 30).timestamp())) + + # A clip 5 min before the dwell is genuinely outside it -> must be kept. + fn_far = "2026_0618_195500_0010F.MP4" + _seed(db, fn_far, recorded_at=int(_utc(19, 55, 0).timestamp())) + + zones = (Place("Home", HOME_LAT, HOME_LON, 30, True),) + skipped = set(geofence.evaluate_day(db, str(rec), "2026-06-18", zones)) + assert fn_dwell in skipped + assert fn_lead in skipped # leading clip now caught by the pad + assert fn_far not in skipped # pad is bounded — far clip untouched + + +def test_sweep_all_seen_skips_unchanged_then_reevaluates(db: Database, tmp_path: Path) -> None: + rec = tmp_path / "rec" + dwell = [ + (_utc(20, m).strftime("%Y-%m-%dT%H:%M:%SZ"), HOME_LAT, HOME_LON) + for m in (0, 2, 4, 6) + ] + fn_a = "2026_0618_200300_0001F.MP4" + _write_skeleton(rec, fn_a, dwell) + _seed(db, fn_a, recorded_at=int(_utc(20, 3).timestamp())) + zones = (Place("Home", HOME_LAT, HOME_LON, 30, True),) + + seen: dict[str, int] = {} + assert geofence.sweep_all(db, str(rec), zones, seen=seen) == 1 + assert seen == {"2026-06-18": 1} + # Nothing new triaged -> day skipped (no re-eval, no new skips). + assert geofence.sweep_all(db, str(rec), zones, seen=seen) == 0 + # A newly-triaged home clip bumps the signature -> day re-evaluated. + fn_b = "2026_0618_200400_0002F.MP4" + _write_skeleton(rec, fn_b, dwell) + _seed(db, fn_b, recorded_at=int(_utc(20, 4).timestamp())) + assert geofence.sweep_all(db, str(rec), zones, seen=seen) == 1 + assert seen == {"2026-06-18": 2} + + +def test_sweep_all_seen_none_is_full_sweep(db: Database, tmp_path: Path) -> None: + rec = tmp_path / "rec" + dwell = [ + (_utc(20, m).strftime("%Y-%m-%dT%H:%M:%SZ"), HOME_LAT, HOME_LON) + for m in (0, 2, 4, 6) + ] + fn = "2026_0618_200300_0001F.MP4" + _write_skeleton(rec, fn, dwell) + _seed(db, fn, recorded_at=int(_utc(20, 3).timestamp())) + zones = (Place("Home", HOME_LAT, HOME_LON, 30, True),) + assert geofence.sweep_all(db, str(rec), zones) == 1 + + +def test_evaluate_day_keeps_departure_clip_in_journey(db: Database, tmp_path: Path) -> None: + """A clip that dwells in the home zone but also falls inside a journey's + padded window is the pull-away/pull-in footage — the journey wins over the + home dwell, so it must NOT be skipped (mirrors the archive grid).""" + rec = tmp_path / "rec" + # Home dwell 20:00-20:06 (one skeleton), then a drive moving east from 20:06:30. + dwell = [(_utc(20, m).strftime("%Y-%m-%dT%H:%M:%SZ"), HOME_LAT, HOME_LON) + for m in range(0, 7)] + drive = [] + for i in range(8): + t = _utc(20, 6, 30) + _dt.timedelta(seconds=30 * i) # 20:06:30..20:10:00 + drive.append((t.strftime("%Y-%m-%dT%H:%M:%SZ"), + HOME_LAT, HOME_LON + 0.001 * (i + 1))) # ~67 m steps east + _write_skeleton(rec, "2026_0618_200000_0001F.MP4", dwell) + _write_skeleton(rec, "2026_0618_200630_0005F.MP4", drive) + _seed(db, "2026_0618_200000_0001F.MP4", recorded_at=int(_utc(20, 0).timestamp())) + _seed(db, "2026_0618_200630_0005F.MP4", recorded_at=int(_utc(20, 6, 30).timestamp())) + + fn_dwell = "2026_0618_200100_0002F.MP4" # deep in the dwell -> skip + fn_departure = "2026_0618_200530_0003F.MP4" # dwell edge + in journey pad -> keep + _seed(db, fn_dwell, recorded_at=int(_utc(20, 1).timestamp())) + _seed(db, fn_departure, recorded_at=int(_utc(20, 5, 30).timestamp())) + + zones = (Place("Home", HOME_LAT, HOME_LON, 30, True),) + skipped = geofence.evaluate_day(db, str(rec), "2026-06-18", zones) + + assert fn_dwell in skipped # genuinely parked at home + assert fn_departure not in skipped # part of the drive -> journey wins diff --git a/tests/test_geofence_backfill.py b/tests/test_geofence_backfill.py new file mode 100644 index 0000000..e25f478 --- /dev/null +++ b/tests/test_geofence_backfill.py @@ -0,0 +1,34 @@ +"""Pure decision helper for the geofence backfill subscriber.""" +from __future__ import annotations + +from types import SimpleNamespace + +from web.app import _geofence_backfill_needed + + +def _snap(**kw): + return SimpleNamespace( + gps_triage=kw.get("gps_triage", True), + locations=kw.get( + "locations", (SimpleNamespace(exclude_recordings=True),) + ), + ) + + +def test_triggers_on_locations_change() -> None: + assert _geofence_backfill_needed({"LOCATIONS"}, _snap()) is True + + +def test_ignores_unrelated_keys() -> None: + assert _geofence_backfill_needed({"ADDRESS"}, _snap()) is False + + +def test_requires_triage_on() -> None: + assert _geofence_backfill_needed({"LOCATIONS"}, _snap(gps_triage=False)) is False + + +def test_requires_an_exclusion_location() -> None: + assert _geofence_backfill_needed( + {"LOCATIONS"}, _snap(locations=(SimpleNamespace(exclude_recordings=False),)) + ) is False + assert _geofence_backfill_needed({"LOCATIONS"}, _snap(locations=())) is False diff --git a/tests/test_gpx_moov_complete.py b/tests/test_gpx_moov_complete.py new file mode 100644 index 0000000..7d897b8 --- /dev/null +++ b/tests/test_gpx_moov_complete.py @@ -0,0 +1,41 @@ +"""has_final_moov detects a reachable top-level moov (finalized clip) vs a +recording-in-progress file that has only a growing mdat and no moov.""" +from __future__ import annotations + +import io +import struct + +from viofosync_lib import TruncatedRead +from viofosync_lib._gpx import has_final_moov + + +def _atom(atom_type: bytes, payload: bytes = b"") -> bytes: + return struct.pack(">I", 8 + len(payload)) + atom_type + payload + + +def test_reachable_moov_is_complete() -> None: + # NB: this ftyp,mdat,moov (moov-at-end) fixture is illustrative of the atom + # walk only. Real A229/A329 clips write moov at the FRONT and keep growing + # it while recording, so a reachable moov does NOT imply finalized on that + # hardware — the active-recording guard uses the cmd=3014 record flag, not + # this check. See has_final_moov's docstring. + data = _atom(b"ftyp", b"isom") + _atom(b"mdat", b"\x00" * 32) + _atom(b"moov") + assert has_final_moov(io.BytesIO(data)) is True + + +def test_mdat_only_is_incomplete() -> None: + data = _atom(b"ftyp", b"isom") + _atom(b"mdat", b"\x00" * 4096) + assert has_final_moov(io.BytesIO(data)) is False + + +def test_zero_size_header_stops_walk() -> None: + data = _atom(b"ftyp", b"isom") + struct.pack(">I", 0) + b"junk" + assert has_final_moov(io.BytesIO(data)) is False + + +def test_truncated_read_means_incomplete() -> None: + class _ShortReader(io.BytesIO): + def read(self, n=-1): + raise TruncatedRead("short read") + + assert has_final_moov(_ShortReader(b"")) is False diff --git a/tests/test_import_skip_list.py b/tests/test_import_skip_list.py new file mode 100644 index 0000000..b6d3574 --- /dev/null +++ b/tests/test_import_skip_list.py @@ -0,0 +1,109 @@ +from __future__ import annotations + +import time +from types import SimpleNamespace +from unittest.mock import MagicMock + +from starlette.requests import Request + +from web.db import Database +from web.routers import imports as imports_router +from web.services import importer + + +def _seed_skipped(db, filename): + with db.write() as c: + c.execute( + "INSERT INTO download_queue (filename, source_dir, state, enqueued_at) " + "VALUES (?, '', 'skipped', ?)", + (filename, int(time.time())), + ) + + +def _req(db, recordings): + snap = SimpleNamespace(recordings=str(recordings), grouping="daily") + provider = SimpleNamespace(get=lambda: snap) + return SimpleNamespace( + app=SimpleNamespace(state=SimpleNamespace(db=db, settings_provider=provider)) + ) + + +def test_present_reports_skip_listed(tmp_path): + db = Database(str(tmp_path / "v.db")) + (tmp_path / "rec").mkdir() + skipped = "2026_0618_120000_0001F.MP4" + wanted = "2026_0618_120100_0002F.MP4" + _seed_skipped(db, skipped) + body = imports_router._FilesBody(files=[ + imports_router._FileRef(name=skipped, size=0), + imports_router._FileRef(name=wanted, size=0), + ]) + out = imports_router.present(_req(db, tmp_path / "rec"), body) + assert out["skipped"] == [skipped] + assert wanted not in out["skipped"] + assert out["present"] == [] # neither is on disk + + +def _fake_app(tmp_path): + snap = MagicMock() + snap.recordings = str(tmp_path / "rec") + snap.grouping = "daily" + snap.import_path = None + snap.retention_disk_pct = 0 + snap.recordings_quota_gb = 0 + snap.retention_protect_ro = True + provider = MagicMock() + provider.get.return_value = snap + db = Database(str(tmp_path / "t.db")) + return SimpleNamespace( + state=SimpleNamespace(settings_provider=provider, db=db) + ) + + +def _upload_request(app, name, body): + messages = [{"type": "http.request", "body": body, "more_body": False}] + + async def receive(): + return messages.pop(0) + + scope = { + "type": "http", "method": "POST", "path": "/api/import/upload", + "query_string": b"", "app": app, + "headers": [ + (b"x-import-path", name.encode()), + (b"x-import-size", str(len(body)).encode()), + ], + } + return Request(scope, receive) + + +async def test_upload_refuses_skip_listed_before_make_room(tmp_path, monkeypatch): + app = _fake_app(tmp_path) + fn = "2026_0618_120000_0001F.MP4" + _seed_skipped(app.state.db, fn) + make_room = MagicMock(return_value=True) + monkeypatch.setattr(imports_router._retention, "make_room_for", make_room) + + res = await imports_router.upload(_upload_request(app, fn, b"x" * 32)) + + assert res["status"] == "skipped" + assert res["filename"] == fn + make_room.assert_not_called() # refused before any eviction + staging = tmp_path / "rec" / importer.STAGING_DIRNAME + assert not staging.exists() or not any(staging.iterdir()) # nothing written + + +async def test_upload_non_skip_listed_reaches_make_room(tmp_path, monkeypatch): + app = _fake_app(tmp_path) + fn = "2026_0618_120000_0001F.MP4" # NOT seeded as skipped + # make_room refuses → over_quota_older proves we passed the skip check. + monkeypatch.setattr( + imports_router._retention, "make_room_for", lambda *a, **k: False + ) + monkeypatch.setattr( + imports_router._retention, "import_exclude_set", lambda *a, **k: set() + ) + + res = await imports_router.upload(_upload_request(app, fn, b"x" * 32)) + + assert res["status"] == "over_quota_older" diff --git a/tests/test_locations.py b/tests/test_locations.py new file mode 100644 index 0000000..8fea790 --- /dev/null +++ b/tests/test_locations.py @@ -0,0 +1,55 @@ +"""Named-location helpers: name_for_point + exclusion_zones.""" +from __future__ import annotations + +from types import SimpleNamespace + +from web.services import locations + + +def _p(name, lat, lon, radius, exclude=False, is_home=False): + return SimpleNamespace( + name=name, lat=lat, lon=lon, radius_m=radius, + exclude_recordings=exclude, is_home=is_home, + ) + + +def test_name_for_point_inside_radius() -> None: + places = [_p("Home", 53.1, -2.0, 30)] + assert locations.name_for_point(places, 53.1, -2.0) == "Home" + + +def test_name_for_point_outside_radius() -> None: + places = [_p("Home", 53.1, -2.0, 30)] + # ~11 km north — well outside 30 m. + assert locations.name_for_point(places, 53.2, -2.0) is None + + +def test_name_for_point_first_match_wins() -> None: + places = [_p("Big", 53.1, -2.0, 100_000), _p("Small", 53.1, -2.0, 30)] + assert locations.name_for_point(places, 53.1, -2.0) == "Big" + + +def test_name_for_point_empty() -> None: + assert locations.name_for_point([], 53.1, -2.0) is None + + +def test_exclusion_zones_filters_by_flag() -> None: + a = _p("A", 0, 0, 30, exclude=True) + b = _p("B", 0, 0, 30, exclude=False) + assert locations.exclusion_zones([a, b]) == [a] + assert locations.exclusion_zones([]) == [] + + +def test_match_for_point_returns_place_with_is_home() -> None: + places = [_p("Home", 53.0, -2.0, 5000, is_home=True)] + p = locations.match_for_point(places, 53.01, -2.0) + assert p is not None and p.name == "Home" and p.is_home is True + + +def test_match_for_point_first_match_wins() -> None: + places = [_p("Big", 53.0, -2.0, 5000), _p("Small", 53.0, -2.0, 50)] + assert locations.match_for_point(places, 53.01, -2.0).name == "Big" + + +def test_match_for_point_none_outside() -> None: + assert locations.match_for_point([_p("Home", 53.0, -2.0, 30)], 60.0, 0.0) is None diff --git a/tests/test_locations_settings.py b/tests/test_locations_settings.py new file mode 100644 index 0000000..0ac8f32 --- /dev/null +++ b/tests/test_locations_settings.py @@ -0,0 +1,209 @@ +"""Settings for the named-locations feature (replaces geofence settings).""" +from __future__ import annotations + +import json +import tempfile + +import pytest +from pydantic import ValidationError + +from web.settings import Place, SettingsProvider +from web.settings_schema import ( + EDITABLE_KEYS, + Location, + SettingsModel, + normalize_locations, + validate_partial, +) + + +def test_defaults_empty() -> None: + assert SettingsModel().LOCATIONS == [] + assert "LOCATIONS" in EDITABLE_KEYS + # Old geofence keys are gone. + assert "GEOFENCE_ENABLED" not in EDITABLE_KEYS + assert "GEOFENCE_ZONES" not in EDITABLE_KEYS + + +def test_location_defaults_and_bounds() -> None: + loc = Location(lat=53.1, lon=-2.0) + assert loc.name == "Home" + assert loc.radius_m == 30 + assert loc.exclude_recordings is False + Location(lat=0, lon=0, radius_m=5) + Location(lat=0, lon=0, radius_m=500) + with pytest.raises(ValidationError): + Location(lat=0, lon=0, radius_m=4) + with pytest.raises(ValidationError): + Location(lat=91, lon=0) + with pytest.raises(ValidationError): + Location(lat=0, lon=181) + + +def test_validate_partial_roundtrips_location_list() -> None: + out = validate_partial( + {"LOCATIONS": [{"lat": 53.1, "lon": -2.0, "exclude_recordings": True}]} + ) + assert out["LOCATIONS"] == [ + {"name": "Home", "lat": 53.1, "lon": -2.0, + "radius_m": 30, "exclude_recordings": True, "is_home": True} + ] + + +def test_snapshot_exposes_locations() -> None: + with tempfile.TemporaryDirectory() as d: + sp = SettingsProvider( + config_path=f"{d}/c.json", env_file_path=f"{d}/v.env", recordings_dir=d + ) + assert sp.get().locations == () + + +def test_location_persists_and_reinflates() -> None: + with tempfile.TemporaryDirectory() as d: + cfg = f"{d}/c.json" + sp = SettingsProvider( + config_path=cfg, env_file_path=f"{d}/v.env", recordings_dir=d + ) + sp.update( + {"LOCATIONS": [{"lat": 53.1, "lon": -2.0, "radius_m": 40, + "exclude_recordings": True}]}, + actor="test", + ) + sp2 = SettingsProvider( + config_path=cfg, env_file_path=f"{d}/v.env", recordings_dir=d + ) + assert sp2.get().locations == ( + Place(name="Home", lat=53.1, lon=-2.0, radius_m=40, + exclude_recordings=True, is_home=True), + ) + + +def test_migrates_geofence_zones_to_locations() -> None: + with tempfile.TemporaryDirectory() as d: + cfg = f"{d}/c.json" + with open(cfg, "w", encoding="utf-8") as f: + json.dump( + {"WEB_PASSWORD_HASH": "x", "GEOFENCE_ENABLED": True, + "GEOFENCE_ZONES": [{"lat": 53.1, "lon": -2.0, "radius_m": 40}]}, + f, + ) + sp = SettingsProvider( + config_path=cfg, env_file_path=f"{d}/v.env", recordings_dir=d + ) + assert sp.get().locations == ( + Place(name="Home", lat=53.1, lon=-2.0, radius_m=40, + exclude_recordings=True, is_home=True), + ) + # Old keys removed from the rewritten config. + with open(cfg, encoding="utf-8") as f: + data = json.load(f) + assert "GEOFENCE_ZONES" not in data + assert "GEOFENCE_ENABLED" not in data + assert "LOCATIONS" in data + + +def test_migration_preserves_disabled_state() -> None: + with tempfile.TemporaryDirectory() as d: + cfg = f"{d}/c.json" + with open(cfg, "w", encoding="utf-8") as f: + json.dump( + {"WEB_PASSWORD_HASH": "x", "GEOFENCE_ENABLED": False, + "GEOFENCE_ZONES": [{"lat": 53.1, "lon": -2.0, "radius_m": 30}]}, + f, + ) + sp = SettingsProvider( + config_path=cfg, env_file_path=f"{d}/v.env", recordings_dir=d + ) + locs = sp.get().locations + assert len(locs) == 1 + assert locs[0].exclude_recordings is False + + +def test_migration_skips_malformed_zone() -> None: + with tempfile.TemporaryDirectory() as d: + cfg = f"{d}/c.json" + with open(cfg, "w", encoding="utf-8") as f: + json.dump( + {"WEB_PASSWORD_HASH": "x", "GEOFENCE_ENABLED": True, + "GEOFENCE_ZONES": [{"radius_m": 30}]}, # no lat/lon + f, + ) + # Must not raise at construction; the malformed zone is dropped. + sp = SettingsProvider( + config_path=cfg, env_file_path=f"{d}/v.env", recordings_dir=d + ) + assert sp.get().locations == () + + +def test_location_is_home_defaults_false() -> None: + assert Location(lat=53.0, lon=-2.0).is_home is False + + +def test_normalize_empty_unchanged() -> None: + assert normalize_locations([]) == [] + + +def test_normalize_promotes_first_when_none_home() -> None: + out = normalize_locations([ + {"name": "A", "lat": 1.0, "lon": 1.0, "radius_m": 30}, + {"name": "B", "lat": 2.0, "lon": 2.0, "radius_m": 30}, + ]) + assert [loc["is_home"] for loc in out] == [True, False] + + +def test_normalize_keeps_first_when_multiple_home() -> None: + out = normalize_locations([ + {"name": "A", "lat": 1.0, "lon": 1.0, "radius_m": 30, "is_home": True}, + {"name": "B", "lat": 2.0, "lon": 2.0, "radius_m": 30, "is_home": True}, + ]) + assert [loc["is_home"] for loc in out] == [True, False] + + +def test_normalize_single_home_unchanged() -> None: + out = normalize_locations([ + {"name": "A", "lat": 1.0, "lon": 1.0, "radius_m": 30, "is_home": False}, + {"name": "B", "lat": 2.0, "lon": 2.0, "radius_m": 30, "is_home": True}, + ]) + assert [loc["is_home"] for loc in out] == [False, True] + + +def test_validate_partial_normalizes_locations() -> None: + out = validate_partial({"LOCATIONS": [ + {"name": "A", "lat": 1.0, "lon": 1.0, "radius_m": 30}, + {"name": "B", "lat": 2.0, "lon": 2.0, "radius_m": 30}, + ]}) + assert [loc["is_home"] for loc in out["LOCATIONS"]] == [True, False] + + +def test_snapshot_exposes_is_home(tmp_path) -> None: + sp = SettingsProvider(config_path=tmp_path / "config.json") + sp.update({"LOCATIONS": [ + {"name": "Home", "lat": 53.0, "lon": -2.0, "radius_m": 30, "is_home": True}, + {"name": "Office", "lat": 54.0, "lon": -1.0, "radius_m": 50}, + ]}, actor="test") + snap = sp.get() + assert snap.locations[0].is_home is True + assert snap.locations[1].is_home is False + + +def test_legacy_location_without_is_home_migrates(tmp_path) -> None: + cfg = tmp_path / "config.json" + cfg.write_text(json.dumps({"LOCATIONS": [ + {"name": "Home", "lat": 53.0, "lon": -2.0, "radius_m": 30, + "exclude_recordings": False}, + ]})) + sp = SettingsProvider(config_path=cfg) + assert sp.get().locations[0].is_home is True + # the invariant is written back to disk, not just applied per-read + on_disk = json.loads(cfg.read_text()) + assert on_disk["LOCATIONS"][0]["is_home"] is True + + +def test_editable_values_includes_is_home(tmp_path) -> None: + from web.routers.settings import _editable_values + sp = SettingsProvider(config_path=tmp_path / "config.json") + sp.update({"LOCATIONS": [ + {"name": "Home", "lat": 53.0, "lon": -2.0, "radius_m": 30, "is_home": True}, + ]}, actor="t") + vals = _editable_values(sp.get()) + assert vals["LOCATIONS"][0]["is_home"] is True diff --git a/tests/test_make_room_for.py b/tests/test_make_room_for.py index fc34654..ba34efe 100644 --- a/tests/test_make_room_for.py +++ b/tests/test_make_room_for.py @@ -173,3 +173,30 @@ def fake_du(path): ) assert ok is True assert _ids(db) == set() # evicted until disk usage dropped under 90% + + +# --------------------------------------------------------------------------- +# locked=1 protection — make_room_for must never evict a locked clip +# --------------------------------------------------------------------------- + +def test_make_room_for_skips_locked_clip_protect_ro_false(env, monkeypatch): + """make_room_for must not evict a locked=1 clip even with protect_ro=False.""" + rec, db = env + locked_id = _clip(rec, db, basename="LOCKED.MP4", ts=100, size=1) + with db.write() as c: + c.execute("UPDATE clip_index SET locked=1 WHERE id=?", (locked_id,)) + + # Always over quota so the loop would evict if it could. + _patch_quota(monkeypatch, used_bytes=2 * (1 << 30)) + monkeypatch.setattr( + "web.services.retention._delete_clip_files", + lambda row, recordings: 1 << 30, + ) + + ok = ret.make_room_for( + db, str(rec), size=0, before_ts=300, + disk_pct=0, quota_gb=1, protect_ro=False, + ) + # Locked clip is the only candidate — nothing can be evicted → False. + assert ok is False + assert _ids(db) == {"LOCKED.MP4"} diff --git a/tests/test_mqtt_state_extraction.py b/tests/test_mqtt_state_extraction.py index c84c690..4e14dd1 100644 --- a/tests/test_mqtt_state_extraction.py +++ b/tests/test_mqtt_state_extraction.py @@ -137,7 +137,8 @@ def test_attrs_sync_status_carries_reason_when_error(): from web.services.mqtt_state import attrs_sync_status hub = _hub_with_state({}) attrs = attrs_sync_status(hub, None, _stub_snapshot(address=None)) - assert attrs == {"reason": "camera address not configured"} + assert attrs["reason"] == "camera address not configured" + assert attrs["triage_active"] is False def test_attrs_sync_status_reason_none_when_not_error(): @@ -148,7 +149,8 @@ def test_attrs_sync_status_reason_none_when_not_error(): "current_item": {"filename": "x.mp4"}, }) attrs = attrs_sync_status(hub, None, _stub_snapshot()) - assert attrs == {"reason": None} + assert attrs["reason"] is None + assert attrs["triage_active"] is False # ---- queue counts diff --git a/tests/test_naming_capture_key.py b/tests/test_naming_capture_key.py new file mode 100644 index 0000000..5dc14d6 --- /dev/null +++ b/tests/test_naming_capture_key.py @@ -0,0 +1,38 @@ +"""capture_key_sql yields the 14-digit YYYYMMDDHHMMSS capture key for both +filename layouts, so MAX() over it picks the true newest capture.""" +from __future__ import annotations + +import sqlite3 + +from web.services.naming import capture_key_sql + + +def _key(filename: str) -> str: + c = sqlite3.connect(":memory:") + try: + c.execute("CREATE TABLE clips (filename TEXT)") + c.execute("INSERT INTO clips (filename) VALUES (?)", (filename,)) + row = c.execute( + f"SELECT {capture_key_sql('filename')} FROM clips" + ).fetchone() + finally: + c.close() + return row[0] + + +def test_standard_layout_strips_underscore() -> None: + assert _key("2026_0628_133416_104430R.MP4") == "20260628133416" + + +def test_compact_layout_passes_through() -> None: + assert _key("20260628133416_000123.MP4") == "20260628133416" + + +def test_same_capture_across_layouts_matches() -> None: + assert _key("2026_0628_133416_0001F.MP4") == _key("20260628133416_9.MP4") + + +def test_siblings_share_key_despite_sequence() -> None: + assert _key("2026_0628_133416_020753PF.MP4") == _key( + "2026_0628_133416_020755PR.MP4" + ) diff --git a/tests/test_progress_disconnect_race.py b/tests/test_progress_disconnect_race.py new file mode 100644 index 0000000..cde41df --- /dev/null +++ b/tests/test_progress_disconnect_race.py @@ -0,0 +1,114 @@ +"""Regression: client disconnects racing the /api/progress handler. + +Starlette marks a WebSocket's application_state DISCONNECTED when a +send fails at the transport (websockets.py: ``except OSError`` → +``WebSocketDisconnect(1006)``). Two windows exposed the route to a +``RuntimeError('WebSocket is not connected. Need to call "accept" +first.')`` escaping as "Exception in ASGI application": + +1. The client vanishes between ``accept()`` and the hub's initial + snapshot send — ``Hub.connect()`` swallowed the failure and the + route entered its receive loop on the dead socket. +2. A hub broadcast fail-sends (evicting the client) while the route + is parked in ``receive_text()``; the loop's next state check + raises RuntimeError instead of WebSocketDisconnect. + +Both use a real starlette WebSocket over a fake ASGI transport so the +production state machine is exercised, not a test double. +""" +from __future__ import annotations + +import asyncio +from types import SimpleNamespace + +from starlette.websockets import WebSocket + +from web.routers.progress import progress +from web.services.hub import Hub + + +class _Auth: + def validate_session(self, token) -> bool: + return True + + +def _make_ws(receive, send, hub: Hub) -> WebSocket: + scope = { + "type": "websocket", + "headers": [(b"host", b"nas:8080")], # no Origin: non-browser path + "app": SimpleNamespace(state=SimpleNamespace(auth=_Auth(), hub=hub)), + } + return WebSocket(scope, receive=receive, send=send) + + +async def test_client_gone_before_snapshot_send_does_not_raise(): + """Window 1: transport drops between accept and the snapshot send.""" + hub = Hub() + got_connect = False + + async def receive(): + nonlocal got_connect + if not got_connect: + got_connect = True + return {"type": "websocket.connect"} + return {"type": "websocket.disconnect", "code": 1006} + + async def send(message): + if message["type"] == "websocket.accept": + return + # First real frame (the snapshot) hits a dead transport; + # starlette converts this to WebSocketDisconnect and flips + # application_state to DISCONNECTED. + raise OSError("connection lost") + + ws = _make_ws(receive, send, hub) + # Must complete without RuntimeError reaching the ASGI layer. + await asyncio.wait_for(progress(ws), timeout=2) + assert not hub._clients, "dead client left registered on the hub" + + +async def test_broadcast_eviction_midsession_does_not_raise(): + """Window 2: a broadcast fail-send flips the socket to + DISCONNECTED while the route loop is parked in receive_text(); + a buffered client frame then re-enters the loop.""" + hub = Hub() + sends = 0 + got_connect = False + text_ready = asyncio.Event() + text_delivered = False + + async def receive(): + nonlocal got_connect, text_delivered + if not got_connect: + got_connect = True + return {"type": "websocket.connect"} + if not text_delivered: + await text_ready.wait() + text_delivered = True + return {"type": "websocket.receive", "text": "ping"} + return {"type": "websocket.disconnect", "code": 1006} + + async def send(message): + nonlocal sends + sends += 1 + if sends <= 2: # accept + snapshot succeed + return + raise OSError("connection lost") # the broadcast fail-send + + ws = _make_ws(receive, send, hub) + task = asyncio.create_task(progress(ws)) + # Let the route connect and park in receive_text(). + for _ in range(10): + await asyncio.sleep(0) + assert ws in hub._clients + + # Broadcast fail-sends: hub evicts the client, starlette marks + # the socket DISCONNECTED under the route's feet. + await hub.broadcast({"type": "dashcam_offline"}) + assert ws not in hub._clients + + # A frame the client sent before dying is still buffered; the + # route loop consumes it and re-enters receive_text() on the + # now-DISCONNECTED socket. + text_ready.set() + await asyncio.wait_for(task, timeout=2) # must not raise diff --git a/tests/test_protocol_moov_probe.py b/tests/test_protocol_moov_probe.py new file mode 100644 index 0000000..2056807 --- /dev/null +++ b/tests/test_protocol_moov_probe.py @@ -0,0 +1,57 @@ +"""extract_remote_gps_points raises IncompleteRecording when the remote clip +has no reachable moov; remote_moov_reachable wraps the atom walk over a +RangeReader without downloading the whole clip.""" +from __future__ import annotations + +import struct + +import pytest + +from viofosync_lib import IncompleteRecording, _gpx +from viofosync_lib._protocol import RangeReader + + +def _atom(atom_type: bytes, payload: bytes = b"") -> bytes: + return struct.pack(">I", 8 + len(payload)) + atom_type + payload + + +def _reader(data: bytes) -> RangeReader: + def _read_range(a, b): + return data[a : b + 1] + + return RangeReader(_read_range, len(data)) + + +def test_has_final_moov_over_rangereader_true() -> None: + data = _atom(b"ftyp", b"isom") + _atom(b"mdat", b"\x00" * 64) + _atom(b"moov") + assert _gpx.has_final_moov(_reader(data)) is True + + +def test_has_final_moov_over_rangereader_false() -> None: + data = _atom(b"ftyp", b"isom") + _atom(b"mdat", b"\x00" * 64) + assert _gpx.has_final_moov(_reader(data)) is False + + +def test_extract_remote_raises_on_no_moov(monkeypatch) -> None: + from viofosync_lib import _protocol + + data = _atom(b"ftyp", b"isom") + _atom(b"mdat", b"\x00" * 64) + monkeypatch.setattr( + _protocol, "open_remote_reader", + lambda *a, **k: _reader(data), + ) + rec = _protocol.Recording("x.MP4", "/DCIM", len(data), None, None, None) + with pytest.raises(IncompleteRecording): + _protocol.extract_remote_gps_points("http://cam", rec) + + +def test_extract_remote_returns_empty_for_moov_without_gps(monkeypatch) -> None: + from viofosync_lib import _protocol + + data = _atom(b"ftyp", b"isom") + _atom(b"moov", _atom(b"udta")) + monkeypatch.setattr( + _protocol, "open_remote_reader", + lambda *a, **k: _reader(data), + ) + rec = _protocol.Recording("x.MP4", "/DCIM", len(data), None, None, None) + assert _protocol.extract_remote_gps_points("http://cam", rec) == [] diff --git a/tests/test_protocol_overage.py b/tests/test_protocol_overage.py new file mode 100644 index 0000000..fea9224 --- /dev/null +++ b/tests/test_protocol_overage.py @@ -0,0 +1,54 @@ +"""If a download streams more bytes than expected (the file was still growing), +download_file aborts early and raises DownloadDeferred instead of looping on +'Incomplete download'. No attempt is burned; the partial is removed.""" +from __future__ import annotations + +import io + +import pytest + +from viofosync_lib import DownloadDeferred, _protocol +from viofosync_lib._archive import Recording + + +class _FakeResp(io.BytesIO): + status = 200 + + def __enter__(self): + return self + + def __exit__(self, *a): + return False + + +def test_overage_aborts_and_defers(tmp_path, monkeypatch): + # HEAD says 60 bytes; GET streams well past the 2 MB overage slack + # (still recording) -- the guard triggers on bytes_done, not on a + # tiny handful of bytes, so the stream must actually outgrow + # expected_size + 2 MiB. + monkeypatch.setattr(_protocol, "get_remote_size", lambda url, t: 60) + + urlopen_calls = {"n": 0} + overage_payload = b"\x00" * (3 * 1024 * 1024) # 3 MB > 60B + 2MB slack + + def fake_urlopen(*a, **k): + urlopen_calls["n"] += 1 + return _FakeResp(overage_payload) + + monkeypatch.setattr(_protocol.urllib.request, "urlopen", fake_urlopen) + + rec = Recording("2026_0628_133416_0002F.MP4", "/DCIM", 60, None, None, None) + # The still-growing file is a deferral, not a failure: download_file + # raises DownloadDeferred so the caller requeues without burning an attempt. + with pytest.raises(DownloadDeferred): + _protocol.download_file( + "http://cam", rec, str(tmp_path), "Group", + max_attempts=3, + ) + # No leftover .part file anywhere under the destination (the + # download lands in destination/Group/). + assert list(tmp_path.rglob("*.part")) == [] + # Proves early abort rather than retry-until-exhausted: if the + # overage weren't caught, the current code would loop on + # "Incomplete download" and call urlopen once per attempt (3x). + assert urlopen_calls["n"] == 1 diff --git a/tests/test_protocol_short_read.py b/tests/test_protocol_short_read.py new file mode 100644 index 0000000..c85617f --- /dev/null +++ b/tests/test_protocol_short_read.py @@ -0,0 +1,73 @@ +"""RangeReader raises on a truncated range response instead of silently +returning partial data (which parse_moov would commit as a valid track).""" +from __future__ import annotations + +import pytest + +from viofosync_lib import TruncatedRead +from viofosync_lib._protocol import RangeReader + + +def test_full_range_ok(): + # read_range returns exactly the requested bytes -> no error. + data = bytes(range(256)) * 8 # 2048 bytes + rr = RangeReader(lambda a, b: data[a : b + 1], len(data), head_len=16) + rr.seek(100) + assert rr.read(50) == data[100:150] + + +def test_short_range_raises(): + # read_range returns fewer bytes than requested, not at EOF -> raise. + size = 2048 + + def short(a, b): + want = b - a + 1 + return b"\x00" * (want // 2) # half what was asked for + + rr = RangeReader(short, size, head_len=16) + rr.seek(100) + with pytest.raises(TruncatedRead): + rr.read(50) + + +def test_short_read_in_head_region_raises(): + # The head prefetch itself is truncated: head_len=16 but the source only + # ever returns half, so the prefetched head is short. A read confined to + # the head's intended region (offset 0, len 16) must still raise — the + # camera closed the body early during the prefetch. + # + # Note: once a head buffer is populated, a pure `_head[start:end]` slice + # can never be short, so the only way a read "inside the head" is short is + # a truncated prefetch like this. With head_n=8 the read crosses into the + # range branch, which is where the guard fires. + def short(a, b): + want = b - a + 1 + return b"\x00" * (want // 2) + + rr = RangeReader(short, 2048, head_len=16) + rr.seek(0) + with pytest.raises(TruncatedRead): + rr.read(16) + + +def test_short_read_spanning_head_boundary_raises(): + size = 2048 + + def part(a, b): + want = b - a + 1 + if a == 0: + return b"\x00" * want # head prefetch is complete + return b"\x00" * (want // 2) # later range read is short + + rr = RangeReader(part, size, head_len=16) + rr.seek(8) # start inside the 16-byte head + with pytest.raises(TruncatedRead): + rr.read(40) # ends well past head_n=16 + + +def test_eof_clamp_is_not_short(): + # Reading past EOF clamps and returns "" — that's legitimate, not short. + data = b"abcd" + rr = RangeReader(lambda a, b: data[a : b + 1], len(data), head_len=16) + rr.seek(4) + assert rr.read(10) == b"" diff --git a/tests/test_queue_active_guard.py b/tests/test_queue_active_guard.py new file mode 100644 index 0000000..c214b85 --- /dev/null +++ b/tests/test_queue_active_guard.py @@ -0,0 +1,58 @@ +"""next_pending holds every lens of the newest (possibly-recording) capture +until remote_complete=1; older captures and superseded captures are free.""" +from __future__ import annotations + +import time + +from web.db import Database +from web.services import queue as q + + +def _seed(db, filename, *, state="pending", remote_complete=None, + triaged_at=1): + now = int(time.time()) + with db.write() as c: + c.execute( + "INSERT INTO download_queue " + "(filename, source_dir, camera, event_type, state, enqueued_at, " + " triaged_at, remote_complete) VALUES (?,?,?,?,?,?,?,?)", + (filename, "/DCIM", filename[-5], "normal", state, now, + triaged_at, remote_complete), + ) + + +def test_newest_capture_front_is_held(tmp_path): + db = Database(str(tmp_path / "v.db")) + _seed(db, "2026_0628_120000_0001F.MP4") # older capture + _seed(db, "2026_0628_133416_0002F.MP4") # newest, unconfirmed + item = q.next_pending(db, active_guard=True) + assert item is not None + assert item.filename == "2026_0628_120000_0001F.MP4" # older one only + + +def test_newest_capture_rear_sibling_is_held(tmp_path): + db = Database(str(tmp_path / "v.db")) + _seed(db, "2026_0628_133416_0002F.MP4") # newest front + _seed(db, "2026_0628_133416_0003R.MP4") # newest rear (diff seq) + assert q.next_pending(db, active_guard=True) is None + + +def test_confirmed_newest_is_released(tmp_path): + db = Database(str(tmp_path / "v.db")) + _seed(db, "2026_0628_133416_0002F.MP4", remote_complete=1) + assert q.next_pending(db, active_guard=True) is not None + + +def test_supersession_releases_previous(tmp_path): + db = Database(str(tmp_path / "v.db")) + _seed(db, "2026_0628_133416_0002F.MP4") # was newest, unconfirmed + _seed(db, "2026_0628_140000_0009F.MP4") # now newest, unconfirmed + item = q.next_pending(db, active_guard=True) + assert item is not None + assert item.filename == "2026_0628_133416_0002F.MP4" # previous now free + + +def test_guard_off_hands_out_newest(tmp_path): + db = Database(str(tmp_path / "v.db")) + _seed(db, "2026_0628_133416_0002F.MP4") + assert q.next_pending(db, active_guard=False) is not None diff --git a/tests/test_queue_delete.py b/tests/test_queue_delete.py new file mode 100644 index 0000000..de12a6d --- /dev/null +++ b/tests/test_queue_delete.py @@ -0,0 +1,194 @@ +"""queue.delete_clips: local file + index delete, mark download_queue skipped.""" +from __future__ import annotations + +import time +from pathlib import Path + +import pytest +from fastapi.testclient import TestClient + + +def _insert_queue(c, filename, state): + c.execute( + "INSERT INTO download_queue (filename, source_dir, state, enqueued_at) " + "VALUES (?, ?, ?, ?)", + (filename, "/DCIM", state, int(time.time())), + ) + + +def _make_clip(rec: Path, db, *, basename, event_type="normal"): + """A fake clip on disk + clip_index row + a 'done' download_queue row.""" + folder = rec / "2026-06-26" + folder.mkdir(exist_ok=True) + path = folder / basename + path.write_bytes(b"x" * 1024) + (folder / (basename + ".gpx")).write_text("") + with db.write() as c: + cur = c.execute( + "INSERT INTO clip_index " + "(path, basename, group_name, timestamp, camera, sequence, " + " event_type, size_bytes, has_gpx, scanned_at) " + "VALUES (?, ?, '2026-06-26', 1, 'F', 1, ?, 1024, 1, ?)", + (str(path), basename, event_type, int(time.time())), + ) + cid = cur.lastrowid + _insert_queue(c, basename, "done") + return cid, path + + +def _env(tmp_path: Path): + from web.db import Database + rec = tmp_path / "rec" + rec.mkdir() + db = Database(str(rec / "v.db")) + return rec, db + + +def test_delete_removes_files_index_and_marks_skipped(tmp_path): + from web.services.queue import delete_clips + rec, db = _env(tmp_path) + cid, path = _make_clip(rec, db, basename="A.MP4") + res = delete_clips(db, ["A.MP4"], str(rec)) + assert res == {"deleted": 1, "skipped": 1, "protected": 0} + assert not path.exists() + assert not (path.parent / "A.MP4.gpx").exists() + with db.conn() as c: + assert c.execute("SELECT COUNT(*) AS n FROM clip_index").fetchone()["n"] == 0 + row = c.execute( + "SELECT state, skip_reason FROM download_queue WHERE filename='A.MP4'" + ).fetchone() + assert row["state"] == "skipped" and row["skip_reason"] == "user" + + +def test_delete_skips_ro_clip(tmp_path): + from web.services.queue import delete_clips + rec, db = _env(tmp_path) + _, path = _make_clip(rec, db, basename="LOCK.MP4", event_type="ro") + res = delete_clips(db, ["LOCK.MP4"], str(rec)) + assert res["deleted"] == 0 + assert res.get("protected") == 1 + assert path.exists() # RO clip retained + + +def test_delete_nonexistent_file_reports_zero(tmp_path): + # A filename with no clip_index AND no download_queue row touches nothing: + # skipped must reflect the actual queue UPDATE rowcount (0), not len(targets). + from web.services.queue import delete_clips + rec, db = _env(tmp_path) + res = delete_clips(db, ["GHOST.MP4"], str(rec)) + assert res == {"deleted": 0, "skipped": 0, "protected": 0} + + +def test_delete_skips_locked_clip(tmp_path): + from web.services.queue import delete_clips + rec, db = _env(tmp_path) + _, path = _make_clip(rec, db, basename="KEEP.MP4") + with db.write() as c: + c.execute("UPDATE clip_index SET locked=1 WHERE basename='KEEP.MP4'") + res = delete_clips(db, ["KEEP.MP4"], str(rec)) + assert res["deleted"] == 0 and res.get("protected") == 1 + assert path.exists() + + +def test_delete_undownloaded_marks_skipped_only(tmp_path): + # A queued-but-not-downloaded clip: no clip_index row, just a pending queue row. + from web.services.queue import delete_clips + rec, db = _env(tmp_path) + with db.write() as c: + _insert_queue(c, "PEND.MP4", "pending") + res = delete_clips(db, ["PEND.MP4"], str(rec)) + assert res == {"deleted": 0, "skipped": 1, "protected": 0} + with db.conn() as c: + assert c.execute( + "SELECT state FROM download_queue WHERE filename='PEND.MP4'" + ).fetchone()["state"] == "skipped" + + +def test_delete_missing_file_does_not_raise(tmp_path): + from web.services.queue import delete_clips + rec, db = _env(tmp_path) + _, path = _make_clip(rec, db, basename="GONE.MP4") + path.unlink() # file already gone; index row remains + res = delete_clips(db, ["GONE.MP4"], str(rec)) + assert res["deleted"] == 1 # the index row was removed + + +def test_delete_empty_is_noop(tmp_path): + from web.services.queue import delete_clips + rec, db = _env(tmp_path) + assert delete_clips(db, [], str(rec)) == {"deleted": 0, "skipped": 0, "protected": 0} + + +def test_delete_logs_info_audit_line(tmp_path, caplog): + # The viofosync.queue logger emits at INFO, which DBLogHandler persists to + # app_log — so a delete leaves an audit trail. + from web.services.queue import delete_clips + rec, db = _env(tmp_path) + _make_clip(rec, db, basename="A.MP4") + with caplog.at_level("INFO", logger="viofosync.queue"): + delete_clips(db, ["A.MP4"], str(rec)) + recs = [r for r in caplog.records if r.name == "viofosync.queue"] + assert any("archive delete" in r.getMessage() and "A.MP4" in r.getMessage() + for r in recs), recs + + +def test_delete_noop_does_not_log(tmp_path, caplog): + # Nothing matched -> no audit line (avoids "removed 0" noise). + from web.services.queue import delete_clips + rec, db = _env(tmp_path) + with caplog.at_level("INFO", logger="viofosync.queue"): + delete_clips(db, ["NOPE.MP4"], str(rec)) + assert not [r for r in caplog.records if r.name == "viofosync.queue"] + + +@pytest.fixture +def authed_client(tmp_config_dir: Path, tmp_recordings_dir: Path, monkeypatch): + from web import app as app_mod + from web import settings as settings_mod + monkeypatch.setenv("VIOFOSYNC_RESTART_DISABLED", "1") + settings_mod.reset_for_tests() + application = app_mod.create_app() + with TestClient(application) as c: + c.post("/setup", data={ + "address": "192.168.1.230", + "password": "twelve-chars-min!", + "confirm": "twelve-chars-min!", + }) + csrf = c.get("/api/auth/csrf").json()["csrf"] + c.headers.update({"x-csrf-token": csrf}) + yield c + + +def test_delete_endpoint(authed_client, tmp_recordings_dir: Path): + db = authed_client.app.state.db + folder = tmp_recordings_dir / "2026-06-26" + folder.mkdir(exist_ok=True) + path = folder / "A.MP4" + path.write_bytes(b"x" * 1024) + with db.write() as c: + c.execute( + "INSERT INTO clip_index " + "(path, basename, group_name, timestamp, camera, sequence, " + " event_type, size_bytes, has_gpx, scanned_at) " + "VALUES (?, 'A.MP4', '2026-06-26', 1, 'F', 1, 'normal', 1024, 0, 1)", + (str(path),), + ) + c.execute( + "INSERT INTO download_queue (filename, source_dir, state, enqueued_at) " + "VALUES ('A.MP4', '/DCIM', 'done', 1)", + ) + r = authed_client.post("/api/queue/delete", json={"filenames": ["A.MP4"]}) + assert r.status_code == 200 + body = r.json() + assert body["ok"] is True and body["deleted"] == 1 and body["skipped"] == 1 + assert not path.exists() + with db.conn() as c: + assert c.execute( + "SELECT state FROM download_queue WHERE filename='A.MP4'" + ).fetchone()["state"] == "skipped" + + +def test_delete_endpoint_empty_body(authed_client): + r = authed_client.post("/api/queue/delete", json={}) + assert r.status_code == 200 + assert r.json()["deleted"] == 0 and r.json()["skipped"] == 0 diff --git a/tests/test_queue_download_next.py b/tests/test_queue_download_next.py new file mode 100644 index 0000000..0e76c7e --- /dev/null +++ b/tests/test_queue_download_next.py @@ -0,0 +1,50 @@ +# tests/test_queue_download_next.py +"""download_next: re-queue selected clips for immediate download.""" +from __future__ import annotations + +import time + +from web.db import Database +from web.services import queue as q + + +def _seed(db, filename, *, state, priority=0): + now = int(time.time()) + with db.write() as c: + c.execute( + "INSERT INTO download_queue " + "(filename, source_dir, state, priority, enqueued_at, attempts) " + "VALUES (?,?,?,?,?,0)", + (filename, "/DCIM/Movie", state, priority, now), + ) + + +def _row(db, filename): + with db.conn() as c: + return dict(c.execute( + "SELECT state, priority FROM download_queue WHERE filename=?", + (filename,), + ).fetchone()) + + +def test_download_next_requeues_and_prioritizes(tmp_path): + db = Database(str(tmp_path / "v.db")) + _seed(db, "skip.MP4", state="skipped") + _seed(db, "fail.MP4", state="failed") + _seed(db, "pend.MP4", state="pending") + _seed(db, "done.MP4", state="done") + _seed(db, "other.MP4", state="pending") # not selected — must stay lower + + n = q.download_next( + db, ["skip.MP4", "fail.MP4", "pend.MP4", "done.MP4"] + ) + + assert _row(db, "skip.MP4")["state"] == "pending" + assert _row(db, "fail.MP4")["state"] == "pending" + assert _row(db, "pend.MP4")["state"] == "pending" + assert _row(db, "done.MP4")["state"] == "done" + sel_prio = min( + _row(db, f)["priority"] for f in ("skip.MP4", "fail.MP4", "pend.MP4") + ) + assert sel_prio > _row(db, "other.MP4")["priority"] + assert n == 3 diff --git a/tests/test_queue_geofence.py b/tests/test_queue_geofence.py new file mode 100644 index 0000000..daf71b0 --- /dev/null +++ b/tests/test_queue_geofence.py @@ -0,0 +1,144 @@ +"""Geofence queue provenance: skip_reason, geofence_skip, release stamping.""" +from __future__ import annotations + +from pathlib import Path + +import pytest + +from web.db import Database +from web.services import queue as q + + +@pytest.fixture +def db(tmp_path: Path) -> Database: + return Database(str(tmp_path / "v.db")) + + +def test_download_queue_has_geofence_columns(db: Database) -> None: + with db.conn() as c: + cols = {r["name"] for r in c.execute("PRAGMA table_info(download_queue)")} + assert "skip_reason" in cols + assert "geofence_released_at" in cols + + +def test_migration_is_idempotent(tmp_path: Path) -> None: + p = str(tmp_path / "v.db") + Database(p) # creates + migrates + db2 = Database(p) # second open re-runs _migrate; must not raise + with db2.conn() as c: + cols = {r["name"] for r in c.execute("PRAGMA table_info(download_queue)")} + assert "skip_reason" in cols + assert "geofence_released_at" in cols + + +def _rows(db: Database) -> dict: + with db.conn() as c: + return { + r["filename"]: dict(r) + for r in c.execute("SELECT * FROM download_queue").fetchall() + } + + +def _seed(db: Database, filename: str, *, state: str = "pending", + skip_reason=None, source_dir="/DCIM/Movie", recorded_at=1_000, + released=None) -> None: + with db.write() as c: + c.execute( + "INSERT INTO download_queue " + "(filename, source_dir, state, skip_reason, recorded_at, " + " geofence_released_at, enqueued_at) VALUES (?,?,?,?,?,?,0)", + (filename, source_dir, state, skip_reason, recorded_at, released), + ) + + +def test_skip_sets_user_reason(db: Database) -> None: + _seed(db, "p_0001F.MP4", state="pending") + q.skip(db, ["p_0001F.MP4"]) + row = _rows(db)["p_0001F.MP4"] + assert row["state"] == "skipped" + assert row["skip_reason"] == "user" + + +def test_unskip_stamps_release_only_for_geofence(db: Database) -> None: + _seed(db, "g_0001F.MP4", state="skipped", skip_reason="geofence") + _seed(db, "u_0001F.MP4", state="skipped", skip_reason="user") + q.unskip(db, ["g_0001F.MP4", "u_0001F.MP4"]) + rows = _rows(db) + assert rows["g_0001F.MP4"]["state"] == "pending" + assert rows["g_0001F.MP4"]["skip_reason"] is None + assert rows["g_0001F.MP4"]["geofence_released_at"] is not None + assert rows["u_0001F.MP4"]["state"] == "pending" + assert rows["u_0001F.MP4"]["geofence_released_at"] is None + + +def test_geofence_skip_flips_only_pending_unreleased(db: Database) -> None: + _seed(db, "a_0001F.MP4", state="pending") + _seed(db, "b_0001F.MP4", state="done") + _seed(db, "c_0001F.MP4", state="pending", released=12345) + n = q.geofence_skip(db, ["a_0001F.MP4", "b_0001F.MP4", "c_0001F.MP4"]) + rows = _rows(db) + assert n == 1 + assert rows["a_0001F.MP4"]["state"] == "skipped" + assert rows["a_0001F.MP4"]["skip_reason"] == "geofence" + assert rows["b_0001F.MP4"]["state"] == "done" # not pending + assert rows["c_0001F.MP4"]["state"] == "pending" # released, untouched + + +def test_geofence_candidates_day_and_guards(db: Database) -> None: + # Real Viofo-prefixed filenames so the day expression matches. + _seed(db, "2026_0618_200000_0001F.MP4", state="pending", recorded_at=6_000) + _seed(db, "2026_0618_200000_0002EF.MP4", state="pending", recorded_at=6_000) + _seed(db, "2026_0618_200000_0003F.MP4", state="pending", + source_dir="/DCIM/Movie/RO", recorded_at=6_000) + _seed(db, "2026_0618_200000_0004F.MP4", state="pending", + recorded_at=6_000, released=1) + _seed(db, "2026_0618_200000_0005F.MP4", state="done", recorded_at=6_000) + names = {c["filename"] for c in q.geofence_candidates(db, "2026-06-18")} + assert names == {"2026_0618_200000_0001F.MP4"} # only the plain pending clip + + +def test_pending_days(db: Database) -> None: + _seed(db, "2026_0618_200000_0001F.MP4", state="pending") + _seed(db, "2026_0619_080000_0001F.MP4", state="pending") + _seed(db, "2026_0617_080000_0001F.MP4", state="done") + assert set(q.pending_days(db)) == {"2026-06-18", "2026-06-19"} + + +def test_geofence_skipped_by_day_counts_front_geofence_only(db: Database) -> None: + from web.routers.archive import geofence_skipped_by_day + # Two geofence-skipped pairs on 2026-06-19 (front clips counted only). + _seed(db, "2026_0619_082048_001F.MP4", state="skipped", skip_reason="geofence") + _seed(db, "2026_0619_082048_001R.MP4", state="skipped", skip_reason="geofence") + _seed(db, "2026_0619_082148_002F.MP4", state="skipped", skip_reason="geofence") + # A user skip and a pending clip must NOT count. + _seed(db, "2026_0619_083000_003F.MP4", state="skipped", skip_reason="user") + _seed(db, "2026_0619_084000_004F.MP4", state="pending") + # A different day. + _seed(db, "2026_0620_090000_005F.MP4", state="skipped", skip_reason="geofence") + assert geofence_skipped_by_day(db) == {"2026-06-19": 2, "2026-06-20": 1} + + +def test_geofence_skipped_by_day_respects_date_range(db: Database) -> None: + from web.routers.archive import geofence_skipped_by_day + _seed(db, "2026_0619_082048_001F.MP4", state="skipped", skip_reason="geofence") + _seed(db, "2026_0620_090000_002F.MP4", state="skipped", skip_reason="geofence") + assert geofence_skipped_by_day(db, "2026-06-20", None) == {"2026-06-20": 1} + assert geofence_skipped_by_day(db, None, "2026-06-19") == {"2026-06-19": 1} + + +def test_locked_columns_exist(db: Database) -> None: + with db.conn() as c: + ci = {r["name"] for r in c.execute("PRAGMA table_info(clip_index)")} + dq = {r["name"] for r in c.execute("PRAGMA table_info(download_queue)")} + assert "locked" in ci + assert "locked" in dq + + +def test_locked_migration_idempotent(tmp_path) -> None: + from pathlib import Path + p = str(Path(tmp_path) / "v.db") + Database(p) # create + migrate + db2 = Database(p) # re-migrate must not raise + with db2.conn() as c: + ci = {r["name"] for r in c.execute("PRAGMA table_info(clip_index)")} + assert "locked" in ci diff --git a/tests/test_queue_geofence_signatures.py b/tests/test_queue_geofence_signatures.py new file mode 100644 index 0000000..d30ef90 --- /dev/null +++ b/tests/test_queue_geofence_signatures.py @@ -0,0 +1,46 @@ +"""queue.geofence_day_signatures: per-day triaged-skeleton counts.""" +from __future__ import annotations + +from pathlib import Path + +import pytest + +from web.db import Database +from web.services import queue + +DETECT = ("pending", "failed", "skipped", "downloading") + + +@pytest.fixture +def db(tmp_path: Path) -> Database: + return Database(str(tmp_path / "v.db")) + + +def _seed(db, filename, *, state="pending", triaged=True, gps=5): + with db.write() as c: + c.execute( + "INSERT INTO download_queue " + "(filename, source_dir, state, recorded_at, triaged_at, " + " gps_points, enqueued_at) VALUES (?,?,?,?,?,?,0)", + (filename, "/DCIM/Movie", state, 0, + 1 if triaged else None, gps if triaged else None), + ) + + +def test_counts_triaged_with_gps_per_day(db: Database) -> None: + _seed(db, "2026_0618_200000_0001F.MP4") # day A, counts + _seed(db, "2026_0618_200100_0002F.MP4") # day A, counts + _seed(db, "2026_0618_200200_0003F.MP4", gps=0) # triaged, no fix -> excluded + _seed(db, "2026_0618_200300_0004F.MP4", triaged=False) # untriaged -> excluded + _seed(db, "2026_0619_080000_0005F.MP4") # day B, counts + sigs = queue.geofence_day_signatures(db, DETECT) + assert sigs == {"2026-06-18": 2, "2026-06-19": 1} + + +def test_skipped_clip_still_counts(db: Database) -> None: + _seed(db, "2026_0618_200000_0001F.MP4", state="skipped") + assert queue.geofence_day_signatures(db, DETECT) == {"2026-06-18": 1} + + +def test_empty_db_is_empty(db: Database) -> None: + assert queue.geofence_day_signatures(db, DETECT) == {} diff --git a/tests/test_queue_gps_state.py b/tests/test_queue_gps_state.py new file mode 100644 index 0000000..2f0fcbc --- /dev/null +++ b/tests/test_queue_gps_state.py @@ -0,0 +1,111 @@ +"""list_page / list_day_items expose a derived gps_state per row.""" +from __future__ import annotations + +import time + +from web.db import Database +from web.services import queue as q +from web.services.triage import TRIAGE_MAX_ATTEMPTS + + +def _seed(db, filename, **cols): + now = int(time.time()) + base = dict(source_dir="/DCIM", camera=filename[-5], event_type="normal", + state="pending", enqueued_at=now, recorded_at=now) + base.update(cols) + keys = ",".join(base) + qmarks = ",".join("?" * len(base)) + with db.write() as c: + c.execute( + f"INSERT INTO download_queue (filename,{keys}) " + f"VALUES (?,{qmarks})", + (filename, *base.values()), + ) + + +def _state_for(items, fn): + return next(it["gps_state"] for it in items if it["filename"] == fn) + + +def test_gps_state_ok_none_pending(tmp_path): + db = Database(str(tmp_path / "v.db")) + now = int(time.time()) + _seed(db, "2026_0618_203643_0001F.MP4", triaged_at=now, gps_points=42) # ok + _seed(db, "2026_0618_203644_0002F.MP4", triaged_at=now, gps_points=0) # none + _seed(db, "2026_0618_203645_0003F.MP4") # pending + _seed(db, "2026_0618_203646_0004F.MP4", + triage_attempts=TRIAGE_MAX_ATTEMPTS) # none (gave up) + + items = q.list_page(db, per_page=100)["items"] + assert _state_for(items, "2026_0618_203643_0001F.MP4") == "ok" + assert _state_for(items, "2026_0618_203644_0002F.MP4") == "none" + assert _state_for(items, "2026_0618_203645_0003F.MP4") == "pending" + assert _state_for(items, "2026_0618_203646_0004F.MP4") == "none" + + +def test_gps_state_present_in_day_items(tmp_path): + db = Database(str(tmp_path / "v.db")) + now = int(time.time()) + fn = "2026_0618_203643_0001F.MP4" + _seed(db, fn, triaged_at=now, gps_points=7) + items = q.list_day_items(db, day="2026-06-18") + assert _state_for(items, fn) == "ok" + + +def test_gps_state_none_for_orphan_non_gps_lenses(tmp_path): + # GPS is a capture-level fact carried by the front lens. A rear/tele/ + # interior row with NO front sibling at its timestamp has no GPS fact to + # inherit — no indicator (None), regardless of its own triage columns. + db = Database(str(tmp_path / "v.db")) + now = int(time.time()) + _seed(db, "2026_0618_203643_0001R.MP4", triaged_at=now, gps_points=9) # rear + _seed(db, "2026_0618_203644_0001T.MP4") # tele + _seed(db, "2026_0618_203645_0001I.MP4") # interior + items = q.list_page(db, per_page=100)["items"] + assert _state_for(items, "2026_0618_203643_0001R.MP4") is None + assert _state_for(items, "2026_0618_203644_0001T.MP4") is None + assert _state_for(items, "2026_0618_203645_0001I.MP4") is None + + +def test_gps_state_none_for_orphan_parking_prefixed_rear(tmp_path): + # Parking/event prefixes (PR, ER) resolve via the registry's last-letter + # rule, so a prefixed rear is still treated as a non-GPS lens. + db = Database(str(tmp_path / "v.db")) + _seed(db, "2026_0618_203643_0001PR.MP4") + items = q.list_page(db, per_page=100)["items"] + assert _state_for(items, "2026_0618_203643_0001PR.MP4") is None + + +def test_gps_state_implied_from_front_sibling(tmp_path): + # Sibling lenses inherit the front's GPS state — matched by timestamp + # prefix (sequences differ per lens), mirroring the archive/triage-gate + # sibling rule. Rear rows are never triaged themselves. + db = Database(str(tmp_path / "v.db")) + now = int(time.time()) + _seed(db, "2026_0618_203643_0001F.MP4", triaged_at=now, gps_points=42) + _seed(db, "2026_0618_203643_0002R.MP4") # → ok + _seed(db, "2026_0618_203744_0003F.MP4", triaged_at=now, gps_points=0) + _seed(db, "2026_0618_203744_0004R.MP4") # → none + _seed(db, "2026_0618_203845_0005F.MP4") + _seed(db, "2026_0618_203845_0006R.MP4") # → pending + _seed(db, "2026_0618_203946_0007F.MP4", + triage_attempts=TRIAGE_MAX_ATTEMPTS) + _seed(db, "2026_0618_203946_0008R.MP4") # → none + + items = q.list_page(db, per_page=100)["items"] + assert _state_for(items, "2026_0618_203643_0002R.MP4") == "ok" + assert _state_for(items, "2026_0618_203744_0004R.MP4") == "none" + assert _state_for(items, "2026_0618_203845_0006R.MP4") == "pending" + assert _state_for(items, "2026_0618_203946_0008R.MP4") == "none" + # The fronts themselves are unaffected by the sibling join. + assert _state_for(items, "2026_0618_203643_0001F.MP4") == "ok" + assert _state_for(items, "2026_0618_203845_0005F.MP4") == "pending" + + +def test_gps_state_implied_in_day_items(tmp_path): + db = Database(str(tmp_path / "v.db")) + now = int(time.time()) + _seed(db, "2026_0618_203643_0001F.MP4", triaged_at=now, gps_points=7) + _seed(db, "2026_0618_203643_0002R.MP4") + items = q.list_day_items(db, day="2026-06-18") + assert _state_for(items, "2026_0618_203643_0002R.MP4") == "ok" diff --git a/tests/test_queue_lock.py b/tests/test_queue_lock.py new file mode 100644 index 0000000..3829a8f --- /dev/null +++ b/tests/test_queue_lock.py @@ -0,0 +1,127 @@ +"""queue.set_locked: flips the user 'locked' flag on both tables + audit log.""" +from __future__ import annotations + +import time +from pathlib import Path + +import pytest +from fastapi.testclient import TestClient + + +def _env(tmp_path: Path): + from web.db import Database + rec = tmp_path / "rec" + rec.mkdir() + return rec, Database(str(rec / "v.db")) + + +def _clip(db, basename): + with db.write() as c: + c.execute( + "INSERT INTO clip_index (path, basename, group_name, timestamp, " + " camera, sequence, event_type, size_bytes, has_gpx, scanned_at) " + "VALUES (?,?, '2026-06-27', 1, 'F', 1, 'normal', 1, 0, 1)", + (f"/x/{basename}", basename), + ) + c.execute( + "INSERT INTO download_queue (filename, source_dir, state, enqueued_at) " + "VALUES (?, '/DCIM', 'done', 1)", (basename,), + ) + + +def test_set_locked_sets_both_tables(tmp_path): + from web.services.queue import set_locked + rec, db = _env(tmp_path) + _clip(db, "A.MP4") + n = set_locked(db, ["A.MP4"], True) + assert n == 1 + with db.conn() as c: + assert c.execute("SELECT locked FROM clip_index WHERE basename='A.MP4'").fetchone()["locked"] == 1 + assert c.execute("SELECT locked FROM download_queue WHERE filename='A.MP4'").fetchone()["locked"] == 1 + + +def test_set_locked_unlock(tmp_path): + from web.services.queue import set_locked + rec, db = _env(tmp_path) + _clip(db, "A.MP4") + set_locked(db, ["A.MP4"], True) + set_locked(db, ["A.MP4"], False) + with db.conn() as c: + assert c.execute("SELECT locked FROM clip_index WHERE basename='A.MP4'").fetchone()["locked"] == 0 + + +def test_set_locked_logs_audit(tmp_path, caplog): + from web.services.queue import set_locked + rec, db = _env(tmp_path) + _clip(db, "A.MP4") + with caplog.at_level("INFO", logger="viofosync.queue"): + set_locked(db, ["A.MP4"], True) + assert any("read-only" in r.getMessage().lower() and "A.MP4" in r.getMessage() + for r in caplog.records if r.name == "viofosync.queue") + + +def test_set_locked_empty_is_noop(tmp_path): + from web.services.queue import set_locked + rec, db = _env(tmp_path) + assert set_locked(db, [], True) == 0 + + +# --- endpoint smoke test --- + + +def _insert_queue(c, filename, state="done"): + now = int(time.time()) + c.execute( + "INSERT INTO download_queue " + "(filename, source_dir, state, enqueued_at) " + "VALUES (?, ?, ?, ?)", + (filename, "/DCIM", state, now), + ) + + +def _insert_index(c, basename): + c.execute( + "INSERT INTO clip_index (path, basename, group_name, timestamp, " + " camera, sequence, event_type, size_bytes, has_gpx, scanned_at) " + "VALUES (?,?, '2026-06-27', 1, 'F', 1, 'normal', 1, 0, 1)", + (f"/x/{basename}", basename), + ) + + +@pytest.fixture +def authed_client(tmp_config_dir: Path, tmp_recordings_dir: Path, monkeypatch): + from web import app as app_mod + from web import settings as settings_mod + monkeypatch.setenv("VIOFOSYNC_RESTART_DISABLED", "1") + settings_mod.reset_for_tests() + application = app_mod.create_app() + with TestClient(application) as c: + c.post("/setup", data={ + "address": "192.168.1.230", + "password": "twelve-chars-min!", + "confirm": "twelve-chars-min!", + }) + csrf = c.get("/api/auth/csrf").json()["csrf"] + c.headers.update({"x-csrf-token": csrf}) + yield c + + +def test_lock_endpoint(authed_client): + db = authed_client.app.state.db + with db.write() as c: + _insert_queue(c, "B.MP4", "done") + _insert_index(c, "B.MP4") + + r = authed_client.post("/api/queue/lock", json={"filenames": ["B.MP4"]}) + assert r.status_code == 200 + body = r.json() + assert body["ok"] is True + assert body["updated"] == 1 + + with db.conn() as c: + assert c.execute( + "SELECT locked FROM clip_index WHERE basename='B.MP4'" + ).fetchone()["locked"] == 1 + assert c.execute( + "SELECT locked FROM download_queue WHERE filename='B.MP4'" + ).fetchone()["locked"] == 1 diff --git a/tests/test_queue_skip.py b/tests/test_queue_skip.py index 0b6edb5..9b8f4aa 100644 --- a/tests/test_queue_skip.py +++ b/tests/test_queue_skip.py @@ -70,6 +70,21 @@ def test_unskip_empty_is_noop(tmp_path): assert unskip(db, []) == 0 +def test_skip_and_unskip_log_info_audit(tmp_path, caplog): + # viofosync.queue INFO is persisted to app_log by DBLogHandler. + from web.db import Database + from web.services.queue import skip, unskip + db = Database(str(tmp_path / "v.db")) + with db.write() as c: + _insert(c, "p.MP4", "pending") + with caplog.at_level("INFO", logger="viofosync.queue"): + skip(db, ["p.MP4"]) + unskip(db, ["p.MP4"]) + msgs = [r.getMessage() for r in caplog.records if r.name == "viofosync.queue"] + assert any("archive skip" in m and "p.MP4" in m for m in msgs), msgs + assert any("archive unskip" in m and "p.MP4" in m for m in msgs), msgs + + def test_next_pending_ignores_skipped(tmp_path): from web.db import Database from web.services.queue import next_pending diff --git a/tests/test_queue_skip_listed.py b/tests/test_queue_skip_listed.py new file mode 100644 index 0000000..0d5df18 --- /dev/null +++ b/tests/test_queue_skip_listed.py @@ -0,0 +1,39 @@ +from __future__ import annotations + +import time + +from web.db import Database +from web.services import queue as q + + +def _seed(db, filename, state): + with db.write() as c: + c.execute( + "INSERT INTO download_queue (filename, source_dir, state, enqueued_at) " + "VALUES (?, '', ?, ?)", + (filename, state, int(time.time())), + ) + + +def test_skip_listed_names_returns_only_skipped(tmp_path): + db = Database(str(tmp_path / "q.db")) + _seed(db, "2026_0618_120000_0001F.MP4", "skipped") # geofence/user skip + _seed(db, "2026_0618_120100_0002F.MP4", "pending") + _seed(db, "2026_0618_120200_0003F.MP4", "done") + _seed(db, "2026_0618_120300_0004F.MP4", "skipped") + names = [ + "2026_0618_120000_0001F.MP4", # skipped + "2026_0618_120100_0002F.MP4", # pending + "2026_0618_120200_0003F.MP4", # done + "2026_0618_120300_0004F.MP4", # skipped + "2026_0618_999999_9999F.MP4", # not in queue at all + ] + assert q.skip_listed_names(db, names) == { + "2026_0618_120000_0001F.MP4", + "2026_0618_120300_0004F.MP4", + } + + +def test_skip_listed_names_empty_input_does_no_query(tmp_path): + db = Database(str(tmp_path / "q.db")) + assert q.skip_listed_names(db, []) == set() diff --git a/tests/test_queue_triage_gate.py b/tests/test_queue_triage_gate.py new file mode 100644 index 0000000..40aff5b --- /dev/null +++ b/tests/test_queue_triage_gate.py @@ -0,0 +1,108 @@ +"""next_pending(triage_gate=True) holds clips whose front sibling is still +awaiting triage, and releases them once triaged / gave-up.""" +from __future__ import annotations + +import time + +from web.db import Database +from web.services import queue as q +from web.services.triage import TRIAGE_MAX_ATTEMPTS + + +def _seed(db, filename, *, state="pending", triaged_at=None, + triage_attempts=0, priority=0): + now = int(time.time()) + with db.write() as c: + c.execute( + "INSERT INTO download_queue " + "(filename, source_dir, camera, event_type, state, enqueued_at, " + " priority, triaged_at, triage_attempts) " + "VALUES (?,?,?,?,?,?,?,?,?)", + (filename, "/DCIM", filename[-5], "normal", state, now, + priority, triaged_at, triage_attempts), + ) + + +def test_untriaged_front_is_gated(tmp_path): + db = Database(str(tmp_path / "v.db")) + _seed(db, "2026_0618_203643_0001F.MP4") # pending-triage + assert q.next_pending(db, triage_gate=True) is None + # Without the gate it would be handed out normally. + assert q.next_pending(db, triage_gate=False) is not None + + +def test_rear_sibling_is_gated_with_front(tmp_path): + db = Database(str(tmp_path / "v.db")) + _seed(db, "2026_0618_203643_0001F.MP4") # front pending-triage + _seed(db, "2026_0618_203643_0001R.MP4") # rear of the SAME pair + # Rear must not be downloaded ahead of its front's triage. + assert q.next_pending(db, triage_gate=True) is None + + +def test_prefixed_parking_pair_is_gated_then_released(tmp_path): + # Parking/event prefixes (PF/PR) must not confuse the sibling match: the + # prefixed rear is held until the prefixed front is triaged. + db = Database(str(tmp_path / "v.db")) + _seed(db, "2026_0618_203643_0001PF.MP4") # parking front, pending + _seed(db, "2026_0618_203643_0001PR.MP4") # parking rear of pair + assert q.next_pending(db, triage_gate=True) is None + # Triage the front → the whole parking pair is released. + with db.write() as c: + c.execute( + "UPDATE download_queue SET triaged_at=? WHERE filename=?", + (int(time.time()), "2026_0618_203643_0001PF.MP4"), + ) + assert q.next_pending(db, triage_gate=True) is not None + + +def test_differing_sequence_pair_is_gated_then_released(tmp_path): + # Real captures can give each lens its own sequence number (observed on + # parking clips: …_020753PF + …_020755PR) — siblings share the timestamp, + # not the full stem. The gate must pair by timestamp, not by rebuilding + # the front's filename from the rear's stem. + db = Database(str(tmp_path / "v.db")) + _seed(db, "2026_0515_023653_020753PF.MP4") # front pending-triage + _seed(db, "2026_0515_023653_020755PR.MP4") # rear, its own sequence + assert q.next_pending(db, triage_gate=True) is None + with db.write() as c: + c.execute( + "UPDATE download_queue SET triaged_at=? WHERE filename=?", + (int(time.time()), "2026_0515_023653_020753PF.MP4"), + ) + assert q.next_pending(db, triage_gate=True) is not None + + +def test_released_once_triaged(tmp_path): + db = Database(str(tmp_path / "v.db")) + _seed(db, "2026_0618_203643_0001F.MP4", + triaged_at=int(time.time())) # fetched/no-fix + _seed(db, "2026_0618_203643_0001R.MP4") + item = q.next_pending(db, triage_gate=True) + assert item is not None # pair released + + +def test_released_once_given_up(tmp_path): + db = Database(str(tmp_path / "v.db")) + _seed(db, "2026_0618_203643_0001F.MP4", + triage_attempts=TRIAGE_MAX_ATTEMPTS) # gave up + item = q.next_pending(db, triage_gate=True) + assert item is not None + + +def test_orphan_rear_not_gated(tmp_path): + # A rear with no matching front has nothing to wait for. + db = Database(str(tmp_path / "v.db")) + _seed(db, "2026_0618_203643_0009R.MP4") + assert q.next_pending(db, triage_gate=True) is not None + + +def test_ungated_clip_still_returned_when_a_gated_clip_present(tmp_path): + # An untriaged front is gated, but a co-queued triaged clip must still be + # handed out — the gate must be selective, not block the whole queue. + db = Database(str(tmp_path / "v.db")) + _seed(db, "2026_0618_203643_0001F.MP4") # gated + _seed(db, "2026_0618_203644_0002F.MP4", + triaged_at=int(time.time())) # ungated + item = q.next_pending(db, triage_gate=True) + assert item is not None + assert item.filename == "2026_0618_203644_0002F.MP4" diff --git a/tests/test_range_reader.py b/tests/test_range_reader.py new file mode 100644 index 0000000..6fb2353 --- /dev/null +++ b/tests/test_range_reader.py @@ -0,0 +1,131 @@ +# tests/test_range_reader.py +"""RangeReader feeds parse_moov; verified against the doc's §9 vector.""" +from __future__ import annotations + +import http.server +import struct +import threading + +from viofosync_lib import parse_moov +from viofosync_lib._archive import Recording +from viofosync_lib._protocol import ( + RangeReader, + extract_remote_gps_points, +) + +# §9 verified freeGPS payload (block offset 16 onward), little-endian. +# lat bytes 9af7a545 = struct.pack(' bytes: + """moov{ gps-index } followed by one freeGPS block at the indexed offset. + + parse_moov: finds the 'gps ' box, reads (offset,size) u32-BE pairs from + box+16 until the box ends, and for each seeks to `offset` expecting a + `free`/`GPS ` atom of `size` bytes, then decodes payload[12:]. + """ + block_body = _PAYLOAD + block_size = 12 + len(block_body) + free_block = struct.pack(">I4s4s", block_size, b"free", b"GPS ") + block_body + + gps_box_size = 8 + 4 + 4 + 8 + moov_size = 8 + gps_box_size + block_offset = moov_size # absolute file offset of free_block + + gps_box = ( + struct.pack(">I4s", gps_box_size, b"gps ") + + struct.pack(">I", 0x00000101) + + struct.pack(">I", 1) + + struct.pack(">II", block_offset, block_size) + ) + moov = struct.pack(">I4s", moov_size, b"moov") + gps_box + return moov + free_block + + +def test_parse_moov_decodes_section9_vector_via_bytesio(): + data = _synthetic_mp4() + reader = RangeReader(lambda a, b: data[a : b + 1], len(data), head_len=64) + pts = parse_moov(reader) + assert len(pts) == 1 + p = pts[0] + assert p["DT"]["DT"] == "2026-06-18T20:36:43Z" + assert round(p["Loc"]["Lat"]["Float"], 4) == 53.1825 + assert round(p["Loc"]["Lon"]["Float"], 4) == -2.865 + + +def test_range_reader_seek_read_eof(): + data = b"0123456789" + r = RangeReader(lambda a, b: data[a : b + 1], len(data), head_len=4) + r.seek(2) + assert r.read(3) == b"234" + r.seek(8) + assert r.read(8) == b"89" + r.seek(10) + assert r.read(4) == b"" + + +def test_extract_remote_gps_points_over_http(): + data = _synthetic_mp4() + + class _Handler(http.server.BaseHTTPRequestHandler): + def log_message(self, *a): + pass + + def do_HEAD(self): + self.send_response(200) + self.send_header("Content-Length", str(len(data))) + self.send_header("Accept-Ranges", "bytes") + self.end_headers() + + def do_GET(self): + rng = self.headers.get("Range") + if rng: + a, b = rng.replace("bytes=", "").split("-") + a = int(a) + b = int(b) if b else len(data) - 1 + b = min(b, len(data) - 1) + chunk = data[a : b + 1] + self.send_response(206) + self.send_header("Content-Range", f"bytes {a}-{b}/{len(data)}") + self.send_header("Content-Length", str(len(chunk))) + self.end_headers() + self.wfile.write(chunk) + else: + self.send_response(200) + self.send_header("Content-Length", str(len(data))) + self.end_headers() + self.wfile.write(data) + + srv = http.server.HTTPServer(("127.0.0.1", 0), _Handler) + port = srv.server_address[1] + t = threading.Thread(target=srv.serve_forever, daemon=True) + t.start() + try: + rec = Recording( + filename="2026_0618_203643_0001F.MP4", + filepath="/DCIM/Movie/2026_0618_203643_0001F.MP4", + size=len(data), + timecode=None, + datetime=None, + attr=None, + ) + pts = extract_remote_gps_points( + f"http://127.0.0.1:{port}", rec, timeout=5.0, head_len=64 + ) + assert len(pts) == 1 + assert pts[0]["DT"]["DT"] == "2026-06-18T20:36:43Z" + finally: + srv.shutdown() + srv.server_close() + t.join(timeout=5.0) + assert not t.is_alive() diff --git a/tests/test_rebuild_grouping.py b/tests/test_rebuild_grouping.py new file mode 100644 index 0000000..b0f8661 --- /dev/null +++ b/tests/test_rebuild_grouping.py @@ -0,0 +1,119 @@ +"""Maintenance flush: un-skip geofence decisions + clear the route cache, then +re-evaluate the geofence (POST /api/archive/rebuild-grouping).""" +from __future__ import annotations + +import os +import types + +from web.db import Database +from web.routers import archive +from web.services import queue as q +from web.services import route_cache +from web.settings import Place + + +def _seed(db, filename, state, skip_reason=None, released=None): + with db.write() as c: + c.execute( + "INSERT INTO download_queue (filename, source_dir, state, " + " skip_reason, geofence_released_at, enqueued_at) VALUES (?,?,?,?,?,0)", + (filename, "/DCIM/Movie", state, skip_reason, released), + ) + + +def test_unskip_geofence_resets_only_geofence_skips(tmp_path): + db = Database(str(tmp_path / "v.db")) + _seed(db, "a_geofence.MP4", "skipped", "geofence") # -> pending + _seed(db, "b_user.MP4", "skipped", "user") # keep (user skip) + _seed(db, "c_released.MP4", "skipped", "geofence", released=1) # keep (released) + _seed(db, "d_pending.MP4", "pending") # keep + _seed(db, "e_done.MP4", "done") # keep + + n = q.unskip_geofence(db) + + assert n == 1 + with db.conn() as c: + st = { + r["filename"]: (r["state"], r["skip_reason"], r["geofence_released_at"]) + for r in c.execute( + "SELECT filename, state, skip_reason, geofence_released_at " + "FROM download_queue" + ) + } + # Reset to pending, skip_reason cleared, and crucially NOT marked released + # (so the next sweep can re-skip it). + assert st["a_geofence.MP4"] == ("pending", None, None) + assert st["b_user.MP4"] == ("skipped", "user", None) + assert st["c_released.MP4"][0] == "skipped" + assert st["d_pending.MP4"][0] == "pending" + assert st["e_done.MP4"][0] == "done" + + +def test_clear_all_removes_route_cache(tmp_path): + rec = str(tmp_path / "rec") + route_cache.store(rec, "2026-06-18", "sig", {"x": 1}) + assert os.path.exists(route_cache._cache_path(rec, "2026-06-18")) + + route_cache.clear_all(rec) + assert not os.path.exists(route_cache._cache_path(rec, "2026-06-18")) + + route_cache.clear_all(rec) # idempotent when already gone + + +def _req(db, rec, *, gps_triage=True, locations=(), worker=None): + snap = types.SimpleNamespace( + recordings=str(rec), gps_triage=gps_triage, locations=locations + ) + provider = types.SimpleNamespace(get=lambda: snap) + state = types.SimpleNamespace(db=db, settings_provider=provider) + if worker is not None: + state.sync_worker = worker + return types.SimpleNamespace(app=types.SimpleNamespace(state=state)) + + +async def test_rebuild_grouping_endpoint(tmp_path, monkeypatch): + db = Database(str(tmp_path / "v.db")) + rec = tmp_path / "rec" + rec.mkdir() + _seed(db, "a_geofence.MP4", "skipped", "geofence") + route_cache.store(str(rec), "2026-06-18", "sig", {"x": 1}) + + # Stub the re-sweep (it needs real skeletons/zones otherwise). + monkeypatch.setattr( + archive.geofence_service, "sweep_all", lambda *a, **k: 4 + ) + worker = types.SimpleNamespace(_geofence_seen={"2026-06-18": 9}) + req = _req( + db, rec, + locations=(Place("Home", 53.1, -2.0, 30, True),), + worker=worker, + ) + + out = await archive.rebuild_grouping(req) + + assert out == {"ok": True, "unskipped": 1, "reskipped": 4} + # geofence clip is back to pending + with db.conn() as c: + row = c.execute( + "SELECT state FROM download_queue WHERE filename='a_geofence.MP4'" + ).fetchone() + assert row["state"] == "pending" + # route cache cleared and worker's seen-cache reset + assert not os.path.exists(route_cache._cache_path(str(rec), "2026-06-18")) + assert worker._geofence_seen == {} + + +async def test_rebuild_grouping_skips_sweep_without_zones(tmp_path, monkeypatch): + db = Database(str(tmp_path / "v.db")) + rec = tmp_path / "rec" + rec.mkdir() + called = {"n": 0} + monkeypatch.setattr( + archive.geofence_service, "sweep_all", + lambda *a, **k: called.__setitem__("n", called["n"] + 1) or 0, + ) + + out = await archive.rebuild_grouping(_req(db, rec, locations=())) + + assert out == {"ok": True, "unskipped": 0, "reskipped": 0} + assert called["n"] == 0 # no zones -> no sweep diff --git a/tests/test_retention.py b/tests/test_retention.py index a3d1d15..2ec310f 100644 --- a/tests/test_retention.py +++ b/tests/test_retention.py @@ -492,3 +492,79 @@ def test_sweep_logs_progress_every_10_deletions(env, caplog) -> None: assert progress[0].startswith("retention sweep: 10/25 clip(s) deleted") assert progress[1].startswith("retention sweep: 20/25 clip(s) deleted") assert "MB freed so far" in progress[0] + + +# --------------------------------------------------------------------------- +# locked=1 protection (independent of protect_ro) +# --------------------------------------------------------------------------- + +def test_eligible_by_time_locked_clip_never_deleted() -> None: + """A clip with locked=1 must be ineligible regardless of protect_ro.""" + clip = _clip(ts=0, event_type="normal") + clip["locked"] = 1 + ok, reason = _eligible_by_time( + clip, now=86400 * 365, max_days=30, protect_ro=False, + ) + assert ok is False + assert reason == "locked" + + +def test_eligible_by_time_unlocked_clip_still_eligible() -> None: + """A clip with locked=0 (or missing) is not shielded by the lock check.""" + clip = _clip(ts=0, event_type="normal") + clip["locked"] = 0 + ok, reason = _eligible_by_time( + clip, now=86400 * 365, max_days=30, protect_ro=False, + ) + assert ok is True + assert reason == "time" + + +def test_sweep_time_protects_locked_clip_with_protect_ro_false(env) -> None: + """Time sweep must NOT delete a locked=1 clip even when protect_ro=False.""" + rec, db = env + now = 86400 * 365 + clip_id = _make_clip(rec, db, basename="LOCKED.MP4", ts=0) + with db.write() as c: + c.execute("UPDATE clip_index SET locked=1 WHERE id=?", (clip_id,)) + # Control: an un-locked clip that SHOULD be deleted. + _make_clip(rec, db, basename="NORMAL.MP4", ts=1) + + summary = sweep( + db, str(rec), max_days=30, disk_pct=0, + protect_ro=False, _now=now, + ) + assert summary["deleted_time"] == 1 + assert _index_count(db) == 1 + # Locked clip file must still exist. + assert (rec / "1970-01-01" / "LOCKED.MP4").exists() + # Un-locked clip must be gone. + assert not (rec / "1970-01-01" / "NORMAL.MP4").exists() + + +def test_sweep_disk_protects_locked_clip_with_protect_ro_false( + env, monkeypatch +) -> None: + """Disk-pressure sweep must NOT evict a locked=1 clip (protect_ro=False).""" + rec, db = env + locked_id = _make_clip(rec, db, basename="LOCKED.MP4", ts=100) + with db.write() as c: + c.execute("UPDATE clip_index SET locked=1 WHERE id=?", (locked_id,)) + _make_clip(rec, db, basename="NORMAL.MP4", ts=200) + + # Disk stays "full" so sweep keeps trying to evict. + monkeypatch.setattr( + "web.services.retention.shutil.disk_usage", + lambda p: DiskUsage(total=100, used=99, free=1), + ) + summary = sweep( + db, str(rec), max_days=0, disk_pct=80, + protect_ro=False, _now=86400 * 365, + ) + assert summary["deleted_disk"] == 1 + # NORMAL deleted; LOCKED survives. + with db.conn() as c: + remaining = c.execute( + "SELECT basename FROM clip_index" + ).fetchone()["basename"] + assert remaining == "LOCKED.MP4" diff --git a/tests/test_scanner_lock_carryover.py b/tests/test_scanner_lock_carryover.py new file mode 100644 index 0000000..77371ee --- /dev/null +++ b/tests/test_scanner_lock_carryover.py @@ -0,0 +1,96 @@ +"""scanner.scan must carry a queued locked=1 into clip_index. + +Two invariants: +1. Carry-over: a download_queue row with locked=1 propagates into + clip_index.locked=1 after a scan that indexes the matching file. +2. Preserve: if clip_index.locked is already 1 (set directly on the + index, e.g. by the archive UI) a rescan must NOT clear it, even + when download_queue.locked=0 for that file. +""" +from __future__ import annotations + +from pathlib import Path + +from web.db import Database +from web.services import scanner + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +_FILENAME = "2026_0603_082421_0001F.MP4" +_GROUP = "2026-06-03" + + +def _make_clip_on_disk(dest: Path) -> Path: + """Create a real Viofo-named file so the scanner can index it.""" + day = dest / _GROUP + day.mkdir(parents=True, exist_ok=True) + f = day / _FILENAME + f.write_bytes(b"\x00" * 16) + return f + + +def _insert_queue_row(db: Database, *, locked: int = 0) -> None: + """Add a download_queue row for _FILENAME.""" + with db.write() as c: + c.execute( + "INSERT INTO download_queue " + "(filename, source_dir, state, priority, attempts, enqueued_at, locked) " + "VALUES (?, '/Movie/Normal/', 'done', 0, 0, 0, ?)", + (_FILENAME, locked), + ) + + +def _get_clip_locked(db: Database) -> int | None: + with db.conn() as c: + row = c.execute( + "SELECT locked FROM clip_index WHERE basename = ?", (_FILENAME,) + ).fetchone() + return row["locked"] if row else None + + +# --------------------------------------------------------------------------- +# Tests +# --------------------------------------------------------------------------- + +def test_scan_carries_queue_lock_into_clip_index(tmp_path: Path) -> None: + """A queued clip with locked=1 must have clip_index.locked=1 after scan.""" + db = Database(str(tmp_path / "t.db")) + dest = tmp_path / "recordings" + _make_clip_on_disk(dest) + _insert_queue_row(db, locked=1) + + scanner.scan(db, str(dest), "daily") + + assert _get_clip_locked(db) == 1, ( + "clip_index.locked should be 1 when download_queue.locked=1" + ) + + +def test_scan_preserves_index_lock_on_rescan(tmp_path: Path) -> None: + """A lock set directly on clip_index must survive a rescan even when + download_queue.locked=0 for that file.""" + db = Database(str(tmp_path / "t.db")) + dest = tmp_path / "recordings" + _make_clip_on_disk(dest) + _insert_queue_row(db, locked=0) + + # First scan — indexes the clip with locked=0 + scanner.scan(db, str(dest), "daily") + assert _get_clip_locked(db) == 0 + + # Simulate an archive-UI lock set directly on clip_index + with db.write() as c: + c.execute( + "UPDATE clip_index SET locked = 1 WHERE basename = ?", (_FILENAME,) + ) + assert _get_clip_locked(db) == 1 + + # Second scan — must NOT clear the lock + scanner.scan(db, str(dest), "daily") + + assert _get_clip_locked(db) == 1, ( + "clip_index.locked must be preserved across rescans " + "(rescan must never clear a lock set on the index)" + ) diff --git a/tests/test_startup_sync_state.py b/tests/test_startup_sync_state.py new file mode 100644 index 0000000..113bacd --- /dev/null +++ b/tests/test_startup_sync_state.py @@ -0,0 +1,77 @@ +"""A started, not-paused worker must 'report in' so the badge shows the real +state, not the 'worker hasn't reported yet → paused' default. + +Regression for the bug where, on first app start, the status badge showed +'paused' even though the worker was running and not paused — so the toggle's +first click (acting on the true not-paused state) paused for real and it took +two clicks to resume. The fix makes start() broadcast a sync_state event, which +the hub folds into its snapshot and uses to recompute sync_status. +""" +from __future__ import annotations + +import asyncio +import types +from unittest.mock import AsyncMock + +from web.db import Database +from web.services.hub import Hub +from web.services.sync_status import compute_sync_status +from web.services.sync_worker import SyncWorker + + +def _snap(tmp_path): + return types.SimpleNamespace( + address="1.2.3.4", recordings=str(tmp_path), disk_critical_pct=95, + ) + + +async def test_started_worker_reports_running_not_paused(tmp_path): + db = Database(str(tmp_path / "t.db")) + snap = _snap(tmp_path) + provider = types.SimpleNamespace(get=lambda: snap) + hub = Hub(settings_provider=provider) + w = SyncWorker(db, provider, hub) + w.bind_loop(asyncio.get_running_loop()) + # Stub the cycle + background sweeps so start() doesn't reach for a real + # dashcam or run retention/geofence; we only care that start() makes the + # worker report its (running, not-paused) state. + w._cycle = AsyncMock(return_value=False) + w._run_retention_sweep = AsyncMock() + w._run_geofence_pass = AsyncMock() + w.start() + try: + # Let the start()-scheduled sync_state broadcast run. + for _ in range(10): + await asyncio.sleep(0) + ss = hub.last_state.get("sync_state") + assert isinstance(ss, dict), "start() did not report sync_state" + assert ss["running"] is True + assert ss["paused"] is False + # And the computed badge is 'waiting', not the 'paused' default. + state, _ = compute_sync_status(hub, None, snap) + assert state == "waiting" + finally: + await w.stop() + + +async def test_started_paused_worker_reports_paused(tmp_path): + # A worker that restored a real pause still reports paused on start. + db = Database(str(tmp_path / "t.db")) + db.kv_set("sync_paused", "1") + snap = _snap(tmp_path) + provider = types.SimpleNamespace(get=lambda: snap) + hub = Hub(settings_provider=provider) + w = SyncWorker(db, provider, hub) + w.bind_loop(asyncio.get_running_loop()) + w._cycle = AsyncMock(return_value=False) + assert w.paused is True + w.start() + try: + for _ in range(10): + await asyncio.sleep(0) + ss = hub.last_state.get("sync_state") + assert isinstance(ss, dict) and ss["paused"] is True + state, _ = compute_sync_status(hub, None, snap) + assert state == "paused" + finally: + await w.stop() diff --git a/tests/test_sync_recording_status.py b/tests/test_sync_recording_status.py new file mode 100644 index 0000000..a20bee2 --- /dev/null +++ b/tests/test_sync_recording_status.py @@ -0,0 +1,58 @@ +"""run_recording_status_release sets remote_complete=1 on every lens of the +newest capture group only when the camera reports record=0; record=1 or an +unknown/None state releases nothing, so unknown-command cameras hold the newest +segment by default.""" +from __future__ import annotations + +import time + +from web.db import Database +from web.services import sync_worker as sw + + +def _seed(db, filename): + now = int(time.time()) + with db.write() as c: + c.execute( + "INSERT INTO download_queue " + "(filename, source_dir, camera, event_type, state, enqueued_at) " + "VALUES (?,?,?,?,?,?)", + (filename, "/DCIM", filename[-5], "normal", "pending", now), + ) + + +def _complete(db): + with db.conn() as c: + return { + r["filename"]: r["remote_complete"] + for r in c.execute( + "SELECT filename, remote_complete FROM download_queue" + ) + } + + +def test_stopped_releases_whole_newest_group(tmp_path): + db = Database(str(tmp_path / "v.db")) + _seed(db, "2026_0628_120000_0001F.MP4") # older capture + _seed(db, "2026_0628_133416_0002F.MP4") # newest front + _seed(db, "2026_0628_133416_0003R.MP4") # newest rear (diff seq) + n = sw.run_recording_status_release(db, recording_state=0) + assert n == 2 + flags = _complete(db) + assert flags["2026_0628_133416_0002F.MP4"] == 1 + assert flags["2026_0628_133416_0003R.MP4"] == 1 + assert flags["2026_0628_120000_0001F.MP4"] is None # older untouched + + +def test_recording_releases_nothing(tmp_path): + db = Database(str(tmp_path / "v.db")) + _seed(db, "2026_0628_133416_0002F.MP4") + assert sw.run_recording_status_release(db, recording_state=1) == 0 + assert _complete(db)["2026_0628_133416_0002F.MP4"] is None + + +def test_unknown_state_releases_nothing(tmp_path): + db = Database(str(tmp_path / "v.db")) + _seed(db, "2026_0628_133416_0002F.MP4") + assert sw.run_recording_status_release(db, recording_state=None) == 0 + assert _complete(db)["2026_0628_133416_0002F.MP4"] is None diff --git a/tests/test_sync_status.py b/tests/test_sync_status.py index 41b8813..006700c 100644 --- a/tests/test_sync_status.py +++ b/tests/test_sync_status.py @@ -1,7 +1,7 @@ """Tests for the pure compute_sync_status() function. Inputs come from hub.last_state and the settings snapshot. Output is -(state, reason) — never raises. Precedence: error > paused > downloading > waiting. +(state, reason) — never raises. Precedence: error > paused > triaging > downloading > waiting. """ from __future__ import annotations @@ -177,3 +177,35 @@ def test_never_raises_on_garbage_state(): current_item=42, dashcam_online="yes") state, _ = compute_sync_status(hub, None, _snap()) assert state in ("downloading", "waiting", "paused", "error") + + +# ---- triaging tier (between paused and downloading) ---- + +def test_triaging_when_active(): + hub = _hub(sync_state={"running": True, "paused": False}, + dashcam_online=True, + triage={"active": True, "triaged": 40, "total": 876}) + state, reason = compute_sync_status(hub, None, _snap()) + assert state == "triaging" + assert reason is None + + +def test_paused_outranks_triaging(): + hub = _hub(sync_state={"running": True, "paused": True}, + triage={"active": True, "triaged": 1, "total": 9}) + state, _ = compute_sync_status(hub, None, _snap()) + assert state == "paused" + + +def test_error_outranks_triaging(): + hub = _hub(sync_state={"running": True, "paused": False}, + triage={"active": True, "triaged": 1, "total": 9}) + state, _ = compute_sync_status(hub, None, _snap(address=None)) + assert state == "error" + + +def test_inactive_triage_falls_through_to_waiting(): + hub = _hub(sync_state={"running": True, "paused": False}, + dashcam_online=True, triage={"active": False}) + state, _ = compute_sync_status(hub, None, _snap()) + assert state == "waiting" diff --git a/tests/test_sync_worker_geofence.py b/tests/test_sync_worker_geofence.py new file mode 100644 index 0000000..b5e1f92 --- /dev/null +++ b/tests/test_sync_worker_geofence.py @@ -0,0 +1,128 @@ +"""SyncWorker geofence pass wiring.""" +from __future__ import annotations + +import datetime as _dt +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from web.db import Database +from web.services.hub import Hub +from web.services.sync_worker import SyncWorker +from web.settings import Place + + +@pytest.fixture +def db(tmp_path: Path) -> Database: + return Database(str(tmp_path / "v.db")) + + +def _utc(h, m, s=0): + return _dt.datetime(2026, 6, 18, h, m, s, tzinfo=_dt.UTC) + + +HOME_LAT, HOME_LON = 53.1, -2.0 + + +def _gpx(points) -> str: + pts = "".join( + f'' + f"00" + for t, lat, lon in points + ) + return ( + '' + f"{pts}" + ) + + +def _provider(rec, **kw): + snap = SimpleNamespace( + recordings=str(rec), + gps_triage=kw.get("gps_triage", True), + locations=kw.get( + "locations", (Place("Home", HOME_LAT, HOME_LON, 30, True),) + ), + ) + return SimpleNamespace(get=lambda: snap) + + +def _seed_home_clip(db, rec): + dwell = [ + (_utc(20, m).strftime("%Y-%m-%dT%H:%M:%SZ"), HOME_LAT, HOME_LON) + for m in (0, 2, 4, 6) + ] + (rec / ".triage").mkdir(parents=True, exist_ok=True) + fn = "2026_0618_200300_0001F.MP4" + (rec / ".triage" / (fn + ".gpx")).write_text(_gpx(dwell)) + with db.write() as c: + c.execute( + "INSERT INTO download_queue " + "(filename, source_dir, state, recorded_at, triaged_at, gps_points, " + " enqueued_at) VALUES (?,?,?,?,?,?,0)", + (fn, "/DCIM/Movie", "pending", int(_utc(20, 3).timestamp()), 1, 5), + ) + return fn + + +def _state(db, fn): + with db.conn() as c: + return c.execute( + "SELECT state FROM download_queue WHERE filename=?", (fn,) + ).fetchone()["state"] + + +async def test_geofence_pass_skips_home_clip(db: Database, tmp_path: Path) -> None: + rec = tmp_path / "rec" + fn = _seed_home_clip(db, rec) + worker = SyncWorker(db, _provider(rec), Hub()) + await worker._run_geofence_pass() + assert _state(db, fn) == "skipped" + + +async def test_geofence_pass_noop_when_not_excluded(db: Database, tmp_path: Path) -> None: + rec = tmp_path / "rec" + fn = _seed_home_clip(db, rec) + worker = SyncWorker( + db, _provider(rec, locations=(Place("Home", HOME_LAT, HOME_LON, 30, False),)), + Hub(), + ) + await worker._run_geofence_pass() + assert _state(db, fn) == "pending" + + +async def test_geofence_pass_noop_when_triage_off(db: Database, tmp_path: Path) -> None: + rec = tmp_path / "rec" + fn = _seed_home_clip(db, rec) + worker = SyncWorker(db, _provider(rec, gps_triage=False), Hub()) + await worker._run_geofence_pass() + assert _state(db, fn) == "pending" + + +async def test_geofence_pass_incremental_caches_day(db: Database, tmp_path: Path) -> None: + rec = tmp_path / "rec" + fn = _seed_home_clip(db, rec) + worker = SyncWorker(db, _provider(rec), Hub()) + + await worker._run_geofence_pass(seen=worker._geofence_seen) + assert _state(db, fn) == "skipped" + assert worker._geofence_seen == {"2026-06-18": 1} + + # Re-arm as pending; unchanged signature -> cached day skipped, not re-skipped. + with db.write() as c: + c.execute("UPDATE download_queue SET state='pending' WHERE filename=?", (fn,)) + await worker._run_geofence_pass(seen=worker._geofence_seen) + assert _state(db, fn) == "pending" + + +async def test_full_pass_clears_cache(db: Database, tmp_path: Path) -> None: + rec = tmp_path / "rec" + fn = _seed_home_clip(db, rec) + worker = SyncWorker(db, _provider(rec), Hub()) + worker._geofence_seen["2026-06-18"] = 1 # stale entry + + await worker._run_geofence_pass() # seen=None -> full sweep, clears cache + assert _state(db, fn) == "skipped" + assert worker._geofence_seen == {} diff --git a/tests/test_timeline_endpoint.py b/tests/test_timeline_endpoint.py index 5d4e502..cde1160 100644 --- a/tests/test_timeline_endpoint.py +++ b/tests/test_timeline_endpoint.py @@ -1,6 +1,7 @@ """Tests for build_route_payload + GET /api/archive/timeline.""" from __future__ import annotations +import datetime as _dt import logging from pathlib import Path @@ -128,15 +129,21 @@ def test_timeline_journey_mode_windows_clips(logged_in_client, monkeypatch): _insert_clip(app, 2, 1_717_312_500, "F", 60.0) _insert_clip(app, 3, 1_717_313_040, "F", 60.0) + cap = archive.MAX_JOURNEY_BUFFER_S fake_route = { "date": "2026-06-02", "point_count": 5, - "journeys": [{"start_ts": 1_717_312_440, "end_ts": 1_717_312_560}], + "journeys": [{ + "start_ts": 1_717_312_440, + "end_ts": 1_717_312_560, + "group_start_ts": 1_717_312_440 - cap, + "group_end_ts": 1_717_312_560 + cap, + }], "stops": [], } monkeypatch.setattr( "web.routers.archive.build_route_payload", - lambda db, recordings, date, geocoder: fake_route, + lambda db, recordings, date, geocoder, places=(): fake_route, ) r = logged_in_client.get("/api/archive/timeline?date=2026-06-02&journey=0") @@ -155,7 +162,7 @@ def test_timeline_journey_out_of_range_404(logged_in_client, monkeypatch): _insert_clip(app, 1, 1_717_312_440, "F", 60.0) monkeypatch.setattr( "web.routers.archive.build_route_payload", - lambda db, recordings, date, geocoder: { + lambda db, recordings, date, geocoder, places=(): { "date": "2026-06-02", "point_count": 0, "journeys": [], "stops": [], }, ) @@ -236,7 +243,7 @@ def test_timeline_keeps_real_duration(logged_in_client): def test_expand_journey_window_no_parking_uses_cap(): cap = archive.MAX_JOURNEY_BUFFER_S - s, e = archive._expand_journey_window(1000.0, 2000.0, []) + s, e = archive.expand_journey_window(1000.0, 2000.0, []) assert s == 1000.0 - cap assert e == 2000.0 + cap @@ -245,7 +252,7 @@ def test_expand_journey_window_parking_bounds_each_side(): # Parking ends 30s before the window and starts 40s after it — both # inside the cap, so they bound the expansion instead of the cap. spans = [(800.0, 970.0), (2040.0, 2200.0)] - s, e = archive._expand_journey_window(1000.0, 2000.0, spans) + s, e = archive.expand_journey_window(1000.0, 2000.0, spans) assert s == 970.0 assert e == 2040.0 @@ -254,7 +261,7 @@ def test_expand_journey_window_distant_parking_falls_back_to_cap(): cap = archive.MAX_JOURNEY_BUFFER_S # Parking is more than the cap away on both sides -> the cap wins. spans = [(0.0, 100.0), (5000.0, 5100.0)] - s, e = archive._expand_journey_window(1000.0, 2000.0, spans) + s, e = archive.expand_journey_window(1000.0, 2000.0, spans) assert s == 1000.0 - cap assert e == 2000.0 + cap @@ -263,16 +270,28 @@ def test_expand_journey_window_overlapping_parking_no_expansion(): # A parking clip straddling an edge means we're already at the boundary; # don't expand into parked footage. spans = [(950.0, 1050.0), (1950.0, 2050.0)] - s, e = archive._expand_journey_window(1000.0, 2000.0, spans) + s, e = archive.expand_journey_window(1000.0, 2000.0, spans) assert s == 1000.0 assert e == 2000.0 -def _fake_journey_route(j0, j1): +def _fake_journey_route(j0, j1, group_start_ts=None, group_end_ts=None): + """Return a minimal fake route payload for journey-mode tests. + + ``group_start_ts`` / ``group_end_ts`` default to the full-cap expansion + when omitted, mirroring what ``_apply_group_windows`` would produce with + no parking clips on the day. + """ + cap = archive.MAX_JOURNEY_BUFFER_S return { "date": "2026-06-02", "point_count": 5, - "journeys": [{"start_ts": j0, "end_ts": j1}], + "journeys": [{ + "start_ts": j0, + "end_ts": j1, + "group_start_ts": j0 - cap if group_start_ts is None else group_start_ts, + "group_end_ts": j1 + cap if group_end_ts is None else group_end_ts, + }], "stops": [], } @@ -289,7 +308,7 @@ def test_timeline_journey_buffer_includes_adjacent_clips( _insert_clip(app, 3, j1 + 30, "F", 40.0) # pull-in [j1+30, j1+70] monkeypatch.setattr( "web.routers.archive.build_route_payload", - lambda db, recordings, date, geocoder: _fake_journey_route(j0, j1), + lambda db, recordings, date, geocoder, places=(): _fake_journey_route(j0, j1), ) body = logged_in_client.get( @@ -303,20 +322,163 @@ def test_timeline_journey_buffer_includes_adjacent_clips( def test_timeline_journey_buffer_stops_at_parking_clip( logged_in_client, monkeypatch, ): - """A parking-mode clip caps how far the start expands — it reaches the - parking boundary (80s back), not the full cap.""" + """A parking-mode clip in the group window caps how far the editor's start + extends — the parking boundary (80 s back) is already baked into + group_start_ts, so the editor picks it up without a second parking query.""" app = logged_in_client.app j0, j1 = 1_000_000, 1_000_300 _insert_clip(app, 1, j0 - 150, "F", 70.0, event_type="parking") # [-150,-80] _insert_clip(app, 2, j0 - 80, "F", 50.0) # pull-away [j0-80, j0-30] _insert_clip(app, 3, j0, "F", 300.0) # the journey itself + # group_start_ts is bounded by the parking clip (ends at j0-80); the route + # payload carries this pre-computed value — the editor must not re-expand. monkeypatch.setattr( "web.routers.archive.build_route_payload", - lambda db, recordings, date, geocoder: _fake_journey_route(j0, j1), + lambda db, recordings, date, geocoder, places=(): _fake_journey_route( + j0, j1, group_start_ts=j0 - 80, + ), ) body = logged_in_client.get( "/api/archive/timeline?date=2026-06-02&journey=0" ).json() - # Expansion stops at the parking boundary, not the 120s cap. + # Editor window starts at the parking boundary baked into group_start_ts, + # then clamped to the earliest clip (also j0-80 here) — not the full cap. assert body["bounds"]["start_ts"] == j0 - 80 + + +# --------------------------------------------------------------------------- +# Queued parking clip bounds the timeline editor window +# --------------------------------------------------------------------------- + +def _moving_gpx(lat0, lon0, n, start, step_lon=0.0008, dt_s=60): + """A GPX track of ``n`` points moving steadily east — no 5-min dwell — + so aggregate_day yields exactly one journey.""" + pts = [] + for i in range(n): + t = start + _dt.timedelta(seconds=i * dt_s) + iso = t.strftime("%Y-%m-%dT%H:%M:%SZ") + pts.append( + f'' + f"1390" + ) + return ( + '' + "" + "".join(pts) + "" + ) + + +def _seed_gpx_clip(app, recordings_dir, clip_id, fn, day, start_dt): + """Seed a clip in clip_index with a real GPX sidecar so build_route_payload + detects a journey for the day.""" + daydir = Path(recordings_dir) / day + daydir.mkdir(parents=True, exist_ok=True) + path = daydir / fn + path.write_bytes(b"x") + (daydir / (fn + ".gpx")).write_text( + _moving_gpx(53.0, -2.0, 6, start_dt) + ) + ts = int(start_dt.replace(tzinfo=_dt.UTC).timestamp()) + with app.state.db.write() as c: + c.execute( + "INSERT INTO clip_index " + "(id, path, basename, group_name, timestamp, camera, " + " sequence, event_type, has_gpx, gps_examined, scanned_at) " + "VALUES (?,?,?,?,?,?,?,?,1,1,?)", + (clip_id, str(path), fn, day, ts, "F", clip_id, "normal", ts), + ) + + +def test_timeline_queued_parking_bounds_editor_window( + logged_in_client, tmp_recordings_dir, monkeypatch, +): + """A parking clip queued but not yet downloaded (download_queue only, no + clip_index entry, no local file) must bound the editor's journey window. + + Under the OLD code, get_timeline re-ran expand_journey_window using a + clip_index-only parking query — the queued clip was invisible, so the + window expanded the full 120 s cap backward past it. Under the new code, + get_timeline reads the journey's group_start_ts (computed by + _apply_group_windows from the unioned day_parking_spans source) and uses + it directly — one definition of the journey's edges, shared with the grid. + """ + app = logged_in_client.app + day = "2026-06-02" + start_dt = _dt.datetime(2026, 6, 2, 8, 0, 0) + fn = "2026_0602_080000_0001F.MP4" + + # Seed a driving clip with a real GPX sidecar so build_route_payload + # detects exactly one journey. + _seed_gpx_clip(app, tmp_recordings_dir, 1, fn, day, start_dt) + + # The journey's raw GPS start (before _apply_group_windows pads it). + route0 = archive.build_route_payload( + app.state.db, str(tmp_recordings_dir), day, None + ) + assert route0["journeys"], "expected one journey from the moving track" + raw_start_ts = route0["journeys"][0]["start_ts"] + + # Seed a parking clip in download_queue only (no clip_index row, no file). + # Place it 60 s before the journey's GPS start — inside the 120 s cap — + # so it bounds the group window instead of the cap taking effect. + prk_recorded_at = int(raw_start_ts) - 60 + with app.state.db.write() as c: + c.execute( + "INSERT INTO download_queue " + "(filename, source_dir, state, event_type, recorded_at, enqueued_at) " + "VALUES (?,?,?,?,?,0)", + ("2026_0602_075900_0009PF.MP4", "/DCIM/Movie", + "pending", "parking", prk_recorded_at), + ) + + # Insert a normal clip that sits INSIDE the old (un-bounded) cap but + # BEFORE the queued parking boundary — i.e. in the gap between + # (raw_start_ts - 120) and prk_recorded_at. Under the old code this clip + # falls inside the re-padded window (start = raw_start_ts - 120) so + # clip-coverage clamping pulls the editor start back to here. Under the + # new code the group window starts at prk_recorded_at so this clip is + # excluded (ts + dur < prk_recorded_at) and the editor start clamps to + # the earliest included clip (raw_start_ts), which is >= group_start_ts. + clip_before_parking = int(raw_start_ts) - 110 # inside old cap, before queued parking + with app.state.db.write() as c: + c.execute( + "INSERT INTO clip_index " + "(id, path, basename, group_name, timestamp, camera, " + " sequence, event_type, has_gpx, gps_examined, scanned_at, duration_s) " + "VALUES (?,?,?,?,?,?,?,?,0,0,?,?)", + (2, f"/rec/pre_{clip_before_parking}.MP4", "pre.MP4", day, + clip_before_parking, "F", 2, "normal", clip_before_parking, 5.0), + ) + + # Step 3: group_start_ts must be bounded by the queued parking clip + # (its recorded_at, which day_parking_spans treats as a point span). + route = archive.build_route_payload( + app.state.db, str(tmp_recordings_dir), day, None + ) + j = route["journeys"][0] + assert j["group_start_ts"] == prk_recorded_at, ( + f"group_start_ts ({j['group_start_ts']}) should equal the queued " + f"parking clip's recorded_at ({prk_recorded_at}), not the full cap " + f"({raw_start_ts - archive.MAX_JOURNEY_BUFFER_S})" + ) + assert j["group_start_ts"] > raw_start_ts - archive.MAX_JOURNEY_BUFFER_S, ( + "group_start_ts must be bounded tighter than the full cap — " + "the queued parking clip must be honoured" + ) + + # Step 4: get_timeline must use the group window, not re-expand raw ts. + r = logged_in_client.get( + f"/api/archive/timeline?date={day}&journey=0" + ) + assert r.status_code == 200 + payload = r.json() + editor_start = payload["bounds"]["start_ts"] + # The editor's start_ts must not reach past the queued parking boundary. + # Under the old code it would expand to raw_start_ts - 120 (the full cap), + # which is beyond the parking boundary — this assertion catches that. + assert editor_start >= j["group_start_ts"], ( + f"editor start_ts ({editor_start}) went behind group_start_ts " + f"({j['group_start_ts']}), suggesting a raw re-pad bypassed the " + f"queued parking boundary" + ) diff --git a/tests/test_timeline_export.py b/tests/test_timeline_export.py index 370ecba..51fb907 100644 --- a/tests/test_timeline_export.py +++ b/tests/test_timeline_export.py @@ -181,6 +181,62 @@ async def fake_probe_res(path): assert not any("apad" in tok for a in calls for tok in a) # no mux +async def test_run_timeline_probes_null_duration_clips(db, tmp_path, monkeypatch): + """A clip overlapping the selection but with a NULL ``duration_s`` (the + duration sweep hasn't caught up — e.g. just after a bulk import) must be + probed on demand, not silently treated as zero-length. Otherwise a valid + selection fails with 'no footage in selection'. The clip here also starts + *before* the window, so the candidate query must reach back for it.""" + monkeypatch.setattr("web.services.exporter.ffmpeg_available", lambda: True) + # Front clip covers [1000, 1060) but its duration is not yet known. + _insert_clip(db, 1, 1000, "F", None, "/rec/f0.mp4") + snap = MagicMock() + snap.recordings = str(tmp_path) + provider = MagicMock() + provider.get.return_value = snap + worker = ExportWorker(db=db, provider=provider, broadcast=_noop) + + async def fake_probe_duration(path): + return 60.0 + + monkeypatch.setattr( + "web.services.durations.probe_duration", fake_probe_duration, + ) + + calls = [] + + async def fake_run_ffmpeg(job_id, args, total, **kw): + calls.append(list(args)) + Path(args[-1]).write_bytes(b"\0") + return 0, "" + + async def fake_probe_res(path): + return (1920, 1080) + + monkeypatch.setattr(worker, "_run_ffmpeg", fake_run_ffmpeg) + monkeypatch.setattr(worker, "_probe_resolution", fake_probe_res) + finishes = [] + monkeypatch.setattr(worker, "_finish", + lambda jid, ok, err, out: finishes.append((ok, err, out))) + + # Selection starts after the clip start, so the old query (which treats a + # NULL-duration clip as zero-length) would exclude the only footage. + segs = [{"channel": "front", "start_ts": 1030, "end_ts": 1050}] + import json as _json + job = {"id": 10, "type": "timeline", + "clip_ids": _json.dumps({"segments": segs, "encoder": "software"})} + await worker._run_job(job) + + assert finishes and finishes[-1][0] is True, finishes + # The probed duration is persisted back to the index so the next export + # (and the rest of the app) sees it. + with db.conn() as c: + row = c.execute( + "SELECT duration_s FROM clip_index WHERE id=1" + ).fetchone() + assert row["duration_s"] == 60.0 + + async def test_run_timeline_no_footage_fails(db, tmp_path, monkeypatch): monkeypatch.setattr("web.services.exporter.ffmpeg_available", lambda: True) snap = MagicMock() @@ -200,6 +256,33 @@ async def test_run_timeline_no_footage_fails(db, tmp_path, monkeypatch): assert finishes[-1][0] is False +async def test_run_timeline_no_footage_logs_diagnostics( + db, tmp_path, monkeypatch, caplog, +): + """The bare 'no footage' job error is unlogged, which is why a failing + timeline export leaves nothing in the logs. The failure path must emit a + diagnostic naming the window so it can be debugged in production.""" + monkeypatch.setattr("web.services.exporter.ffmpeg_available", lambda: True) + snap = MagicMock() + snap.recordings = str(tmp_path) + provider = MagicMock() + provider.get.return_value = snap + worker = ExportWorker(db=db, provider=provider, broadcast=_noop) + monkeypatch.setattr(worker, "_finish", lambda *a: None) + import json as _json + job = {"id": 11, "type": "timeline", + "clip_ids": _json.dumps( + {"segments": [{"channel": "front", "start_ts": 1, "end_ts": 9}], + "encoder": "software"})} + import logging + with caplog.at_level(logging.WARNING, logger="viofosync.exporter"): + await worker._run_job(job) + assert any( + "no footage in selection" in rec.message and "candidate clip" in rec.message + for rec in caplog.records + ), caplog.records + + class _FakeMqttService: def __init__(self, **k): pass def start(self): pass diff --git a/tests/test_triage_archive_merge.py b/tests/test_triage_archive_merge.py new file mode 100644 index 0000000..b809955 --- /dev/null +++ b/tests/test_triage_archive_merge.py @@ -0,0 +1,508 @@ +# tests/test_triage_archive_merge.py +"""Archive endpoints union queued (triaged) clips with clip_index.""" +from __future__ import annotations + +import datetime as _dt +import time +import types + +from web.db import Database +from web.routers import archive + + +def _req(db, *, gps_triage): + """Minimal stand-in for a FastAPI Request: the archive endpoints only read + request.app.state.db and request.app.state.settings_provider.get().""" + snap = types.SimpleNamespace(gps_triage=gps_triage) + provider = types.SimpleNamespace(get=lambda: snap) + state = types.SimpleNamespace(db=db, settings_provider=provider) + return types.SimpleNamespace(app=types.SimpleNamespace(state=state)) + + +def _gpx(lat, lon, t="2026-06-18T20:36:43Z"): + return ( + '' + f'' + f"00" + "" + ) + + +def _seed_queue(db, filename, *, camera, state="pending", recorded_at=None, + triaged=True, gps_points=1): + """Seed a queue row. Defaults model a triaged clip with a GPS trace + (``triaged=True, gps_points=1``). Pass ``gps_points=0`` for a triaged + clip with no GPS fix (e.g. garage parking) and ``triaged=False, + gps_points=None`` for a clip triage hasn't reached yet.""" + triaged_at = int(time.time()) if triaged else None + with db.write() as c: + c.execute( + "INSERT INTO download_queue " + "(filename, source_dir, camera, event_type, state, enqueued_at, " + " recorded_at, triaged_at, gps_points) " + "VALUES (?,?,?,?,?,?,?,?,?)", + (filename, "/DCIM/Movie", camera, "normal", state, + int(time.time()), recorded_at, triaged_at, gps_points), + ) + + +def test_route_includes_skeleton_for_queued_clip(tmp_path): + db = Database(str(tmp_path / "v.db")) + rec = tmp_path / "rec" + (rec / ".triage").mkdir(parents=True) + fn = "2026_0618_203643_0001F.MP4" + (rec / ".triage" / (fn + ".gpx")).write_text(_gpx(53.0, -2.0)) + ts = int(_dt.datetime(2026, 6, 18, 20, 36, 43).timestamp()) + _seed_queue(db, fn, camera="F", recorded_at=ts) + + payload = archive.build_route_payload(db, str(rec), "2026-06-18", None) + assert payload["point_count"] >= 1 + + +def test_route_excludes_downloaded_clip_skeleton(tmp_path): + """A clip already downloaded (state done) must not also be counted from .triage.""" + db = Database(str(tmp_path / "v.db")) + rec = tmp_path / "rec" + (rec / ".triage").mkdir(parents=True) + fn = "2026_0618_203643_0001F.MP4" + (rec / ".triage" / (fn + ".gpx")).write_text(_gpx(53.0, -2.0)) + _seed_queue(db, fn, camera="F", state="done") + payload = archive.build_route_payload(db, str(rec), "2026-06-18", None) + assert payload["point_count"] == 0 + + +def test_days_includes_remote_only_day(tmp_path): + db = Database(str(tmp_path / "v.db")) + _seed_queue(db, "2026_0618_203643_0001F.MP4", camera="F") # nothing in clip_index + out = archive.remote_day_summaries(db) + assert "2026-06-18" in {d["day"] for d in out} + d = next(d for d in out if d["day"] == "2026-06-18") + assert d["remote_count"] >= 1 + + +def test_day_includes_remote_clip_placeholder(tmp_path): + db = Database(str(tmp_path / "v.db")) + ts = int(_dt.datetime(2026, 6, 18, 20, 36, 43).timestamp()) + _seed_queue(db, "2026_0618_203643_0001F.MP4", camera="F", recorded_at=ts) + # Only the GPS-bearing lens is ever triaged — the rear row realistically + # keeps triaged_at NULL and must inherit its front sibling's triage. + _seed_queue(db, "2026_0618_203643_0002R.MP4", camera="R", recorded_at=ts, + triaged=False, gps_points=None) + clips = archive.remote_day_clips(db, "2026-06-18") + assert len(clips) == 1 # F+R paired + pair = clips[0] + assert pair["remote"] is True + assert pair["front"]["remote"] is True + assert pair["rear"]["remote"] is True + assert "id" not in pair["front"] # not downloaded → no clip id + + +def test_remote_day_clips_collapses_event_kind_to_normal(tmp_path): + """Impact (E-prefix) clips are 'event' in the queue but 'normal' once + downloaded (scanner._event_type_for). remote_day_clips must collapse to + 'normal' so get_day's de-dup key matches and no ghost placeholder remains.""" + db = Database(str(tmp_path / "v.db")) + ts = int(_dt.datetime(2026, 6, 18, 20, 36, 43).timestamp()) + with db.write() as c: + c.execute( + "INSERT INTO download_queue " + "(filename, source_dir, camera, event_type, state, enqueued_at, " + " recorded_at, triaged_at, gps_points) " + "VALUES (?,?,?,?,?,?,?,?,?)", + ("2026_0618_203643_0001EF.MP4", "/DCIM/Movie", "F", "event", + "pending", int(time.time()), ts, int(time.time()), 1), + ) + clips = archive.remote_day_clips(db, "2026-06-18") + assert len(clips) == 1 + assert clips[0]["event_type"] == "normal" + + +def test_remote_day_clips_excludes_skipped_and_done(tmp_path): + """Skipped clips are hidden from the archive grid (they snap onto journeys). + Done and gone clips are already downloaded so also not shown as placeholders.""" + db = Database(str(tmp_path / "v.db")) + ts = int(_dt.datetime(2026, 6, 18, 20, 36, 43).timestamp()) + _seed_queue(db, "2026_0618_203643_0001F.MP4", camera="F", + state="pending", recorded_at=ts) + _seed_queue(db, "2026_0618_203644_0002F.MP4", camera="F", + state="skipped", recorded_at=ts + 60) + _seed_queue(db, "2026_0618_203645_0003F.MP4", camera="F", + state="downloading", recorded_at=ts + 120) + _seed_queue(db, "2026_0618_203646_0004F.MP4", camera="F", + state="done", recorded_at=ts + 180) + _seed_queue(db, "2026_0618_203647_0005F.MP4", camera="F", + state="gone", recorded_at=ts + 240) + + clips = archive.remote_day_clips(db, "2026-06-18") + by_state = {c["front"]["state"]: c for c in clips if c.get("front")} + assert set(by_state) == {"pending", "downloading"} + assert "skipped" not in by_state + assert "done" not in by_state + assert "gone" not in by_state + + +def test_get_day_hides_remote_clips_when_triage_off(tmp_path): + db = Database(str(tmp_path / "v.db")) + ts = int(_dt.datetime(2026, 6, 18, 20, 36, 43).timestamp()) + _seed_queue(db, "2026_0618_203643_0001F.MP4", camera="F", + state="pending", recorded_at=ts) + kw = dict(time_from=None, time_to=None, driving=True, parking=True, ro=True) + + # Triage off → no placeholder pairs in the day view (they would otherwise + # snap onto the nearest journey with no GPS of their own). + off = archive.get_day(_req(db, gps_triage=False), "2026-06-18", **kw) + assert off["clips"] == [] + + # Triage on → the queued clip appears as a remote placeholder pair. + on = archive.get_day(_req(db, gps_triage=True), "2026-06-18", **kw) + assert any(c.get("remote") for c in on["clips"]) + + +def test_list_days_hides_remote_only_day_when_triage_off(tmp_path): + db = Database(str(tmp_path / "v.db")) + _seed_queue(db, "2026_0618_203643_0001F.MP4", camera="F", state="pending") + kw = dict(date_from=None, date_to=None, driving=True, parking=True, ro=True, + sort="desc", page=1, per_page=20) + + off = archive.list_days(_req(db, gps_triage=False), **kw) + assert off["days"] == [] + + on = archive.list_days(_req(db, gps_triage=True), **kw) + assert "2026-06-18" in {d["day"] for d in on["days"]} + + +def test_get_day_hides_skipped_clips(tmp_path): + db = Database(str(tmp_path / "v.db")) + drive_ts = int(_dt.datetime(2026, 6, 18, 12, 0, 0).timestamp()) + home_ts = int(_dt.datetime(2026, 6, 18, 2, 0, 0).timestamp()) + # A pending drive clip (must still show) and a geofence-skipped home clip + # (must be hidden — it would otherwise snap onto a journey). + _seed_queue(db, "2026_0618_120000_0001F.MP4", camera="F", + state="pending", recorded_at=drive_ts) + _seed_queue(db, "2026_0618_020000_0009F.MP4", camera="F", + state="skipped", recorded_at=home_ts) + + out = archive.get_day(_req(db, gps_triage=True), "2026-06-18", + time_from=None, time_to=None, + driving=True, parking=True, ro=True) + times = {c["timestamp"] for c in out["clips"]} + assert drive_ts in times # pending drive clip still shown + assert home_ts not in times # skipped home clip hidden from the grid + + +def test_remote_day_clips_excludes_trackless_clip(tmp_path): + """A triaged clip with no GPS fix (gps_points=0, e.g. garage parking) is not + a journey placeholder — it lives only in the download queue. A GPS-bearing + clip on the same day still shows.""" + db = Database(str(tmp_path / "v.db")) + drive_ts = int(_dt.datetime(2026, 6, 18, 20, 36, 43).timestamp()) + garage_ts = int(_dt.datetime(2026, 6, 18, 2, 15, 0).timestamp()) + _seed_queue(db, "2026_0618_203643_0001F.MP4", camera="F", + recorded_at=drive_ts, gps_points=1) # has a trace + _seed_queue(db, "2026_0618_021500_0009F.MP4", camera="F", + recorded_at=garage_ts, gps_points=0) # garage, no trace + clips = archive.remote_day_clips(db, "2026-06-18") + times = {c["timestamp"] for c in clips} + assert drive_ts in times + assert garage_ts not in times + + +def test_remote_day_clips_excludes_untriaged_clip(tmp_path): + """A clip triage hasn't reached yet (no skeleton on disk) is not shown as a + placeholder — it appears only once triage confirms a GPS trace.""" + db = Database(str(tmp_path / "v.db")) + ts = int(_dt.datetime(2026, 6, 18, 20, 36, 43).timestamp()) + _seed_queue(db, "2026_0618_203643_0001F.MP4", camera="F", + recorded_at=ts, triaged=False, gps_points=None) + assert archive.remote_day_clips(db, "2026-06-18") == [] + + +def test_get_day_excludes_trackless_remote_clip(tmp_path): + """End to end: with triage on, a queued garage clip with no GPS is excluded + from the day grid; a GPS-bearing drive clip is included.""" + db = Database(str(tmp_path / "v.db")) + drive_ts = int(_dt.datetime(2026, 6, 18, 12, 0, 0).timestamp()) + garage_ts = int(_dt.datetime(2026, 6, 18, 2, 0, 0).timestamp()) + _seed_queue(db, "2026_0618_120000_0001F.MP4", camera="F", + state="pending", recorded_at=drive_ts, gps_points=1) + _seed_queue(db, "2026_0618_020000_0009F.MP4", camera="F", + state="pending", recorded_at=garage_ts, gps_points=0) + out = archive.get_day(_req(db, gps_triage=True), "2026-06-18", + time_from=None, time_to=None, + driving=True, parking=True, ro=True) + times = {c["timestamp"] for c in out["clips"]} + assert drive_ts in times + assert garage_ts not in times + + +def test_list_days_skips_remote_only_day_without_gps(tmp_path): + """A remote-only day whose queued clips have no GPS trace must not inject an + (empty) archive day card — that footage stays in the queue until it has a + place on a journey.""" + db = Database(str(tmp_path / "v.db")) + _seed_queue(db, "2026_0618_020000_0009F.MP4", camera="F", state="pending", + recorded_at=int(_dt.datetime(2026, 6, 18, 2, 0).timestamp()), + gps_points=0) + kw = dict(date_from=None, date_to=None, driving=True, parking=True, ro=True, + sort="desc", page=1, per_page=20) + out = archive.list_days(_req(db, gps_triage=True), **kw) + assert "2026-06-18" not in {d["day"] for d in out["days"]} + + +def test_route_includes_downloading_skeleton(tmp_path): + """A clip mid-download keeps its skeleton and still shows as a grid tile, so + its track must remain on the journey map — otherwise the tile is orphaned + (shown with nowhere to sit).""" + db = Database(str(tmp_path / "v.db")) + rec = tmp_path / "rec" + (rec / ".triage").mkdir(parents=True) + fn = "2026_0618_203643_0001F.MP4" + (rec / ".triage" / (fn + ".gpx")).write_text(_gpx(53.0, -2.0)) + ts = int(_dt.datetime(2026, 6, 18, 20, 36, 43).timestamp()) + _seed_queue(db, fn, camera="F", state="downloading", recorded_at=ts) + payload = archive.build_route_payload(db, str(rec), "2026-06-18", None) + assert payload["point_count"] >= 1 + + +def test_remote_day_summaries_excludes_skipped(tmp_path): + db = Database(str(tmp_path / "v.db")) + _seed_queue(db, "2026_0618_120000_0001F.MP4", camera="F", + state="pending", recorded_at=int(_dt.datetime(2026, 6, 18, 12, 0).timestamp())) + _seed_queue(db, "2026_0618_020000_0009F.MP4", camera="F", + state="skipped", recorded_at=int(_dt.datetime(2026, 6, 18, 2, 0).timestamp())) + rows = archive.remote_day_summaries(db, "2026-06-18", "2026-06-18") + assert len(rows) == 1 + assert rows[0]["remote_count"] == 1 # only the pending clip counted + + +# --- Sibling lenses inherit triage from the GPS-bearing clip --- + + +def test_remote_day_clips_rear_hidden_when_front_has_no_gps(tmp_path): + """No trace on the front → the whole capture stays out of the archive + (it has no journey to sit on); the rear must not leak in either.""" + db = Database(str(tmp_path / "v.db")) + ts = int(_dt.datetime(2026, 6, 18, 20, 36, 43).timestamp()) + _seed_queue(db, "2026_0618_203643_0001F.MP4", camera="F", recorded_at=ts, + gps_points=0) # triaged, no GPS fix + _seed_queue(db, "2026_0618_203643_0002R.MP4", camera="R", recorded_at=ts, + triaged=False, gps_points=None) + assert archive.remote_day_clips(db, "2026-06-18") == [] + + +def test_remote_day_clips_orphan_rear_hidden(tmp_path): + """A rear with no GPS-bearing sibling at its timestamp has no trace to + inherit — it stays in the Queue tab (and is not gated from download).""" + db = Database(str(tmp_path / "v.db")) + ts = int(_dt.datetime(2026, 6, 18, 20, 36, 43).timestamp()) + _seed_queue(db, "2026_0618_203643_0009R.MP4", camera="R", recorded_at=ts, + triaged=False, gps_points=None) + assert archive.remote_day_clips(db, "2026-06-18") == [] + + +def test_remote_day_clips_pairs_differing_sequences(tmp_path): + """Real captures can give each lens its own sequence number (observed on + parking clips: …_020753PF + …_020755PR); siblings share the timestamp.""" + db = Database(str(tmp_path / "v.db")) + ts = int(_dt.datetime(2026, 5, 15, 2, 36, 53).timestamp()) + _seed_queue(db, "2026_0515_023653_020753PF.MP4", camera="PF", + recorded_at=ts) + _seed_queue(db, "2026_0515_023653_020755PR.MP4", camera="PR", + recorded_at=ts, triaged=False, gps_points=None) + clips = archive.remote_day_clips(db, "2026-05-15") + assert len(clips) == 1 + assert clips[0]["front"] is not None + assert clips[0]["rear"] is not None + + +def test_remote_day_clips_rear_survives_front_download(tmp_path): + """Front already downloaded (state done; its real sidecar now carries the + trace) with the rear still pending: the rear placeholder must remain, or + it can never be downloaded from the archive screen.""" + db = Database(str(tmp_path / "v.db")) + ts = int(_dt.datetime(2026, 6, 18, 20, 36, 43).timestamp()) + _seed_queue(db, "2026_0618_203643_0001F.MP4", camera="F", recorded_at=ts, + state="done") + _seed_queue(db, "2026_0618_203643_0002R.MP4", camera="R", recorded_at=ts, + triaged=False, gps_points=None) + clips = archive.remote_day_clips(db, "2026-06-18") + assert len(clips) == 1 + assert clips[0]["front"] is None # done → not a placeholder + assert clips[0]["rear"]["state"] == "pending" + + +def test_remote_day_clips_rear_hidden_when_front_gone(tmp_path): + """A front that rotated off the card untriaged leaves no trace source at + all (its skeleton is swept) — the rear must not surface as a placeholder.""" + db = Database(str(tmp_path / "v.db")) + ts = int(_dt.datetime(2026, 6, 18, 20, 36, 43).timestamp()) + _seed_queue(db, "2026_0618_203643_0001F.MP4", camera="F", recorded_at=ts, + state="gone") + _seed_queue(db, "2026_0618_203643_0002R.MP4", camera="R", recorded_at=ts, + triaged=False, gps_points=None) + assert archive.remote_day_clips(db, "2026-06-18") == [] + + +def test_get_day_merges_pending_rear_into_downloaded_pair(tmp_path): + """When the front of a capture is downloaded but the rear is still queued, + get_day must merge the rear placeholder into the downloaded pair's empty + slot instead of dropping it with the ghost-placeholder de-dup.""" + db = Database(str(tmp_path / "v.db")) + day = "2026-06-18" + ts = int(_dt.datetime(2026, 6, 18, 20, 36, 43).timestamp()) + _seed_clip(db, "2026_0618_203643_0001F.MP4", camera="F", day=day, ts=ts) + _seed_queue(db, "2026_0618_203643_0001F.MP4", camera="F", recorded_at=ts, + state="done") + _seed_queue(db, "2026_0618_203643_0002R.MP4", camera="R", recorded_at=ts, + triaged=False, gps_points=None) + + kw = dict(time_from=None, time_to=None, driving=True, parking=True, ro=True) + out = archive.get_day(_req(db, gps_triage=True), day, **kw) + assert len(out["clips"]) == 1 + pair = out["clips"][0] + assert pair["front"]["id"] # downloaded slot intact + assert pair["rear"]["remote"] is True # queued rear merged in + assert pair["rear"]["state"] == "pending" + + +def test_remote_day_clips_rear_qualifies_via_downloaded_front_sidecar(tmp_path): + """A front downloaded before triage existed (queue row done, triaged_at + NULL) still proves the capture's GPS via its clip_index sidecar + (has_gpx=1) — the queued rear must surface in the archive.""" + db = Database(str(tmp_path / "v.db")) + day = "2026-06-18" + ts = int(_dt.datetime(2026, 6, 18, 20, 36, 43).timestamp()) + _seed_clip(db, "2026_0618_203643_0001F.MP4", camera="F", day=day, ts=ts, + has_gpx=1) + _seed_queue(db, "2026_0618_203643_0001F.MP4", camera="F", recorded_at=ts, + state="done", triaged=False, gps_points=None) # legacy row + _seed_queue(db, "2026_0618_203643_0002R.MP4", camera="R", recorded_at=ts, + triaged=False, gps_points=None) + clips = archive.remote_day_clips(db, day) + assert len(clips) == 1 + assert clips[0]["rear"]["state"] == "pending" + + +def test_remote_day_clips_rear_qualifies_without_any_front_queue_row(tmp_path): + """Ancient captures may have no queue row for the front at all — the + downloaded sidecar alone must qualify the rear.""" + db = Database(str(tmp_path / "v.db")) + day = "2026-06-18" + ts = int(_dt.datetime(2026, 6, 18, 20, 36, 43).timestamp()) + _seed_clip(db, "2026_0618_203643_0001F.MP4", camera="F", day=day, ts=ts, + has_gpx=1) + _seed_queue(db, "2026_0618_203643_0002R.MP4", camera="R", recorded_at=ts, + triaged=False, gps_points=None) + clips = archive.remote_day_clips(db, day) + assert len(clips) == 1 + assert clips[0]["rear"] is not None + + +def test_remote_day_clips_rear_hidden_when_downloaded_front_lacks_gps(tmp_path): + """A downloaded front with no sidecar (has_gpx=0) proves nothing — the + rear stays out of the journey-organised archive.""" + db = Database(str(tmp_path / "v.db")) + day = "2026-06-18" + ts = int(_dt.datetime(2026, 6, 18, 20, 36, 43).timestamp()) + _seed_clip(db, "2026_0618_203643_0001F.MP4", camera="F", day=day, ts=ts, + has_gpx=0) + _seed_queue(db, "2026_0618_203643_0002R.MP4", camera="R", recorded_at=ts, + triaged=False, gps_points=None) + assert archive.remote_day_clips(db, day) == [] + + +# --- RO-7: locked flag in day payloads --- + + +def _seed_clip(db, filename, *, camera, day, ts, locked=0, has_gpx=0): + """Insert a clip_index row (no real file needed for payload tests).""" + with db.write() as c: + c.execute( + "INSERT INTO clip_index " + "(path, basename, group_name, timestamp, camera, sequence, " + " event_type, size_bytes, has_gpx, gps_examined, scanned_at, locked) " + "VALUES (?,?,?,?,?,?,?,?,?,1,?,?)", + (f"/rec/{day}/{filename}", filename, day, ts, + camera, 1, "normal", 1, has_gpx, ts, locked), + ) + + +def test_get_day_locked_clip_sets_pair_locked(tmp_path): + """A downloaded clip with locked=1 makes the returned pair carry locked=1.""" + db = Database(str(tmp_path / "v.db")) + day = "2026-06-18" + ts = int(_dt.datetime(2026, 6, 18, 12, 0, 0).timestamp()) + _seed_clip(db, "2026_0618_120000_0001F.MP4", camera="F", + day=day, ts=ts, locked=1) + + kw = dict(time_from=None, time_to=None, driving=True, parking=True, ro=True) + out = archive.get_day(_req(db, gps_triage=False), day, **kw) + assert len(out["clips"]) == 1 + assert out["clips"][0]["locked"] == 1 + + +def test_get_day_unlocked_clip_sets_pair_locked_zero(tmp_path): + """A downloaded clip with locked=0 produces pair locked=0.""" + db = Database(str(tmp_path / "v.db")) + day = "2026-06-18" + ts = int(_dt.datetime(2026, 6, 18, 12, 0, 0).timestamp()) + _seed_clip(db, "2026_0618_120000_0001F.MP4", camera="F", + day=day, ts=ts, locked=0) + + kw = dict(time_from=None, time_to=None, driving=True, parking=True, ro=True) + out = archive.get_day(_req(db, gps_triage=False), day, **kw) + assert len(out["clips"]) == 1 + assert not out["clips"][0]["locked"] + + +def test_get_day_locked_rear_makes_pair_locked(tmp_path): + """If either camera in the pair is locked the pair-level locked is truthy.""" + db = Database(str(tmp_path / "v.db")) + day = "2026-06-18" + ts = int(_dt.datetime(2026, 6, 18, 12, 0, 0).timestamp()) + _seed_clip(db, "2026_0618_120000_0001F.MP4", camera="F", + day=day, ts=ts, locked=0) + _seed_clip(db, "2026_0618_120000_0002R.MP4", camera="R", + day=day, ts=ts, locked=1) + + kw = dict(time_from=None, time_to=None, driving=True, parking=True, ro=True) + out = archive.get_day(_req(db, gps_triage=False), day, **kw) + assert len(out["clips"]) == 1 + assert out["clips"][0]["locked"] == 1 + + +def test_remote_day_clips_carries_locked(tmp_path): + """A queued clip with locked=1 produces a remote pair with locked=1.""" + db = Database(str(tmp_path / "v.db")) + ts = int(_dt.datetime(2026, 6, 18, 12, 0, 0).timestamp()) + with db.write() as c: + c.execute( + "INSERT INTO download_queue " + "(filename, source_dir, camera, event_type, state, enqueued_at, " + " recorded_at, triaged_at, gps_points, locked) " + "VALUES (?,?,?,?,?,?,?,?,?,?)", + ("2026_0618_120000_0001F.MP4", "/DCIM/Movie", "F", "normal", + "pending", int(time.time()), ts, int(time.time()), 1, 1), + ) + clips = archive.remote_day_clips(db, "2026-06-18") + assert len(clips) == 1 + assert clips[0]["locked"] == 1 + + +def test_remote_day_clips_unlocked_pair_locked_zero(tmp_path): + """A queued clip with locked=0 produces a remote pair with locked=0.""" + db = Database(str(tmp_path / "v.db")) + ts = int(_dt.datetime(2026, 6, 18, 12, 0, 0).timestamp()) + with db.write() as c: + c.execute( + "INSERT INTO download_queue " + "(filename, source_dir, camera, event_type, state, enqueued_at, " + " recorded_at, triaged_at, gps_points, locked) " + "VALUES (?,?,?,?,?,?,?,?,?,?)", + ("2026_0618_120000_0001F.MP4", "/DCIM/Movie", "F", "normal", + "pending", int(time.time()), ts, int(time.time()), 1, 0), + ) + clips = archive.remote_day_clips(db, "2026-06-18") + assert len(clips) == 1 + assert not clips[0]["locked"] diff --git a/tests/test_triage_incomplete.py b/tests/test_triage_incomplete.py new file mode 100644 index 0000000..32c94b1 --- /dev/null +++ b/tests/test_triage_incomplete.py @@ -0,0 +1,71 @@ +"""Triage never mis-marks the unconfirmed newest capture: select_targets +excludes it, and an IncompleteRecording during triage_one leaves it untriaged +without burning an attempt, while an older truncated clip is marked no-GPS.""" +from __future__ import annotations + +import time + +from viofosync_lib import IncompleteRecording +from web.db import Database +from web.services import triage + + +def _seed(db, filename, *, recorded_at): + with db.write() as c: + c.execute( + "INSERT INTO download_queue " + "(filename, source_dir, camera, event_type, state, enqueued_at, " + " recorded_at) VALUES (?,?,?,?,?,?,?)", + (filename, "/DCIM", filename[-5], "normal", "pending", + int(time.time()), recorded_at), + ) + + +def _row(db, filename): + with db.conn() as c: + return dict(c.execute( + "SELECT * FROM download_queue WHERE filename=?", (filename,) + ).fetchone()) + + +def test_select_targets_excludes_unconfirmed_newest(tmp_path): + db = Database(str(tmp_path / "v.db")) + old = int(time.time()) - 10_000 # past the settle window + _seed(db, "2026_0628_120000_0001F.MP4", recorded_at=old) # older + _seed(db, "2026_0628_133416_0002F.MP4", recorded_at=old) # newest, unconfirmed + names = {t["filename"] for t in triage.select_targets(db)} + assert "2026_0628_120000_0001F.MP4" in names + assert "2026_0628_133416_0002F.MP4" not in names + + +def test_triage_one_incomplete_newest_defers_without_attempt(tmp_path, monkeypatch): + db = Database(str(tmp_path / "v.db")) + old = int(time.time()) - 10_000 + _seed(db, "2026_0628_133416_0002F.MP4", recorded_at=old) # newest + monkeypatch.setattr( + triage.vfs, "extract_remote_gps_points", + lambda *a, **k: (_ for _ in ()).throw(IncompleteRecording("x")), + ) + row = _row(db, "2026_0628_133416_0002F.MP4") + n = triage.triage_one(db, "http://cam", row, str(tmp_path), timeout=5) + assert n == -1 + after = _row(db, "2026_0628_133416_0002F.MP4") + assert after["triaged_at"] is None + assert after["triage_attempts"] == 0 + assert after["triage_last_attempt_at"] is not None + + +def test_triage_one_incomplete_old_marks_no_gps(tmp_path, monkeypatch): + db = Database(str(tmp_path / "v.db")) + old = int(time.time()) - 10_000 + _seed(db, "2026_0628_120000_0001F.MP4", recorded_at=old) # older + _seed(db, "2026_0628_140000_0009F.MP4", recorded_at=old) # newest (so 1st is old) + monkeypatch.setattr( + triage.vfs, "extract_remote_gps_points", + lambda *a, **k: (_ for _ in ()).throw(IncompleteRecording("x")), + ) + row = _row(db, "2026_0628_120000_0001F.MP4") + triage.triage_one(db, "http://cam", row, str(tmp_path), timeout=5) + after = _row(db, "2026_0628_120000_0001F.MP4") + assert after["triaged_at"] is not None + assert after["gps_points"] == 0 diff --git a/tests/test_triage_lifecycle.py b/tests/test_triage_lifecycle.py new file mode 100644 index 0000000..a799569 --- /dev/null +++ b/tests/test_triage_lifecycle.py @@ -0,0 +1,101 @@ +# tests/test_triage_lifecycle.py +"""Wiring of triage into the app lifespan. + +Two behaviours pinned here: +- startup sweep: orphaned skeleton sidecars are removed when the app starts. +- purge on disable: turning GPS_TRIAGE off via settings triggers purge_all, + removing the .triage directory and clearing triage columns. +""" +from __future__ import annotations + +import bcrypt +from fastapi.testclient import TestClient + +from web import settings as settings_mod + + +class _FakeMqtt: + def __init__(self, **kwargs): + pass + + def start(self): + pass + + async def stop(self): + pass + + async def on_settings_changed(self, keys, snap): + pass + + +def _setup_provider_with_password(): + """Configure a fresh provider with a hashed password so the app + doesn't stay stuck in setup mode.""" + settings_mod.reset_for_tests() + p = settings_mod.get_provider() + data = p._store.load() + data["WEB_PASSWORD_HASH"] = bcrypt.hashpw(b"pw" * 8, bcrypt.gensalt()).decode() + p._store.write(data) + settings_mod.reset_for_tests() + + +def test_startup_sweep_removes_orphaned_skeleton( + tmp_config_dir, tmp_recordings_dir, monkeypatch +): + """Skeleton sidecars with no live queue row must be swept at boot.""" + from web.app import create_app + from web.services.sync_worker import SyncWorker + + _setup_provider_with_password() + monkeypatch.setattr(SyncWorker, "start", lambda self: None) + monkeypatch.setattr("web.app.MqttService", _FakeMqtt) + + # Write an orphaned skeleton (no matching queue row exists). + triage_dir = tmp_recordings_dir / ".triage" + triage_dir.mkdir(parents=True) + orphan = triage_dir / "2026_0618_120000_0001F.MP4.gpx" + orphan.write_text("") + + app = create_app() + with TestClient(app): + assert not orphan.exists(), ( + "orphaned skeleton sidecar was not swept on startup" + ) + + settings_mod.reset_for_tests() + + +def test_purge_on_disable_removes_triage_dir( + tmp_config_dir, tmp_recordings_dir, monkeypatch +): + """Setting GPS_TRIAGE=False must trigger purge_all, removing .triage.""" + from web.app import create_app + from web.services.sync_worker import SyncWorker + + _setup_provider_with_password() + monkeypatch.setattr(SyncWorker, "start", lambda self: None) + monkeypatch.setattr("web.app.MqttService", _FakeMqtt) + + # Enable GPS_TRIAGE in the stored config so we can turn it off inside + # the lifespan and trigger the subscriber. + settings_mod.reset_for_tests() + p = settings_mod.get_provider() + p.update({"GPS_TRIAGE": True}, actor="test") + + app = create_app() + with TestClient(app): + # Create a skeleton sidecar so there's something to purge. + triage_dir = tmp_recordings_dir / ".triage" + triage_dir.mkdir(parents=True, exist_ok=True) + skel = triage_dir / "2026_0618_120000_0001F.MP4.gpx" + skel.write_text("") + + # Disable triage — the subscriber must call purge_all synchronously. + provider = settings_mod.get_provider() + provider.update({"GPS_TRIAGE": False}, actor="test") + + assert not triage_dir.exists(), ( + ".triage directory was not purged when GPS_TRIAGE was disabled" + ) + + settings_mod.reset_for_tests() diff --git a/tests/test_triage_migration.py b/tests/test_triage_migration.py new file mode 100644 index 0000000..50d9adc --- /dev/null +++ b/tests/test_triage_migration.py @@ -0,0 +1,50 @@ +"""download_queue gains triage columns, idempotently.""" +from __future__ import annotations + +import time + +from web.db import Database + + +def _columns(db, table): + with db.conn() as c: + return {r["name"] for r in c.execute(f"PRAGMA table_info({table})")} + + +def test_triage_columns_present_on_fresh_db(tmp_path): + db = Database(str(tmp_path / "v.db")) + cols = _columns(db, "download_queue") + assert "triaged_at" in cols + assert "gps_points" in cols + + +def test_migration_idempotent_on_existing_db(tmp_path): + p = str(tmp_path / "v.db") + Database(p) # creates + migrates once + db2 = Database(p) # re-open: ALTERs must be no-ops, not raise + cols = _columns(db2, "download_queue") + assert {"triaged_at", "gps_points"} <= cols + + +def test_triage_retry_columns_present(tmp_path): + db = Database(str(tmp_path / "v.db")) + cols = _columns(db, "download_queue") + assert "triage_attempts" in cols + assert "triage_last_attempt_at" in cols + + +def test_triage_attempts_defaults_to_zero(tmp_path): + db = Database(str(tmp_path / "v.db")) + now = int(time.time()) + with db.write() as c: + c.execute( + "INSERT INTO download_queue " + "(filename, source_dir, camera, event_type, state, enqueued_at) " + "VALUES (?,?,?,?,?,?)", + ("2026_0618_203643_0001F.MP4", "/DCIM", "F", "normal", "pending", now), + ) + with db.conn() as c: + r = c.execute( + "SELECT triage_attempts FROM download_queue" + ).fetchone() + assert r["triage_attempts"] == 0 diff --git a/tests/test_triage_service.py b/tests/test_triage_service.py new file mode 100644 index 0000000..5151502 --- /dev/null +++ b/tests/test_triage_service.py @@ -0,0 +1,426 @@ +# tests/test_triage_service.py +"""Triage service: select F clips, write skeleton sidecars, track state.""" +from __future__ import annotations + +import time +import urllib.error + +from web.db import Database +from web.services import triage + + +def _seed(db, filename, *, camera, source="/DCIM/Movie", state="pending", + remote_complete=1): + now = int(time.time()) + with db.write() as c: + c.execute( + "INSERT INTO download_queue " + "(filename, source_dir, camera, event_type, state, enqueued_at, " + " remote_complete) " + "VALUES (?,?,?,?,?,?,?)", + (filename, source, camera, "normal", state, now, remote_complete), + ) + + +def test_select_targets_front_only_untriaged(tmp_path): + db = Database(str(tmp_path / "v.db")) + _seed(db, "2026_0618_203643_0001F.MP4", camera="F") + _seed(db, "2026_0618_203644_0002R.MP4", camera="R") + targets = triage.select_targets(db, limit=10) + names = [t["filename"] for t in targets] + assert names == ["2026_0618_203643_0001F.MP4"] # R excluded + + +def test_triage_one_writes_sidecar_and_sets_columns(tmp_path, monkeypatch): + db = Database(str(tmp_path / "v.db")) + rec_dir = tmp_path / "recordings" + rec_dir.mkdir() + fn = "2026_0618_203643_0001F.MP4" + _seed(db, fn, camera="F") + + point = { + "DT": {"DT": "2026-06-18T20:36:43Z"}, + "Loc": { + "Lat": {"Float": 53.1825}, "Lon": {"Float": -2.865}, + "Speed": 0.0, "Bearing": 0.0, + }, + } + monkeypatch.setattr( + triage.vfs, "extract_remote_gps_points", + lambda *a, **k: [point], + ) + + row = triage.select_targets(db, limit=1)[0] + triage.triage_one(db, "http://cam", row, str(rec_dir), timeout=5.0) + + sidecar = rec_dir / ".triage" / (fn + ".gpx") + assert sidecar.exists() + assert "53.1825" in sidecar.read_text() + with db.conn() as c: + r = c.execute( + "SELECT triaged_at, gps_points FROM download_queue WHERE filename=?", + (fn,), + ).fetchone() + assert r["triaged_at"] is not None + assert r["gps_points"] == 1 + + +def test_triage_one_no_fix_sets_zero_points_no_sidecar(tmp_path, monkeypatch): + db = Database(str(tmp_path / "v.db")) + rec_dir = tmp_path / "recordings" + rec_dir.mkdir() + fn = "2026_0618_203643_0001F.MP4" + _seed(db, fn, camera="F") + monkeypatch.setattr( + triage.vfs, "extract_remote_gps_points", lambda *a, **k: [] + ) + row = triage.select_targets(db, limit=1)[0] + triage.triage_one(db, "http://cam", row, str(rec_dir), timeout=5.0) + assert not (rec_dir / ".triage" / (fn + ".gpx")).exists() + with db.conn() as c: + r = c.execute( + "SELECT triaged_at, gps_points FROM download_queue WHERE filename=?", + (fn,), + ).fetchone() + assert r["triaged_at"] is not None + assert r["gps_points"] == 0 + + +def test_sweep_orphans_removes_skeletons_for_done_and_absent(tmp_path): + db = Database(str(tmp_path / "v.db")) + rec_dir = tmp_path / "recordings" + triage_dir = rec_dir / ".triage" + triage_dir.mkdir(parents=True) + keep = "2026_0618_203643_0001F.MP4" + drop_done = "2026_0618_203644_0002F.MP4" + drop_absent = "2026_0618_203645_0003F.MP4" + for fn in (keep, drop_done, drop_absent): + (triage_dir / (fn + ".gpx")).write_text("x") + _seed(db, keep, camera="F", state="pending") + _seed(db, drop_done, camera="F", state="done") + removed = triage.sweep_orphans(db, str(rec_dir)) + assert removed == 2 + assert (triage_dir / (keep + ".gpx")).exists() + assert not (triage_dir / (drop_done + ".gpx")).exists() + assert not (triage_dir / (drop_absent + ".gpx")).exists() + + +def test_purge_all_removes_dir_and_clears_columns(tmp_path): + db = Database(str(tmp_path / "v.db")) + rec_dir = tmp_path / "recordings" + (rec_dir / ".triage").mkdir(parents=True) + (rec_dir / ".triage" / "x.gpx").write_text("x") + _seed(db, "2026_0618_203643_0001F.MP4", camera="F") + with db.write() as c: + c.execute("UPDATE download_queue SET triaged_at=1, gps_points=1") + triage.purge_all(db, str(rec_dir)) + assert not (rec_dir / ".triage").exists() + with db.conn() as c: + r = c.execute( + "SELECT triaged_at, gps_points FROM download_queue" + ).fetchone() + assert r["triaged_at"] is None and r["gps_points"] is None + + +def test_sweep_orphans_keeps_downloading(tmp_path): + db = Database(str(tmp_path / "v.db")) + rec_dir = tmp_path / "recordings" + (rec_dir / ".triage").mkdir(parents=True) + fn = "2026_0618_203643_0001F.MP4" + (rec_dir / ".triage" / (fn + ".gpx")).write_text("x") + _seed(db, fn, camera="F", state="downloading") + removed = triage.sweep_orphans(db, str(rec_dir)) + assert removed == 0 + assert (rec_dir / ".triage" / (fn + ".gpx")).exists() + + +def test_triage_one_read_error_leaves_row_untriaged(tmp_path, monkeypatch): + db = Database(str(tmp_path / "v.db")) + rec_dir = tmp_path / "recordings" + rec_dir.mkdir() + fn = "2026_0618_203643_0001F.MP4" + _seed(db, fn, camera="F") + + def _boom(*a, **k): + raise OSError("network down") + + monkeypatch.setattr(triage.vfs, "extract_remote_gps_points", _boom) + row = triage.select_targets(db, limit=1)[0] + rc = triage.triage_one(db, "http://cam", row, str(rec_dir), timeout=5.0) + assert rc == -1 + with db.conn() as c: + r = c.execute( + "SELECT triaged_at, gps_points FROM download_queue WHERE filename=?", + (fn,), + ).fetchone() + assert r["triaged_at"] is None # untriaged -> retried next cycle + assert r["gps_points"] is None + + +def test_run_pass_triages_and_counts(tmp_path, monkeypatch): + db = Database(str(tmp_path / "v.db")) + rec_dir = tmp_path / "recordings" + rec_dir.mkdir() + _seed(db, "2026_0618_203643_0001F.MP4", camera="F") + _seed(db, "2026_0618_203644_0002F.MP4", camera="F") + _seed(db, "2026_0618_203644_0003R.MP4", camera="R") # ignored (rear) + + point = { + "DT": {"DT": "2026-06-18T20:36:43Z"}, + "Loc": { + "Lat": {"Float": 53.1825}, "Lon": {"Float": -2.865}, + "Speed": 0.0, "Bearing": 0.0, + }, + } + monkeypatch.setattr( + triage.vfs, "extract_remote_gps_points", lambda *a, **k: [point] + ) + summary = triage.run_pass(db, "http://cam", str(rec_dir), timeout=5.0) + assert summary == {"triaged": 2, "with_gps": 2} + with db.conn() as c: + n = c.execute( + "SELECT COUNT(*) AS n FROM download_queue WHERE triaged_at IS NOT NULL" + ).fetchone()["n"] + assert n == 2 # only the two F clips + + +def test_run_pass_triages_entire_backlog_not_capped(tmp_path, monkeypatch): + """A single pass must triage the WHOLE pending front backlog, not a fixed + batch — otherwise the long download drain starves triage and the journey + map never fills in (the reported bug: ~50 triaged of ~900 pending).""" + db = Database(str(tmp_path / "v.db")) + rec_dir = tmp_path / "recordings" + rec_dir.mkdir() + # 45 front clips — deliberately more than the old 40-per-pass cap. + for i in range(45): + _seed(db, f"2026_0619_2030{i:02d}_{1000 + i:04d}F.MP4", camera="F") + point = { + "DT": {"DT": "2026-06-19T20:30:00Z"}, + "Loc": { + "Lat": {"Float": 53.1}, "Lon": {"Float": -2.0}, + "Speed": 0.0, "Bearing": 0.0, + }, + } + monkeypatch.setattr( + triage.vfs, "extract_remote_gps_points", lambda *a, **k: [point] + ) + summary = triage.run_pass(db, "http://cam", str(rec_dir), timeout=5.0) + assert summary["triaged"] == 45 # all of them, not capped at 40 + with db.conn() as c: + untriaged = c.execute( + "SELECT COUNT(*) AS n FROM download_queue " + "WHERE state='pending' AND triaged_at IS NULL " + "AND upper(substr(filename,-5,1))='F'" + ).fetchone()["n"] + assert untriaged == 0 # nothing left for the drain to starve + + +def test_run_pass_reports_incremental_progress(tmp_path, monkeypatch): + """A long full-backlog pass must report progress as it goes (every + PROGRESS_EVERY clips) so the UI can fill the map in, not only at the end.""" + db = Database(str(tmp_path / "v.db")) + rec_dir = tmp_path / "rec" + rec_dir.mkdir() + for i in range(45): + _seed(db, f"2026_0619_2031{i:02d}_{2000 + i:04d}F.MP4", camera="F") + point = { + "DT": {"DT": "2026-06-19T20:31:00Z"}, + "Loc": { + "Lat": {"Float": 53.1}, "Lon": {"Float": -2.0}, + "Speed": 0.0, "Bearing": 0.0, + }, + } + monkeypatch.setattr( + triage.vfs, "extract_remote_gps_points", lambda *a, **k: [point] + ) + calls = [] + triage.run_pass( + db, "http://cam", str(rec_dir), timeout=5.0, + progress_cb=lambda t, total, eta: calls.append((t, total, eta)), + ) + # start (0/45, no ETA) + reports at 20 and 40, all against total=45. + assert [c[0] for c in calls] == [0, 20, 40] + assert all(c[1] == 45 for c in calls) + assert calls[0][2] is None # no ETA at the start + assert all(isinstance(c[2], int) for c in calls[1:]) # ETA computed after + + +def test_sweep_orphans_keeps_skipped(tmp_path): + # A geofence-skipped clip must keep its skeleton: the detector re-reads + # skipped skeletons to keep seeing the full home dwell. Deleting it + # fragments the dwell and strands the dwell's edge clips. + db = Database(str(tmp_path / "v.db")) + rec_dir = tmp_path / "recordings" + (rec_dir / ".triage").mkdir(parents=True) + fn = "2026_0618_203643_0001F.MP4" + (rec_dir / ".triage" / (fn + ".gpx")).write_text("x") + _seed(db, fn, camera="F", state="skipped") + removed = triage.sweep_orphans(db, str(rec_dir)) + assert removed == 0 + assert (rec_dir / ".triage" / (fn + ".gpx")).exists() + + +def test_run_pass_aborts_when_camera_drops(tmp_path, monkeypatch): + """If reads keep failing (camera offline mid-pass), the breaker stops the + pass early instead of timing out on every remaining clip.""" + db = Database(str(tmp_path / "v.db")) + rec_dir = tmp_path / "rec" + rec_dir.mkdir() + for i in range(30): + _seed(db, f"2026_0619_2032{i:02d}_{3000 + i:04d}F.MP4", camera="F") + + attempts = {"n": 0} + + def _boom(*a, **k): + attempts["n"] += 1 + raise OSError("camera offline") + + monkeypatch.setattr(triage.vfs, "extract_remote_gps_points", _boom) + summary = triage.run_pass(db, "http://cam", str(rec_dir), timeout=5.0) + assert summary["triaged"] == 0 + # Aborted well before attempting all 30 (breaker trips at 8 consecutive, + # plus at most a couple already in flight). + assert attempts["n"] < 30 + + +def test_select_targets_defers_in_progress_clips(tmp_path): + db = Database(str(tmp_path / "v.db")) + now = 1_800_000_000 + def seed(fn, *, recorded_at, event_type="normal"): + with db.write() as c: + c.execute( + "INSERT INTO download_queue " + "(filename, source_dir, camera, event_type, state, recorded_at, enqueued_at) " + "VALUES (?,?,?,?,?,?,0)", + (fn, "/DCIM/Movie", "F", event_type, "pending", recorded_at), + ) + # normal clip recorded 90s ago -> still within the 120s settle -> deferred + seed("2026_0618_200000_0001F.MP4", recorded_at=now - 90, event_type="normal") + # normal clip recorded 5 min ago -> settled -> selected + seed("2026_0618_195000_0002F.MP4", recorded_at=now - 300, event_type="normal") + # parking clip recorded 10 min ago -> within 31-min parking settle -> deferred + seed("2026_0618_193000_0003F.MP4", recorded_at=now - 600, event_type="parking") + # parking clip recorded 40 min ago -> settled -> selected + seed("2026_0618_191000_0004F.MP4", recorded_at=now - 2400, event_type="parking") + # clip with NULL recorded_at -> not gated (selected) + seed("2026_0618_190000_0005F.MP4", recorded_at=None, event_type="normal") + + got = {r["filename"] for r in triage.select_targets(db, now=now)} + assert "2026_0618_195000_0002F.MP4" in got # settled normal + assert "2026_0618_191000_0004F.MP4" in got # settled parking + assert "2026_0618_190000_0005F.MP4" in got # null recorded_at, ungated + assert "2026_0618_200000_0001F.MP4" not in got # in-progress normal + assert "2026_0618_193000_0003F.MP4" not in got # in-progress parking + + +def _attempts(db, fn): + with db.conn() as c: + return c.execute( + "SELECT triage_attempts, triaged_at, triage_last_attempt_at " + "FROM download_queue WHERE filename=?", + (fn,), + ).fetchone() + + +def test_offline_failure_does_not_count(tmp_path, monkeypatch): + db = Database(str(tmp_path / "v.db")) + rec_dir = tmp_path / "rec" + rec_dir.mkdir() + fn = "2026_0618_203643_0001F.MP4" + _seed(db, fn, camera="F") + + def boom(*a, **k): + raise urllib.error.URLError("connection refused") + + monkeypatch.setattr(triage.vfs, "extract_remote_gps_points", boom) + row = triage.select_targets(db, limit=1)[0] + n = triage.triage_one(db, "http://cam", row, str(rec_dir), timeout=5.0) + + assert n == -1 + r = _attempts(db, fn) + assert r["triage_attempts"] == 0 # offline: not counted + assert r["triaged_at"] is None # still un-triaged -> retries + + +def test_unreadable_failure_counts(tmp_path, monkeypatch): + db = Database(str(tmp_path / "v.db")) + rec_dir = tmp_path / "rec" + rec_dir.mkdir() + fn = "2026_0618_203643_0001F.MP4" + _seed(db, fn, camera="F") + + def http_err(*a, **k): + raise urllib.error.HTTPError("http://cam", 500, "boom", {}, None) + + monkeypatch.setattr(triage.vfs, "extract_remote_gps_points", http_err) + row = triage.select_targets(db, limit=1)[0] + n = triage.triage_one(db, "http://cam", row, str(rec_dir), timeout=5.0) + + assert n == -1 + r = _attempts(db, fn) + assert r["triage_attempts"] == 1 # unreadable: counted + assert r["triaged_at"] is None # not done; retries until cap + assert r["triage_last_attempt_at"] is not None + + +def test_truncated_read_counts_as_unreadable(tmp_path, monkeypatch): + from viofosync_lib import TruncatedRead + db = Database(str(tmp_path / "v.db")) + rec_dir = tmp_path / "rec" + rec_dir.mkdir() + fn = "2026_0618_203643_0001F.MP4" + _seed(db, fn, camera="F") + + def trunc(*a, **k): + raise TruncatedRead("short read") + + monkeypatch.setattr(triage.vfs, "extract_remote_gps_points", trunc) + row = triage.select_targets(db, limit=1)[0] + triage.triage_one(db, "http://cam", row, str(rec_dir), timeout=5.0) + r = _attempts(db, fn) + assert r["triage_attempts"] == 1 + assert r["triage_last_attempt_at"] is not None + + +def test_select_skips_given_up_clip(tmp_path): + db = Database(str(tmp_path / "v.db")) + fn = "2026_0618_203643_0001F.MP4" + _seed(db, fn, camera="F") + with db.write() as c: + c.execute( + "UPDATE download_queue SET triage_attempts=? WHERE filename=?", + (triage.TRIAGE_MAX_ATTEMPTS, fn), + ) + now = 1_900_000_000 + assert triage.select_targets(db, limit=10, now=now) == [] # gave up -> excluded + + +def test_select_max_minus_one_still_selectable(tmp_path): + db = Database(str(tmp_path / "v.db")) + fn = "2026_0618_203643_0001F.MP4" + _seed(db, fn, camera="F") + now = 1_900_000_000 + with db.write() as c: + c.execute( + "UPDATE download_queue SET triage_attempts=?, " + "triage_last_attempt_at=? WHERE filename=?", + (triage.TRIAGE_MAX_ATTEMPTS - 1, now - 700, fn), # past 600s backoff + ) + assert len(triage.select_targets(db, limit=10, now=now)) == 1 + + +def test_select_respects_backoff(tmp_path): + db = Database(str(tmp_path / "v.db")) + fn = "2026_0618_203643_0001F.MP4" + _seed(db, fn, camera="F") + now = 1_900_000_000 + # 1 failure -> 10s backoff. Last attempt 5s ago -> still backing off. + with db.write() as c: + c.execute( + "UPDATE download_queue SET triage_attempts=1, " + "triage_last_attempt_at=? WHERE filename=?", + (now - 5, fn), + ) + assert triage.select_targets(db, limit=10, now=now) == [] + # 15s later -> past the 10s window -> selectable again. + assert len(triage.select_targets(db, limit=10, now=now + 15)) == 1 diff --git a/tests/test_triage_settings.py b/tests/test_triage_settings.py new file mode 100644 index 0000000..c96f6b1 --- /dev/null +++ b/tests/test_triage_settings.py @@ -0,0 +1,19 @@ +# tests/test_triage_settings.py +"""GPS_TRIAGE is an opt-in boolean exposed through the settings provider.""" +from __future__ import annotations + +from web.settings import SettingsProvider +from web.settings_schema import DEFAULT_VALUES, EDITABLE_KEYS, validate_partial + + +def test_default_off_and_editable(): + assert DEFAULT_VALUES["GPS_TRIAGE"] is False + assert "GPS_TRIAGE" in EDITABLE_KEYS + assert validate_partial({"GPS_TRIAGE": True}) == {"GPS_TRIAGE": True} + + +def test_snapshot_exposes_gps_triage(tmp_config_dir, tmp_recordings_dir): + p = SettingsProvider() + assert p.get().gps_triage is False + snap = p.update({"GPS_TRIAGE": True}, actor="test") + assert snap.gps_triage is True diff --git a/tests/test_triage_status.py b/tests/test_triage_status.py new file mode 100644 index 0000000..2f57285 --- /dev/null +++ b/tests/test_triage_status.py @@ -0,0 +1,67 @@ +"""Triage as a first-class sync status: hub substate + MQTT exposure.""" +from __future__ import annotations + +import types + +from web.services.hub import Hub + + +async def test_hub_stores_and_clears_triage_substate(): + hub = Hub() + assert hub.last_state["triage"] == {"active": False} + + await hub.broadcast({ + "type": "triage_progress", "active": True, + "triaged": 40, "total": 876, "eta_s": 720, + }) + assert hub.last_state["triage"] == { + "active": True, "triaged": 40, "total": 876, "eta_s": 720, + } + + await hub.broadcast({"type": "triage_progress", "active": False}) + assert hub.last_state["triage"] == {"active": False} + + +def _triage_hub(): + return types.SimpleNamespace(last_state={ + "sync_state": {"running": True, "paused": False}, + "dashcam_online": True, + "triage": {"active": True, "triaged": 40, "total": 876, "eta_s": 720}, + }) + + +def _snap(): + return types.SimpleNamespace( + address="192.168.1.50", recordings="/recordings", disk_critical_pct=95, + ) + + +def test_mqtt_state_sync_status_is_triaging(): + from web.services import mqtt_state + assert mqtt_state.state_sync_status(_triage_hub(), None, _snap()) == "triaging" + + +def test_mqtt_attrs_carry_triage_progress(): + from web.services import mqtt_state + attrs = mqtt_state.attrs_sync_status(_triage_hub(), None, _snap()) + assert attrs["triage_active"] is True + assert attrs["triaged"] == 40 + assert attrs["triage_total"] == 876 + assert attrs["triage_eta_s"] == 720 + + +def test_mqtt_attrs_triage_inactive_by_default(): + from web.services import mqtt_state + hub = types.SimpleNamespace(last_state={ + "sync_state": {"running": True, "paused": False}, + "dashcam_online": True, "current_item": None, + "triage": {"active": False}, + }) + attrs = mqtt_state.attrs_sync_status(hub, None, _snap()) + assert attrs["triage_active"] is False + + +def test_sync_status_entity_republishes_on_triage_progress(): + from web.services.mqtt_topology import TOPOLOGY + ent = next(e for e in TOPOLOGY if e.object_id == "sync_status") + assert "triage_progress" in ent.affected_by_hub_events diff --git a/tests/test_triage_sync_worker.py b/tests/test_triage_sync_worker.py new file mode 100644 index 0000000..12b098d --- /dev/null +++ b/tests/test_triage_sync_worker.py @@ -0,0 +1,314 @@ +# tests/test_triage_sync_worker.py +"""SyncWorker runs the triage pass only when GPS_TRIAGE is on.""" +from __future__ import annotations + +import time + +from web.db import Database +from web.services.hub import Hub +from web.services.sync_worker import SyncWorker + + +class _Provider: + def __init__(self, snap): + self._snap = snap + + def get(self): + return self._snap + + +class _Snap: + def __init__(self, recordings, gps_triage): + self.recordings = recordings + self.gps_triage = gps_triage + self.timeout = 5.0 + + +def _seed(db, fn, camera="F"): + with db.write() as c: + c.execute( + "INSERT INTO download_queue " + "(filename, source_dir, camera, event_type, state, enqueued_at) " + "VALUES (?,?,?,?,?,?)", + (fn, "/DCIM/Movie", camera, "normal", "pending", int(time.time())), + ) + + +async def test_pass_runs_when_enabled(tmp_path, monkeypatch): + db = Database(str(tmp_path / "v.db")) + rec = tmp_path / "rec" + rec.mkdir() + _seed(db, "2026_0618_203643_0001F.MP4") + snap = _Snap(str(rec), gps_triage=True) + sw = SyncWorker(db, _Provider(snap), Hub()) + sw._active_address = "1.2.3.4" + + called = {"n": 0} + from web.services import triage + monkeypatch.setattr( + triage, "run_pass", + lambda *a, **k: called.__setitem__("n", called["n"] + 1) or {"triaged": 1}, + ) + await sw._run_triage_pass() + assert called["n"] == 1 + + +async def test_pass_skipped_when_disabled(tmp_path, monkeypatch): + db = Database(str(tmp_path / "v.db")) + rec = tmp_path / "rec" + rec.mkdir() + snap = _Snap(str(rec), gps_triage=False) + sw = SyncWorker(db, _Provider(snap), Hub()) + sw._active_address = "1.2.3.4" + from web.services import triage + monkeypatch.setattr( + triage, "run_pass", + lambda *a, **k: (_ for _ in ()).throw(AssertionError("must not run")), + ) + await sw._run_triage_pass() # no exception = skipped + + +async def test_pass_skipped_when_no_active_address(tmp_path, monkeypatch): + db = Database(str(tmp_path / "v.db")) + rec = tmp_path / "rec" + rec.mkdir() + snap = _Snap(str(rec), gps_triage=True) # enabled, but no address + sw = SyncWorker(db, _Provider(snap), Hub()) + # _active_address defaults to None; do NOT set it. + from web.services import triage + monkeypatch.setattr( + triage, "run_pass", + lambda *a, **k: (_ for _ in ()).throw(AssertionError("must not run")), + ) + await sw._run_triage_pass() # no exception = skipped at the address guard + + +async def test_triage_pass_clears_stale_cancel(tmp_path, monkeypatch): + db = Database(str(tmp_path / "v.db")) + rec = tmp_path / "rec" + rec.mkdir() + snap = _Snap(str(rec), gps_triage=True) + sw = SyncWorker(db, _Provider(snap), Hub()) + sw._active_address = "1.2.3.4" + sw._cancel_current.set() # stale cancel left over from a prior pause/skip + from web.services import triage + monkeypatch.setattr(triage, "run_pass", lambda *a, **k: {"triaged": 0}) + await sw._run_triage_pass() + assert not sw._cancel_current.is_set() # pass cleared the stale flag + + +async def test_triage_pass_emits_progress_and_always_clears(tmp_path, monkeypatch): + import asyncio + + from web.services import triage as triage_module + + db = Database(str(tmp_path / "v.db")) + rec = tmp_path / "rec" + rec.mkdir() + snap = _Snap(str(rec), gps_triage=True) + sw = SyncWorker(db, _Provider(snap), Hub()) + sw._active_address = "1.2.3.4" + sw.bind_loop(asyncio.get_running_loop()) + + events = [] + + async def _bcast(ev): + events.append(ev) + + def _sched(loop, ev): + events.append(ev) + + monkeypatch.setattr(sw.hub, "broadcast", _bcast) + monkeypatch.setattr(sw.hub, "schedule_broadcast", _sched) + + def _fake_run_pass(db_, base, recordings, *, timeout, cancel_check, + progress_cb): + progress_cb(0, 5, None) # start + progress_cb(5, 5, 0) # progress + return {"triaged": 5, "with_gps": 5} + + monkeypatch.setattr(triage_module, "run_pass", _fake_run_pass) + + await sw._run_triage_pass() + + tp = [e for e in events if e.get("type") == "triage_progress"] + assert any(e.get("active") is True for e in tp) # start/progress fired + assert tp[-1] == {"type": "triage_progress", "active": False} # always clears + + +async def test_triage_pass_clears_even_on_error(tmp_path, monkeypatch): + import asyncio + + from web.services import triage as triage_module + + db = Database(str(tmp_path / "v.db")) + rec = tmp_path / "rec" + rec.mkdir() + snap = _Snap(str(rec), gps_triage=True) + sw = SyncWorker(db, _Provider(snap), Hub()) + sw._active_address = "1.2.3.4" + sw.bind_loop(asyncio.get_running_loop()) + + events = [] + + async def _bcast(ev): # hub.broadcast is a coroutine + events.append(ev) + + monkeypatch.setattr(sw.hub, "broadcast", _bcast) + monkeypatch.setattr(sw.hub, "schedule_broadcast", lambda loop, ev: None) + + def _boom(*a, **k): + raise RuntimeError("triage blew up") + + monkeypatch.setattr(triage_module, "run_pass", _boom) + + await sw._run_triage_pass() # must not raise (except-guarded) + assert events[-1] == {"type": "triage_progress", "active": False} + + +async def test_drain_passes_triage_gate(tmp_path, monkeypatch): + db = Database(str(tmp_path / "v.db")) + rec = tmp_path / "rec" + rec.mkdir() + snap = _Snap(str(rec), gps_triage=True) + snap.sync_ro_only = False + snap.disk_critical_pct = 95 + sw = SyncWorker(db, _Provider(snap), Hub()) + + async def _noop(*a, **k): + return None + + async def _true(*a, **k): + return True + + async def _addr(*a, **k): + return ("1.2.3.4", "primary") + + # Stub everything _cycle does before the drain so it reaches next_pending. + monkeypatch.setattr(sw, "_emit_disk_pct", _noop) + monkeypatch.setattr(sw, "_check_recordings_writable", _true) + monkeypatch.setattr(sw, "_select_active_address", _addr) + monkeypatch.setattr(sw, "_refresh_listing_and_reconcile", _true) + monkeypatch.setattr(sw, "_run_recording_status_pass", _noop) + monkeypatch.setattr(sw, "_run_triage_pass", _noop) + monkeypatch.setattr(sw, "_run_geofence_pass", _noop) + + captured = {} + from web.services import queue as q + + def spy(_db, *, ro_only=False, triage_gate=False, active_guard=False): + captured["ro_only"] = ro_only + captured["triage_gate"] = triage_gate + return None # empty queue -> drain ends immediately + + monkeypatch.setattr(q, "next_pending", spy) + + await sw._cycle() + assert captured == {"ro_only": False, "triage_gate": True} + + +def _seed_dl(db, filename, *, recorded_at, triaged_at=None, gps_points=None, + remote_complete=None): + with db.write() as c: + c.execute( + "INSERT INTO download_queue (filename, source_dir, camera, " + "event_type, state, enqueued_at, recorded_at, triaged_at, " + "gps_points, remote_complete) VALUES (?,?,?,?,?,?,?,?,?,?)", + (filename, "/DCIM", filename[-5], "normal", "pending", + recorded_at, recorded_at, triaged_at, gps_points, remote_complete), + ) + + +def _cycle_stubs(sw, monkeypatch): + async def _noop(*a, **k): + return None + + async def _true(*a, **k): + return True + + async def _addr(*a, **k): + return ("1.2.3.4", "primary") + + async def _triaged_nothing(*a, **k): + return 0 # a pass that made no progress (e.g. pause/resume abort) + + monkeypatch.setattr(sw, "_emit_disk_pct", _noop) + monkeypatch.setattr(sw, "_check_recordings_writable", _true) + monkeypatch.setattr(sw, "_select_active_address", _addr) + monkeypatch.setattr(sw, "_refresh_listing_and_reconcile", _true) + monkeypatch.setattr(sw, "_run_recording_status_pass", _noop) + monkeypatch.setattr(sw, "_run_triage_pass", _triaged_nothing) + monkeypatch.setattr(sw, "_run_geofence_pass", _noop) + monkeypatch.setattr(sw, "_probe_one", _true) + + +async def test_drain_held_until_triage_complete(tmp_path, monkeypatch): + # A settled, un-triaged clip means triage is incomplete: NOTHING should + # download this cycle — not even a co-queued already-triaged clip. This is + # the pause-during-triage / resume case the user hit. + db = Database(str(tmp_path / "v.db")) + rec = tmp_path / "rec" + rec.mkdir() + snap = _Snap(str(rec), gps_triage=True) + snap.sync_ro_only = False + snap.disk_critical_pct = 95 + sw = SyncWorker(db, _Provider(snap), Hub()) + + old = int(time.time()) - 10_000 # well past the settle window + # Both clips are finalized (remote_complete=1); 0002F is finalized but not + # yet triaged, so triage is genuinely incomplete and the whole drain must + # hold — the pause-during-triage / resume case. (A clip that is the + # *unconfirmed* newest capture is instead excluded from triage as + # actively-recording; that is a different case, covered elsewhere.) + _seed_dl(db, "2026_0618_203643_0001F.MP4", + recorded_at=old, triaged_at=old, gps_points=5, + remote_complete=1) # triaged + _seed_dl(db, "2026_0618_203644_0002F.MP4", recorded_at=old, + remote_complete=1) # finalized, NOT triaged + + _cycle_stubs(sw, monkeypatch) + downloaded = [] + + async def _dl(item): + downloaded.append(item.filename) + return True + + monkeypatch.setattr(sw, "_download_one", _dl) + + await sw._cycle() + assert downloaded == [] # held: triage not complete + + +async def test_drain_runs_when_triage_complete(tmp_path, monkeypatch): + # With every settled clip triaged AND finalized (remote_complete=1), + # neither the triage gate nor the active-recording guard fires, so the + # drain downloads normally. + db = Database(str(tmp_path / "v.db")) + rec = tmp_path / "rec" + rec.mkdir() + snap = _Snap(str(rec), gps_triage=True) + snap.sync_ro_only = False + snap.disk_critical_pct = 95 + sw = SyncWorker(db, _Provider(snap), Hub()) + + old = int(time.time()) - 10_000 + _seed_dl(db, "2026_0618_203643_0001F.MP4", + recorded_at=old, triaged_at=old, gps_points=5, remote_complete=1) + _seed_dl(db, "2026_0618_203644_0002F.MP4", + recorded_at=old, triaged_at=old, gps_points=0, remote_complete=1) + + _cycle_stubs(sw, monkeypatch) + from web.services import queue as q + downloaded = [] + + async def _dl(item): + downloaded.append(item.filename) + q.mark_done(db, item.id) # advance state so the drain doesn't loop + return True + + monkeypatch.setattr(sw, "_download_one", _dl) + + await sw._cycle() + assert set(downloaded) == { + "2026_0618_203643_0001F.MP4", "2026_0618_203644_0002F.MP4", + } diff --git a/viofosync_lib/__init__.py b/viofosync_lib/__init__.py index a133dcb..c8e8451 100644 --- a/viofosync_lib/__init__.py +++ b/viofosync_lib/__init__.py @@ -30,15 +30,21 @@ get_group_name, ) from ._gpx import ( + IncompleteRecording, extract_gps_data, generate_gpx, + has_final_moov, parse_moov, ) from ._protocol import ( DownloadCancelled, + DownloadDeferred, + TruncatedRead, download_file, + extract_remote_gps_points, get_dashcam_filenames, get_dashcam_filenames_html, + remote_moov_reachable, ) from .progress import ProgressSink @@ -103,6 +109,9 @@ def delete_dashcam_file( __all__ = [ "DownloadCancelled", + "DownloadDeferred", + "IncompleteRecording", + "TruncatedRead", "Recording", "ProgressSink", "delete_dashcam_file", @@ -110,11 +119,14 @@ def delete_dashcam_file( "download_file_with", "downloaded_filename_re", "extract_gps_data", + "extract_remote_gps_points", "generate_gpx", "get_dashcam_filenames", "get_dashcam_filenames_html", "get_downloaded_recordings", "get_filepath", "get_group_name", + "has_final_moov", "parse_moov", + "remote_moov_reachable", ] diff --git a/viofosync_lib/_archive.py b/viofosync_lib/_archive.py index 3c754a6..b3159ed 100644 --- a/viofosync_lib/_archive.py +++ b/viofosync_lib/_archive.py @@ -32,19 +32,30 @@ "yearly": "[0-9][0-9][0-9][0-9]", } -# Downloaded recording filename glob pattern. The trailing -# letter is the camera (see cameras.py for the registry). -downloaded_filename_glob = ( +# Downloaded recording filename glob patterns. Two on-disk layouts exist: +# - standard: YYYY_MMDD_HHMMSS_NNNN[PE]?.MP4 — the trailing letter +# is the camera (see cameras.py for the registry). +# - compact: YYYYMMDDHHMMSS_NNNNNN.MP4 — some single-channel units list +# recordings with no datetime separators and no camera suffix; the sole +# lens is the GPS-bearing one. +downloaded_filename_globs = ( "[0-9][0-9][0-9][0-9]_[0-9][0-9][0-9][0-9]" "_[0-9][0-9][0-9][0-9][0-9][0-9]" - f"_*[{CAMERA_LETTERS}].MP4" + f"_*[{CAMERA_LETTERS}].MP4", + "[0-9]" * 14 + "_*.MP4", ) -# Downloaded recording filename regular expression. +# Kept for callers of the historical single-glob name (standard layout only). +downloaded_filename_glob = downloaded_filename_globs[0] + +# Downloaded recording filename regular expression, matching both layouts: +# the datetime separators are optional and the camera suffix may be absent +# (compact names put a sequence digit where the letter would be, so an empty +# ``camera`` group identifies them — callers default it to the GPS lens). downloaded_filename_re = re.compile( - r"^(?P\d{4})_(?P\d{2})(?P\d{2})" - r"_(?P\d{2})(?P\d{2})(?P\d{2})" - r"_(?P\d+)(?P.+)\.MP4$", + r"^(?P\d{4})_?(?P\d{2})(?P\d{2})" + r"_?(?P\d{2})(?P\d{2})(?P\d{2})" + r"_(?P\d+)(?P[A-Za-z]*)\.MP4$", re.IGNORECASE, ) @@ -75,10 +86,11 @@ def get_group_name(recording_datetime, grouping): def get_downloaded_recordings(destination, grouping): """Reads destination dir and returns set of (filename, date).""" group_name_glob = group_name_globs[grouping] - filepath_glob = get_filepath( - destination, group_name_glob, downloaded_filename_glob - ) - downloaded_filepaths = glob.glob(filepath_glob) + downloaded_filepaths = [] + for name_glob in downloaded_filename_globs: + downloaded_filepaths.extend(glob.glob( + get_filepath(destination, group_name_glob, name_glob) + )) recordings = set() for filepath in downloaded_filepaths: diff --git a/viofosync_lib/_control.py b/viofosync_lib/_control.py index 56c7e00..de3dce3 100644 --- a/viofosync_lib/_control.py +++ b/viofosync_lib/_control.py @@ -470,6 +470,25 @@ def _record_state(address: str) -> int | None: return None +def record_state(address: str) -> int | None: + """Public, resilient record-flag read for the active-recording guard. + + Returns ``1`` (recording) or ``0`` (stopped) from the camera's + ``cmd=3014`` dump (pair ``2001``); returns ``None`` for ANY uncertainty — + camera unreachable, ``cmd=3014`` unsupported, the ``2001`` pair absent from + this model's dump, or an unparseable response. Callers MUST treat ``None`` + as "assume recording" and hold the newest capture, so cameras with + different or unknown command sets stay safe by default.""" + try: + return _record_state(address) + except Exception: + # Deliberately broad: a malformed/garbage cmd=3014 response can raise + # beyond CameraUnreachable (e.g. ValueError from int() on a bad pair). + # Any uncertainty must degrade to None so the guard holds the newest + # capture — never narrow this to CameraUnreachable. + return None + + def _apply_and_verify(address: str, cmd: int, index: int): """Send ``cmd&par=index`` then poll cmd=3014 to confirm it applied. diff --git a/viofosync_lib/_gpx.py b/viofosync_lib/_gpx.py index 5f04106..8056909 100644 --- a/viofosync_lib/_gpx.py +++ b/viofosync_lib/_gpx.py @@ -15,6 +15,13 @@ logger = logging.getLogger("viofosync_lib.gpx") + +class IncompleteRecording(Exception): + """A clip has no reachable top-level ``moov`` atom — the camera is still + recording it (or it was truncated). Raised by the remote GPS path so the + caller can defer the clip instead of caching a bogus "no GPS" result.""" + + # GPS sanity bounds for the spike filter. GPS_MAX_REASONABLE_SPEED_MPS = 85.0 GPS_OUTLIER_MIN_JUMP_M = 2000.0 @@ -295,6 +302,42 @@ def get_gps_atom(gps_atom_info, f): return get_gps_data(data[12:]) +def has_final_moov(in_fh) -> bool: + """True if a top-level ``moov`` atom is reachable by walking the atom + chain from the start. + + WARNING — this does NOT prove a clip is finalized on current A229/A329 + firmware. Field-verified 2026-07-02: these cameras write the ``moov`` at + the FRONT (``ftyp,skip,moov,skip,mdat``), inside a fixed ~65 KB reserved + region, and keep growing it in place while recording — so a + still-recording clip has a reachable, populated ``moov`` and this returns + True mid-recording. Use the camera's ``cmd=3014`` record flag + (``viofosync_lib._control.record_state``) to decide "is it still + recording?"; see the active-recording guard docs. What this function + genuinely tells you: a ``moov`` atom is present and walkable (still useful + for the GPS path — no ``moov`` means no GPS box to parse yet). A + short/garbage/zero-size header returns False (treat "can't confirm" as + not-final), as does a ``TruncatedRead`` from the reader.""" + from ._protocol import TruncatedRead + try: + while True: + header = in_fh.read(8) + if len(header) < 8: + return False + atom_size, atom_type = get_atom_info(header) + if atom_type == 'moov': + return True + # size==1 (64-bit extended size) also lands here and returns + # False. Parity with parse_moov: these cameras don't emit 64-bit + # atoms (clips are well under 4 GB). See the "out of scope" note in + # docs/superpowers/specs/2026-07-02-active-recording-guard-design.md. + if atom_size < 8: # zero/garbage/64-bit header — stop + return False + in_fh.seek(atom_size - 8, 1) # skip this atom's payload + except TruncatedRead: + return False + + def parse_moov(in_fh): gps_data = [] offset = 0 diff --git a/viofosync_lib/_protocol.py b/viofosync_lib/_protocol.py index 6b2c3ce..aa74cc1 100644 --- a/viofosync_lib/_protocol.py +++ b/viofosync_lib/_protocol.py @@ -42,6 +42,27 @@ class DownloadCancelled(Exception): """ +class DownloadDeferred(Exception): + """Raised when ``download_file`` aborts because the file is still being + recorded (the stream outgrew its expected size — the stale-size firmware + under-reports the active file). Like :class:`DownloadCancelled`, this is + NOT a failed attempt: the caller must requeue the item without burning a + retry, so the completeness guard can hold it until the segment finalizes.""" + + +class TruncatedRead(Exception): + """A byte-range read returned fewer bytes than requested while not at + EOF — the camera closed the body early. Distinct from a connection + error: the camera *was* reachable, so the caller treats this as a + clip-specific read failure, not a camera-offline condition.""" + + +class _StillRecording(Exception): + """Internal: the file grew past its expected size mid-download (active + recording). Caught in download_file and re-raised as the public + :class:`DownloadDeferred` so the caller refunds the attempt.""" + + # Defaults used when download_file gets no per-call override. socket_timeout = 10.0 DEFAULT_DOWNLOAD_ATTEMPTS = 1 @@ -405,6 +426,17 @@ def download_file(base_url, recording, destination, group_name, break out.write(chunk) bytes_done += len(chunk) + # Defense-in-depth for the stale-size firmware: if the + # stream outgrows the expected size by more than the + # MB-rounding slack, the file is still being recorded. + # Abort and defer rather than loop on "Incomplete + # download" (the camera under-reports the active file). + if (expected_size is not None + and bytes_done > expected_size + 2 * 1024 * 1024): + raise _StillRecording( + f"file still growing: streamed " + f"{bytes_done} > expected {expected_size}" + ) if sink is not None: now = time.perf_counter() if now - last_emit >= 0.25: @@ -427,6 +459,14 @@ def download_file(base_url, recording, destination, group_name, f"Download of {recording.filename} cancelled" ) raise + except _StillRecording as e: + logger.info(f"Deferring {recording.filename}: {e}") + # Treat like a cancellation: no attempt burned, partial cleaned + # up by the finally. Propagate as the public DownloadDeferred so + # the caller requeues without penalty (a plain False return + # would be counted as a failed attempt). The completeness guard + # holds it until the segment finalizes. + raise DownloadDeferred(str(e)) from e except Exception as e: if isinstance(e, OSError) and e.errno == errno.ENOSPC: # Full disk: retrying can only fail the same way @@ -499,3 +539,131 @@ def download_file(base_url, recording, destination, group_name, os.remove(tmp_path) except OSError: pass + + +class RangeReader: + """Seekable, read-only file-like object backed by a byte-range source. + + ``read_range(a, b)`` returns the inclusive byte range ``[a, b]``. A head + buffer of ``head_len`` bytes is prefetched on construction so a consumer + that does many small sequential reads near the start (e.g. + :func:`viofosync_lib.parse_moov` walking the leading ``moov`` atom) is + served from memory; only reads outside the buffer hit ``read_range``. + + Designed to drop straight into ``parse_moov`` so the GPS decode + spike + filter are reused for remote clips without duplicating any parsing. + """ + + def __init__(self, read_range, size, head_len=65536): + self._read_range = read_range + self._size = int(size) + self._pos = 0 + n = min(head_len, self._size) + self._head = read_range(0, n - 1) if n > 0 else b"" + + def seek(self, pos, whence=0): + if whence == 0: + self._pos = pos + elif whence == 1: + self._pos += pos + elif whence == 2: + self._pos = self._size + pos + else: + raise ValueError(f"bad whence: {whence}") + return self._pos + + def tell(self): + return self._pos + + def read(self, n=-1): + if n is None or n < 0: + n = self._size - self._pos + start = self._pos + end = min(start + n, self._size) # exclusive, clamped to EOF + if end <= start: + return b"" + head_n = len(self._head) + if end <= head_n: + data = self._head[start:end] + elif start >= head_n: + data = self._read_range(start, end - 1) + else: + data = self._head[start:head_n] + self._read_range(head_n, end - 1) + # A range response shorter than the (EOF-clamped) request means the + # camera closed the body early — raise so triage doesn't commit a + # truncated track. `end` is already clamped to EOF, so a short slice + # here is never a legitimate end-of-file read. + if len(data) < (end - start): + raise TruncatedRead( + f"short read: got {len(data)} of {end - start} bytes " + f"at offset {start}" + ) + self._pos = start + len(data) + return data + + +def _remote_clip_url(base_url, recording): + """Build the camera URL for a clip, matching download_file's rules + (strip the A:/B: drive letter, swap '\\' -> '/').""" + cleaned = re.sub(r"^[A-Z]:", "", recording.filepath).replace("\\", "/") + return f"{base_url}/{cleaned.lstrip('/')}" + + +def open_remote_reader(base_url, recording, *, timeout=None, head_len=65536): + """Return a RangeReader over a clip on the dashcam's HTTP server. + + Uses HEAD for the total size (needed for EOF), then serves each read via + a one-shot ``Range`` request (the camera closes the socket per request + and resets above ~3 concurrent connections, so one in-flight request per + reader is deliberate).""" + t = timeout if timeout is not None else globals()["socket_timeout"] + url = _remote_clip_url(base_url, recording) + size = get_remote_size(url, t) + if not size: + size = getattr(recording, "size", None) or 0 + if not size: + logger.warning( + "remote reader: unknown size for %s; GPS triage will read nothing", + url, + ) + + def _read_range(a, b): + req = urllib.request.Request(url, headers={"Range": f"bytes={a}-{b}"}) + with urllib.request.urlopen(req, timeout=t) as resp: + status = getattr(resp, "status", 206) + body = resp.read() + # A server that ignores Range answers 200 with the whole body; slice. + if status == 200 and len(body) > (b - a + 1): + return body[a : b + 1] + return body + + return RangeReader(_read_range, size, head_len=head_len) + + +def extract_remote_gps_points(base_url, recording, *, timeout=None, head_len=65536): + """Decode a remote clip's embedded GPS track without downloading it. + + Raises :class:`IncompleteRecording` if the clip has no reachable top-level + ``moov`` (still recording / truncated) — the caller must defer, not cache + a bogus empty track. A present ``moov`` with no ``gps `` box or no fix + returns ``[]`` (a real "no GPS" result).""" + from ._gpx import parse_moov, has_final_moov, IncompleteRecording + reader = open_remote_reader( + base_url, recording, timeout=timeout, head_len=head_len + ) + if not has_final_moov(reader): + raise IncompleteRecording(recording.filename) + reader.seek(0) + return parse_moov(reader) + + +def remote_moov_reachable(base_url, recording, *, timeout=None, head_len=65536): + """True if a finalized ``moov`` is reachable on the remote clip (proof the + segment has closed). Costs one HEAD + a few small range reads. Camera- + offline errors propagate so the caller can treat them as "unknown" rather + than "incomplete".""" + from ._gpx import has_final_moov + reader = open_remote_reader( + base_url, recording, timeout=timeout, head_len=head_len + ) + return has_final_moov(reader) diff --git a/viofosync_lib/cameras.py b/viofosync_lib/cameras.py index 168a1cf..334d906 100644 --- a/viofosync_lib/cameras.py +++ b/viofosync_lib/cameras.py @@ -23,11 +23,17 @@ class Camera: letter: str # filename suffix letter (the F in ``…0001F.MP4``) channel: str # stable machine key: timeline channel / pair slot label: str # human-facing label + gps: bool = False # True for the lens whose stream carries the GPS track # Declaration order is display order (timeline tracks, UI cycles). +# ``gps=True`` marks the lens that embeds the GPS track: Viofo records GPS +# once, in the front camera's stream, and the other lenses of the same +# capture share it. GPS Triage reads only this lens (see web/services/ +# triage.py); the download gate holds a clip until its GPS-bearing sibling +# is triaged (see web/services/queue.py::next_pending). CAMERAS: tuple[Camera, ...] = ( - Camera("F", "front", "Front"), + Camera("F", "front", "Front", gps=True), Camera("R", "rear", "Rear"), Camera("T", "tele", "Tele"), Camera("I", "interior", "Interior"), @@ -38,6 +44,21 @@ class Camera: CHANNEL_FOR_LETTER = {c.letter: c.channel for c in CAMERAS} +# The single GPS-bearing lens (the front). Derived, not hard-coded, so the +# whole app's triage/gate/indicator logic follows the registry. +GPS_CAMERA_LETTER = next(c.letter for c in CAMERAS if c.gps) + + +def is_gps_camera(camera: str | None) -> bool: + """True if a clip's ``camera`` code is the GPS-bearing lens. + + The lens is the trailing letter of the code, so parking/event prefixes + (``PF``, ``EF``) still resolve correctly; a bare letter works too. + Unknown / empty codes are not GPS-bearing.""" + if not camera: + return False + return camera[-1].upper() == GPS_CAMERA_LETTER + def channel_of(camera: str | None) -> str: """Map a clip's ``camera`` code to its channel key. diff --git a/web/app.py b/web/app.py index bd2cdd7..1331143 100644 --- a/web/app.py +++ b/web/app.py @@ -53,6 +53,7 @@ from .services.sync_worker import SyncWorker from .services import derive_worker from .services.derive_worker import DeriveWorker +from .services import locations as _locations from .setup_mode import SetupModeMiddleware log = logging.getLogger("viofosync.web") @@ -74,6 +75,17 @@ def _sync_worker_action(keys: set, snap) -> str | None: return "stop" +def _geofence_backfill_needed(keys: set, snap) -> bool: + """True when a settings change should trigger an immediate geofence + re-evaluation of the existing queue: a LOCATIONS change while GPS triage is + on and at least one location is flagged for exclusion.""" + if "LOCATIONS" not in keys: + return False + if not getattr(snap, "gps_triage", False): + return False + return bool(_locations.exclusion_zones(getattr(snap, "locations", ()) or ())) + + @asynccontextmanager async def lifespan(app: FastAPI): """Startup / shutdown hook. @@ -128,6 +140,14 @@ def _on_settings_changed( except Exception: # pragma: no cover — non-fatal log.exception("export orphan sweep failed") + # GPS Triage: drop skeleton sidecars orphaned by a crash (clip since + # downloaded / rotated off / row gone). + try: + from .services import triage as _triage_mod + _triage_mod.sweep_orphans(app.state.db, s.recordings) + except Exception: # pragma: no cover — non-fatal + log.exception("triage startup sweep failed") + log.info( "viofosync web UI ready on http://%s:%d", s.host, s.port ) @@ -279,6 +299,33 @@ def _on_derive_settings_changed(keys, snap) -> None: provider.subscribe(_on_derive_settings_changed) ) + def _on_triage_settings_changed(keys, snap) -> None: + # When GPS triage is turned off, drop the .triage skeleton cache and + # clear the triage columns so stale skeletons don't linger. + if "GPS_TRIAGE" in keys and not snap.gps_triage: + from .services import triage as _triage_mod + try: + _triage_mod.purge_all(app.state.db, snap.recordings) + except Exception: # pragma: no cover — non-fatal + log.exception("triage purge on disable failed") + + app.state.settings_unsubscribes.append( + provider.subscribe(_on_triage_settings_changed) + ) + + def _on_geofence_settings_changed(keys, snap) -> None: + # Re-evaluate the existing queue immediately when the home zone is + # set/changed, so the backlog is skipped without waiting for a cycle. + if _geofence_backfill_needed(keys, snap): + _tasks.spawn( + app.state.sync_worker._run_geofence_pass(), + name="geofence-backfill", + ) + + app.state.settings_unsubscribes.append( + provider.subscribe(_on_geofence_settings_changed) + ) + app.state.mqtt = MqttService( db=app.state.db, provider=provider, @@ -343,7 +390,7 @@ def create_app() -> FastAPI: app = FastAPI( title="Viofosync", - version="2.4", + version="2.5", lifespan=lifespan, docs_url=None, # no swagger in prod build redoc_url=None, diff --git a/web/db.py b/web/db.py index e4d0da6..f22f001 100644 --- a/web/db.py +++ b/web/db.py @@ -105,6 +105,7 @@ def migrate_legacy_db_path(new_path: str) -> None: size_bytes INTEGER, has_gpx INTEGER NOT NULL DEFAULT 0, gps_examined INTEGER NOT NULL DEFAULT 0, + locked INTEGER NOT NULL DEFAULT 0, -- user "retain indefinitely" duration_s REAL, scanned_at INTEGER NOT NULL ); @@ -112,6 +113,10 @@ def migrate_legacy_db_path(new_path: str) -> None: ON clip_index(timestamp); CREATE INDEX IF NOT EXISTS idx_clip_index_group ON clip_index(group_name); +-- basename lookups: the archive's GPS-sibling probe (prefix range) and the +-- delete/lock paths all address clips by basename. +CREATE INDEX IF NOT EXISTS idx_clip_index_basename + ON clip_index(basename); CREATE TABLE IF NOT EXISTS download_queue ( id INTEGER PRIMARY KEY AUTOINCREMENT, @@ -129,7 +134,13 @@ def migrate_legacy_db_path(new_path: str) -> None: enqueued_at INTEGER NOT NULL, started_at INTEGER, finished_at INTEGER, - manual INTEGER NOT NULL DEFAULT 0 + manual INTEGER NOT NULL DEFAULT 0, + skip_reason TEXT, -- 'geofence' (auto) | 'user' | NULL + geofence_released_at INTEGER, -- set when a geofence-skip is manually released + locked INTEGER NOT NULL DEFAULT 0, -- user "retain indefinitely" + triage_attempts INTEGER NOT NULL DEFAULT 0, -- failed-triage retry counter + triage_last_attempt_at INTEGER, -- last triage attempt (backoff) + remote_complete INTEGER -- 1 = finalized moov seen; NULL = unconfirmed ); CREATE INDEX IF NOT EXISTS idx_queue_state ON download_queue(state, priority DESC, enqueued_at ASC); @@ -252,6 +263,40 @@ def _add_column(table: str, col: str, ddl: str) -> None: _add_column("export_jobs", "output_size", "INTEGER") _add_column("export_jobs", "output_duration_s", "REAL") + # GPS Triage: skeleton-track state for queued, not-yet-downloaded + # clips. triaged_at NULL = not triaged; gps_points NULL = not + # triaged, 0 = triaged but no fix (never retried), >0 = points found. + _add_column("download_queue", "triaged_at", "INTEGER") + _add_column("download_queue", "gps_points", "INTEGER") + + # Triage retry bookkeeping (resilient-gps-triage): attempt counter + + # last-attempt timestamp so failed triage reads back off and give up + # after a bounded number of *unreadable* attempts. triage_attempts is + # bumped only for clip-specific failures, never for camera-offline. + _add_column( + "download_queue", "triage_attempts", + "INTEGER NOT NULL DEFAULT 0", + ) + _add_column("download_queue", "triage_last_attempt_at", "INTEGER") + + # Geofence exclusion: skip provenance + permanent-release marker. + # skip_reason 'geofence' = auto-skipped while parked at home, + # 'user' = manually skipped, NULL = legacy / not skipped. + # geofence_released_at: when a user un-skips a geofence clip, so the + # geofence never re-skips it. + _add_column("download_queue", "skip_reason", "TEXT") + _add_column("download_queue", "geofence_released_at", "INTEGER") + + # User "retain indefinitely" flag. Distinct from event_type='ro' + # (dashcam-locked): a user pin that retention always honours. + _add_column("clip_index", "locked", "INTEGER NOT NULL DEFAULT 0") + _add_column("download_queue", "locked", "INTEGER NOT NULL DEFAULT 0") + + # Active-recording guard: 1 once a finalized moov is confirmed on the + # camera; NULL = unconfirmed (newest capture may still be recording). + # One-way — finalization is irreversible, so it never needs clearing. + _add_column("download_queue", "remote_complete", "INTEGER") + @contextmanager def conn(self) -> Iterator[sqlite3.Connection]: """Yield a connection with row-factory set. diff --git a/web/routers/archive.py b/web/routers/archive.py index 0e6ea59..3fdba15 100644 --- a/web/routers/archive.py +++ b/web/routers/archive.py @@ -24,11 +24,19 @@ from ..services import durations, filmstrip, route_cache, scanner, thumbs from ..services import tasks as _tasks from ..services import gps as gps_service +from ..services import day_tracks +from ..services import geofence as geofence_service +from ..services import queue as q +from ..services.gps import MAX_JOURNEY_BUFFER_S, expand_journey_window, _haversine_ll +from ..services import locations as locations_service from ..services.naming import ( CAMERAS, CHANNEL_LABELS, CHANNEL_ORDER, + GPS_CAMERA_LETTER, channel_of, + day_key_sql, + gps_lens_sql, pair_slot_of, ) @@ -43,16 +51,8 @@ FALLBACK_MAX_S = 300.0 FALLBACK_DEFAULT_S = 60.0 -# Journey-window buffer. GPS stop boundaries land ~STOP_RADIUS_M (50m) inside -# the real drive, so the pull-away clip at the start and the pull-in clip at -# the end sit outside the raw journey window — "missing the start" / "cut short -# on arrival". Pad each edge outward, but no further than the nearest -# parking-mode clip on that side (the genuine "car is parked" boundary) and -# never more than this cap. The cap matters because the dashcam can be knocked -# out of parking mode by the car's electrics waking, producing spurious driving -# clips — so we don't trust an arbitrary parking->driving switch to mark the -# journey edge; we only lean on a *parking* clip as a hard stop. -MAX_JOURNEY_BUFFER_S = 120.0 +# MAX_JOURNEY_BUFFER_S and expand_journey_window live in services/gps.py so the +# geofence service can share them (a service can't import from this router). router = APIRouter( prefix="/api/archive", @@ -152,12 +152,47 @@ def list_days( params + [per_page, offset], ).fetchall() - return { - "total": total, - "page": page, - "per_page": per_page, - "days": [dict(r) for r in rows], - } + days = [dict(r) for r in rows] + # GPS Triage: only fold in queued (not-downloaded) clips when triage is on. + # With triage off, the archive shows downloaded clips only — no placeholder + # tiles and no remote-only day cards. + if _settings(request).gps_triage: + # Annotate every page's days with a remote (queued) count, and on page 1 + # fold in days that ONLY have queued clips (so brand-new footage gets a + # card to open). Page-1-only keeps remote-only days from repeating on + # every page; with the default newest-first sort they belong at the top. + # NOTE: remote-only days are injected/counted only on page 1, so `total` + # is page-1-inclusive only — the frontend must not derive an exact page + # count from it across pages. A fully paginated clip_index + queue union + # is deliberately out of scope (see the plan). + # GPS-excluded (geofence auto-skipped) clips per day, so the day card + # can report how much parked-at-home footage was held back. + gskip = geofence_skipped_by_day(_db(request), date_from, date_to) + for d in days: + d["geofence_skipped_count"] = gskip.get(d["day"], 0) + by_day = {d["day"]: d for d in days} + for rd in remote_day_summaries(_db(request), date_from, date_to): + d = by_day.get(rd["day"]) + if d is not None: + d["remote_count"] = rd["remote_count"] + d["remote_gps_count"] = rd["remote_gps_count"] + elif page == 1 and rd["remote_gps_count"]: + # Only inject a remote-ONLY day once it has GPS-bearing footage + # to place on a journey. A day of track-less clips (no GPS lock) + # would otherwise open to an empty/Ungrouped card; that footage + # stays in the Queue tab until downloaded. + days.append({ + "day": rd["day"], "clip_count": 0, + "driving_count": 0, "parking_count": 0, "ro_count": 0, + "gpx_count": 0, "first_ts": rd["first_ts"], + "last_ts": rd["last_ts"], "total_bytes": 0, + "remote_count": rd["remote_count"], + "remote_gps_count": rd["remote_gps_count"], + "geofence_skipped_count": gskip.get(rd["day"], 0), + }) + total += 1 + days.sort(key=lambda dd: dd["day"], reverse=(sort == "desc")) + return {"total": total, "page": page, "per_page": per_page, "days": days} @router.get("/day/{date}") @@ -169,6 +204,7 @@ def get_day( driving: bool = Query(True), parking: bool = Query(True), ro: bool = Query(True), + show_geofenced: bool = Query(False), ) -> dict: """All clips for a date, paired front+rear by (timestamp, sequence).""" @@ -187,7 +223,7 @@ def get_day( rows = c.execute( f""" SELECT id, basename, path, timestamp, camera, - sequence, event_type, size_bytes, has_gpx + sequence, event_type, size_bytes, has_gpx, locked FROM clip_index WHERE {' AND '.join(where)} ORDER BY timestamp DESC, sequence DESC @@ -243,14 +279,226 @@ def _in_range(ts: int) -> bool: "timestamp": ts, "sequence": pair["sequence"], "event_type": kind, + "locked": 1 if any( + pair[s] and pair[s].get("locked") for s in slots + ) else 0, "iso": _dt.datetime.fromtimestamp(ts).isoformat(), **{s: pair[s] for s in slots}, }) + # GPS Triage: only when enabled, append queued (not-downloaded) clips as + # placeholder pairs (respecting the same kind + time-range filters). + # remote_day_clips only returns clips with a known GPS trace, so track-less + # footage never snaps onto a journey — it stays in the Queue tab until + # downloaded. When a (timestamp, event_type) is already covered by a + # downloaded pair, the remote pair's slots are merged into that pair's + # EMPTY slots (a half-downloaded capture: front done, rear still queued) + # rather than dropped — dropping would make the queued rear invisible and + # undownloadable from the archive. Occupied slots always win, so a + # downloaded clip never regresses to a ghost placeholder. With triage off, + # the day view shows downloaded clips only. + if _settings(request).gps_triage: + kind_ok = {"normal": driving, "parking": parking, "ro": ro} + by_key = {(cl["timestamp"], cl["event_type"]): cl for cl in clips} + for rc in remote_day_clips(_db(request), date, show_geofenced): + if not kind_ok.get(rc["event_type"], True): + continue + if not _in_range(rc["timestamp"]): + continue + have = by_key.get((rc["timestamp"], rc["event_type"])) + if have is not None: + for s in slots: + if have[s] is None and rc[s] is not None: + have[s] = rc[s] + have["locked"] = 1 if any( + have[s] and have[s].get("locked") for s in slots + ) else 0 + continue + clips.append(rc) + clips.sort(key=lambda cl: cl["timestamp"], reverse=True) return {"date": date, "clips": clips} -def build_route_payload(db, recordings, date: str, geocoder) -> dict: +def _queue_day_expr() -> str: + # YYYY-MM-DD from a Viofo filename, format-aware (standard + compact). + return day_key_sql() + + +def remote_day_summaries(db, date_from=None, date_to=None) -> list[dict]: + """Per-day counts of queued, not-yet-downloaded clips, so a day with no + downloaded clips still gets a card. Counts pairs by front clips only, + matching the archive's pair-oriented view.""" + day = _queue_day_expr() + where = [ + # Skipped clips are excluded from download AND dropped from the journey + # map, so they must not be counted into the archive grid either — they'd + # otherwise snap onto a journey. They stay visible in the Queue tab. + "state IN ('pending','failed','downloading')", + gps_lens_sql(), + ] + params: list = [] + if date_from: + where.append(f"{day} >= ?") + params.append(date_from) + if date_to: + where.append(f"{day} <= ?") + params.append(date_to) + with db.conn() as c: + rows = c.execute( + f""" + SELECT {day} AS day, + COUNT(*) AS remote_count, + SUM(CASE WHEN triaged_at IS NOT NULL AND gps_points > 0 + THEN 1 ELSE 0 END) AS remote_gps_count, + MIN(recorded_at) AS first_ts, + MAX(recorded_at) AS last_ts + FROM download_queue + WHERE {' AND '.join(where)} + GROUP BY {day} + """, + params, + ).fetchall() + return [dict(r) for r in rows] + + +def geofence_skipped_by_day(db, date_from=None, date_to=None) -> dict[str, int]: + """Per-day count of clips auto-skipped by the home geofence + (``skip_reason='geofence'``), keyed by ``YYYY-MM-DD``. Counts front clips + only, matching the archive's pair-oriented day stats. Lets the day card show + how much footage was excluded as parked-at-home.""" + day = _queue_day_expr() + where = [ + "state = 'skipped'", + "skip_reason = 'geofence'", + gps_lens_sql(), + ] + params: list = [] + if date_from: + where.append(f"{day} >= ?") + params.append(date_from) + if date_to: + where.append(f"{day} <= ?") + params.append(date_to) + with db.conn() as c: + rows = c.execute( + f"SELECT {day} AS day, COUNT(*) AS n FROM download_queue " + f"WHERE {' AND '.join(where)} GROUP BY {day}", + params, + ).fetchall() + return {r["day"]: r["n"] for r in rows} + + +def remote_day_clips( + db, date: str, include_geofenced: bool = False, +) -> list[dict]: + """Queued, not-downloaded clips for a day, paired front+rear and flagged + remote. Mirrors get_day's pair shape so the frontend renders them in the + same grid/journey layout. + + Only captures with a known GPS trace are returned: the archive is + journey-organised, so a clip with no track has nowhere to sit and would + land in "Ungrouped" (or snap onto the wrong journey). Only the GPS-bearing + lens is ever triaged, so every lens qualifies via its GPS *sibling* + (``queue.GPS_SIBLING_SQL``: same timestamp prefix, GPS letter) — the + sibling's ``triaged_at``/``gps_points`` speak for the whole capture. The + sibling may already be downloaded (``done`` — its real sidecar carries the + trace) but not ``gone`` (skeleton swept, no trace left). A sibling whose + queue row was never triaged (downloaded before triage existed, or while it + was off — ``purge_all`` clears the columns) still qualifies via its + ``clip_index`` sidecar (``has_gpx=1``), so legacy captures surface their + queued lenses too. Track-less captures (no GPS lock — e.g. garage + parking — or not yet triaged) and orphan non-GPS clips stay in the Queue + tab until they're downloaded. This mirrors ``day_tracks.day_gpx_paths``' + skeleton filter so a grid tile always has a matching track on the journey + map. + + With ``include_geofenced`` (the archive's opt-in "GPS-excluded" view), + geofence-auto-skipped rows are returned too; user-skipped rows never are. + """ + day = _queue_day_expr() + state_clause = "dq.state IN ('pending','failed','downloading')" + if include_geofenced: + state_clause = ( + "(dq.state IN ('pending','failed','downloading') " + "OR (dq.state = 'skipped' AND dq.skip_reason = 'geofence'))" + ) + with db.conn() as c: + rows = c.execute( + f""" + SELECT dq.filename, dq.source_dir, dq.camera, dq.event_type, + dq.recorded_at, dq.triaged_at, dq.gps_points, dq.state, + dq.skip_reason, dq.locked + FROM download_queue dq + WHERE {state_clause} + AND {day.replace('filename', 'dq.filename')} = ? + AND ( + EXISTS ( + SELECT 1 FROM download_queue f + WHERE {q.GPS_SIBLING_SQL} + AND f.state <> 'gone' + AND f.triaged_at IS NOT NULL AND f.gps_points > 0 + ) + OR EXISTS ( + SELECT 1 FROM clip_index ci + WHERE ci.basename >= substr(dq.filename, 1, 16) + AND ci.basename < substr(dq.filename, 1, 16) || '~' + AND {gps_lens_sql('ci.basename')} + AND ci.has_gpx = 1 + ) + ) + ORDER BY dq.recorded_at DESC, dq.filename DESC + """, + (date,), + ).fetchall() + + slots = [cam.channel for cam in CAMERAS] + pairs: dict = defaultdict(lambda: dict.fromkeys(slots)) + for r in rows: + src = (r["source_dir"] or "").upper() + raw = r["event_type"] or "normal" + # Match scanner._event_type_for, which classifies downloaded clips as + # only 'ro' / 'parking' / 'normal' (impact 'event' clips collapse to + # 'normal'). Aligning here keeps get_day's (ts, event_type) de-dup key + # matching so a downloaded clip never leaves a ghost placeholder. + kind = "ro" if "/RO/" in src or src.endswith("/RO") else ( + "parking" if raw == "parking" else "normal") + ts = r["recorded_at"] or 0 + key = (ts, kind) + slot = pair_slot_of(r["camera"] or "") + if slot not in slots: + continue + pairs[key][slot] = { + "remote": True, + "filename": r["filename"], + "basename": r["filename"], + "camera": r["camera"], + "triaged": r["triaged_at"] is not None, + "gps_points": r["gps_points"] or 0, + "state": r["state"], + "skip_reason": r["skip_reason"], + "locked": r["locked"] or 0, + } + + out = [] + for (ts, kind), pair in sorted(pairs.items(), reverse=True): + out.append({ + "remote": True, + "timestamp": ts, + "sequence": 0, + "event_type": kind, + "locked": 1 if any( + pair[s] and pair[s].get("locked") for s in slots + ) else 0, + "iso": _dt.datetime.fromtimestamp(ts).isoformat() if ts else "", + **{s: pair[s] for s in slots}, + }) + return out + + +def build_route_payload( + db, recordings, date: str, geocoder, places=(), + include_geofenced: bool = False, +) -> dict: """Merged GPS track for a day plus detected journeys/stops, as a JSON-able dict. Shared by GET /day/{date}/route and GET /timeline. @@ -262,21 +510,28 @@ def build_route_payload(db, recordings, date: str, geocoder) -> dict: ``geocoder`` is the app's geocoder (or None); only its synchronous ``cache_lookup`` is used here — uncached labels are fetched lazily by the UI via /geocode after first paint. + + ``include_geofenced`` weaves the day's geofence-skipped skeleton tracks + into journey detection (the archive's opt-in "GPS-excluded" view), and + adds per-clip overlay geometry under ``geofenced_tracks`` so the UI can + draw the recovered stretches distinctly from the rest of the journey. """ - with db.conn() as c: - rows = c.execute( - """ - SELECT path FROM clip_index - WHERE group_name = ? AND has_gpx = 1 - ORDER BY timestamp ASC - """, - (date,), - ).fetchall() + gpx_paths = day_tracks.day_gpx_paths(db, recordings, date) + geo_paths: list[str] = [] + if include_geofenced: + # The archive's "GPS-excluded" view: geofence-skipped skeletons + # join journey detection. The cache signature covers the combined + # file set, so on/off variants are distinct cache entries. + geo_paths = [ + p for p in day_tracks.geofence_skipped_gpx_paths( + db, recordings, date) + if p not in gpx_paths + ] + gpx_paths = gpx_paths + geo_paths - gpx_paths = [r["path"] + ".gpx" for r in rows] sig = route_cache.signature(gpx_paths) payload = route_cache.load(recordings, date, sig) - if payload is None: + if payload is None or "points" not in payload: log.info( "route: aggregating %d GPX file(s) for %s", len(gpx_paths), date ) @@ -288,7 +543,17 @@ def build_route_payload(db, recordings, date: str, geocoder) -> dict: payload = _assemble_route(date, points, stops, journeys) route_cache.store(recordings, date, sig, payload) - _apply_labels(payload, geocoder) + if include_geofenced: + # Per-clip overlay geometry, applied on read (never cached): lets + # the UI draw recovered stretches dashed over the journey lines. + payload["geofenced_tracks"] = _geofenced_tracks(geo_paths) + + _apply_group_windows(db, date, payload) + _trim_stops_to_journeys(payload) + _reframe_journeys(payload) + _apply_completion(db, date, payload) + payload.pop("points", None) # server-only scratch; don't ship it to clients + _apply_labels(payload, geocoder, places) return payload @@ -299,6 +564,7 @@ def _assemble_route(date: str, points, stops, journeys) -> dict: return { "date": date, "point_count": len(points), + "points": [[p.lon, p.lat, p.t.timestamp()] for p in points], "journeys": [ { "start_time": j.start_time.isoformat(), @@ -344,21 +610,247 @@ def _assemble_route(date: str, points, stops, journeys) -> dict: } -def _apply_labels(payload: dict, geocoder) -> None: - """Fill journey/stop labels from the geocode cache (synchronous, no - network). Mutates ``payload`` in place. Uncached labels stay None and - are fetched lazily by the UI via /geocode after first paint.""" - def _lbl(lat, lon): - return geocoder.cache_lookup(lat, lon) if geocoder else None +def _geofenced_tracks(paths: list[str]) -> list[dict]: + """Per-clip skeleton traces for geofence-skipped clips. Unparseable or + single-point skeletons are dropped — nothing to draw. Each track carries + a parallel ``times`` array (one epoch-second per coordinate) so the client + can scope it to the owning journey's window — without it, every journey + map on a multi-journey day would draw every day's recovered stretch.""" + tracks = [] + for p in paths: + try: + pts = gps_service._parse_gpx(p) + except Exception: + continue + if len(pts) < 2: + continue + name = os.path.basename(p) + if name.endswith(".gpx"): + name = name[:-4] + tracks.append({ + "filename": name, + "times": [pt.t.timestamp() for pt in pts], + "geojson": { + "type": "Feature", + "geometry": { + "type": "LineString", + "coordinates": [[pt.lon, pt.lat] for pt in pts], + }, + }, + }) + return tracks + + +def _apply_labels(payload: dict, geocoder, places=()) -> None: + """Fill journey/stop labels. A point inside a named location's radius is + labelled with the location name regardless of geocoding; otherwise the + geocode cache is used (uncached labels stay None and are fetched lazily via + /geocode). Mutates ``payload`` in place; runs on every read so it stays + current. + + Flags set per point: + - ``home``/``start_home``/``end_home`` — True only when the matched place + has ``is_home=True`` (i.e. the designated Home location). + - ``named``/``start_named``/``end_named`` — True whenever *any* named place + matched, regardless of whether it is the Home; False for geocoded or + unresolved labels.""" + def _lookup(lat, lon): + p = locations_service.match_for_point(places, lat, lon) + if p is not None: + return p.name, p.is_home, True + return (geocoder.cache_lookup(lat, lon) if geocoder else None), False, False for j in payload.get("journeys", []): - j["start_label"] = _lbl(j["start_lat"], j["start_lon"]) - j["end_label"] = _lbl(j["end_lat"], j["end_lon"]) + j["start_label"], j["start_home"], j["start_named"] = _lookup(j["start_lat"], j["start_lon"]) + j["end_label"], j["end_home"], j["end_named"] = _lookup(j["end_lat"], j["end_lon"]) + for s in payload.get("stops", []): + s["label"], s["home"], s["named"] = _lookup(s["lat"], s["lon"]) + + +def _trim_stops_to_journeys(payload: dict) -> None: + """Shrink each parking stop's window so it no longer overlaps an adjacent + journey's PADDED window (``group_start_ts``/``group_end_ts`` from + :func:`_apply_group_windows`). The padding claims the pull-away/pull-in + clips at a drive's edges, so the stop must not also advertise that time — + otherwise a card reads "stopped for 20 min" while its clips start a couple + minutes in (the arrival clips render on the drive). Run AFTER + ``_apply_group_windows``; mutates ``payload['stops']`` in place. + + A stop entirely absorbed by journey padding (trimmed to a non-positive + span) is dropped: its clips, if any, belong to the flanking drive(s). The + displayed duration and ISO times are recomputed to match the trimmed + window, so the card and its clips agree.""" + windows = [ + (j.get("group_start_ts", j["start_ts"]), + j.get("group_end_ts", j["end_ts"])) + for j in payload.get("journeys", []) + ] + if not windows: + return + kept = [] for s in payload.get("stops", []): - s["label"] = _lbl(s["lat"], s["lon"]) + ss0, se0 = s["start_ts"], s["end_ts"] + ss, se = ss0, se0 + for gs, ge in windows: + if gs <= ss0 < ge: # padded drive covers the stop's start + ss = max(ss, ge) + if gs < se0 <= ge: # padded drive covers the stop's end + se = min(se, gs) + if ss >= se: # fully absorbed by the drive(s) — drop it + continue + if ss != ss0 or se != se0: + s["start_ts"], s["end_ts"] = ss, se + s["start_time"], s["end_time"] = _iso_utc(ss), _iso_utc(se) + s["duration_s"] = int(se - ss) + kept.append(s) + payload["stops"] = kept + + +def _apply_group_windows(db, date: str, payload: dict) -> None: + """Pad each journey's window outward for the archive day-grid grouping, so + pull-away / pull-in clips that sit just outside the raw GPS journey (the GPS + stop boundary lands ~STOP_RADIUS_M inside the real drive) attach to the + journey card instead of the neighbouring stop. Adds ``group_start_ts`` / + ``group_end_ts`` to each journey (= the raw window when there's nothing to + pad). Mirrors the timeline's ``expand_journey_window`` and is bounded the + same way by the day's parking clips. Applied on read (not cached) so it + tracks parking clips, which the cached payload's signature doesn't cover.""" + journeys = payload.get("journeys") + if not journeys: + return + # Parking clips bound the pad; under triage they live in the queue, not + # clip_index, so use the unioned source (see day_tracks.day_parking_spans). + parking_spans = day_tracks.day_parking_spans(db, date) + for j in journeys: + gs, ge = expand_journey_window(j["start_ts"], j["end_ts"], parking_spans) + j["group_start_ts"] = gs + j["group_end_ts"] = ge + + +def _iso_utc(ts: float) -> str: + return _dt.datetime.fromtimestamp(ts, tz=_dt.timezone.utc).isoformat() + + +def _reframe_journeys(payload: dict) -> None: + """Re-derive each journey's geometry, endpoints, time-range, duration and + distance from the merged-point slice inside its padded group window, so the + trace, the start/end markers, the geocoded labels and the displayed time + all describe the SAME window the day grid uses to group clips. + + The window only *selects* points; it never fabricates them. When fixes exist + before the raw GPS start (low-speed maneuvering the stop detector swallowed) + the start moves back to them. When none do (a cold GPS start with no fix + until a minute in) the start stays at the first real fix — the line can't be + drawn where nothing was recorded. + + Runs on read, after ``_apply_group_windows`` (which sets the window from the + *raw* start/end) and before ``_apply_labels`` (which reads the new + endpoints). ``payload['points']`` must be present and sorted ascending by + timestamp.""" + pts = payload.get("points") + if not pts: + return + for j in payload.get("journeys", []): + gs = j.get("group_start_ts", j["start_ts"]) + ge = j.get("group_end_ts", j["end_ts"]) + sl = [p for p in pts if gs <= p[2] <= ge] + if len(sl) < 2: + continue + j["geojson"] = { + "type": "Feature", + "geometry": { + "type": "LineString", + "coordinates": [[p[0], p[1]] for p in sl], + }, + } + j["times"] = [p[2] for p in sl] + j["start_ts"], j["end_ts"] = sl[0][2], sl[-1][2] + j["start_time"], j["end_time"] = _iso_utc(sl[0][2]), _iso_utc(sl[-1][2]) + j["start_lon"], j["start_lat"] = sl[0][0], sl[0][1] + j["end_lon"], j["end_lat"] = sl[-1][0], sl[-1][1] + j["duration_s"] = int(sl[-1][2] - sl[0][2]) + dist = 0.0 + for i in range(1, len(sl)): + dist += _haversine_ll(sl[i - 1][1], sl[i - 1][0], sl[i][1], sl[i][0]) + j["distance_m"] = round(dist, 1) + + +def _apply_completion(db, date: str, payload: dict) -> None: + """Attach per-journey data-completeness so the archive can show a pie: + ``completion`` (downloaded duration / total, 0-1) and ``completion_detail`` + for the tooltip. Front clips only (the GPS-bearing lens; counting the rear + too would double the duration). Skipped clips are already absent from both + sources, so they're excluded from the denominator — a fully-geofenced + journey reads complete. Empty window -> None (the UI omits the pie). + + Runs on read, in the chain ``_apply_group_windows`` -> ``_reframe_journeys`` + -> this. It needs the padded window; it reads ``group_start_ts`` / + ``group_end_ts``, which ``_reframe_journeys`` doesn't touch, so order with + that step is safe. + Duration is estimated uniformly via gap-to-next-front-clip (the + ``_effective_durations`` fallback) so it works before ffprobe and for + not-yet-downloaded clips: real ``duration_s`` is preferred when present.""" + journeys = payload.get("journeys") + if not journeys: + return + qday = _queue_day_expr() + with db.conn() as c: + dl = c.execute( + "SELECT timestamp AS ts, duration_s FROM clip_index " + "WHERE group_name = ? AND upper(substr(camera, -1, 1)) = ? " + "ORDER BY timestamp ASC", + (date, GPS_CAMERA_LETTER), + ).fetchall() + pend = c.execute( + f"SELECT recorded_at AS ts FROM download_queue " + f"WHERE {qday} = ? AND {gps_lens_sql()} " + f"AND state IN ('pending','failed','downloading') " + f"AND triaged_at IS NOT NULL AND gps_points > 0 " + f"ORDER BY recorded_at ASC", + (date,), + ).fetchall() + + clips = [(r["ts"], r["duration_s"], True) for r in dl if r["ts"] is not None] + clips += [(r["ts"], None, False) for r in pend if r["ts"] is not None] + clips.sort(key=lambda x: x[0]) + + eff = [] + for i, (ts, real, dled) in enumerate(clips): + if real and real > 0: + d = float(real) + elif i + 1 < len(clips): + gap = clips[i + 1][0] - ts + d = float(min(gap, FALLBACK_MAX_S)) if gap > 0 else FALLBACK_DEFAULT_S + else: + d = FALLBACK_DEFAULT_S + eff.append((ts, d, dled)) + + for j in journeys: + gs = j.get("group_start_ts", j["start_ts"]) + ge = j.get("group_end_ts", j["end_ts"]) + total_s = dl_s = 0.0 + n_total = n_dl = 0 + for ts, d, dled in eff: + if gs <= ts <= ge: + total_s += d + n_total += 1 + if dled: + dl_s += d + n_dl += 1 + j["completion"] = ( + None if total_s <= 0 else round(dl_s / total_s, 4) + ) + j["completion_detail"] = { + "downloaded_s": round(dl_s), + "total_s": round(total_s), + "downloaded_clips": n_dl, + "total_clips": n_total, + } @router.get("/day/{date}/route") -def get_route(request: Request, date: str) -> dict: +def get_route( + request: Request, date: str, show_geofenced: bool = Query(False), +) -> dict: """Merged GPS track for the day plus detected journeys.""" try: _dt.date.fromisoformat(date) @@ -366,7 +858,8 @@ def get_route(request: Request, date: str) -> dict: raise HTTPException(400, "bad date format") geocoder = getattr(request.app.state, "geocode", None) return build_route_payload( - _db(request), _settings(request).recordings, date, geocoder + _db(request), _settings(request).recordings, date, geocoder, + _settings(request).locations, include_geofenced=show_geofenced, ) @@ -397,30 +890,6 @@ def _effective_durations(rows) -> dict[int, float]: return eff -def _expand_journey_window( - start_ts: float, end_ts: float, - parking_spans: list[tuple[float, float]], -) -> tuple[float, float]: - """Pad a GPS journey window ``[start_ts, end_ts]`` outward to catch the - pull-away / pull-in footage that sits just outside the GPS stop radius. - - Each edge expands by at most ``MAX_JOURNEY_BUFFER_S``, but stops at the - nearest parking-mode clip on that side so we never swallow parked footage. - ``parking_spans`` is ``[(start_ts, end_ts), ...]`` for the day's parking - clips. A parking clip straddling an edge means we're already at the - boundary, so that edge doesn't expand at all. See ``MAX_JOURNEY_BUFFER_S`` - for why the cap, not the mode switch, is the backstop. - """ - new_start = start_ts - MAX_JOURNEY_BUFFER_S - new_end = end_ts + MAX_JOURNEY_BUFFER_S - for ps, pe in parking_spans: - if ps < start_ts: # parking (partly) before the window - new_start = max(new_start, min(pe, start_ts)) - if pe > end_ts: # parking (partly) after the window - new_end = min(new_end, max(ps, end_ts)) - return new_start, new_end - - @router.get("/timeline") def get_timeline( request: Request, @@ -442,7 +911,8 @@ def get_timeline( geocoder = getattr(request.app.state, "geocode", None) db = _db(request) route = build_route_payload( - db, _settings(request).recordings, date, geocoder + db, _settings(request).recordings, date, geocoder, + _settings(request).locations, ) log.info( "timeline: route built (%d GPS point(s)) — querying clips", @@ -456,23 +926,13 @@ def get_timeline( if journey >= len(journeys): raise HTTPException(404, "journey index out of range") j = journeys[journey] - start_ts, end_ts = j["start_ts"], j["end_ts"] - # Pad the window so the pull-away / pull-in footage isn't lost. Parking - # clips are queried unfiltered (the display kind-filter must not hide a - # boundary) and bound how far each edge may travel. - with db.conn() as c: - prk = c.execute( - "SELECT timestamp, duration_s FROM clip_index " - "WHERE group_name = ? AND event_type = 'parking'", - [date], - ).fetchall() - parking_spans = [ - (r["timestamp"], r["timestamp"] + (r["duration_s"] or 0.0)) - for r in prk - ] - start_ts, end_ts = _expand_journey_window( - start_ts, end_ts, parking_spans - ) + # The journey already carries its padded window (group_start_ts/ + # group_end_ts from _apply_group_windows), bounded by the unioned + # clip_index+download_queue parking source. Use it directly so the + # editor's clip window matches the day grid and the reframed trace — + # one definition of the journey's edges, no second (divergent) re-pad. + start_ts = j["group_start_ts"] + end_ts = j["group_end_ts"] where = ["group_name = ?"] params: list = [date] @@ -551,11 +1011,16 @@ async def geocode( lat: float = Query(...), lon: float = Query(...), ) -> dict: + # A named location wins over reverse-geocoding (local, always available). + p = locations_service.match_for_point(_settings(request).locations, lat, lon) + if p is not None: + return {"lat": lat, "lon": lon, "label": p.name, + "home": p.is_home, "named": True} geocoder = getattr(request.app.state, "geocode", None) if geocoder is None: - return {"lat": lat, "lon": lon, "label": None} + return {"lat": lat, "lon": lon, "label": None, "home": False, "named": False} label = await geocoder.reverse(lat, lon) - return {"lat": lat, "lon": lon, "label": label} + return {"lat": lat, "lon": lon, "label": label, "home": False, "named": False} # --- Clip bytes --- @@ -661,6 +1126,34 @@ async def rescan(request: Request) -> JSONResponse: return JSONResponse({"ok": True, "indexed": n}) +@router.post( + "/rebuild-grouping", + dependencies=[Depends(require_csrf)], +) +async def rebuild_grouping(request: Request) -> dict: + """Maintenance: flush stale geofence decisions and the journey/route cache, + then re-evaluate the geofence from the GPS data already on disk. The + ``.triage`` skeletons are NOT touched, so this needs no camera and triggers + no re-triage. Clips the current logic no longer re-skips return to + ``pending`` and may be downloaded on the next drain — that's intended.""" + s = _settings(request) + db = _db(request) + unskipped = await asyncio.to_thread(q.unskip_geofence, db) + route_cache.clear_all(s.recordings) + # Reset the worker's per-day signature cache so its incremental sweeps + # re-evaluate too (not just this one-shot full sweep). + worker = getattr(request.app.state, "sync_worker", None) + if worker is not None: + worker._geofence_seen.clear() + reskipped = 0 + zones = locations_service.exclusion_zones(s.locations) + if s.gps_triage and zones: + reskipped = await asyncio.to_thread( + geofence_service.sweep_all, db, s.recordings, zones, seen=None + ) + return {"ok": True, "unskipped": unskipped, "reskipped": reskipped} + + # --- GPS extraction for existing clips --- diff --git a/web/routers/imports.py b/web/routers/imports.py index b9007d1..f933da6 100644 --- a/web/routers/imports.py +++ b/web/routers/imports.py @@ -17,6 +17,7 @@ from ..auth import require_csrf, require_session from ..services import exporter, importer +from ..services import queue as _queue from ..services import retention as _retention log = logging.getLogger("viofosync.import") @@ -87,12 +88,17 @@ def scan(request: Request, body: _PathBody) -> dict: @router.post("/present", dependencies=[Depends(require_csrf)]) def present(request: Request, body: _FilesBody) -> dict: - """Report which clips already have a complete copy in the archive, so - the browser-upload tab can skip re-sending them. Size-matched: a - truncated archive copy is reported absent so the upload redoes it.""" + """Report which clips the browser-upload tab should not send: clips already + in the archive (``present``, size-matched — a truncated copy is reported + absent so the upload redoes it) and clips currently marked ``skipped`` + (``skipped`` — the triage geofence or the user excluded them). The browser + excludes the union of both from the upload batch.""" snap = _snap(request) sizes = {f.name: f.size for f in body.files} - return {"present": sorted(importer.present_in_archive(snap, sizes))} + return { + "present": sorted(importer.present_in_archive(snap, sizes)), + "skipped": sorted(_queue.skip_listed_names(_db(request), list(sizes.keys()))), + } @router.post("/ingest", dependencies=[Depends(require_csrf)]) @@ -146,6 +152,14 @@ async def upload(request: Request) -> dict: if importer.has_complete_copy(dest, item.size_bytes): return {"status": "already_present", "filename": name} + # Honour the skip list: a clip the triage geofence (or the user) marked + # skipped is refused here, BEFORE make_room_for, so a clip we won't keep can + # never evict existing footage. The browser already drops these via /present; + # this is the server-side backstop. To force one in, unskip it in the Queue + # tab first (that clears the skipped state). + if _queue.skip_listed_names(db, [name]): + return {"status": "skipped", "filename": name} + # Evict to fit BEFORE writing bytes (size known from the header). # Off the loop: in quota mode this walks the whole archive (and # may delete files) — seconds of blocking I/O on a NAS. diff --git a/web/routers/progress.py b/web/routers/progress.py index 7c6d1e4..b69c834 100644 --- a/web/routers/progress.py +++ b/web/routers/progress.py @@ -41,7 +41,8 @@ async def progress(ws: WebSocket) -> None: return hub = ws.app.state.hub - await hub.connect(ws) + if not await hub.connect(ws): + return # client vanished during the handshake try: while True: # Read messages just to detect disconnects; we @@ -49,5 +50,12 @@ async def progress(ws: WebSocket) -> None: await ws.receive_text() except WebSocketDisconnect: pass + except RuntimeError: + # A hub broadcast can fail-send while we're parked in + # receive_text(), which makes starlette mark the socket + # DISCONNECTED under our feet; the loop's next state + # check then raises RuntimeError rather than + # WebSocketDisconnect. Same meaning: client is gone. + pass finally: await hub.disconnect(ws) diff --git a/web/routers/queue.py b/web/routers/queue.py index ffed8db..66f8620 100644 --- a/web/routers/queue.py +++ b/web/routers/queue.py @@ -108,6 +108,21 @@ def prioritize(body: Prioritize, request: Request) -> dict: return {"ok": True, "updated": n} +class DownloadNext(BaseModel): + filenames: List[str] = Field(default_factory=list) + + +@router.post("/queue/download-next", dependencies=[Depends(require_csrf)]) +def download_next(body: DownloadNext, request: Request) -> dict: + n = q.download_next(request.app.state.db, body.filenames) + q.emit_queue_changed(request.app.state.db, request.app.state.hub) + # Kick the worker so the prioritized clips download right away. + worker = getattr(request.app.state, "sync_worker", None) + if worker is not None: + worker.kick() + return {"ok": True, "updated": n} + + class Retry(BaseModel): # Omit/empty to retry every failed file; otherwise retry just these. filenames: List[str] = Field(default_factory=list) @@ -141,6 +156,37 @@ class Unskip(BaseModel): filenames: List[str] = Field(default_factory=list) +class Lock(BaseModel): + filenames: List[str] = Field(default_factory=list) + + +class DeleteClips(BaseModel): + filenames: List[str] = Field(default_factory=list) + + +class DeleteFromCamera(BaseModel): + filenames: List[str] = Field(default_factory=list) + + +@router.post("/queue/delete-from-camera", dependencies=[Depends(require_csrf)]) +def delete_from_camera_route(body: DeleteFromCamera, request: Request) -> dict: + snap = request.app.state.settings_provider.get() + addr = snap.address + if not addr: + return {"ok": False, "error": "no dashcam address configured"} + res = q.delete_from_camera(request.app.state.db, body.filenames, f"http://{addr}") + q.emit_queue_changed(request.app.state.db, request.app.state.hub) + return {"ok": True, **res} + + +@router.post("/queue/delete", dependencies=[Depends(require_csrf)]) +def delete_clips(body: DeleteClips, request: Request) -> dict: + recordings = request.app.state.settings_provider.get().recordings + res = q.delete_clips(request.app.state.db, body.filenames, recordings) + q.emit_queue_changed(request.app.state.db, request.app.state.hub) + return {"ok": True, **res} + + @router.post("/queue/unskip", dependencies=[Depends(require_csrf)]) def unskip(body: Unskip, request: Request) -> dict: n = q.unskip(request.app.state.db, body.filenames) @@ -152,6 +198,13 @@ def unskip(body: Unskip, request: Request) -> dict: return {"ok": True, "updated": n} +@router.post("/queue/lock", dependencies=[Depends(require_csrf)]) +def lock(body: Lock, request: Request) -> dict: + n = q.set_locked(request.app.state.db, body.filenames, True) + q.emit_queue_changed(request.app.state.db, request.app.state.hub) + return {"ok": True, "updated": n} + + @router.get("/sync/status") def sync_status(request: Request) -> dict: worker = getattr(request.app.state, "sync_worker", None) diff --git a/web/routers/settings.py b/web/routers/settings.py index 0b63e0d..f1cecc7 100644 --- a/web/routers/settings.py +++ b/web/routers/settings.py @@ -38,10 +38,17 @@ def _editable_values(snap) -> dict[str, Any]: "GROUPING": snap.grouping, "HTML": snap.use_html_listing, "GPS_EXTRACT": snap.gps_extract, + "GPS_TRIAGE": snap.gps_triage, "DERIVE_THUMBS_EAGER": snap.derive_thumbs_eager, "DERIVE_FILMSTRIPS_EAGER": snap.derive_filmstrips_eager, "DELETE_AFTER_DOWNLOAD": snap.delete_after_download, "SYNC_RO_ONLY": snap.sync_ro_only, + "LOCATIONS": [ + {"name": p.name, "lat": p.lat, "lon": p.lon, + "radius_m": p.radius_m, "exclude_recordings": p.exclude_recordings, + "is_home": p.is_home} + for p in snap.locations + ], "RETENTION_MAX_DAYS": snap.retention_max_days, "RETENTION_DISK_PCT": snap.retention_disk_pct, "RETENTION_PROTECT_RO": snap.retention_protect_ro, diff --git a/web/services/day_tracks.py b/web/services/day_tracks.py new file mode 100644 index 0000000..36ae5c9 --- /dev/null +++ b/web/services/day_tracks.py @@ -0,0 +1,117 @@ +"""Shared GPX path gathering for a day. + +The archive route view and the geofence evaluator both need "the GPX tracks +for this day": downloaded clips' real ``.gpx`` sidecars plus the ``.triage/`` +skeletons of queued, not-yet-downloaded clips. Defining it once keeps the +geofence's stop detection aligned with what the journey map shows. + +The route view wants the in-flight skeletons (``pending``/``failed``/ +``downloading``) so every clip shown as a grid tile has a matching track on the +map; skipped clips drop off the map unless the archive's opt-in "GPS-excluded" +view adds geofence-skipped skeletons back via +:func:`geofence_skipped_gpx_paths`. The geofence evaluator passes a broader +``queue_states`` so a clip already auto-skipped still contributes its track to +dwell detection. +""" +from __future__ import annotations + +import os +from typing import Sequence + +from ..db import Database +from . import triage as triage_service +from .naming import day_key_sql + + +def day_gpx_paths( + db: Database, + recordings: str, + date: str, + queue_states: Sequence[str] = ("pending", "failed", "downloading"), +) -> list[str]: + """Return the GPX paths for ``date`` (``YYYY-MM-DD``): downloaded sidecars + from ``clip_index`` plus triage skeletons for queued clips in + ``queue_states`` (with a GPS fix), de-duped against downloaded clips by + basename.""" + with db.conn() as c: + rows = c.execute( + "SELECT path FROM clip_index " + "WHERE group_name = ? AND has_gpx = 1 " + "ORDER BY timestamp ASC", + (date,), + ).fetchall() + gpx_paths = [r["path"] + ".gpx" for r in rows] + downloaded_names = {os.path.basename(r["path"]) for r in rows} + + placeholders = ",".join("?" * len(queue_states)) + day_expr = day_key_sql() + with db.conn() as c: + q_rows = c.execute( + f"SELECT filename FROM download_queue " + f"WHERE state IN ({placeholders}) " + f" AND triaged_at IS NOT NULL AND gps_points > 0 " + f" AND {day_expr} = ?", + (*queue_states, date), + ).fetchall() + for qr in q_rows: + if qr["filename"] in downloaded_names: + continue + sk = triage_service.skeleton_path(recordings, qr["filename"]) + if os.path.exists(sk): + gpx_paths.append(sk) + return gpx_paths + + +def geofence_skipped_gpx_paths( + db: Database, recordings: str, date: str, +) -> list[str]: + """Triage-skeleton paths for ``date``'s geofence-skipped clips + (``state='skipped' AND skip_reason='geofence'``, with a GPS fix). + Powering the archive's opt-in "GPS-excluded" view; kept separate from + :func:`day_gpx_paths` so the defaults every other caller relies on + (geofence evaluator included) cannot drift. User-skipped clips are + never returned — a manual skip means "don't show me this".""" + day_expr = day_key_sql() + with db.conn() as c: + rows = c.execute( + f"SELECT filename FROM download_queue " + f"WHERE state = 'skipped' AND skip_reason = 'geofence' " + f" AND triaged_at IS NOT NULL AND gps_points > 0 " + f" AND {day_expr} = ?", + (date,), + ).fetchall() + out: list[str] = [] + for r in rows: + sk = triage_service.skeleton_path(recordings, r["filename"]) + if os.path.exists(sk): + out.append(sk) + return out + + +def day_parking_spans(db: Database, date: str) -> list[tuple[float, float]]: + """``(start, end)`` spans for ``date``'s parking-mode clips, unioned across + downloaded clips (``clip_index``, with their real ``duration_s``) and + queued-but-not-yet-downloaded clips (``download_queue``, a *point* span at + ``recorded_at`` since the queue has no duration). Feeds the journey-window + parking bound (``gps.expand_journey_window``) so the "car is parked" hard + edge is honoured even before the parking clips are downloaded — the common + case under GPS triage, where the day's parking clips live only in the queue. + Overlapping/duplicate spans are harmless: the consumer reduces with min/max. + """ + day_expr = day_key_sql() + spans: list[tuple[float, float]] = [] + with db.conn() as c: + for r in c.execute( + "SELECT timestamp, duration_s FROM clip_index " + "WHERE group_name = ? AND event_type = 'parking'", + (date,), + ): + spans.append((r["timestamp"], r["timestamp"] + (r["duration_s"] or 0.0))) + for r in c.execute( + f"SELECT recorded_at FROM download_queue " + f"WHERE event_type = 'parking' AND recorded_at IS NOT NULL " + f" AND {day_expr} = ?", + (date,), + ): + spans.append((r["recorded_at"], r["recorded_at"])) + return spans diff --git a/web/services/exporter.py b/web/services/exporter.py index dc2e65e..0548fc6 100644 --- a/web/services/exporter.py +++ b/web/services/exporter.py @@ -1171,24 +1171,55 @@ async def _run_timeline( with self.db.conn() as c: rows = c.execute( """ - SELECT path, camera, timestamp, duration_s + SELECT id, path, camera, timestamp, duration_s FROM clip_index WHERE timestamp < ? - AND timestamp + COALESCE(duration_s, 0) > ? + AND ( + timestamp + COALESCE(duration_s, 0) > ? + OR (duration_s IS NULL AND timestamp > ?) + ) """, - (hi, lo), + (hi, lo, lo - _TIMELINE_PROTECT_MARGIN_S), ).fetchall() + + # A clip whose duration_s hasn't been filled in yet — the async + # duration sweep lags indexing, most visibly right after a bulk + # SD-card import — is invisible to build_switch_pieces, which treats + # an unknown duration as zero-length and would drop the selection's + # only footage ("no footage in selection"). The NULL branch in the + # query above reaches back one clip-length so we can probe those + # candidates on demand and persist the result, rather than failing an + # export of footage that is actually present. + probed: dict[int, Optional[float]] = {} + for r in rows: + if r["duration_s"] is None: + probed[r["id"]] = await durations.probe_and_store( + self.db, r["id"], r["path"], + ) clips = [ { "path": r["path"], "channel": channel_of(r["camera"]), "start_ts": r["timestamp"], - "duration_s": r["duration_s"], + "duration_s": ( + probed[r["id"]] if r["duration_s"] is None + else r["duration_s"] + ), } for r in rows ] pieces = build_switch_pieces(segments, clips) if not pieces: + # Leave a breadcrumb: the bare "no footage" job error is otherwise + # unlogged, so a failed timeline export is invisible in the logs. + log.warning( + "timeline export %s: no footage in selection [%.0f, %.0f] — " + "%d candidate clip(s), channels=%s, %d with unknown duration " + "after probing", + job["id"], lo, hi, len(clips), + sorted({c["channel"] for c in clips}), + sum(1 for c in clips if not c["duration_s"]), + ) self._finish(job["id"], False, "no footage in selection", None) return diff --git a/web/services/geofence.py b/web/services/geofence.py new file mode 100644 index 0000000..047c361 --- /dev/null +++ b/web/services/geofence.py @@ -0,0 +1,124 @@ +"""GPS geofence exclusion — auto-skip queued clips parked at home. + +Reuses the stop detector in :mod:`web.services.gps` (a stop is already a +>= 5-min dwell) over a day's GPX tracks. A *home stop* is a detected stop +whose centre lies within a configured zone's radius; the queued clips whose +``recorded_at`` falls inside a home stop are marked ``skipped``/``geofence``. + +Pure-ish: detection here, mutations delegated to :mod:`web.services.queue`. +Off / inert unless at least one location is flagged for exclusion (and +GPS_TRIAGE, checked by the caller — without triaged skeletons there are no +tracks). +""" +from __future__ import annotations + +import logging +from collections.abc import Sequence + +from ..db import Database +from . import day_tracks, gps +from . import queue as q +from . import triage as triage_service + +log = logging.getLogger("viofosync.geofence") + +# Geofence detection considers a clip's track even after it has been +# auto-skipped, so re-evaluation still sees the full home dwell. Shared with +# the orphan sweep so skipped skeletons aren't deleted out from under us. +_DETECT_STATES = triage_service.SKELETON_KEEP_STATES + +# A clip's recorded_at (filename second) precedes its first GPS fix by the +# receiver's acquisition lag (≈1 s driving, tens of seconds for a parking clip +# that re-acquires). A home stop's window starts at that first fix, so the +# dwell's leading clip lands just before it. Pad the leading edge by one +# nominal clip length to pull it in. Larger would risk swallowing the +# preceding pull-in drive clip (which sits a full clip + lag earlier). +LEADING_EDGE_PAD_S = 60 + + +def home_stops(stops: Sequence[gps.Stop], zones: Sequence) -> list[gps.Stop]: + """Stops whose centre is within any zone's radius (metres).""" + out: list[gps.Stop] = [] + for s in stops: + for z in zones: + if gps._haversine_ll( + s.center_lat, s.center_lon, z.lat, z.lon + ) <= z.radius_m: + out.append(s) + break + return out + + +def evaluate_day(db: Database, recordings: str, date: str, zones: Sequence) -> list[str]: + """Auto-skip queued clips on ``date`` that dwell inside a home zone. + Returns the filenames skipped (empty when no zones / no home stop).""" + if not zones: + return [] + candidates = q.geofence_candidates(db, date) + if not candidates: + return [] + paths = day_tracks.day_gpx_paths( + db, recordings, date, queue_states=_DETECT_STATES + ) + _points, stops, journeys = gps.aggregate_day(paths) + home = home_stops(stops, zones) + if not home: + return [] + windows = [(s.start_time.timestamp(), s.end_time.timestamp()) for s in home] + # A clip that's part of a drive must not be skipped even if its timestamp + # dwells in a home zone — the journey wins over the home dwell, mirroring the + # archive grid's "journey beats stop" grouping. Use the same padded journey + # windows (parking-bounded via gps.expand_journey_window) so the pull-away / + # pull-in clips at the dwell edge survive; only genuinely-parked clips skip. + parking = day_tracks.day_parking_spans(db, date) + drives = [ + gps.expand_journey_window( + j.start_time.timestamp(), j.end_time.timestamp(), parking + ) + for j in journeys + ] + + def _in_drive(ts: float) -> bool: + return any(lo <= ts <= hi for lo, hi in drives) + + to_skip = [ + c["filename"] + for c in candidates + if any( + lo - LEADING_EDGE_PAD_S <= c["recorded_at"] <= hi + for lo, hi in windows + ) + and not _in_drive(c["recorded_at"]) + ] + if to_skip: + q.geofence_skip(db, to_skip) + return to_skip + + +def sweep_all( + db: Database, recordings: str, zones: Sequence, *, seen: dict | None = None, +) -> int: + """Evaluate every day that currently has pending clips. Returns the total + number auto-skipped. Used for the per-cycle pass and the on-enable backfill. + + ``seen`` (optional) is a caller-owned ``{day: signature}`` cache. When + given, a day is only re-evaluated if its triaged-skeleton signature changed + since the cached value, so steady-state ticks skip the GPX re-parse. When + ``None`` (a full sweep), every pending day is evaluated.""" + if not zones: + return 0 + sigs = ( + q.geofence_day_signatures(db, _DETECT_STATES) + if seen is not None else None + ) + total = 0 + for day in q.pending_days(db): + if seen is not None: + sig = sigs.get(day, 0) + if seen.get(day) == sig: + continue + seen[day] = sig + total += len(evaluate_day(db, recordings, day, zones)) + if total: + log.info("geofence: auto-skipped %d clip(s) parked at home", total) + return total diff --git a/web/services/gps.py b/web/services/gps.py index 8bb521a..21b1049 100644 --- a/web/services/gps.py +++ b/web/services/gps.py @@ -38,6 +38,15 @@ # parked elsewhere) can't be drawn as a single journey. SESSION_GAP_SECONDS = 1800 # 30 minutes +# Invariant relied on by the archive's journey reframe: a confirmed stop is +# >= MIN_STOP_DURATION_S (300 s) and journeys are separated by a stop or a +# session gap (1800 s). expand_journey_window pads each edge by at most +# MAX_JOURNEY_BUFFER_S (120 s), and 120 + 120 < 300, so two journeys' padded +# windows can never overlap. If MIN_STOP_DURATION_S ever drops below +# 2 * MAX_JOURNEY_BUFFER_S, the reframe needs a per-journey overlap clamp. +# (Locked by tests/test_archive_group_window.py:: +# test_adjacent_journey_windows_never_overlap.) + @dataclass class Point: @@ -415,3 +424,43 @@ def aggregate_day( all_journeys.extend(journeys) return merged, all_stops, all_journeys + + +# Journey-window buffer. GPS stop boundaries land ~STOP_RADIUS_M (50m) inside the +# real drive, so the pull-away clip at the start and the pull-in clip at the end +# sit outside the raw journey window — "missing the start" / "cut short on +# arrival". Pad each edge outward, but no further than the nearest parking-mode +# clip on that side (the genuine "car is parked" boundary) and never more than +# this cap. The cap matters because the dashcam can be knocked out of parking +# mode by the car's electrics waking, producing spurious driving clips — so we +# don't trust an arbitrary parking->driving switch to mark the journey edge; we +# only lean on a *parking* clip as a hard stop. +MAX_JOURNEY_BUFFER_S = 120.0 +# Cold-start residual: this pad moves the journey's *time* window, but the +# trace/markers can only follow GPS fixes that exist. A journey whose GPS fix +# came a minute after power-on stays anchored at that first fix; the grid may +# still reach an earlier clip via the padded window. That gap is expected. + + +def expand_journey_window( + start_ts: float, end_ts: float, + parking_spans: list[tuple[float, float]], +) -> tuple[float, float]: + """Pad a GPS journey window ``[start_ts, end_ts]`` outward to catch the + pull-away / pull-in footage that sits just outside the GPS stop radius. + + Each edge expands by at most ``MAX_JOURNEY_BUFFER_S``, but stops at the + nearest parking-mode clip on that side so we never swallow parked footage. + ``parking_spans`` is ``[(start_ts, end_ts), ...]`` for the day's parking + clips. A parking clip straddling an edge means we're already at the + boundary, so that edge doesn't expand at all. See ``MAX_JOURNEY_BUFFER_S`` + for why the cap, not the mode switch, is the backstop. + """ + new_start = start_ts - MAX_JOURNEY_BUFFER_S + new_end = end_ts + MAX_JOURNEY_BUFFER_S + for ps, pe in parking_spans: + if ps < start_ts: # parking (partly) before the window + new_start = max(new_start, min(pe, start_ts)) + if pe > end_ts: # parking (partly) after the window + new_end = min(new_end, max(ps, end_ts)) + return new_start, new_end diff --git a/web/services/hub.py b/web/services/hub.py index 7c5a685..8f66474 100644 --- a/web/services/hub.py +++ b/web/services/hub.py @@ -81,9 +81,18 @@ def __init__(self, settings_provider: Any = None, session: Any = None) -> None: # is "error". "sync_status": None, "sync_status_reason": None, + # GPS triage progress substate (see triage.py / sync_status.py). + # active=False when no pass is running. + "triage": {"active": False}, } - async def connect(self, ws: WebSocket) -> None: + async def connect(self, ws: WebSocket) -> bool: + """Accept the socket and send the state snapshot. Returns + False when the client vanished during the handshake — the + socket is already marked disconnected by starlette at that + point, so the caller must not enter its receive loop + (receive_text on it raises RuntimeError, not + WebSocketDisconnect).""" await ws.accept() async with self._lock: self._clients.add(ws) @@ -93,15 +102,14 @@ async def connect(self, ws: WebSocket) -> None: ) except (WebSocketDisconnect, RuntimeError, OSError): # Client closed during the handshake (e.g. tab hot- - # reloaded between accept and the first send). The - # route's finally-clause will remove us from - # _clients via disconnect(); no need to raise out of - # the route handler as a 500. + # reloaded between accept and the first send). log.debug( "client disconnected before initial snapshot", ) async with self._lock: self._clients.discard(ws) + return False + return True async def disconnect(self, ws: WebSocket) -> None: async with self._lock: @@ -154,6 +162,16 @@ async def broadcast(self, event: Dict[str, Any]) -> None: pct = event.get("pct") if isinstance(pct, (int, float)): self.last_state["disk_pct"] = float(pct) + elif t == "triage_progress": + if event.get("active"): + self.last_state["triage"] = { + "active": True, + "triaged": event.get("triaged"), + "total": event.get("total"), + "eta_s": event.get("eta_s"), + } + else: + self.last_state["triage"] = {"active": False} # Feed the session tracker (reads the event; mutates the tracker). self._feed_session(t, event) diff --git a/web/services/importer.py b/web/services/importer.py index 76a5990..a9911fd 100644 --- a/web/services/importer.py +++ b/web/services/importer.py @@ -19,6 +19,7 @@ from dataclasses import asdict, dataclass, field import viofosync_lib as vfs +from viofosync_lib.cameras import GPS_CAMERA_LETTER from ..db import Database from . import retention as _retention @@ -78,7 +79,9 @@ def scan_item_from_match( int(m.group("year")), int(m.group("month")), int(m.group("day")), int(m.group("hour")), int(m.group("minute")), int(m.group("second")), ).timestamp()) - cam = m.group("camera") + # Compact suffix-less names have an empty camera group; the sole lens + # is the GPS-bearing one (mirrors scanner._clip_meta_for). + cam = m.group("camera") or GPS_CAMERA_LETTER return ScanItem( src_path=src_path, source_rel_path=source_rel_path, basename=name, timestamp=ts, camera=cam.upper(), sequence=int(m.group("sequence")), diff --git a/web/services/locations.py b/web/services/locations.py new file mode 100644 index 0000000..ae34bde --- /dev/null +++ b/web/services/locations.py @@ -0,0 +1,33 @@ +"""Named locations (e.g. "Home"). + +A location is any object exposing ``name``, ``lat``, ``lon``, ``radius_m`` (and, +for exclusion, ``exclude_recordings``; and ``is_home`` for the designated Home) +— the ``Place`` snapshot dataclass. Two consumers: the archive label override +(``match_for_point``, with ``name_for_point`` as a name-only convenience wrapper) +and the geofence exclusion (``exclusion_zones``). Pure; no DB, no settings import. +""" +from __future__ import annotations + +from collections.abc import Sequence + +from . import gps + + +def match_for_point(places: Sequence, lat: float, lon: float): + """First place whose centre is within its radius (metres) of (lat, lon), + else None. The returned object exposes ``.name`` and ``.is_home``.""" + for p in places: + if gps._haversine_ll(p.lat, p.lon, lat, lon) <= p.radius_m: + return p + return None + + +def name_for_point(places: Sequence, lat: float, lon: float) -> str | None: + """Name of the first place matching (lat, lon), else None.""" + p = match_for_point(places, lat, lon) + return p.name if p is not None else None + + +def exclusion_zones(places: Sequence) -> list: + """Places flagged ``exclude_recordings`` — the geofence's input zones.""" + return [p for p in places if p.exclude_recordings] diff --git a/web/services/mqtt_state.py b/web/services/mqtt_state.py index 3cf53a1..8134698 100644 --- a/web/services/mqtt_state.py +++ b/web/services/mqtt_state.py @@ -64,11 +64,21 @@ def state_sync_status(hub, db, snapshot) -> Optional[str]: def attrs_sync_status(hub, db, snapshot) -> Optional[dict]: - """JSON attributes payload for the sync_status sensor. Always - returns a dict with a ``reason`` key so HA templating doesn't have - to guard for a missing attribute.""" + """JSON attributes payload for the sync_status sensor. Always returns a + dict with ``reason`` and ``triage_active`` so HA templating needn't guard + for missing keys. When a triage pass is running, the progress fields are + included so automations can read it.""" _state, reason = compute_sync_status(hub, db, snapshot) - return {"reason": reason} + attrs: dict = {"reason": reason, "triage_active": False} + triage = (getattr(hub, "last_state", None) or {}).get("triage") or {} + if triage.get("active"): + attrs.update( + triage_active=True, + triaged=triage.get("triaged"), + triage_total=triage.get("total"), + triage_eta_s=triage.get("eta_s"), + ) + return attrs # ---- queue counts diff --git a/web/services/mqtt_topology.py b/web/services/mqtt_topology.py index ec0fc63..38ac9dd 100644 --- a/web/services/mqtt_topology.py +++ b/web/services/mqtt_topology.py @@ -169,6 +169,7 @@ def build_discovery_payload(entity: EntityDef, cfg: dict) -> dict: "queue_changed", "dashcam_online", "dashcam_offline", "disk_pct", "sync_error", + "triage_progress", ), attrs_fn=_st.attrs_sync_status, ), diff --git a/web/services/naming.py b/web/services/naming.py index ab94155..70567c8 100644 --- a/web/services/naming.py +++ b/web/services/naming.py @@ -27,7 +27,9 @@ from viofosync_lib.cameras import ( # noqa: F401 — re-exported CAMERAS, + GPS_CAMERA_LETTER, channel_of, + is_gps_camera, pair_slot_of, ) @@ -126,6 +128,86 @@ def export_download_name( return f"{build_basename(clips, label)}.mp4" +# --- Filename-derived SQL fragments --------------------------------------- +# +# Two on-disk filename layouts exist (see docs/ARCHITECTURE.md): +# standard: ``YYYY_MMDD_HHMMSS_NNNN[PE]?.MP4`` +# compact: ``YYYYMMDDHHMMSS_NNNNNN.MP4`` — some single-channel units list +# recordings with no datetime separators and no camera suffix; +# the sole lens is the GPS-bearing one. +# Discriminators: standard has ``_`` at position 5 and a camera letter at -5; +# compact has digits in both places. The GPS letter is interpolated rather +# than bound (it is a one-character registry constant, not user input) so the +# fragments carry no SQL parameters — callers can't get param order wrong. + + +def suffixless_sql(col: str = "filename") -> str: + """SQL: the name is compact/suffix-less (a digit where the camera + letter would sit).""" + return f"substr({col}, -5, 1) BETWEEN '0' AND '9'" + + +def gps_lens_sql(col: str = "filename") -> str: + """SQL: the clip is the GPS-bearing lens — the registry's GPS letter, + or a suffix-less single-channel name (which IS that lens).""" + return ( + f"(upper(substr({col}, -5, 1)) = '{GPS_CAMERA_LETTER}' " + f"OR {suffixless_sql(col)})" + ) + + +def camera_letter_sql(col: str = "filename") -> str: + """SQL: the clip's camera letter; suffix-less names default to the + GPS lens.""" + return ( + f"CASE WHEN {suffixless_sql(col)} THEN '{GPS_CAMERA_LETTER}' " + f"ELSE upper(substr({col}, -5, 1)) END" + ) + + +def day_key_sql(col: str = "filename") -> str: + """SQL: the clip's ``YYYY-MM-DD`` day key from the filename, for both + layouts. Derived from the name rather than ``recorded_at`` so grouping + is stable for rows missing a timestamp and immune to unixepoch/localtime + conversion drift.""" + return ( + f"CASE WHEN substr({col}, 5, 1) = '_' " + f"THEN substr({col}, 1, 4) || '-' || substr({col}, 6, 2) " + f" || '-' || substr({col}, 8, 2) " + f"ELSE substr({col}, 1, 4) || '-' || substr({col}, 5, 2) " + f" || '-' || substr({col}, 7, 2) END" + ) + + +def gps_sibling_sql(col: str = "f.filename") -> str: + """Correlated SQL: ``col`` names the GPS-bearing sibling of the row + aliased ``dq``. Same-capture lenses share the filename's 16-char + timestamp prefix but NOT necessarily the sequence number (parking + captures give each lens its own), so the sibling is matched by prefix + range — usable with an index on ``col`` — never by rebuilding a sibling + filename from the stem. For a compact name the prefix spans the + timestamp + first sequence digit and the only possible match is the row + itself: a compact clip is its own GPS sibling.""" + return ( + f"{col} >= substr(dq.filename, 1, 16)" + f" AND {col} < substr(dq.filename, 1, 16) || '~'" + f" AND {gps_lens_sql(col)}" + ) + + +def capture_key_sql(col: str = "filename") -> str: + """SQL: the clip's 14-digit ``YYYYMMDDHHMMSS`` capture key, normalized + across both filename layouts so ``MAX()`` and grouping are layout-safe + (raw prefixes don't collate — ``'_'`` sorts after ``'9'``). Same-capture + lenses share this key; sequence numbers do not affect it.""" + return ( + f"CASE WHEN substr({col}, 5, 1) = '_' " + f"THEN substr({col}, 1, 4) || substr({col}, 6, 2) " + f" || substr({col}, 8, 2) || substr({col}, 11, 6) " + f"ELSE substr({col}, 1, 14) END" + ) + + # --- Timeline camera channels ------------------------------------------- # Channel keys/labels come straight from the registry; "other" is the diff --git a/web/services/queue.py b/web/services/queue.py index d857cf2..c3dfcb1 100644 --- a/web/services/queue.py +++ b/web/services/queue.py @@ -20,13 +20,71 @@ from __future__ import annotations +import logging import time from dataclasses import dataclass -from typing import Iterable, List, Optional +from typing import Iterable, List, Optional, Sequence -from viofosync_lib.cameras import CAMERA_LETTERS +import viofosync_lib as vfs +from viofosync_lib.cameras import ( + CAMERA_LETTERS, + GPS_CAMERA_LETTER, + is_gps_camera, +) from ..db import Database +from .naming import camera_letter_sql, capture_key_sql, day_key_sql, gps_sibling_sql +from .triage import TRIAGE_MAX_ATTEMPTS + +# INFO-level here is persisted to the app_log table by DBLogHandler (the +# "viofosync.*" namespace is captured at INFO) — so user-initiated archive +# mutations (delete/skip/unskip/retry/download-next/prioritize) leave an +# audit trail in the activity log, not just the console. +log = logging.getLogger("viofosync.queue") + + +def _names(filenames: List[str], limit: int = 20) -> str: + """Compact, log-friendly rendering of a filename list (truncated).""" + shown = ", ".join(filenames[:limit]) + extra = len(filenames) - limit + return f"{shown} (+{extra} more)" if extra > 0 else shown + + +def _gps_state(d: dict) -> Optional[str]: + """Derive the per-file GPS indicator from triage columns. + + 'ok' = GPS fetched (gps_points > 0) + 'none' = triaged with no fix, OR gave up after MAX unreadable attempts + 'pending' = still awaiting triage + None = non-GPS lens with no GPS sibling at its timestamp (orphan) — + there is no capture-level GPS fact to imply + + GPS is a capture-level fact carried by one lens (the front): that row's + own triage columns speak for it, and the other lenses (rear/tele/interior, + which are never triaged themselves) *inherit* the state from their GPS + sibling's columns — selected as ``sib_*`` by the listing queries via + ``GPS_SIBLING_SQL``. + + Camera identity is taken from the filename via the registry, not the + ``camera`` column, so historical rows with a NULL ``camera`` still resolve. + """ + if is_gps_camera(_camera_from_filename(d.get("filename") or "")): + triaged_at = d.get("triaged_at") + gps_points = d.get("gps_points") + attempts = d.get("triage_attempts") + elif d.get("sib_id") is not None: + triaged_at = d.get("sib_triaged_at") + gps_points = d.get("sib_gps_points") + attempts = d.get("sib_triage_attempts") + else: + return None + if (gps_points or 0) > 0: + return "ok" + if triaged_at is not None: + return "none" # triaged, no GPS fix + if (attempts or 0) >= TRIAGE_MAX_ATTEMPTS: + return "none" # gave up after MAX unreadable attempts + return "pending" @dataclass @@ -183,18 +241,27 @@ def reconcile( } +# Compact single-channel names (``YYYYMMDDHHMMSS_NNNNNN.MP4``) carry no +# event prefix or camera suffix; the sole lens is the GPS-bearing one. +_COMPACT_FILENAME_RE = r"^\d{14}_\d+\.MP4$" + + def _camera_from_filename(filename: str) -> Optional[str]: - # Handles both ``…_0001F.MP4`` and ``…_0001PF.MP4`` / - # ``…_0001EF.MP4`` — the optional prefix letter encodes the - # event type (P=parking, E=event); the camera letter set - # comes from the registry. + # Handles ``…_0001F.MP4`` and ``…_0001PF.MP4`` / ``…_0001EF.MP4`` — + # the optional prefix letter encodes the event type (P=parking, + # E=event); the camera letter set comes from the registry. Compact + # suffix-less names default to the GPS-bearing lens. import re as _re m = _re.match( rf"^\d{{4}}_\d{{4}}_\d{{6}}_\d+[PE]?([{CAMERA_LETTERS}])\.MP4$", filename, _re.IGNORECASE, ) - return m.group(1).upper() if m else None + if m: + return m.group(1).upper() + if _re.match(_COMPACT_FILENAME_RE, filename, _re.IGNORECASE): + return GPS_CAMERA_LETTER + return None def _event_from_filename(filename: str) -> Optional[str]: @@ -204,36 +271,95 @@ def _event_from_filename(filename: str) -> Optional[str]: filename, _re.IGNORECASE, ) - if not m: - return None - prefix = (m.group(1) or "").upper() - return {"P": "parking", "E": "event"}.get(prefix, "normal") + if m: + prefix = (m.group(1) or "").upper() + return {"P": "parking", "E": "event"}.get(prefix, "normal") + if _re.match(_COMPACT_FILENAME_RE, filename, _re.IGNORECASE): + return "normal" + return None # SQL expressions for deriving camera / event type straight # from the filename. Used for filtering so we don't depend on # historical rows having ``camera`` / ``event_type`` populated. -# Filenames end in ``…NNNNN[PE]?[FRTI].MP4`` — the camera letter -# is the character immediately before ``.MP4``, and the byte -# before that is either a digit (normal) or P/E. -_CAM_SQL = "upper(substr(filename, -5, 1))" +# The camera letter comes from naming.camera_letter_sql (format-aware: +# suffix-less compact names default to the GPS lens); for standard names +# the byte before the letter is either a digit (normal) or P/E, and for +# compact names it is always a digit — which the CASE consumers already +# read as 'normal'. +_CAM_SQL = camera_letter_sql() _EVT_PREFIX_SQL = "upper(substr(filename, -6, 1))" +# Correlated match for a row's GPS-bearing sibling: ``f`` is the sibling +# candidate, ``dq`` the row being tested. See naming.gps_sibling_sql for the +# pairing rule (timestamp-prefix range, format-aware GPS-lens test; binds no +# parameters — the registry letter is interpolated). +GPS_SIBLING_SQL = gps_sibling_sql() + +# GPS-sibling join + columns for the listing queries: exposes the sibling's +# triage columns as ``sib_*`` so :func:`_gps_state` can imply a non-GPS lens's +# badge. Binds no parameters. Assumes one GPS-lens file per capture +# timestamp — the same uniqueness every pairing consumer relies on (get_day, +# the triage gate, remote_day_clips). +_SIB_COLS_SQL = ( + "f.id AS sib_id, f.triaged_at AS sib_triaged_at, " + "f.gps_points AS sib_gps_points, f.triage_attempts AS sib_triage_attempts" +) +_SIB_JOIN_SQL = f"LEFT JOIN download_queue f ON {GPS_SIBLING_SQL}" + +# The capture key of the newest non-gone queue row — the only capture that +# can still be actively recording. Its lenses are held until remote_complete=1. +_CAPTURE_KEY_SQL = capture_key_sql("dq.filename") +_NEWEST_CAPTURE_SQL = ( + "SELECT MAX(" + capture_key_sql("filename") + ") " + "FROM download_queue WHERE state <> 'gone'" +) + def next_pending( - db: Database, *, ro_only: bool = False, + db: Database, *, ro_only: bool = False, triage_gate: bool = False, + active_guard: bool = False, ) -> Optional[QueueItem]: - """Highest priority, oldest enqueue time. If ``ro_only`` is - set, only consider rows whose source_dir is under /RO/.""" - sql = ( - "SELECT * FROM download_queue " - "WHERE state='pending'" - ) + """Highest priority, oldest enqueue time. If ``ro_only`` is set, only + consider rows whose source_dir is under /RO/. + + If ``triage_gate`` is set (GPS_TRIAGE on), a row is held back while its + GPS-bearing sibling is still awaiting triage (``triaged_at IS NULL AND + triage_attempts < TRIAGE_MAX_ATTEMPTS``) — so we never download a clip, or + its paired lenses, ahead of triage. The sibling is the GPS-bearing lens + (letter from the registry) sharing the filename's timestamp prefix — NOT + the full stem: same-capture lenses share the recording second but can carry + different sequence numbers (see ``GPS_SIBLING_SQL``). The GPS lens is its + own sibling, and an orphan non-GPS clip (no GPS sibling) is not gated. + If ``active_guard`` is set, every lens of the newest capture group is held + until its ``remote_complete`` flag is set, since that capture may still be + recording.""" + sql = "SELECT dq.* FROM download_queue dq WHERE dq.state='pending'" + params: List[object] = [] if ro_only: - sql += " AND (source_dir LIKE '%/RO/%' OR source_dir LIKE '%/RO')" - sql += " ORDER BY priority DESC, enqueued_at ASC LIMIT 1" + sql += ( + " AND (dq.source_dir LIKE '%/RO/%' " + "OR dq.source_dir LIKE '%/RO')" + ) + if triage_gate: + sql += ( + " AND NOT EXISTS (" + f" SELECT 1 FROM download_queue f" + f" WHERE {GPS_SIBLING_SQL}" + f" AND f.state='pending'" + f" AND f.triaged_at IS NULL" + f" AND f.triage_attempts < ?" + " )" + ) + params.append(TRIAGE_MAX_ATTEMPTS) + if active_guard: + sql += ( + f" AND NOT ({_CAPTURE_KEY_SQL} = ({_NEWEST_CAPTURE_SQL})" + f" AND dq.remote_complete IS NULL)" + ) + sql += " ORDER BY dq.priority DESC, dq.enqueued_at ASC LIMIT 1" with db.conn() as c: - row = c.execute(sql).fetchone() + row = c.execute(sql, params).fetchone() if row is None: return None return QueueItem( @@ -423,35 +549,36 @@ def list_page( CASE WHEN dq.state = 'downloading' THEN 0 ELSE p.queue_position - END AS queue_position + END AS queue_position, + {_SIB_COLS_SQL} FROM download_queue dq LEFT JOIN positions p ON dq.id = p.id + {_SIB_JOIN_SQL} {where.replace("filename", "dq.filename") if where else ""} ORDER BY {order} LIMIT ? OFFSET ? """, params + [per_page, (page - 1) * per_page], ).fetchall() + items = [dict(r) for r in rows] + for d in items: + d["gps_state"] = _gps_state(d) return { "total": total, "page": page, "per_page": per_page, "sort_by": sort_by or "priority", "sort_dir": sort_dir, - "items": [dict(r) for r in rows], + "items": items, } def _day_expr() -> str: - """SQL expression for the YYYY-MM-DD day key derived from - the filename (``YYYY_MMDD_HHMMSS_NN.MP4``). Uses the - filename rather than ``recorded_at`` so grouping is - consistent even for rows missing a timestamp.""" - return ( - "substr(filename,1,4) || '-' || " - "substr(filename,6,2) || '-' || " - "substr(filename,8,2)" - ) + """SQL expression for the YYYY-MM-DD day key derived from the filename + (format-aware — see naming.day_key_sql). Uses the filename rather than + ``recorded_at`` so grouping is consistent even for rows missing a + timestamp.""" + return day_key_sql() _RO_SQL = "source_dir LIKE '%/RO/%'" @@ -568,7 +695,7 @@ def list_day_items( download order (priority + enqueued_at) so the client can show "next up" cues independent of display order. """ - day_expr = _day_expr() + day_expr = _day_expr().replace("filename", "dq.filename") clauses = [f"{day_expr} = ?"] params: List[object] = [day] if query: @@ -607,15 +734,20 @@ def list_day_items( WHEN 'E' THEN 'event' ELSE 'normal' END AS kind_event, - CASE WHEN {ro_dq} THEN 1 ELSE 0 END AS kind_ro + CASE WHEN {ro_dq} THEN 1 ELSE 0 END AS kind_ro, + {_SIB_COLS_SQL} FROM download_queue dq LEFT JOIN positions p ON dq.id = p.id + {_SIB_JOIN_SQL} {where} ORDER BY dq.filename DESC """, params, ).fetchall() - return [dict(r) for r in rows] + items = [dict(r) for r in rows] + for d in items: + d["gps_state"] = _gps_state(d) + return items def pending_bytes(db: Database) -> int: @@ -671,6 +803,9 @@ def prioritize( f"WHERE filename IN ({ph}) AND state='pending'", [target] + filenames, ) + if cur.rowcount: + log.info("archive prioritize (%s): %d clip(s) — %s", + position, cur.rowcount, _names(filenames)) return cur.rowcount @@ -685,6 +820,9 @@ def retry(db: Database, filenames: List[str]) -> int: f"WHERE filename IN ({ph}) AND state='failed'", filenames, ) + if cur.rowcount: + log.info("archive retry: %d failed clip(s) requeued — %s", + cur.rowcount, _names(filenames)) return cur.rowcount @@ -698,14 +836,63 @@ def skip(db: Database, filenames: List[str]) -> int: with db.write() as c: ph = ",".join("?" * len(filenames)) cur = c.execute( - f"UPDATE download_queue SET state='skipped' " + f"UPDATE download_queue SET state='skipped', skip_reason='user' " f"WHERE filename IN ({ph}) " f"AND state IN ('pending', 'failed')", filenames, ) + if cur.rowcount: + log.info("archive skip: %d clip(s) skipped — %s", + cur.rowcount, _names(filenames)) return cur.rowcount +def delete_clips(db: Database, filenames: List[str], recordings: str) -> dict: + """User-initiated delete: remove downloaded files + clip_index rows and mark + the queue rows skipped. Clips the user has pinned read-only (clip_index or + download_queue locked=1) or dashcam-locked (event_type='ro') are skipped and + reported as 'protected'. Returns {deleted, skipped, protected}.""" + if not filenames: + return {"deleted": 0, "skipped": 0, "protected": 0} + from . import retention as _retention + ph = ",".join("?" * len(filenames)) + with db.conn() as c: + protected = { + r["name"] for r in c.execute( + f"SELECT basename AS name FROM clip_index " + f"WHERE basename IN ({ph}) " + f"AND (COALESCE(locked,0)=1 OR COALESCE(event_type,'')='ro') " + f"UNION " + f"SELECT filename AS name FROM download_queue " + f"WHERE filename IN ({ph}) AND COALESCE(locked,0)=1", + [*filenames, *filenames], + ).fetchall() + } + targets = [f for f in filenames if f not in protected] + deleted = 0 + skipped = 0 + if targets: + tph = ",".join("?" * len(targets)) + with db.conn() as c: + rows = c.execute( + f"SELECT id, path, basename, event_type FROM clip_index " + f"WHERE basename IN ({tph})", targets, + ).fetchall() + for r in rows: + _retention.delete_clip(db, dict(r), recordings) + deleted += 1 + with db.write() as c: + cur = c.execute( + f"UPDATE download_queue SET state='skipped', skip_reason='user' " + f"WHERE filename IN ({tph})", targets, + ) + skipped = cur.rowcount # rows actually marked, not len(targets) + if deleted or skipped or protected: + log.info("archive delete: removed %d clip(s), %d protected — %s", + deleted, len(protected), _names(filenames)) + return {"deleted": deleted, "skipped": skipped, "protected": len(protected)} + + def unskip(db: Database, filenames: List[str]) -> int: """Return ``skipped`` files to ``pending`` for downloading again, resetting attempts/last_error for a fresh try (mirrors ``retry``). @@ -716,13 +903,201 @@ def unskip(db: Database, filenames: List[str]) -> int: ph = ",".join("?" * len(filenames)) cur = c.execute( f"UPDATE download_queue SET state='pending', " - f"attempts=0, last_error=NULL " + f"attempts=0, last_error=NULL, skip_reason=NULL, " + f"geofence_released_at=CASE WHEN skip_reason='geofence' " + f" THEN ? ELSE geofence_released_at END " f"WHERE filename IN ({ph}) AND state='skipped'", + [int(time.time())] + filenames, + ) + if cur.rowcount: + log.info("archive unskip: %d clip(s) returned to pending — %s", + cur.rowcount, _names(filenames)) + return cur.rowcount + + +def geofence_skip(db: Database, filenames: List[str]) -> int: + """Auto-skip the given clips as parked-at-home: ``pending`` → + ``skipped``/``geofence``. Never touches non-pending rows or clips the user + has permanently released (``geofence_released_at`` set). Returns the count.""" + if not filenames: + return 0 + with db.write() as c: + ph = ",".join("?" * len(filenames)) + cur = c.execute( + f"UPDATE download_queue SET state='skipped', skip_reason='geofence' " + f"WHERE filename IN ({ph}) AND state='pending' " + f"AND geofence_released_at IS NULL", filenames, ) return cur.rowcount +def unskip_geofence(db: Database) -> int: + """Reset every geofence auto-skipped clip back to ``pending`` so a fresh + sweep can re-evaluate it. Unlike :func:`unskip`, this deliberately does NOT + set ``geofence_released_at`` — we want the geofence to re-skip the genuine + home clips. Leaves user skips, user-released clips, and downloaded/other + rows untouched. Returns the count reset. Used by the maintenance flush.""" + with db.write() as c: + cur = c.execute( + "UPDATE download_queue SET state='pending', " + "attempts=0, last_error=NULL, skip_reason=NULL " + "WHERE state='skipped' AND skip_reason='geofence' " + "AND geofence_released_at IS NULL" + ) + return cur.rowcount + + +def geofence_candidates(db: Database, day: str) -> List[dict]: + """Pending, time-stamped clips on ``day`` eligible for geofence auto-skip. + + Excludes RO/locked clips and event recordings (the sparse, important + 'something happened while parked' footage) and clips a user has + permanently released. Returns ``[{filename, recorded_at}, ...]``.""" + day_expr = _day_expr() + with db.conn() as c: + rows = c.execute( + f"SELECT filename, recorded_at FROM download_queue " + f"WHERE {day_expr} = ? AND state='pending' " + f"AND recorded_at IS NOT NULL " + f"AND geofence_released_at IS NULL " + # RO clips can be stored with or without a trailing slash + # (see next_pending); exclude both forms. + f"AND source_dir NOT LIKE '%/RO/%' AND source_dir NOT LIKE '%/RO' " + f"AND {_EVT_PREFIX_SQL} <> 'E'", + (day,), + ).fetchall() + return [dict(r) for r in rows] + + +def geofence_day_signatures(db: Database, states: tuple) -> dict: + """Per-day count of clips that carry a GPS triage skeleton, across the + given queue ``states``. Monotonic as triage progresses (a skipped clip + stays counted), so an unchanged count means a day has no new detection + input — letting the geofence sweep skip re-parsing it. + + Returns ``{ 'YYYY-MM-DD': count, ... }``.""" + day_expr = _day_expr() + ph = ",".join("?" * len(states)) + with db.conn() as c: + rows = c.execute( + f"SELECT {day_expr} AS day, COUNT(*) AS n FROM download_queue " + f"WHERE state IN ({ph}) " + f" AND triaged_at IS NOT NULL AND gps_points > 0 " + f"GROUP BY day", + tuple(states), + ).fetchall() + return {r["day"]: r["n"] for r in rows} + + +def set_locked(db: Database, filenames: List[str], locked: bool = True) -> int: + """Set the user 'retain indefinitely' flag on the given clips, in BOTH + clip_index (by basename) and download_queue (by filename), so the state is + stable whether the clip is downloaded or still queued. Returns the count of + distinct filenames affected.""" + if not filenames: + return 0 + val = 1 if locked else 0 + ph = ",".join("?" * len(filenames)) + with db.write() as c: + c.execute( + f"UPDATE clip_index SET locked=? WHERE basename IN ({ph})", + [val, *filenames], + ) + c.execute( + f"UPDATE download_queue SET locked=? WHERE filename IN ({ph})", + [val, *filenames], + ) + verb = "read-only" if locked else "writable" + log.info("archive mark %s: %d clip(s) — %s", verb, len(filenames), + _names(filenames)) + return len(filenames) + + +def delete_from_camera( + db: Database, filenames: List[str], base_url: str, *, timeout: float = 10.0, +) -> dict: + """Delete the given clips from the dashcam SD card (cmd=4003). Skips clips + the user has locked (retain indefinitely) and dashcam RO clips (source_dir + under /RO/ — firmware write-protected). On a successful delete the queue row + becomes 'gone' (no longer on the camera). Network failures are counted, not + raised. Returns {deleted, skipped, errors}.""" + if not filenames: + return {"deleted": 0, "skipped": 0, "errors": 0} + ph = ",".join("?" * len(filenames)) + with db.conn() as c: + rows = c.execute( + f"SELECT filename, source_dir, locked FROM download_queue " + f"WHERE filename IN ({ph})", filenames, + ).fetchall() + deleted = skipped = errors = 0 + gone: List[str] = [] + for r in rows: + sd = r["source_dir"] or "" + if r["locked"] or "/RO/" in sd or sd.endswith("/RO"): + skipped += 1 + continue + if vfs.delete_dashcam_file(base_url, sd, r["filename"], timeout=timeout): + gone.append(r["filename"]) + deleted += 1 + else: + errors += 1 + if gone: + gph = ",".join("?" * len(gone)) + with db.write() as c: + c.execute( + f"UPDATE download_queue SET state='gone' WHERE filename IN ({gph})", + gone, + ) + log.info("delete from camera: %d deleted, %d skipped, %d error(s) — %s", + deleted, skipped, errors, _names(filenames)) + return {"deleted": deleted, "skipped": skipped, "errors": errors} + + +def skip_listed_names(db: Database, names: Sequence[str]) -> set[str]: + """Subset of ``names`` whose download_queue row is currently ``state='skipped'`` + (any ``skip_reason`` — geofence auto-skip or a user skip). Empty input → empty + set with no query. ``filename`` is UNIQUE-indexed, so the lookup is cheap. + + Single source of truth for the manual-import skip gate: a clip the triage + geofence (or the user) marked skipped is not re-imported via browser upload.""" + names = list(names) + if not names: + return set() + ph = ",".join("?" * len(names)) + with db.conn() as c: + rows = c.execute( + f"SELECT filename FROM download_queue " + f"WHERE state='skipped' AND filename IN ({ph})", + names, + ).fetchall() + return {r["filename"] for r in rows} + + +def pending_days(db: Database) -> List[str]: + """Distinct YYYY-MM-DD day keys that currently have ``pending`` clips.""" + day_expr = _day_expr() + with db.conn() as c: + rows = c.execute( + f"SELECT DISTINCT {day_expr} AS day FROM download_queue " + f"WHERE state='pending'" + ).fetchall() + return [r["day"] for r in rows] + + +def download_next(db: Database, filenames: List[str]) -> int: + """Re-queue the given clips for immediate download: un-skip and retry any + that were skipped/failed (both → ``pending``), then bump all of them to the + top of the queue. Returns the number prioritized. ``done`` clips are + untouched (``prioritize`` only affects ``pending``). Order matters: the + state moves must run before ``prioritize`` so the rows are ``pending``.""" + if not filenames: + return 0 + unskip(db, filenames) + retry(db, filenames) + return prioritize(db, filenames, "top") + + def retry_failed(db: Database) -> int: """Move every queue item in state ``failed`` back to ``pending``, resetting attempts. Returns the count updated.""" diff --git a/web/services/retention.py b/web/services/retention.py index f2be904..db02c6b 100644 --- a/web/services/retention.py +++ b/web/services/retention.py @@ -41,6 +41,8 @@ def _eligible_by_time( window or rule disabled), or ``'ro_protected'`` (no, RO clip and protection is on). """ + if clip.get("locked"): + return False, "locked" if protect_ro and (clip.get("event_type") or "") == "ro": return False, "ro_protected" if max_days <= 0: @@ -88,6 +90,15 @@ def _delete_index_row(db: Database, clip_id: int) -> None: c.execute("DELETE FROM clip_index WHERE id = ?", (clip_id,)) +def delete_clip(db: Database, rec: dict, recordings: str) -> int: + """Delete one downloaded clip's files (mp4 + .gpx + thumb/sprite/meta) and its + clip_index row. Returns bytes freed. Public wrapper over the same plumbing the + retention sweep uses, for the user-initiated archive delete.""" + freed = _delete_clip_files(rec, recordings) + _delete_index_row(db, rec["id"]) + return freed + + def _broadcast(sink, filename: str, reason: str) -> None: if sink is None: return @@ -129,7 +140,7 @@ def sweep( with db.conn() as c: rows = [ dict(r) for r in c.execute( - "SELECT id, path, basename, timestamp, event_type " + "SELECT id, path, basename, timestamp, event_type, locked " f"FROM clip_index WHERE timestamp < ?{guard_sql}", [now - max_days * 86400, *guard_params], ).fetchall() @@ -396,7 +407,7 @@ def _disk_pressure_pass( recordings, disk_pct=disk_pct, quota_gb=quota_gb, refresh=True, exclude=exclude, ): - conds = ["1=1"] + conds = ["COALESCE(locked, 0) = 0"] params: list = [] if protect_ro: conds.append("COALESCE(event_type, '') != 'ro'") @@ -480,7 +491,7 @@ def _over() -> bool: return True return False - where = "WHERE timestamp < ?" + where = "WHERE timestamp < ? AND COALESCE(locked, 0) = 0" params: list = [before_ts] if protect_ro: where += " AND COALESCE(event_type, '') != 'ro'" diff --git a/web/services/route_cache.py b/web/services/route_cache.py index 06b8a1f..a300c78 100644 --- a/web/services/route_cache.py +++ b/web/services/route_cache.py @@ -16,6 +16,7 @@ import hashlib import json import os +import shutil from typing import Any, Iterable, Optional @@ -25,6 +26,12 @@ def _cache_dir(recordings: str) -> str: return d +def clear_all(recordings: str) -> None: + """Drop the entire per-day route cache; journeys recompute from the on-disk + GPX on the next request. Best-effort and idempotent (no-op when absent).""" + shutil.rmtree(os.path.join(recordings, ".route_cache"), ignore_errors=True) + + def _cache_path(recordings: str, date: str) -> str: return os.path.join(_cache_dir(recordings), f"{date}.json") diff --git a/web/services/scanner.py b/web/services/scanner.py index 1f97114..fbac95a 100644 --- a/web/services/scanner.py +++ b/web/services/scanner.py @@ -25,6 +25,7 @@ from typing import Iterable, List import viofosync_lib as vfs +from viofosync_lib.cameras import GPS_CAMERA_LETTER from ..db import Database @@ -91,7 +92,10 @@ def _clip_meta_for( if not os.path.isfile(path): return None - camera_field = m.group("camera") + # Compact single-channel names have no camera suffix (empty group); + # the sole lens is the GPS-bearing one, so default to the registry's + # GPS letter. Event type falls out naturally: no P/E prefix → normal. + camera_field = m.group("camera") or GPS_CAMERA_LETTER return ClipMeta( path=path, basename=filename, @@ -256,6 +260,16 @@ def scan(db: Database, destination: str, grouping: str, hub=None, loop=None) -> c.execute("ROLLBACK") raise + # Carry a queued "retain indefinitely" lock into the freshly-indexed clip. + # Only sets locked 0->1 (never clears), so a lock set on the index after + # download — or a later unlock — is preserved across rescans. + with db.write() as c: + c.execute( + "UPDATE clip_index SET locked = 1 " + "WHERE locked = 0 AND basename IN " + "(SELECT filename FROM download_queue WHERE locked = 1)" + ) + if hub is not None: event = {"type": "clip_indexed", "total": len(seen_paths)} try: diff --git a/web/services/sync_status.py b/web/services/sync_status.py index 0629549..fbd245e 100644 --- a/web/services/sync_status.py +++ b/web/services/sync_status.py @@ -1,6 +1,6 @@ -"""Single source of truth for the four-state sync status. +"""Single source of truth for the sync status. -State precedence (highest wins): error > paused > downloading > waiting. +State precedence (highest wins): error > paused > triaging > downloading > waiting. Inputs come from ``hub.last_state`` (populated by the Hub.broadcast side-effects in ``hub.py``) and from the settings ``Snapshot``. @@ -13,7 +13,7 @@ from typing import Any, Optional, Tuple -STATES = ("downloading", "waiting", "paused", "error") +STATES = ("triaging", "downloading", "waiting", "paused", "error") def compute_sync_status(hub, db, snapshot) -> Tuple[str, Optional[str]]: @@ -67,6 +67,13 @@ def _compute(hub, snapshot) -> Tuple[str, Optional[str]]: if sync_state.get("paused"): return "paused", None + # ---- triaging ---- + # GPS triage runs (running, not paused) before the download drain. Surface + # it as its own state so the UI doesn't read as "waiting"/stalled. + triage = last.get("triage") + if isinstance(triage, dict) and triage.get("active"): + return "triaging", None + # ---- downloading ---- current_item = last.get("current_item") diff --git a/web/services/sync_worker.py b/web/services/sync_worker.py index c1112fc..674266f 100644 --- a/web/services/sync_worker.py +++ b/web/services/sync_worker.py @@ -33,13 +33,18 @@ from typing import Optional import viofosync_lib as vfs +from viofosync_lib import _control as control from ..db import Database from ..settings import SettingsProvider from . import queue as q from . import retention as _retention from . import scanner +from . import geofence as _geofence +from . import locations as _locations +from . import triage as _triage from .hub import Hub +from .naming import capture_key_sql as _capture_key_sql log = logging.getLogger("viofosync.sync_worker") @@ -50,6 +55,7 @@ # still sweeps immediately after a drain; this loop is what keeps the # archive bounded when the camera is offline or has nothing new. RETENTION_INTERVAL_SECONDS = 300 # 5 minutes +GEOFENCE_INTERVAL_SECONDS = 60 # incremental home-skip cadence during triage # Upper bound on how long stop() waits for the cycle/retention tasks # before abandoning them — keeps SIGTERM shutdown bounded even when a @@ -305,6 +311,37 @@ def __init__(self, use_html: bool, gps_extract: bool) -> None: self.gps_extract = gps_extract +def run_recording_status_release(db, *, recording_state) -> int: + """Release the newest capture group for triage + download when the camera + reports it is NOT recording. + + ``recording_state`` is the camera's record flag: ``0`` stopped, ``1`` + recording, ``None`` unknown (no/unsupported status command, camera + unreachable, or unparseable). Only an explicit ``0`` releases — it sets + ``remote_complete=1`` on every still-unconfirmed lens of the newest capture + group. ``1`` or ``None`` release nothing, so the newest segment stays held + by default; a later supersession (a newer capture appears, so the held one + is no longer ``MAX(capture_key)``) frees the previous one via + ``next_pending``'s guard. Returns the number of rows released. + + Why not per-file: the record flag is camera-global, and on this firmware + every lens of the newest capture is the one being written, so the whole + newest group is released together. Belt-and-suspenders against a stopped + read landing mid-finalize: the download drain's outgrows-expected-size + abort still defers a clip that turns out to still be growing.""" + if recording_state != 0: + return 0 + with db.write() as c: + cur = c.execute( + "UPDATE download_queue SET remote_complete=1 " + "WHERE state IN ('pending','failed') AND remote_complete IS NULL " + " AND " + _capture_key_sql("filename") + " = (" + " SELECT MAX(" + _capture_key_sql("filename") + ") " + " FROM download_queue WHERE state <> 'gone')" + ) + return cur.rowcount + + class SyncWorker: def __init__( self, @@ -317,6 +354,12 @@ def __init__( self.hub = hub self._task: Optional[asyncio.Task] = None self._retention_task: Optional[asyncio.Task] = None + self._geofence_task: Optional[asyncio.Task] = None + # Per-day {day: signature} cache shared by the periodic loop and the + # per-cycle sweep, so an idle tick is two queries, not a GPX re-parse. + self._geofence_seen: dict[str, int] = {} + # Serialises the periodic loop against the per-cycle / backfill sweeps. + self._geofence_lock = asyncio.Lock() self._stop = asyncio.Event() self._cancel_current = threading.Event() self._kick = asyncio.Event() @@ -375,6 +418,18 @@ def start(self) -> None: self._retention_task = asyncio.run_coroutine_threadsafe( self._retention_loop(), self._loop ) + # Skip home-parked clips incrementally as triage fills in skeletons, + # rather than only after the whole backlog finishes. + self._geofence_task = asyncio.run_coroutine_threadsafe( + self._geofence_loop(), self._loop + ) + # Report in immediately so the hub's sync_state (and the computed + # status badge) reflect the real running/paused state from the first + # snapshot. Without this, a running-but-idle worker never emits a + # sync_state until its first download, so compute_sync_status falls back + # to its "worker hasn't reported yet → paused" default — which made the + # toggle's first click pause for real (two clicks to resume). + self._broadcast_sync_state() def _is_running(self) -> bool: if self._task is None: @@ -388,7 +443,7 @@ async def stop(self) -> None: self._cancel_current.set() self._kick.set() import concurrent.futures - for task in (self._task, self._retention_task): + for task in (self._task, self._retention_task, self._geofence_task): if task is None: continue # ``task`` may be an asyncio.Task or a @@ -453,6 +508,9 @@ def resume(self) -> None: log.info("sync resumed") self._paused.clear() self.db.kv_set(_PAUSED_KV_KEY, "0") + # Drop the cancel the pause set, so the resumed cycle's probe/listing + # and triage pass aren't suppressed by a stale abort flag. + self._cancel_current.clear() self._broadcast_sync_state() self.kick() @@ -605,6 +663,120 @@ async def _retention_loop(self) -> None: except asyncio.TimeoutError: pass + async def _run_triage_pass(self) -> int: + """Extract skeleton GPS tracks for queued clips (GPS_TRIAGE). Triages + the whole un-triaged backlog before the download drain so the journey + map is complete first; results are cached on the queue row, so + steady-state cost is only newly-listed clips. + + Returns the number of clips triaged this pass (0 if disabled, no + camera, or the pass made no progress) so the cycle can choose a quick + retry vs back-off when triage is still incomplete.""" + snap = self._provider.get() + if not getattr(snap, "gps_triage", False): + return 0 + if not self._active_address: + return 0 + # Clear any stale cancel from a prior pause/skip so a resumed cycle's + # triage pass isn't suppressed (the flag is otherwise only cleared by + # _download_one, which may not run if the queue has nothing to download). + self._cancel_current.clear() + base = f"http://{self._active_address}" + + def _progress(triaged: int, total: int, eta_s) -> None: + # Called from the triage worker thread; schedule_broadcast hops back + # onto the loop thread-safely. + self.hub.schedule_broadcast( + self._loop, + {"type": "triage_progress", "active": True, + "triaged": triaged, "total": total, "eta_s": eta_s}, + ) + + triaged = 0 + try: + summary = await asyncio.to_thread( + _triage.run_pass, + self.db, base, snap.recordings, + timeout=snap.timeout, + cancel_check=self._cancel_current.is_set, + progress_cb=_progress, + ) + if isinstance(summary, dict): + triaged = int(summary.get("triaged", 0)) + except Exception: # pragma: no cover — never kill the cycle + log.exception("triage pass failed") + finally: + # Always clear the triaging substate (completion, pause-abort, or + # error) so the status never gets stuck on "triaging". + await self.hub.broadcast({"type": "triage_progress", "active": False}) + return triaged + + async def _run_recording_status_pass(self) -> None: + """Release the newest capture group once the camera reports it is no + longer recording, so triage + the drain can pick it up. Runs + unconditionally each cycle. A camera with no/unknown record command + yields ``None`` and the newest capture stays held (safe default); a + newer segment later supersedes and frees it. See + docs/superpowers/specs/2026-07-02-active-recording-guard-design.md.""" + if not self._active_address: + return + state = await asyncio.to_thread( + control.record_state, self._active_address + ) + await asyncio.to_thread( + run_recording_status_release, self.db, recording_state=state, + ) + + async def _run_geofence_pass(self, *, seen: dict | None = None) -> None: + """Auto-skip queued clips dwelling in a location flagged for exclusion. + Runs after the triage pass, on the periodic loop while triage fills in + skeletons, and on a settings change. Works on local data, so it doesn't + need the camera online. + + Pass ``seen=self._geofence_seen`` for the cheap incremental sweeps + (loop + per-cycle): only days with newly-triaged skeletons are + re-parsed. A full sweep (``seen=None``, the settings-change backfill) + drops that cache so every day is re-evaluated under the new zones.""" + snap = self._provider.get() + if not getattr(snap, "gps_triage", False): + return + zones = _locations.exclusion_zones(getattr(snap, "locations", ()) or ()) + if not zones: + return + async with self._geofence_lock: + # Clear inside the lock: a full sweep (backfill, seen=None) must not + # reset the cache while the periodic loop's worker thread is still + # mutating it under the lock. + if seen is None: + self._geofence_seen.clear() + try: + n = await asyncio.to_thread( + _geofence.sweep_all, self.db, snap.recordings, zones, + seen=seen, + ) + except Exception: # pragma: no cover — never kill the cycle + log.exception("geofence pass failed") + return + if n: + q.emit_queue_changed(self.db, self.hub) + + async def _geofence_loop(self) -> None: + """Re-skip home-parked clips on a fixed cadence while a triage pass is + filling in skeletons, instead of only once after the whole backlog. + The per-day signature cache keeps an idle tick to a couple of indexed + queries, so this is cheap to run often. Same lifecycle as the worker.""" + while not self._stop.is_set(): + try: + await self._run_geofence_pass(seen=self._geofence_seen) + except Exception: # pragma: no cover — never kill the loop + log.exception("periodic geofence sweep failed") + try: + await asyncio.wait_for( + self._stop.wait(), timeout=GEOFENCE_INTERVAL_SECONDS + ) + except asyncio.TimeoutError: + pass + async def _run_retention_sweep(self) -> None: """Run one retention pass against the live settings, then refresh the broadcast disk %. Offloaded to a thread because @@ -739,6 +911,33 @@ async def _cycle(self) -> bool: }) return False + # Release the newest capture for triage + the drain once the camera + # reports record=0; hold it otherwise (it may still be recording, and + # the camera lies about size). See _run_recording_status_pass. + await self._run_recording_status_pass() + + # Triage queued clips (if enabled) before downloading any, so the + # journey view shows where clips were recorded first. + triaged_n = await self._run_triage_pass() + + # Auto-skip clips parked at home (if enabled) once skeleton tracks + # exist, so they never enter the download drain below. + await self._run_geofence_pass(seen=self._geofence_seen) + + # Never start downloads while settled clips still need triage. The + # per-clip gate (next_pending) only holds the un-triaged clips, but a + # partial/aborted triage pass (e.g. after a pause/resume or a camera + # hiccup) would otherwise let the drain download the already-triaged + # clips ahead of the rest. Hold the whole drain until triage is done; + # a clip merely settling or given up does not count, so this can't + # block forever. Quick retry if we made progress, else back off. + snap = self._provider.get() + if getattr(snap, "gps_triage", False) and _triage.has_pending_targets( + self.db + ): + log.info("holding downloads until GPS triage completes") + return bool(triaged_n) + # Drain the queue. After each successful download the # loop re-checks ``next_pending`` so a priority update # mid-cycle takes effect immediately. @@ -746,9 +945,12 @@ async def _cycle(self) -> bool: while not self._stop.is_set(): if self._paused.is_set(): break + snap = self._provider.get() item = q.next_pending( self.db, - ro_only=self._provider.get().sync_ro_only, + ro_only=snap.sync_ro_only, + triage_gate=getattr(snap, "gps_triage", False), + active_guard=True, ) if item is None: break @@ -894,6 +1096,8 @@ def _blocking(): "gpx extract failed for %s: %s", item.filename, e, ) + # The real sidecar supersedes any triage skeleton. + _triage.remove_skeleton(snap.recordings, item.filename) _maybe_delete_from_dashcam( item=item, dest_path=dest_path, @@ -906,6 +1110,12 @@ def _blocking(): # Deliberate abort (pause/stop/unreachable): not a # failure, so the caller must not burn an attempt. return False, None, True, False + except vfs.DownloadDeferred: + # The file is still recording (stream outgrew its expected + # size). Like a cancellation: requeue without burning an + # attempt — the completeness guard holds it until it finalizes, + # so it must never exhaust max_attempts and get stuck 'failed'. + return False, None, True, False except OSError as e: if e.errno == _errno.ENOSPC: return False, str(e), False, True diff --git a/web/services/triage.py b/web/services/triage.py new file mode 100644 index 0000000..76a801f --- /dev/null +++ b/web/services/triage.py @@ -0,0 +1,383 @@ +# web/services/triage.py +"""GPS Triage — skeleton-track extraction for queued, not-yet-downloaded clips. + +When GPS_TRIAGE is on, the sync worker calls :func:`run_pass` each cycle +(before draining downloads). For each pending front-camera clip we read its +embedded GPS track straight off the camera over HTTP byte-range requests +(reusing ``parse_moov`` + the spike-filtering ``generate_gpx``), write a +skeleton ``$RECORDINGS/.triage/.gpx`` sidecar, and record the +result on the queue row (``triaged_at`` / ``gps_points``). + +The skeletons feed the archive journey view (see web/routers/archive.py); the +real per-clip sidecar written on download supersedes them. +""" +from __future__ import annotations + +import http.client +import logging +import os +import shutil +import socket +import threading +import time +import urllib.error +from concurrent.futures import ThreadPoolExecutor + +import viofosync_lib as vfs + +from ..db import Database +from .naming import gps_lens_sql, capture_key_sql + +log = logging.getLogger("viofosync.triage") + +# The camera's HTTP server resets above ~3 concurrent connections. +MAX_CONCURRENCY = 3 +# How often run_pass reports progress so the UI can fill the journey map in +# during a long full-backlog pass (one report per this many triaged clips). +PROGRESS_EVERY = 20 +# Stop a full-backlog pass after this many consecutive read failures — that +# many in a row means the camera dropped (car drove off), and every remaining +# read would otherwise block to timeout. The skipped clips stay un-triaged and +# retry on the next cycle once the camera is back. +ABORT_AFTER_FAILURES = 8 + +# Give up triaging a single clip after this many *unreadable* read failures +# (camera answered but the clip can't be decoded). Camera-offline failures +# never count, so a car that drove away does not exhaust this budget. +TRIAGE_MAX_ATTEMPTS = 5 + +# Per-clip backoff between unreadable retries, indexed by prior attempt count +# (1st retry waits steps[0], etc.). Mirrors the download backoff schedule. +# IMPORTANT: these values are duplicated in the SQL CASE in select_targets — +# update both together. +_TRIAGE_BACKOFF_S = (10, 30, 120, 600) + + +def _is_offline_error(exc: Exception) -> bool: + """True if the failure is a camera-unreachable condition (no completed + HTTP exchange) rather than a clip-specific read problem. An HTTPError + means the camera answered with a status, so it is NOT offline.""" + if isinstance(exc, urllib.error.HTTPError): + return False + # A mid-transfer short body (http.client.IncompleteRead, and our own + # TruncatedRead) is deliberately treated as *unreadable* (returns False + # here), not offline — the camera did respond, so crediting one attempt + # per mid-transfer drop is acceptable. + return isinstance(exc, ( + urllib.error.URLError, + socket.timeout, + TimeoutError, + http.client.RemoteDisconnected, + ConnectionError, + )) + + +# Don't triage a clip until its recording window has surely closed, or we read +# a half-written file with no GPS yet and cache a wrong gps_points=0 that's +# never revisited. Parking clips are time-lapse (~30 real min observed); normal +# and RO clips are <=60 s. +TRIAGE_SETTLE_S = 120 # normal / RO: 60 s clip + margin +TRIAGE_SETTLE_PARKING_S = 1860 # parking time-lapse window (~31 min) + +# States whose ``.triage`` skeleton must be kept on disk. This is the single +# source of truth shared with the geofence detector (geofence._DETECT_STATES): +# a skipped clip keeps its skeleton so re-evaluation still sees the full home +# dwell. 'done'/'gone' clips are superseded by the real sidecar / never return, +# so their skeletons are swept. +SKELETON_KEEP_STATES = ("pending", "failed", "skipped", "downloading") + + +def triage_dir(recordings: str) -> str: + return os.path.join(recordings, ".triage") + + +def skeleton_path(recordings: str, filename: str) -> str: + return os.path.join(triage_dir(recordings), filename + ".gpx") + + +def select_targets( + db: Database, *, limit: int | None = None, now: int | None = None, +) -> list[dict]: + """Pending front-camera rows not yet triaged and past their recording + settle window, newest first. + + Clips are deferred until their recording window has surely closed so we + don't read a half-written file with no GPS yet and cache a wrong + gps_points=0 that is never revisited (``triaged_at IS NULL`` prevents + re-triage). NULL ``recorded_at`` bypasses the gate (legacy rows). + + Clips at or past ``TRIAGE_MAX_ATTEMPTS`` unreadable failures are excluded + (they have given up); clips that failed recently wait out a per-attempt + backoff mirroring ``_TRIAGE_BACKOFF_S`` before they are selected again. + + ``limit=None`` (the default) returns the entire un-triaged backlog so one + pass can triage everything before the download drain runs — otherwise the + long drain starves triage and the journey map never fills in. SQLite + treats ``LIMIT -1`` as unbounded. + + Only the GPS-bearing lens is triaged: Viofo embeds the GPS track in one + stream (the front), and the other lenses of the same capture share it, so + triaging just that one covers the whole group. Which lens that is comes + from the camera registry via ``naming.gps_lens_sql`` (compact suffix-less + names are single-channel and count as the GPS lens), not a hard-coded + letter; the sibling clips still show as placeholder tiles.""" + if now is None: + now = int(time.time()) + with db.conn() as c: + rows = c.execute( + f"SELECT id, filename, source_dir, remote_size, recorded_at " + f"FROM download_queue " + f"WHERE state='pending' AND triaged_at IS NULL " + f"AND {gps_lens_sql()} " + f"AND NOT ({capture_key_sql('filename')} = (" + f" SELECT MAX({capture_key_sql('filename')}) " + f" FROM download_queue WHERE state <> 'gone') " + f" AND remote_complete IS NULL) " + f"AND (recorded_at IS NULL OR ? - recorded_at >= " + f" (CASE WHEN event_type='parking' THEN ? ELSE ? END)) " + f"AND triage_attempts < ? " + f"AND (triage_last_attempt_at IS NULL OR ? - triage_last_attempt_at " + f" >= (CASE triage_attempts WHEN 1 THEN 10 WHEN 2 THEN 30 " + f" WHEN 3 THEN 120 ELSE 600 END)) " + f"ORDER BY recorded_at DESC, id DESC LIMIT ?", + (now, TRIAGE_SETTLE_PARKING_S, TRIAGE_SETTLE_S, + TRIAGE_MAX_ATTEMPTS, now, + -1 if limit is None else limit), + ).fetchall() + return [dict(r) for r in rows] + + +def has_pending_targets(db: Database, *, now: int | None = None) -> bool: + """True if any clip still needs triage *right now* — the same predicate as + :func:`select_targets` (settled, un-triaged, under the give-up cap, past + its backoff). The sync cycle uses this to hold downloads until triage is + complete: a clip that is merely deferred (still settling) or given up does + not count, so the drain isn't blocked forever.""" + return bool(select_targets(db, limit=1, now=now)) + + +def _write_skeleton(recordings: str, filename: str, gpx: str) -> str: + d = triage_dir(recordings) + os.makedirs(d, exist_ok=True) + final = skeleton_path(recordings, filename) + tmp = final + ".part" + try: + with open(tmp, "w", encoding="utf-8") as f: + f.write(gpx) + os.replace(tmp, final) + except OSError: + # Don't leave a half-written .part behind (sweep_orphans only + # cleans .gpx files). + try: + os.remove(tmp) + except OSError: + pass + raise + return final + + +def triage_one( + db: Database, base_url: str, row: dict, recordings: str, *, timeout: float +) -> int: + """Triage one queued clip. Returns the number of GPS points found + (0 = no fix, -1 = read error). Sets ``triaged_at`` + ``gps_points`` on + success/no-fix so a fixless clip is never re-attempted; a read error + leaves the row untriaged for a later retry.""" + filename = row["filename"] + rec = vfs.Recording( + filename=filename, + filepath=row["source_dir"] or "", + size=row.get("remote_size"), + timecode=None, + datetime=None, + attr=None, + ) + try: + points = vfs.extract_remote_gps_points(base_url, rec, timeout=timeout) + except vfs.IncompleteRecording: + # No finalized moov yet. If this is the unconfirmed newest capture it + # is still recording — defer without burning an attempt (attempts stay + # 0 so the backoff CASE rechecks in ~600 s and has_pending_targets + # stays quiet). An older clip with no moov is truncated-final: mark it + # no-GPS so its pair's download gate releases. + now = int(time.time()) + with db.conn() as c: + newest = c.execute( + f"SELECT MAX({capture_key_sql('filename')}) FROM " + f"download_queue WHERE state <> 'gone'" + ).fetchone()[0] + key = c.execute( + f"SELECT {capture_key_sql('filename')} FROM download_queue " + f"WHERE id=?", (row["id"],) + ).fetchone()[0] + is_unconfirmed_newest = (key == newest and not row.get("remote_complete")) + with db.write() as c: + if is_unconfirmed_newest: + c.execute( + "UPDATE download_queue SET triage_last_attempt_at=? " + "WHERE id=?", (now, row["id"]), + ) + else: + c.execute( + "UPDATE download_queue SET triaged_at=?, gps_points=0 " + "WHERE id=?", (now, row["id"]), + ) + log.info("triage deferred (recording in progress) for %s", filename) + return -1 + except Exception as e: + if _is_offline_error(e): + # Camera unreachable (car drove away) — not the clip's fault. + # Leave the row untriaged with no attempt bump; it retries + # whenever the camera returns. + log.info("triage skipped (camera offline) for %s: %s", filename, e) + else: + # Camera answered but the clip can't be triaged (HTTP error, + # corrupt/truncated data) — count toward the give-up cap. + now = int(time.time()) + with db.write() as c: + c.execute( + "UPDATE download_queue SET " + "triage_attempts=triage_attempts+1, " + "triage_last_attempt_at=? WHERE id=?", + (now, row["id"]), + ) + log.info("triage failed (clip unreadable) for %s: %s", filename, e) + return -1 + if points: + gpx = vfs.generate_gpx(points, os.path.basename(filename) + ".gpx") + _write_skeleton(recordings, filename, gpx) + now = int(time.time()) + with db.write() as c: + c.execute( + "UPDATE download_queue SET triaged_at=?, gps_points=? WHERE id=?", + (now, len(points), row["id"]), + ) + return len(points) + + +def run_pass( + db: Database, base_url: str, recordings: str, *, timeout: float, + cancel_check=None, progress_cb=None, +) -> dict: + """Triage every un-triaged pending front clip concurrently (<=3), so the + journey map fully populates before the download drain runs. A read error + leaves a clip un-triaged for retry next cycle. Returns a summary dict. + + ``cancel_check`` (optional) aborts between clips (pause/stop). + ``progress_cb(triaged, total, eta_s)`` (optional) is invoked once at the + start (``0, total, None``) and every ``PROGRESS_EVERY`` clips, so the UI can + show live progress + ETA during a long full-backlog pass.""" + sweep_orphans(db, recordings) + targets = select_targets(db) + if not targets: + return {"triaged": 0, "with_gps": 0} + + total = len(targets) + started = time.perf_counter() + log.info("triage: starting pass — %d clip(s) to triage", total) + if progress_cb is not None: + progress_cb(0, total, None) + + triaged = 0 + with_gps = 0 + + # Circuit breaker for a mid-pass camera drop: count consecutive read + # failures (triage_one returns -1 only on an I/O error — a clip with no GPS + # returns 0 and counts as success). Once tripped, queued _work calls return + # immediately so the pass finishes promptly instead of timing out on every + # remaining clip. + abort = threading.Event() + fail_streak = {"n": 0} + fail_lock = threading.Lock() + + def _work(row): + if abort.is_set() or (cancel_check is not None and cancel_check()): + return None + n = triage_one(db, base_url, row, recordings, timeout=timeout) + with fail_lock: + if n < 0: + fail_streak["n"] += 1 + if fail_streak["n"] >= ABORT_AFTER_FAILURES: + abort.set() + else: + fail_streak["n"] = 0 + return n + + with ThreadPoolExecutor(max_workers=MAX_CONCURRENCY) as ex: + for n in ex.map(_work, targets): + if n is None or n < 0: + continue + triaged += 1 + if n > 0: + with_gps += 1 + if triaged % PROGRESS_EVERY == 0: + elapsed = time.perf_counter() - started + log.info( + "triage: %d/%d clip(s) done (%.0fs elapsed, %.1fs/clip)", + triaged, total, elapsed, elapsed / triaged, + ) + if progress_cb is not None: + eta_s = round((total - triaged) * (elapsed / triaged)) + progress_cb(triaged, total, eta_s) + elapsed = time.perf_counter() - started + if abort.is_set(): + log.warning( + "triage: stopped early after %d consecutive read failures " + "(camera offline?); remaining clips retry next cycle", + ABORT_AFTER_FAILURES, + ) + log.info( + "triage: pass complete — %d/%d clip(s), %d with GPS, in %.0fs", + triaged, total, with_gps, elapsed, + ) + return {"triaged": triaged, "with_gps": with_gps} + + +def remove_skeleton(recordings: str, filename: str) -> None: + try: + os.remove(skeleton_path(recordings, filename)) + except OSError: + pass + + +def sweep_orphans(db: Database, recordings: str) -> int: + """Remove skeleton sidecars whose queue row is no longer in a keep state + (downloaded, gone, skipped-but-released, or absent). Returns the number + removed.""" + d = triage_dir(recordings) + if not os.path.isdir(d): + return 0 + with db.conn() as c: + ph = ",".join("?" * len(SKELETON_KEEP_STATES)) + live = { + r["filename"] + for r in c.execute( + f"SELECT filename FROM download_queue " + f"WHERE state IN ({ph})", + SKELETON_KEEP_STATES, + ) + } + removed = 0 + for entry in os.listdir(d): + if not entry.endswith(".gpx"): + continue + filename = entry[: -len(".gpx")] + if filename in live: + continue + try: + os.remove(os.path.join(d, entry)) + removed += 1 + except OSError: + pass + return removed + + +def purge_all(db: Database, recordings: str) -> None: + """Drop the entire .triage cache and clear triage columns (used when + GPS_TRIAGE is turned off).""" + shutil.rmtree(triage_dir(recordings), ignore_errors=True) + with db.write() as c: + c.execute( + "UPDATE download_queue SET triaged_at=NULL, gps_points=NULL " + "WHERE triaged_at IS NOT NULL OR gps_points IS NOT NULL" + ) diff --git a/web/settings.py b/web/settings.py index b08b1f1..d234574 100644 --- a/web/settings.py +++ b/web/settings.py @@ -21,6 +21,7 @@ from .settings_schema import ( DEFAULT_VALUES, SettingsModel, + normalize_locations, validate_new_password, validate_partial, ) @@ -36,6 +37,18 @@ def _config_dir() -> Path: Subscriber = Callable[[set[str], "Snapshot"], None] +@dataclass(frozen=True) +class Place: + """Immutable snapshot view of one named location (metres for radius).""" + + name: str + lat: float + lon: float + radius_m: int + exclude_recordings: bool + is_home: bool = False + + @dataclass(frozen=True) class Snapshot: """Immutable view of every setting the running app may need.""" @@ -47,6 +60,7 @@ class Snapshot: grouping: str use_html_listing: bool gps_extract: bool + gps_triage: bool derive_thumbs_eager: bool derive_filmstrips_eager: bool delete_after_download: bool @@ -61,6 +75,7 @@ class Snapshot: retention_protect_ro: bool recordings_quota_gb: int disk_critical_pct: int + locations: tuple[Place, ...] password_hash: str session_secret: str @@ -208,6 +223,34 @@ def _load_snapshot(self) -> Snapshot: # disk; otherwise every restart would invalidate every # session cookie. Has to run before the snapshot is built. data = self._store.load() + # One-shot: fold a pre-existing GEOFENCE_ZONES (the earlier geofence + # design) into the LOCATIONS list, preserving whether exclusion was on. + if "LOCATIONS" not in data and data.get("GEOFENCE_ZONES"): + excl = bool(data.get("GEOFENCE_ENABLED")) + valid = [ + z for z in data["GEOFENCE_ZONES"] + if z.get("lat") is not None and z.get("lon") is not None + ] + data["LOCATIONS"] = [ + { + "name": "Home" if i == 0 else f"Location {i + 1}", + "lat": z["lat"], + "lon": z["lon"], + "radius_m": z.get("radius_m", 30), + "exclude_recordings": excl, + } + for i, z in enumerate(valid) + ] + data.pop("GEOFENCE_ZONES", None) + data.pop("GEOFENCE_ENABLED", None) + self._store.write(data) + # One-shot: back-fill is_home on locations written by the previous + # design (no is_home key). normalize_locations promotes the first place + # to Home; persist so the invariant holds on disk, not just per-read. + locs = data.get("LOCATIONS") + if locs and any("is_home" not in loc for loc in locs): + data["LOCATIONS"] = normalize_locations(locs) + self._store.write(data) if "SESSION_SECRET" not in data: data["SESSION_SECRET"] = secrets.token_hex(32) self._store.write(data) @@ -246,6 +289,7 @@ def _make_snapshot(self, data: dict) -> Snapshot: grouping=m.GROUPING, use_html_listing=m.HTML, gps_extract=m.GPS_EXTRACT, + gps_triage=m.GPS_TRIAGE, derive_thumbs_eager=m.DERIVE_THUMBS_EAGER, derive_filmstrips_eager=m.DERIVE_FILMSTRIPS_EAGER, delete_after_download=m.DELETE_AFTER_DOWNLOAD, @@ -260,6 +304,11 @@ def _make_snapshot(self, data: dict) -> Snapshot: retention_protect_ro=m.RETENTION_PROTECT_RO, recordings_quota_gb=m.RECORDINGS_QUOTA_GB, disk_critical_pct=m.DISK_CRITICAL_PCT, + locations=tuple( + Place(loc.name, loc.lat, loc.lon, loc.radius_m, + loc.exclude_recordings, loc.is_home) + for loc in m.LOCATIONS + ), password_hash=m.WEB_PASSWORD_HASH, session_secret=m.SESSION_SECRET, host=m.WEB_HOST, diff --git a/web/settings_schema.py b/web/settings_schema.py index 768876f..113e370 100644 --- a/web/settings_schema.py +++ b/web/settings_schema.py @@ -22,6 +22,45 @@ _MQTT_NODE_ID_RE = re.compile(r"^[a-z0-9_]{1,32}$") +class Location(BaseModel): + """A named place (e.g. "Home"). ``radius_m`` (metres) is used for the + geocode-label match and, when ``exclude_recordings`` is on, the geofence + dwell match. The stop cluster radius is 50 m, so raise the radius if a + genuine match is missed.""" + + model_config = ConfigDict(extra="forbid") + + name: str = "Home" + lat: float = Field(ge=-90, le=90) + lon: float = Field(ge=-180, le=180) + radius_m: int = Field(default=30, ge=5, le=500) + exclude_recordings: bool = False + is_home: bool = False + + +def normalize_locations(items: list[dict]) -> list[dict]: + """Enforce the 'exactly one Home' invariant on a LOCATIONS list of dicts. + + - empty list -> unchanged (no Home) + - exactly one is_home True -> unchanged + - multiple is_home True -> keep the first in list order, clear the rest + - none is_home True and list non-empty -> promote the first element + + Tolerates dicts missing the ``is_home`` key (treated as False). Returns a new + list of shallow-copied dicts; never mutates the input. + """ + out = [dict(it) for it in items] + for it in out: + it.setdefault("is_home", False) + homes = [i for i, it in enumerate(out) if it["is_home"]] + if out and not homes: + out[0]["is_home"] = True + elif len(homes) > 1: + for i in homes[1:]: + out[i]["is_home"] = False + return out + + class SettingsModel(BaseModel): """Strict-typed view of every persisted setting.""" @@ -36,6 +75,9 @@ class SettingsModel(BaseModel): GROUPING: Literal["none", "daily", "weekly", "monthly", "yearly"] = "daily" HTML: bool = True GPS_EXTRACT: bool = True + # Pre-download GPS triage: extract a skeleton track for queued clips so + # the journey view can show where they were recorded before downloading. + GPS_TRIAGE: bool = False DERIVE_THUMBS_EAGER: bool = True # Filmstrips are ~10-25x the ffmpeg work of a thumbnail, so eager # pre-generation is opt-in; off means they build on demand on first @@ -48,6 +90,11 @@ class SettingsModel(BaseModel): SYNC_INTERVAL: int = Field(default=600, ge=60, le=86400) ENABLE_SCHEDULED_SYNC: bool = True SYNC_RO_ONLY: bool = False + # Named locations (Home by default). Used to label journey/stop endpoints + # by name, and — for locations flagged exclude_recordings (with GPS_TRIAGE + # on) — to auto-skip clips that dwell there. Multiple locations are + # supported; exactly one is the designated Home (is_home). + LOCATIONS: list[Location] = Field(default_factory=list) RETENTION_MAX_DAYS: int = Field(default=0, ge=0, le=3650) RETENTION_DISK_PCT: int = Field(default=0, ge=0, le=99) RETENTION_PROTECT_RO: bool = True @@ -161,12 +208,14 @@ def _validate_disk_critical(self): # Public taxonomy used by the API + UI. EDITABLE_KEYS = { "ADDRESS", "ADDRESS_FALLBACK", "IMPORT_PATH", "GROUPING", "HTML", "GPS_EXTRACT", + "GPS_TRIAGE", "DERIVE_THUMBS_EAGER", "DERIVE_FILMSTRIPS_EAGER", "DELETE_AFTER_DOWNLOAD", "TIMEOUT", "DOWNLOAD_ATTEMPTS", "MAX_DOWNLOAD_ATTEMPTS", "SYNC_INTERVAL", "ENABLE_SCHEDULED_SYNC", "WEB_HOST", "WEB_PORT", "EXPORT_ENCODER", "NOMINATIM_EMAIL", "GEOCODE_ENABLED", - "SYNC_RO_ONLY", "RETENTION_MAX_DAYS", "RETENTION_DISK_PCT", + "SYNC_RO_ONLY", "LOCATIONS", + "RETENTION_MAX_DAYS", "RETENTION_DISK_PCT", "RETENTION_PROTECT_RO", "RECORDINGS_QUOTA_GB", "DISK_CRITICAL_PCT", "DISTANCE_UNITS", "PIP_POSITION", @@ -218,6 +267,12 @@ def validate_partial(patch: dict) -> dict: out: dict[str, Any] = {} for k in patch: out[k] = getattr(model, k) + # LOCATIONS coerces to a list of Location models; persist + echo them as + # plain dicts (normalised so exactly one is the Home). + if "LOCATIONS" in out: + out["LOCATIONS"] = normalize_locations( + [loc.model_dump() for loc in out["LOCATIONS"]] + ) return out diff --git a/web/static/app.js b/web/static/app.js index 4fa779f..73e43fb 100644 --- a/web/static/app.js +++ b/web/static/app.js @@ -39,21 +39,32 @@ const state = { queueExpanded: new Set(), queueHoursExpanded: new Set(), // keys: "YYYY-MM-DD HH" (HH may be "??") queueSelected: new Set(),// filenames ticked - filters: { driving: true, parking: true, ro: true }, + filters: { driving: true, parking: true, ro: true, location: "", + geofenced: localStorage.getItem("vfs.showGeofenced") === "1" }, showMaps: localStorage.getItem("vfs.showMaps") !== "0", archiveSelected: new Map(), // pair_id → { ts, front, rear, tele, interior } archiveExpanded: new Set(), // open archive day keys ("YYYY-MM-DD"); persists // open days across in-app navigation (re-render) + archiveStale: false, // a change event arrived while the archive was + // hidden; reload the day list on next tab entry + journeyExpanded: new Set(), // open journey/stop card keys ("date|kind|ts"); + // restores expansion across day-body re-renders + journeyMapViews: new Map(), // card key → {center, zoom}; restores a map's + // view across re-renders instead of re-running + // the animated fitBounds (a visible zoom jump) map: null, routeLayer: null, ws: null, syncRunning: false, syncPaused: false, + triage: { active: false }, // GPS triage progress: {active, triaged, total, eta_s} currentFilename: null, // Mirrored from /api/settings on login + on Save so display // helpers (fmtDistance) don't need to read from settingsState // (which is only loaded when the Settings tab is visited). distanceUnits: "km", + locations: [], // named locations mirrored from /api/settings, for the + // archive location filter dropdown logsFilter: null, // { level, logger, q } currently shown in Logs tab logsOldestId: null, // smallest id loaded, for "Load older" pagination }; @@ -128,16 +139,14 @@ async function showApp() { // The WS snapshot will deliver the server-computed sync_status // shortly; no direct updateSyncState call needed here. } catch {} - try { - const gs = await api("/api/archive/extract-gps/status"); - setExtractButton(gs); - } catch {} } async function refreshDisplayPrefs() { try { const body = await api("/api/settings"); state.distanceUnits = body.editable.DISTANCE_UNITS || "km"; + state.locations = body.editable.LOCATIONS || []; + buildLocationFilter(); } catch { /* keep defaults */ } } @@ -208,6 +217,26 @@ document.getElementById("sync-toggle").addEventListener("click", async () => { // updateSyncState here — the server-computed status is the truth. }); +// Format a seconds-remaining estimate for the triage badge. +function fmtTriageEta(s) { + if (s == null || !isFinite(s) || s < 0) return ""; + if (s < 60) return `~${Math.round(s)}s`; + const m = Math.round(s / 60); + if (m < 60) return `~${m} min`; + const h = Math.floor(m / 60); + return `~${h}h ${m % 60}m`; +} + +// Build the "Triaging GPS 40/876 · ~12 min left" badge text from state.triage. +function triageBadgeText() { + const t = state.triage || {}; + let s = "Triaging GPS"; + if (t.total) s += ` ${t.triaged || 0}/${t.total}`; + const eta = fmtTriageEta(t.eta_s); + if (eta) s += ` · ${eta} left`; + return s; +} + // state.syncStatus is one of: "downloading", "waiting", "paused", "error", null // state.syncStatusReason is a short human-readable string (only set in error) function updateSyncState(status) { @@ -228,6 +257,8 @@ function updateSyncState(status) { let show, title, klass; if (status === "downloading") { show = iconSync; title = "Pause downloading"; klass = "active"; + } else if (status === "triaging") { + show = iconSync; title = "Pause"; klass = "triaging"; } else if (status === "waiting") { show = iconSync; title = "Pause downloading"; klass = "waiting"; } else if (status === "paused") { @@ -249,7 +280,7 @@ function updateSyncState(status) { setVisible(iconPause, show === iconPause); setVisible(iconSync, show === iconSync); setVisible(iconWarn, show === iconWarn); - btn.classList.remove("active", "paused", "waiting", "error"); + btn.classList.remove("active", "paused", "waiting", "error", "triaging"); if (klass) btn.classList.add(klass); btn.title = title; } @@ -289,7 +320,13 @@ function routeTo(hash) { } } if (tab === "archive") { - loadDays(); + // Rebuild the day list only when something actually changed while the + // view was hidden (archiveStale, set by the WS handlers below) or it was + // never rendered. Skipping the reload preserves open days, expanded + // journey cards and their live Leaflet maps — switching to the timeline + // and back should feel instant, not re-fetch the world. + const days = document.getElementById("days"); + if (state.archiveStale || !days.children.length) loadDays(); refreshExportJobs(); } if (tab === "downloads") loadQueue(); @@ -327,12 +364,112 @@ function refreshArchiveOnIndexChange() { if (_archiveRefreshTimer) clearTimeout(_archiveRefreshTimer); _archiveRefreshTimer = setTimeout(() => { _archiveRefreshTimer = null; - if (document.getElementById("view-archive").hidden) return; + if (document.getElementById("view-archive").hidden) { + // Don't drop the event: remember the list is out of date so the + // next visit to the tab reloads it (routeTo checks this flag). + state.archiveStale = true; + return; + } if (document.querySelector("#days .day .day-body:not([hidden])")) return; loadDays(); }, 400); } +// ---- FLIP animation for day-body re-renders -------------------------- +// A day-body refresh (skip, download-next, WS queue_changed) rebuilds its +// DOM, which reads as an abrupt snap: tiles vanish and neighbours teleport. +// FLIP smooths it: capture each keyed element's position before the swap, +// then animate survivors from their old position to the new one; elements +// that only exist in the new DOM fade in. Keys live in data-flip-key +// (clip pairs, journey/stop/ungrouped cards). Positions are stored relative +// to the day body so a page-scroll shift between capture and play can't +// skew the deltas. Disabled under prefers-reduced-motion. +const _REDUCED_MOTION = window.matchMedia("(prefers-reduced-motion: reduce)"); +const FLIP_MS = 220; + +function flipCapture(body) { + if (_REDUCED_MOTION.matches) return null; + const ref = body.getBoundingClientRect(); + const rects = new Map(); + for (const el of body.querySelectorAll("[data-flip-key]")) { + const r = el.getBoundingClientRect(); + rects.set(el.dataset.flipKey, + { x: r.left - ref.left, y: r.top - ref.top }); + } + return rects.size ? rects : null; // null = first render, nothing to play +} + +function flipPlay(body, prev) { + if (!prev) return; + const ref = body.getBoundingClientRect(); + for (const el of body.querySelectorAll("[data-flip-key]")) { + const r = el.getBoundingClientRect(); + const was = prev.get(el.dataset.flipKey); + if (!was) { + // Newcomer (e.g. a clip that just finished downloading). + el.animate( + [{ opacity: 0, transform: "scale(0.97)" }, + { opacity: 1, transform: "none" }], + { duration: FLIP_MS, easing: "ease-out" }); + continue; + } + const dx = was.x - (r.left - ref.left); + const dy = was.y - (r.top - ref.top); + if (dx || dy) { + el.animate( + [{ transform: `translate(${dx}px, ${dy}px)` }, + { transform: "none" }], + { duration: FLIP_MS, easing: "ease-in-out" }); + } + } +} + +// Fade the selected clip tiles out before an action that removes them from +// the archive (skip, delete), so the removal reads as intentional rather +// than a glitch. Resolves when the fade completes; the follow-up re-render +// then FLIPs the surviving neighbours into the freed space. +function animateOutSelectedPairs() { + if (_REDUCED_MOTION.matches) return Promise.resolve(); + const anims = []; + for (const pairId of state.archiveSelected.keys()) { + document + .querySelectorAll(`.clip-pair[data-pair-id="${CSS.escape(pairId)}"]`) + .forEach((el) => { + anims.push( + el.animate( + [{ opacity: 1, transform: "none" }, + { opacity: 0, transform: "scale(0.95)" }], + { duration: 160, easing: "ease-in", fill: "forwards" } + ).finished.catch(() => {}) + ); + }); + } + return Promise.all(anims); +} + +// Re-render any currently-open archive day so per-clip status icons reflect +// the latest queue state. Unlike refreshArchiveOnIndexChange (which skips open +// days to avoid disrupting browsing), this targets the open day directly — +// used after a Download Action and on queue_changed. +let _openDayRefreshTimer = null; +function refreshOpenArchiveDays() { + document.querySelectorAll("#days .day").forEach((dayEl) => { + const body = dayEl.querySelector(".day-body"); + if (body && !body.hidden) renderDayBody(body, dayEl.dataset.day); + }); +} +function scheduleOpenArchiveRefresh() { + if (_openDayRefreshTimer) clearTimeout(_openDayRefreshTimer); + _openDayRefreshTimer = setTimeout(() => { + _openDayRefreshTimer = null; + if (document.getElementById("view-archive").hidden) { + state.archiveStale = true; // reload on next tab entry instead + return; + } + refreshOpenArchiveDays(); + }, 300); +} + // ---------- Archive ---------- function wireArchiveFilter(id, key) { @@ -346,6 +483,59 @@ wireArchiveFilter("f-driving", "driving"); wireArchiveFilter("f-parking", "parking"); wireArchiveFilter("f-ro", "ro"); +// "GPS-excluded": reveal clips auto-skipped by the home geofence and +// weave their recovered skeleton tracks into the day's journeys. +// Persisted — it's a mode, like the maps toggle, not a per-visit filter. +(() => { + const cb = document.getElementById("f-geofenced"); + cb.checked = state.filters.geofenced; + cb.addEventListener("change", (e) => { + state.filters.geofenced = e.target.checked; + localStorage.setItem("vfs.showGeofenced", e.target.checked ? "1" : "0"); + state.page = 1; + loadDays(); + }); +})(); + +// Location filter: client-side, narrows the open day timeline to journeys +// (start or end) and stops at the selected named location. Populated from +// state.locations (mirrored from /api/settings); hidden when none are defined. +function buildLocationFilter() { + const wrap = document.getElementById("f-location-wrap"); + const sel = document.getElementById("f-location"); + if (!wrap || !sel) return; + const locs = state.locations || []; + if (!locs.length) { + wrap.hidden = true; + state.filters.location = ""; + sel.value = ""; + return; + } + // Home first, then alphabetical; name doubles as the option value (filtering + // matches by place name). + const ordered = [...locs].sort((a, b) => + (b.is_home ? 1 : 0) - (a.is_home ? 1 : 0) || + (a.name || "").localeCompare(b.name || "")); + const prev = state.filters.location; + const names = ordered.map((l) => l.name); + sel.innerHTML = + '' + + ordered.map((l) => + ``).join(""); + if (prev && names.includes(prev)) { + sel.value = prev; + } else { + sel.value = ""; + state.filters.location = ""; + } + wrap.hidden = false; +} + +document.getElementById("f-location").addEventListener("change", (e) => { + state.filters.location = e.target.value; + refreshOpenArchiveDays(); +}); + // "GPS Journey Splits" is a view option, not a filter: it gates the // journey machinery (Leaflet, route fetch, reverse-geocode) on // expansion but doesn't change what's fetched. Persisted to @@ -380,48 +570,12 @@ document.getElementById("rescan").addEventListener("click", async () => { } }); -function setExtractButton({ running, done, total, extracted, empty, errors }) { - const btn = document.getElementById("extract-gps"); - if (running) { - btn.disabled = true; - btn.textContent = total - ? `Extracting GPS ${done}/${total}…` - : "Extracting GPS…"; - } else { - btn.disabled = false; - btn.textContent = "Extract GPS"; - } - if (!running && total > 0 && (extracted != null)) { - btn.title = `Last run: ${extracted} extracted · ${empty} empty · ${errors} errors`; - } -} - -document.getElementById("extract-gps").addEventListener("click", async (e) => { - // Shift+click forces re-extraction of clips that already - // have a sidecar — use this after tweaking the spike filter. - const force = e.shiftKey; - if (force && !confirm( - "Re-extract GPS for every clip in the archive? This overwrites existing .gpx sidecars." - )) return; - try { - const url = force ? "/api/archive/extract-gps?force=true" : "/api/archive/extract-gps"; - const r = await api(url, { method: "POST" }); - if (!r.started) { - const btn = document.getElementById("extract-gps"); - btn.textContent = r.total === 0 ? "No clips need GPS" : "Extract GPS"; - setTimeout(() => { btn.textContent = "Extract GPS"; }, 2000); - return; - } - setExtractButton({ running: true, done: 0, total: r.total }); - } catch (e) { - console.warn("extract-gps failed", e); - } -}); function archiveKindParams(q) { q.set("driving", state.filters.driving ? "true" : "false"); q.set("parking", state.filters.parking ? "true" : "false"); q.set("ro", state.filters.ro ? "true" : "false"); + if (state.filters.geofenced) q.set("show_geofenced", "true"); } // Tear down Leaflet instances under `root` before its innerHTML is @@ -438,6 +592,14 @@ function destroyJourneyMaps(root) { } } +function destroyLocationMaps(root) { + for (const div of root.querySelectorAll(".loc-map")) { + const m = div._locMap; + if (m) { try { m.remove(); } catch {} } + div._locMap = null; + } +} + async function loadDays() { // Stale-response guard (same token pattern as loadQueue): WS // pushes, filter changes, and pagination can race; only the most @@ -451,6 +613,7 @@ async function loadDays() { const data = await api("/api/archive/days?" + q); if (reqId !== state.daysRequestId) return; // superseded + state.archiveStale = false; // rendering fresh data const container = document.getElementById("days"); destroyJourneyMaps(container); container.innerHTML = ""; @@ -474,6 +637,8 @@ function renderDayCard(d) { const open = state.archiveExpanded.has(d.day); el.innerHTML = `
+

${d.day}

${d.clip_count} clips${ @@ -482,7 +647,13 @@ function renderDayCard(d) { d.parking_count ? `${d.parking_count} parking` : null, d.ro_count ? `${d.ro_count} read-only` : null, ].filter(Boolean).map((s) => ` · ${s}`).join("") - } · ${fmtBytes(d.total_bytes)}${d.gpx_count ? " · GPS" : ""} + } · ${fmtBytes(d.total_bytes)}${d.gpx_count ? " · GPS" : ""}${ + d.remote_count ? ` · ${d.remote_count} on camera` : "" + }${ + d.geofence_skipped_count + ? ` · ${d.geofence_skipped_count} GPS-excluded` + : "" + }
@@ -496,18 +667,61 @@ function renderDayCard(d) { } body.hidden = false; state.archiveExpanded.add(d.day); - body.innerHTML = "

Loading…

"; - await renderDayBody(body, d.day); + await loadDayBody(el); }); + wireDayCheck(el); // Restore a remembered-open day: populate its body immediately. Async; the // card is returned synchronously and fills in when the fetch resolves. if (open) { - body.innerHTML = "

Loading…

"; - renderDayBody(body, d.day); + loadDayBody(el); } return el; } +// Load (or reload) a day's body, then sync its select-all checkbox. Centralised +// so the day-header toggle, a remembered-open day, and the day-level select-all +// all populate the body the same way and mark it loaded for refreshDayCheck. +async function loadDayBody(dayEl) { + const body = dayEl.querySelector(".day-body"); + // Only show the placeholder on a body that has never rendered; a refresh + // of an already-populated day keeps the old content on screen until the + // new payload arrives (renderDayBody swaps it in one go). + if (!body.dataset.loaded) body.innerHTML = "

Loading…

"; + await renderDayBody(body, dayEl.dataset.day); + body.dataset.loaded = "1"; + refreshDayCheck(dayEl); +} + +// A pair carries GPS if it's a remote tile (already gated to gps_points>0) or +// any downloaded camera slot has a sidecar. Used to decide whether a clip that +// snaps to no journey/stop is worth an Ungrouped card or should be dropped. +function pairHasGps(pair) { + if (pair.remote) return true; + return CAMERAS.some((c) => pair[c.channel] && pair[c.channel].has_gpx); +} + +// True if a timeline event involves the named location: a journey whose start +// OR end is that place, or a stop at that place. Matches on the server's +// `named` flag + the place name, so a coincidental geocoded label can't match. +function eventMatchesLocation(ev, name) { + if (ev.kind === "journey") { + const j = ev.data; + return (j.start_named && j.start_label === name) || + (j.end_named && j.end_label === name); + } + const s = ev.data; + return !!(s.named && s.label === name); +} + +// Placeholder shown in an expanded day that has nothing to/from the filtered +// location (uses textContent — `name` is user-supplied). +function renderNoLocationActivity(name) { + const note = document.createElement("p"); + note.style.cssText = "color:var(--muted);font-size:12px"; + note.textContent = `No activity to/from ${name} on this day.`; + return note; +} + async function renderDayBody(body, date) { const q = new URLSearchParams(); archiveKindParams(q); @@ -521,7 +735,8 @@ async function renderDayBody(body, date) { const promises = [api(`/api/archive/day/${date}?` + q)]; if (state.showMaps) { promises.push( - api(`/api/archive/day/${date}/route`) + api(`/api/archive/day/${date}/route` + + (state.filters.geofenced ? "?show_geofenced=true" : "")) .catch((e) => { console.warn("route failed", e); return null; }), ); } else { @@ -534,13 +749,23 @@ async function renderDayBody(body, date) { return; } + // FLIP: snapshot keyed positions before the swap, animate after (below). + const flipPrev = flipCapture(body); + destroyJourneyMaps(body); body.innerHTML = ""; const journeys = (route && route.journeys) || []; const stops = (route && route.stops) || []; const hasGps = journeys.length > 0 || stops.length > 0; + const locFilter = state.filters.location; if (!hasGps) { + // With a location filter active, a day with no journeys/stops has nothing + // to/from that place — show the note rather than an unfiltered flat grid. + if (locFilter) { + body.appendChild(renderNoLocationActivity(locFilter)); + return; + } // No journeys and no stops — fall back to a flat grid. const grid = document.createElement("div"); grid.className = "clip-grid"; @@ -552,6 +777,7 @@ async function renderDayBody(body, date) { note.textContent = "No GPS data for this day."; body.appendChild(note); } + flipPlay(body, flipPrev); return; } @@ -561,43 +787,79 @@ async function renderDayBody(body, date) { // there directly; the rest snap to whichever event is // closest in time, so everything is anchored to *some* // visible context instead of a flat "other" pile. - const events = [ - ...journeys.map((j, idx) => ({ - kind: "journey", data: j, clips: [], idx, - start: j.start_ts, end: j.end_ts, - })), - ...stops.map((s, idx) => ({ - kind: "stop", data: s, clips: [], idx, - start: s.start_ts, end: s.end_ts, - })), - ]; - // Newest first — most recent activity at the top of the day. - events.sort((a, b) => b.start - a.start); + // Journeys group with a PADDED window (group_start_ts/group_end_ts from the + // server — the same _expand_journey_window the timeline uses) so the pull-away + // and pull-in clips that sit just outside the raw GPS journey (the stop + // boundary lands ~50 m inside the real drive) land on the journey card. Fall + // back to the raw window for older payloads. + const journeyEvents = journeys.map((j, idx) => ({ + kind: "journey", data: j, clips: [], idx, + start: j.group_start_ts ?? j.start_ts, + end: j.group_end_ts ?? j.end_ts, + })); + const stopEvents = stops.map((s, idx) => ({ + kind: "stop", data: s, clips: [], idx, + start: s.start_ts, end: s.end_ts, + })); + // Newest first — most recent activity at the top of the day. Render order and + // the snap fallback below scan this combined list. + const events = [...journeyEvents, ...stopEvents].sort((a, b) => b.start - a.start); + + // A loose clip (no event contains its timestamp) snaps to the nearest event + // only if it's within this gap; otherwise it collects in an "Ungrouped" card + // so a not-yet-skipped home clip can't drag across the whole day onto a journey. + const SNAP_MAX_GAP_S = 1800; // 30 min — matches the GPS session-gap splitter + const ungrouped = []; for (const pair of data.clips) { const ts = pair.timestamp; - let target = events.find((e) => ts >= e.start && ts <= e.end); + // Journeys win over an overlapping stop at the boundary: a pull-in clip that + // falls inside both the padded journey and the following stop belongs to the + // drive, not the dwell. + let target = + journeyEvents.find((e) => ts >= e.start && ts <= e.end) || + stopEvents.find((e) => ts >= e.start && ts <= e.end); if (!target) { - // Snap to the closest event by time gap. let best = null, bestGap = Infinity; for (const e of events) { const gap = ts < e.start ? e.start - ts : ts - e.end; if (gap < bestGap) { bestGap = gap; best = e; } } - target = best; + target = bestGap <= SNAP_MAX_GAP_S ? best : null; } if (target) target.clips.push(pair); + else if (pairHasGps(pair)) ungrouped.push(pair); + // A no-GPS pair with nothing within snap range has no place in the + // journey-organised view (its real context — e.g. a home dwell built from + // skipped clips — is invisible here), so drop it rather than float it in an + // Ungrouped card. Extends the trace-gate principle to downloaded clips. A + // no-GPS clip that DID snap to a journey/stop is kept (handled above). } - for (const ev of events) { + const shown = locFilter + ? events.filter((ev) => eventMatchesLocation(ev, locFilter)) + : events; + + for (const ev of shown) { if (ev.kind === "journey") { - body.appendChild(renderJourneyCard(ev.data, ev.clips, ev.idx, date)); + body.appendChild(renderJourneyCard( + ev.data, ev.clips, ev.idx, date, + (route && route.geofenced_tracks) || [], + )); } else { - body.appendChild(renderStopCard(ev.data, ev.clips, ev.idx)); + body.appendChild(renderStopCard(ev.data, ev.clips, ev.idx, date)); } } + if (locFilter) { + // The Ungrouped pile isn't tied to a place, so it's hidden while filtering. + if (!shown.length) body.appendChild(renderNoLocationActivity(locFilter)); + } else if (ungrouped.length) { + body.appendChild(renderUngroupedCard(ungrouped, date)); + } + wireClipPairMapClicks(body); + flipPlay(body, flipPrev); } // Clicking anywhere on a clip-pair (except the thumbnail image, @@ -687,7 +949,7 @@ function fmtEta(seconds) { return fmtDuration(seconds); } -function renderStopCard(stop, clips, idx) { +function renderStopCard(stop, clips, idx, date) { const startT = new Date(stop.start_time).toLocaleTimeString(); const endT = new Date(stop.end_time).toLocaleTimeString(); const hasClips = clips && clips.length > 0; @@ -698,10 +960,11 @@ function renderStopCard(stop, clips, idx) { if (!hasClips) { const el = document.createElement("div"); el.className = "stop-banner"; + el.dataset.flipKey = `${date}|sb|${stop.start_ts}`; el.innerHTML = ` Stopped for ${fmtDuration(stop.duration_s)} - at ${escHtml(placeLabel)} + at ${placeGlyph(stop.home, stop.named)}${escHtml(placeLabel)} ${startT} – ${endT} `; if (!stop.label) { @@ -722,14 +985,14 @@ function renderStopCard(stop, clips, idx) { ${startT} – ${endT} - - ${escHtml(placeLabel)} + ${placeGlyph(stop.home, stop.named)}${escHtml(placeLabel)} ${fmtDuration(stop.duration_s)} · ${clips.length} clip${clips.length === 1 ? "" : "s"} +
-
- Originals: - - -
-
- Joined: - - -
-
- PIP: - - -
- - -
diff --git a/web/static/styles.css b/web/static/styles.css index 484af08..5ca05ee 100644 --- a/web/static/styles.css +++ b/web/static/styles.css @@ -179,12 +179,17 @@ input[type="radio"] { /* Waiting: match the sync-status badge's amber so the button and badge * read as the same state. */ .icon-btn.waiting { color: var(--warn); border-color: var(--warn); } +.icon-btn.triaging { color: var(--accent); border-color: var(--accent); } /* When the syncing icon is the visible one, give it a gentle * continuous spin so the button reads as "in progress". */ .icon-btn.active #sync-icon-sync:not([hidden]) { animation: vfs-spin 1.6s linear infinite; transform-origin: 50% 50%; } +.icon-btn.triaging #sync-icon-sync:not([hidden]) { + animation: vfs-spin 1.6s linear infinite; + transform-origin: 50% 50%; +} @keyframes vfs-spin { from { transform: rotate(0deg); } to { transform: rotate(360deg); } @@ -309,6 +314,7 @@ nav a.active { background: var(--panel-2); } * dark/light themes stay coherent — extend the palette only if a * future revision adds new accent tokens. */ .status.downloading { color: var(--ok); border-color: var(--ok); } +.status.triaging { color: var(--accent); border-color: var(--accent); } .status.waiting { color: var(--warn); border-color: var(--warn); } .status.paused { color: var(--err-text); border-color: var(--err); } .status.error { @@ -337,6 +343,8 @@ main { } .archive-filters .spacer { flex: 1; } .archive-filters .kind-label { flex-direction: row; } +.archive-filters .loc-filter { align-items: center; gap: 6px; } +.archive-filters .loc-filter select { max-width: 170px; } .archive-actions { display: flex; align-items: center; gap: 8px; @@ -550,6 +558,13 @@ main { align-items: baseline; } .day-header:hover { background: rgba(255, 255, 255, 0.015); } +.day-header .day-check { + flex: 0 0 auto; + align-self: center; + margin: 0; + cursor: pointer; + accent-color: var(--accent); +} /* The date is the strongest landmark on the archive page; * it deserves more weight than the metadata that follows it. * Pre-bolder this was 15px/600 (≈1.07× the 14px body — flat). @@ -636,16 +651,47 @@ main { /* Filenames are debug-y data; fade them and clip to one line. * Hover reveals the full string via the title attribute on the * label element (set in the renderer). */ -.clip-pair .label { - font-size: 10px; - color: var(--muted); - font-family: "SF Mono", "Menlo", monospace; - white-space: nowrap; - overflow: hidden; - text-overflow: ellipsis; - opacity: 0.75; +/* Per-clip filename labels are noise in the archive grid — the new + * Download Actions selection + journey context identify clips instead. */ +.clip-pair .label { display: none; } +.clip-pair .thumb.remote { + display: flex; + align-items: center; + justify-content: center; + aspect-ratio: 16 / 9; /* match a real thumb's footprint */ + background: repeating-linear-gradient( + 45deg, var(--panel), var(--panel) 8px, var(--bg) 8px, var(--bg) 16px); + opacity: 0.85; +} +.clip-pair .thumb.remote .remote-badge { + position: absolute; + top: 4px; + right: 6px; + font-size: 14px; + opacity: 0.8; +} +.clip-pair .thumb.remote .remote-badge svg { display: block; } +.clip-pair .thumb.remote.state-pending .remote-badge { color: var(--accent); } +.clip-pair .thumb.remote.state-downloading .remote-badge { color: var(--accent); } +.clip-pair .thumb.remote.state-failed .remote-badge { color: var(--err); } +.clip-pair .thumb.remote.state-skipped .remote-badge { color: var(--muted); } +.clip-pair .thumb.remote.state-skipped { opacity: 0.5; } +.clip-pair .thumb.remote.geofence .remote-badge { color: var(--accent); } +.clip-pair .thumb.remote .remote-badge .spin { animation: vfs-spin 1.6s linear infinite; transform-origin: 50% 50%; } +.clip-pair .thumb.remote .unskip-btn { + position: absolute; + bottom: 6px; + right: 6px; + font-size: 11px; + padding: 2px 8px; + background: var(--panel); + color: var(--text); + border: 1px solid var(--muted); + border-radius: 4px; + cursor: pointer; } -.clip-pair:hover .label { opacity: 1; } +.clip-pair .thumb.remote .unskip-btn:hover { border-color: var(--accent); } +.clip-pair .thumb.remote .unskip-btn:disabled { opacity: 0.5; cursor: default; } .clip-pair.flash { outline: 2px solid var(--accent); animation: flash-pulse 1.5s ease-out; @@ -656,6 +702,11 @@ main { } #map { height: 340px; border-radius: 8px; margin: 16px 0; } +.home-marker { color: var(--accent); display: block; } +.home-marker-wrap { display: flex; align-items: center; justify-content: center; } +.journey-title .start-label svg, +.journey-title .end-label svg, +.stop-label svg { vertical-align: -2px; margin-right: 3px; color: var(--accent); } .journey-table { width: 100%; border-collapse: collapse; @@ -680,6 +731,11 @@ main { display: flex; align-items: center; gap: 12px; padding: 10px 14px; flex-wrap: wrap; + /* Reserve the Timeline button's row height (only drive cards carry it) so a + button-less stop / ungrouped header is the same height as a drive's. With + box-sizing: border-box this clamps every header — whatever its padding — to + one height; align-items: center keeps the content vertically centred. */ + min-height: 50px; } .journey-card.collapsible .journey-header { cursor: pointer; @@ -715,7 +771,13 @@ main { .start-label, .end-label, .stop-label { overflow-wrap: anywhere; } -.journey-header .journey-times { color: var(--text); font-size: 13px; } +/* tabular-nums keeps every HH:MM:SS range the same width, so the journey-title + column starts at the same x on drive and stop cards. */ +.journey-header .journey-times { + color: var(--text); font-size: 13px; + font-variant-numeric: tabular-nums; + flex: 0 0 auto; +} .journey-header .journey-meta { color: var(--muted); font-size: 12px; margin-left: auto; @@ -728,9 +790,24 @@ main { .journey-card .clip-grid { padding: 12px; } .stop-card .journey-header { background: rgba(241, 196, 15, 0.05); } -.stop-card .stop-icon { font-size: 14px; color: var(--warn); } +/* In the stop-card header the icon sits in the right-hand action slot — the + same place the Timeline button occupies on drive cards (8px past the meta, + matching the button's margin). */ +.stop-card .journey-header .stop-icon { + font-size: 14px; color: var(--warn); + flex: 0 0 auto; margin-left: 8px; +} .stop-card .journey-map { height: 200px; } +/* Drives are the spine of the day: accent rail + graded header wash + bold title. + Stops recede (the existing faint amber header stays; just dim + tighten it). */ +.journey-card.drive { border-left: 3px solid var(--accent); } +.journey-card.drive .journey-header { + background: linear-gradient(to right, var(--accent-18), transparent 55%), var(--panel-2); +} +.journey-card.drive .journey-title { font-weight: 700; } +.stop-card .journey-header { opacity: 0.85; padding: 6px 14px; } + .stop-banner { display: flex; align-items: center; gap: 10px; padding: 8px 14px; @@ -1483,6 +1560,7 @@ main { .queue-filters, .archive-actions { padding: 10px; } .filters .spacer, + .queue-filters .spacer, .archive-actions .spacer, .archive-filters .spacer { display: none; } @@ -1670,6 +1748,7 @@ main { border: none; } .status.downloading { background: var(--ok); } + .status.triaging { background: var(--accent); } .status.waiting { background: var(--warn); } .status.paused { background: var(--err); } .status.error { background: #7f1d1d; } @@ -2417,4 +2496,103 @@ main { .switch:checked { background: var(--accent-45); border-color: var(--accent); } .switch:checked::before { transform: translateX(20px); background: var(--on-accent); } .switch:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; } + +/* --- Named locations list (Settings → GPS) --- */ +.loc-list { margin: 8px 0 4px; } +.loc { + border: 1px solid var(--border); + border-radius: 8px; + margin: 7px 0; + background: var(--panel); + overflow: hidden; +} +.loc.open { border-color: var(--accent); } +.loc-row { + display: flex; + align-items: center; + gap: 10px; + padding: 10px 12px; + cursor: pointer; +} +.loc-ico { color: var(--accent); display: inline-flex; } +.loc-name { font-weight: 600; color: var(--text); } +.loc-badge { + font-size: 10px; + padding: 1px 7px; + border-radius: 10px; + border: 1px solid transparent; +} +.loc-badge.home { color: var(--warn); border-color: var(--warn); } +.loc-badge.excl { color: var(--accent); border-color: var(--accent-45); } +.loc-meta { + margin-left: auto; + color: var(--muted); + font-size: 12px; + font-variant-numeric: tabular-nums; +} +.loc-caret { color: var(--muted); } +.loc-editor { + border-top: 1px solid var(--border); + padding: 12px; + background: var(--panel-2); +} +.loc-map { height: 220px; border-radius: 8px; margin: 0 0 10px; } +.loc-actions { display: flex; gap: 8px; margin-top: 12px; } +.loc-del-btn { margin-left: auto; color: var(--err-text); border-color: var(--err); } +.loc-add { + width: 100%; + border: 1px dashed var(--border-hover); + border-radius: 8px; + padding: 11px; + margin-top: 9px; + background: transparent; + color: var(--muted); +} +.loc-add:hover { background: var(--panel-2); color: var(--text); } +.loc-empty { margin: 8px 2px; } .switch:disabled { opacity: 0.5; cursor: default; } +.clip-action-group { display: flex; align-items: center; gap: 6px; } +.clip-action-group select { max-width: 190px; } +/* The Downloads bar reuses .clip-action-group; give its label the same muted + treatment as the archive action label so the two bars read identically. */ +.queue-filters .action-label { + font-size: 12px; color: var(--muted); + font-weight: 500; margin-right: 2px; +} + +/* GPS triage badge — per-file indicator in the Downloads queue table. */ +.gps-badge { + display: inline-block; + min-width: 1.2em; + text-align: center; + font-weight: 600; + border-radius: 4px; + padding: 0 4px; +} +.gps-badge.gps-ok { color: var(--ok); } +.gps-badge.gps-none { color: var(--muted); } +.gps-badge.gps-pending { color: var(--muted); opacity: 0.7; } +.queue-items td.gps-cell { text-align: center; } + +/* Journey completion pie — small conic-gradient circle in the journey header. */ +.journey-pie { + display: inline-block; + width: 14px; + height: 14px; + border-radius: 50%; + vertical-align: -2px; + margin-right: 8px; + background: var(--panel); + box-shadow: inset 0 0 0 1px var(--border); +} +.journey-pie::before { + content: ""; + display: block; + width: 100%; + height: 100%; + border-radius: 50%; + background: conic-gradient(var(--muted) calc(var(--pp) * 1%), transparent 0); +} +.journey-pie.is-complete::before { + background: var(--ok); +}