Skip to content

sxm.pl: persist channel list to disk cache with expiry and background refresh#70

Merged
paul-1 merged 7 commits intomainfrom
copilot/save-channel-list-to-cache
Mar 1, 2026
Merged

sxm.pl: persist channel list to disk cache with expiry and background refresh#70
paul-1 merged 7 commits intomainfrom
copilot/save-channel-list-to-cache

Conversation

Copy link
Contributor

Copilot AI commented Feb 22, 2026 •
edited
Loading

The proxy fetched the channel list from the SiriusXM API on every cold start and never persisted it, adding unnecessary latency and API calls. This adds a full cache lifecycle: save on fetch, load at startup, and async background refresh on expiry.

Cache format

channels.json is written to the same directory as the cookie file:

{
  "fetched_at": 1740000000,
  "expires_at": 1740086400,
  "jsessionid_expires": 1740014400,
  "channels": [ ... ]
}

Changes

  • CHANNEL_CACHE_TTL — new constant (86400 s / 24 h)
  • SiriusXM::new() — derives channel_cache_file path (<cookie_dir>/channels.json), initialises channel_cache_expires and jsessionid_expires fields, calls load_channel_cache() at construction time
  • load_channel_cache() — reads and validates channels.json on startup; restores both the channel list and the tracked jsessionid_expires timestamp; if the channel cache is expired it still populates $self->{channels} so requests are served immediately and sets channel_cache_expires = 0 to trigger an immediate background refresh
  • save_channel_cache() — serialises the channel array with fetched_at/expires_at/jsessionid_expires envelope so the tracked session expiry survives proxy restarts
  • authenticate() — when a global-context authentication succeeds, records jsessionid_expires = time() + SESSION_MAX_LIFE (4 hours); JSESSIONID carries no explicit cookie timestamp so we track it ourselves
  • get_channels() — calls save_channel_cache() after every successful API fetch
  • refresh_channel_cache_if_expired() — background refresh called from the server loop; checks jsessionid_expires (our own tracked timestamp) and only calls authenticate() when the session has elapsed, avoiding unnecessary re-authentication; keeps old channel list in place while fetching, falls back to old data on any failure and retries in 5 minutes
  • start_http_daemon() loop — wires refresh_channel_cache_if_expired() alongside the existing refresh_expired_playlists() / process_segment_queues() calls
  • start_server() — calls get_channels() after authentication to ensure the channel list is always available at startup, covering: no cache file, corrupt cache, and expired cache
  • /channel/all HTTP handler — calls get_channels() (cached) instead of refresh_channels() so client requests always return the in-memory list without triggering a live API fetch
Original prompt

This section details on the original issue you should resolve

<issue_title>Save channel list to cache</issue_title>
<issue_description>In the proxy sxm.pl

Save channel listing to the cache. Make sure its in a format where we know the expiry
Use the same folder we are using for cookies

Load the channel listing from cache on startup
Run a background event to refresh the cache when it expires. Do this asynchronously.
Continue to use and serve channel data until the new new channel data arrives.</issue_description>

Comments on the Issue (you are @copilot in this section)

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

@paul-1
Add channel list disk cache to sxm.pl proxy
288d79d 
- Add CHANNEL_CACHE_TTL constant (24 hours) to SiriusXM package
- Add File::Basename and File::Spec imports to SiriusXM package
- Add channel_cache_file and channel_cache_expires fields to new()
- Load channel cache from disk at startup via load_channel_cache()
- New load_channel_cache(): reads channels.json, handles expired cache
  by still serving old data while scheduling immediate background refresh
- New save_channel_cache(): persists channel list as JSON with
  fetched_at/expires_at envelope in the same dir as the cookie file
- get_channels() now calls save_channel_cache() after successful API fetch
- New refresh_channel_cache_if_expired(): background refresh that keeps
  old channel list available for serving during the refresh; retries in
  5 minutes on failure
- Wire refresh_channel_cache_if_expired() into start_http_daemon() loop

Co-authored-by: paul-1 <6473457+paul-1@users.noreply.github.com>
Copilot AI changed the title [WIP] Save channel list to cache and manage expiration sxm.pl: persist channel list to disk cache with expiry and background refresh Feb 22, 2026
Copilot AI requested a review from paul-1 February 22, 2026 00:47
paul-1
paul-1 requested changes Feb 22, 2026
View reviewed changes
Copy link
Owner

@paul-1 paul-1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot make sure that we automatically get channel data at startup if.

  • The cache file does not exist
  • The cache file is expired
  • the cache file is corrupt.

@paul-1
Ensure channel list is fetched at startup for missing/corrupt cache
16dce82 
Call get_channels() in start_server() after authentication so that:
- No cache file: channels are fetched from the API and saved to disk
- Corrupt cache: channels are fetched from the API and saved to disk
- Expired cache: channels were already loaded by load_channel_cache()
  and the background refresh loop updates them asynchronously

Co-authored-by: paul-1 <6473457+paul-1@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Feb 22, 2026

@copilot make sure that we automatically get channel data at startup if.

  • The cache file does not exist...

Fixed in 16dce82. start_server() now calls get_channels() after authentication, which covers all three cases:

  • No cache file — $self->{channels} is undef after load_channel_cache(), so get_channels() fetches from the API and saves a fresh channels.json
  • Corrupt cache — same as above; the JSON parse error leaves $self->{channels} as undef, triggering an API fetch
  • Expired cache — load_channel_cache() already populates $self->{channels} with the stale data (so it's immediately serveable), get_channels() returns it as-is, and the background refresh loop (refresh_channel_cache_if_expired) updates it asynchronously

Copilot AI requested a review from paul-1 February 22, 2026 18:48
paul-1
paul-1 requested changes Feb 23, 2026
View reviewed changes
Copy link
Owner

@paul-1 paul-1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot When the client requests channel data, just return the cached channel information, do not expire the cache.

See:
[22.02.2026 22:22:17.656] DEBUG SiriusHandler::handle_http_request:2720: GET request: /channel/all
[22.02.2026 22:22:17.656] DEBUG SiriusHandler::handle_http_request:2811: Channel info request for: all
[22.02.2026 22:22:17.656] DEBUG SiriusXM::refresh_channels:2410: Refreshing channel data (clearing cache)

@paul-1
Return cached channel data on /channel/all requests instead of forcin…
3f95ce3 
…g refresh

Replace refresh_channels() with get_channels() in the /channel/all HTTP
handler so client requests always return the cached list. The background
refresh loop (refresh_channel_cache_if_expired) is the only path that
triggers a live API fetch.

Co-authored-by: paul-1 <6473457+paul-1@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Feb 23, 2026

@copilot When the client requests channel data, just return the cached channel information, do not expire the cache.

See:...

Fixed in 3f95ce3. The /channel/all handler now calls get_channels() instead of refresh_channels(), so it always returns the in-memory cached list without touching the API. Only the background refresh loop (refresh_channel_cache_if_expired) triggers live fetches.

Copilot AI requested a review from paul-1 February 23, 2026 03:28
@paul-1
Copy link
Owner

paul-1 commented Feb 26, 2026

@copilot can you check to make sure that before trying to refresh channel data, that we check the session cookies in the global cookie to make sure they are not expired.

@paul-1
Check global session cookies before background channel cache refresh
8eb328e 
In refresh_channel_cache_if_expired(), check is_session_authenticated()
against the global cookie jar before attempting the API call. If the
session cookies are expired, re-authenticate first; if re-auth fails,
skip this cycle and let the 5-minute retry handle it.

Co-authored-by: paul-1 <6473457+paul-1@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Feb 26, 2026

@copilot can you check to make sure that before trying to refresh channel data, that we check the session cookies in the global cookie to make sure they are not expired.

Done in 8eb328e. refresh_channel_cache_if_expired() now calls is_session_authenticated(undef) against the global cookie jar before making any API call. If the session cookies are expired it calls set_channel_context(undef) + authenticate(undef) to renew them first; if re-authentication fails the refresh cycle is skipped and the 5-minute retry picks it up next time.

paul-1
paul-1 requested changes Feb 28, 2026
View reviewed changes
Copy link
Owner

@paul-1 paul-1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot Something is wrong with checking session cookie expiration.

The log shows the system was idle for almost 24 hours, when the system tried to update the channel data. There is no way a Session Cookie lasts 24hours, infact we manually set the expiration at 14400 seconds.

[26.02.2026 22:25:49.746] DEBUG <SiriusHandler::start_http_daemon>:2717: Client connection closed
[27.02.2026 19:41:11.087]  INFO <SiriusXM::refresh_channel_cache_if_expired>:2204: Channel cache expired – fetching fresh channel list in background...
[27.02.2026 19:41:11.612]  WARN <SiriusXM::refresh_channel_cache_if_expired>:2254: Background channel refresh: invalid or empty response – keeping existing channel list
[27.02.2026 19:46:11.934]  INFO <SiriusXM::refresh_channel_cache_if_expired>:2204: Channel cache expired – fetching fresh channel list in background...
[27.02.2026 19:46:12.229]  WARN <SiriusXM::refresh_channel_cache_if_expired>:2254: Background channel refresh: invalid or empty response – keeping existing channel list
[27.02.2026 19:51:11.544]  INFO <SiriusXM::refresh_channel_cache_if_expired>:2204: Channel cache expired – fetching fresh channel list in background...
[27.02.2026 19:51:11.890]  WARN <SiriusXM::refresh_channel_cache_if_expired>:2254: Background channel refresh: invalid or empty response – keeping existing channel list
[27.02.2026 19:56:11.204]  INFO <SiriusXM::refresh_channel_cache_if_expired>:2204: Channel cache expired – fetching fresh channel list in background...

This same log messages go on for a whole day.. Restating sxm.pl the system properly renews the session and updates the channels.

[28.02.2026 11:13:08.399]  INFO <main::signal_handler>: 349: Received signal TERM, shutting down gracefully...
[28.02.2026 11:13:10.538]  INFO <main::init_logging>: 264: SiriusXM Proxy logging initialized with level: DEBUG
[28.02.2026 11:13:10.538]  INFO <main::init_logging>: 265: File logging enabled: /opt/slimserver/Logs/sxm-proxy.log
[28.02.2026 11:13:10.538]  INFO <main::init_logging>: 272: Cookie file: /opt/slimserver/Cache/siriusxm/sxm.txt
[28.02.2026 11:13:10.538]  INFO <main::main>:3089: Starting SiriusXM Perl proxy v1.1.0
[28.02.2026 11:13:10.538]  INFO <SiriusXM::new>: 442: Loaded cookies from: /opt/slimserver/Cache/siriusxm/sxm.txt
[28.02.2026 11:13:10.538]  INFO <SiriusXM::__ANON__>: 518: Cookie SXMDATA (global): expires 2094-03-17 04:07:52 UTC (in 24853d 11h 54m)
[28.02.2026 11:13:10.538]  INFO <SiriusXM::__ANON__>: 518: Cookie AWSALB (global): expires 2026-03-06 00:54:47 UTC (in 5d 8h 41m)
[28.02.2026 11:13:10.538]  INFO <SiriusXM::__ANON__>: 507: Cookie JSESSIONID (global): no expiration set, estimated lifetime ~4h (expires ~2026-02-28 20:13:10 UTC)
[28.02.2026 11:13:10.539] DEBUG <SiriusXM::new>: 465: SiriusXM object created for user: phermann2@outlook.com, region: US
[28.02.2026 11:13:10.539] DEBUG <SiriusXM::new>: 471: Channel cache file: /opt/slimserver/Cache/siriusxm/channels.json
[28.02.2026 11:13:10.562]  INFO <SiriusXM::load_channel_cache>:2143: Loaded 435 channels from expired cache (expired: 2026-02-28 00:41:11 UTC) – background refresh will run
[28.02.2026 11:13:10.563]  INFO <main::start_server>:3048: Starting HTTP server on port 8888
[28.02.2026 11:13:10.563] DEBUG <SiriusXM::__ANON__>: 876: SXMDATA cookie valid for 24853d 11h
[28.02.2026 11:13:10.563] DEBUG <SiriusXM::authenticate>:1123: Attempting to authenticate session for global
[28.02.2026 11:13:10.940]  INFO <SiriusXM::authenticate>:1158: Session authentication successful for global
[28.02.2026 11:13:10.941]  INFO <SiriusXM::__ANON__>: 518: Cookie SXMDATA (global): expires 2094-03-18 19:27:17 UTC (in 24855d 3h 14m)
[28.02.2026 11:13:10.941]  INFO <SiriusXM::__ANON__>: 518: Cookie AWSALB (global): expires 2026-03-07 16:13:10 UTC (in 7d 0h 0m)
[28.02.2026 11:13:10.941]  INFO <SiriusXM::__ANON__>: 507: Cookie JSESSIONID (global): no expiration set, estimated lifetime ~4h (expires ~2026-02-28 20:13:10 UTC)
[28.02.2026 11:13:10.941]  INFO <main::start_server>:3056: Authentication successful - starting server
[28.02.2026 11:13:10.941]  INFO <main::start_server>:3068: Channel list ready at startup: 435 channels
[28.02.2026 11:13:10.941]  INFO <SiriusHandler::start_http_daemon>:2654: HTTP server started on port 8888
[28.02.2026 11:13:10.941]  INFO <SiriusHandler::start_http_daemon>:2655: Access channels at: http://127.0.0.1:8888/channel.m3u8
[28.02.2026 11:13:11.942]  INFO <SiriusXM::refresh_channel_cache_if_expired>:2204: Channel cache expired – fetching fresh channel list in background...
[28.02.2026 11:13:12.556]  INFO <SiriusXM::refresh_channel_cache_if_expired>:2260: Background channel refresh complete: 435 channels loaded
[28.02.2026 11:13:12.571]  INFO <SiriusXM::save_channel_cache>:2184: Saved 435 channels to cache /opt/slimserver/Cache/siriusxm/channels.json (expires: 2026-03-01 16:13:12 UTC)

@paul-1
Fix background channel refresh failing after idle session expiry
84ef5e2 
Replace the conditional is_session_authenticated() guard with an
unconditional authenticate() call in refresh_channel_cache_if_expired().

Root cause: JSESSIONID has no explicit expiry timestamp in the cookie
jar, so is_session_authenticated() always returned true even after the
server had invalidated the session (after ~4h of idle). This caused the
background refresh to call the API with a dead session, getting an empty
response every 5 minutes indefinitely.

Fix: always call authenticate(undef) before the channel fetch, mirroring
what start_server() does on every startup. This ensures a live session
regardless of cookie timestamp availability.

Co-authored-by: paul-1 <6473457+paul-1@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Feb 28, 2026

@copilot Something is wrong with checking session cookie expiration.

The log shows the system was idle for almost 24 hours, when the system tried to update the channel data. There is no way a Sessi...

Fixed in 84ef5e2.

Root cause: JSESSIONID has no explicit expiry timestamp in the cookie jar (the logs confirm "no expiration set"). is_session_authenticated() checks if ($expires && $expires < $now) — with $expires being undef, the expired-cookie branch never fires and the function always returns true, even after the server has invalidated the session after hours of idle.

Fix: replaced the conditional is_session_authenticated() guard with an unconditional authenticate(undef) call at the start of refresh_channel_cache_if_expired(). This mirrors exactly what start_server() does on every restart (which is why a restart fixes it). authenticate() always does the live resume POST to get fresh AWSALB/JSESSIONID tokens regardless of what cookie timestamps say.

Copilot AI requested a review from paul-1 February 28, 2026 16:17
@paul-1
Copy link
Owner

paul-1 commented Feb 28, 2026

@copilot I do not like unconditional authentication, this can lead to rate blocking (Especially when debugging). The JSESSIONID cookie does not contain an expiry, but can we not track it ourselves. If would be best if this tracking is retained (saved on disk) across sessions either in the cookie itself, or a separate cache file

@paul-1
Track JSESSIONID expiry ourselves and persist it in channels.json
6093243 
Replace unconditional authenticate() in background channel refresh with
a conditional check against a tracked jsessionid_expires timestamp.

Changes:
- Add jsessionid_expires field to SiriusXM object (default 0 = expired)
- In authenticate(): when global context (channel_id undef) succeeds,
  set jsessionid_expires = time() + SESSION_MAX_LIFE (4 hours)
- In save_channel_cache(): include jsessionid_expires in channels.json
  so the tracked expiry survives proxy restarts
- In load_channel_cache(): restore jsessionid_expires from channels.json
  (defaults to 0 if absent, triggering re-auth on first background refresh)
- In refresh_channel_cache_if_expired(): only call authenticate() when
  time() >= jsessionid_expires; otherwise log remaining validity and skip

Co-authored-by: paul-1 <6473457+paul-1@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Feb 28, 2026

@copilot I do not like unconditional authentication, this can lead to rate blocking (Especially when debugging). The JSESSIONID cookie does not contain an expiry, but can we not tr...

Done in 6093243. Instead of unconditional authentication, we now track the JSESSIONID lifetime ourselves:

  • authenticate() — when a global-context (channel_id = undef) authentication succeeds, records jsessionid_expires = time() + SESSION_MAX_LIFE (4 hours)
  • save_channel_cache() — includes jsessionid_expires in channels.json so it survives proxy restarts
  • load_channel_cache() — restores jsessionid_expires from channels.json on startup (defaults to 0 if absent, which safely triggers a one-time re-auth on the first background refresh)
  • refresh_channel_cache_if_expired() — only calls authenticate() when time() >= jsessionid_expires; otherwise logs remaining validity and skips re-auth entirely

The updated channels.json format:

{
  "fetched_at": 1740000000,
  "expires_at": 1740086400,
  "jsessionid_expires": 1740014400,
  "channels": [ ... ]
}

@paul-1 paul-1 marked this pull request as ready for review March 1, 2026 19:31
@paul-1 paul-1 merged commit 3b884f3 into main Mar 1, 2026
1 check passed
@paul-1 paul-1 deleted the copilot/save-channel-list-to-cache branch March 1, 2026 19:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@paul-1 paul-1 paul-1 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@paul-1 paul-1

Copilot code review Copilot

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Save channel list to cache

2 participants

@paul-1