[🐛 Bug]: Selenium Manager telemetry usage

[titusfortner]

Description

Selenium's Plausible telemetry usage has exceeded our plan's limit, which locks the dashboard.
Typical usage is well within our limits until we hit outliers. The last time this happened we had Plausible filter out everything coming from a specific version of Selenium that we think was bundled into someone's .NET application.
We need to find a more reliable solution to minimizing large signals that do not provide useful information.

Essentially we've been relying on TTL values in a local cache to minimize reporting volume, with the goal of one update per configuration per hour per system.

The primary issue is containerized systems that run single sessions and close. If Selenium Manager is called, then each execution will send telemetry data.

While digging into possible solutions, I've found a number of things that I think we should address, I'm listing everything here so we can keep track of it. The first two are the ones I'll address first as a priority.

Reduce excessive reporting

1. Don't report on the first run. Environments that don't keep the cache (ephemeral per-session containers, caches wiped on teardown) cannot be throttled by cache/ttl; we can avoid all of it by ensuring it is the second run instead of the first.

2. Don't report if the cache can't be read. If the cache is unreadable or get_cache_path() returns None, get_metadata returns empty and telemetry is sent by default.

Bugs (unintended defects)

3. Panics on an existing-but-unreadable metadata file. File::open(&metadata_path).unwrap() (rust/src/metadata.rs:102): exists() can be true while the open fails with EACCES (write-only mode, cross-UID between runs, SELinux/mount policy), crashing the whole tool. The serde parse error immediately below is handled; the open error is not.

4. Unsynchronized, non-atomic metadata writes. stats() rewrites the whole se-metadata.json with fs::write and holds no lock (the existing sm.lock only guards per-asset downloads, and runs later than stats()). Concurrent processes (a launch burst, or stats() racing another run's setup()) can lose updates or read a truncated/empty file.

Assumptions we should stop relying on

5. The only persistence record is discarded when the send-TTL expires. get_metadata prunes the stats list with retain(stats_ttl > now), deleting a config's entire entry at expiry. This welds "suppress sending" to "we've run here before," so there is no durable signal that a cache actually persists across runs.

6. Persistence is tracked per configuration, though it's really a per-machine property. Whether a cache persists is a property of the machine, but the entry is keyed per browser + browser_version + os + arch + lang + selenium_version. A brand-new config on a durable machine is indistinguishable from a first-ever run on an ephemeral one.

7. A write→read round-trip of se-metadata.json isn't guaranteed across runs. Violated by write-only, eventually-consistent (network FS), or wiped-between-runs caches. Persistence detection built on this must fail closed (send nothing, don't crash), not assume the read reflects the write.

8. Telemetry cadence is coupled to the version-discovery TTL. get_ttl() also governs how quickly SM notices new driver/browser releases (browser_ttl / driver_ttl), so retuning it to change telemetry frequency would degrade driver resolution. Telemetry cadence must be decoupled from the version-discovery TTL.

9. A fire-and-forget send isn't a confirmed delivery. The send is a detached thread::spawn with a 3s timeout; the process can exit before the POST completes, so "decided to send" ≠ "delivered." Any accounting is about decisions, not deliveries.

10. The stats schema can't change freely once SM releases independently. With Selenium Manager moving to independent releases (see [adr] Selenium Manager released API #17741), two SM versions can share one cache. Renaming, removing, or repurposing fields in se-metadata.json (e.g. changing the meaning of stats_ttl) would break older readers; only additive, serde-default fields are safe.

Status / follow-ups

The immediate fix for the overage is one fail-closed change: defer reporting until an entry written on a prior run is found still present. That resolves #1 and #2 together (with no readable, persisted cache there is never a prior entry, so nothing is sent). Nothing else here is required to get back under the limit.

The rest is tracked for later, not needed now: the robustness bugs (#3, #4), and the design assumptions (#5 to #10) to revisit only if we want more headroom, chiefly decoupling telemetry cadence from the version-discovery TTL and moving persistence detection to an environment-level signal. Two ideas from the investigation, a longer default reporting cadence and an approximate count in the payload, are optional; note Plausible can't sum a numeric property, so count is low value for now, and once-a-day vs once-an-hour barely changes the "what percent uses what" signal.

[diemol]

The immediate fix for the overage is one fail-closed change: defer reporting until an entry written on a prior run is found still present.

This makes sense. Thank you for looking into this, @titusfortner!

[Parth021105]

The main issue appears to be excessive telemetry reporting from ephemeral/containerized environments where the local cache does not persist between runs.

The proposed immediate fix makes sense: fail closed and only report when metadata from a previous run is still present and readable. This addresses both cases where the cache is unavailable and where a new/ephemeral environment would otherwise report on its first execution.

This should significantly reduce unnecessary telemetry without affecting Selenium Manager's version-discovery behavior. The other concerns around metadata file handling, atomic writes, persistence detection, and decoupling telemetry TTL from version-discovery TTL seem reasonable to track separately as follow-up improvements.

For the immediate overage, I agree that #1 and #2 are the highest priority and should be sufficient without changing the broader telemetry design.

[Riteshyadav04]

This looks like the right approach for the immediate overage issue. The fail-closed behavior is a good, minimal fix: only report when we can confirm that metadata from a previous run is still present, which should naturally eliminate telemetry from ephemeral/containerized environments without changing the existing TTL behavior.

I also agree with keeping the robustness and longer-term design changes separate from this fix. In particular, decoupling telemetry cadence from version-discovery TTL and moving persistence detection to an environment-level signal seem worth revisiting once we have the immediate reporting volume under control.

One thing I’d suggest making explicit in the implementation/tests is that any failure to read or validate the prior metadata should result in no telemetry and no crash. That gives us the fail-closed behavior we want across unreadable, missing, truncated, or non-persistent caches.

[binaryan005]

I think the immediate fix should stay narrowly focused on reducing excessive reporting rather than trying to address all of the robustness/design issues at once.

The key change is to fail closed: only report telemetry when we can confirm that a metadata entry written on a previous run is still present. This addresses both immediate problems:

1. Don't report on the first run — ephemeral/containerized environments won't generate telemetry on every Selenium Manager invocation.
2. Don't report when the cache can't be read — an unreadable or unavailable cache won't fall back to sending telemetry.

This should bring reporting back under control without changing the existing TTL semantics or coupling telemetry behavior to version discovery.

The other issues identified here (#3–#10) are worth tracking, particularly the metadata read/write robustness and the longer-term question of separating telemetry cadence from version-discovery TTL. However, I don't think they need to be part of this immediate fix.

One important constraint for the follow-ups is that se-metadata.json may be shared by independently released Selenium Manager versions, so schema changes should remain additive and use serde defaults.

Overall, I'd merge the fail-closed/previous-run change first, then address the robustness and persistence model separately if we need additional headroom.

[BotDrip]

Hi, I'd like to work on this issue, particularly the immediate fix for #1 and #2. I'll work on preventing telemetry from being sent on the first run and when the cache cannot be persisted.

[titusfortner]

@BotDrip I'm going to propose 3 options for addressing this. The more comprehensive one needs to start with recording the problems instead of fixing them. I'll update this issue shortly.