Faster count implementation that's still quite accurate

[brandur]

I wasn't particularly surprised to pop open our PlanetScale report this
morning and see that the count-by-state query used in River UI is now
the demo's most expensive query by cumulative time:

select state, count(*) from river_job group by state
Count: 59,386   ·   p99: 13,131 ms   ·   Cache hit: 87.5%

This has been a known problem for quite some time both in Postgres and
in River UI. The demo's now up to 1.6M completed rows, so counts are
getting slower by the day.

I was having Codex help brainstorm ways that this could be improved, and
it came up with what I think is quite a clever strategy that should be
very fast with minimum downsides:

• The count endpoint starts out with an optimistic query that tries to
  do a full count by all states, but puts a limit of 10k rows on any
  particular one.

• If only the constrained 10k+ information is available, that's what's
  shown, but we immediately try to get a full exact count of all rows
  because even if you have a lot of rows, it's still better to know that
  you have 10,001 versus 50k versus 200k, versus 5M. This longer count
  is kicked off in the background, and is refreshed every 1-30 minutes,
  depending on how long the count is taking. Its results are used when a
  reasonably fresh cache value is available so we can show users the
  best available number. Even when a cached value is available, we still
  prefer a more fresh capped count for states that don't exceed 10k.

• In Postgres, if no cached exactly count is available (most commonly
  right after startup), we use a planner estimate to find a rough
  number. This value will only be in play for a short time until an
  exact count is available.

The type of count (exact, exact_cached, estimated, lower_bound)
is communicated o the UI so that it can give context on counts in
tooltips. For example, it might show 12.3M, ≈987.7K, or 10K+ depending
on the situation, along with source and freshness.

I ran a benchmark and you can see that at large numbers doing a bounded
count stays orders of magnitude more responsive. This might seem like a
small thing, but it keeps the UI more up-to-date and responsive even for
very large users, which is very good.

Rows	Table + indexes	Existing exact count	Bounded count	Planner estimate	Bounded speedup
100K	17 MB	7.16 ms	0.93 ms	0.47 ms	7.7×
1M	174 MB	24.45 ms	1.09 ms	0.66 ms	22×
10M	1.7 GB	203.54 ms	1.01 ms	0.59 ms	201×

I'm sort of hoping that this is a nice compromise for all things -- i.e.
fast at small numbers, reasonably fast at large numbers, and still keeps
precise numbers so we don't have to get too abstract. The downside is
more code complexity, but Codex seems to have done a decent job of
implementation (and I tweaked a bunch of stuff for style) and we have
pretty good tests.

[brandur]

@bgentry Want to take a look at this one? I think it's an improvement UI-wise, but should also be some great blog post material.