feat: Cloud cost anomaly detection (backport #7251) - #7258
Open
mergify[bot] wants to merge 5 commits into
Open
Conversation
The newest month in the window is only billed up to today, so measuring it against a full previous month reported every service as collapsing and the notable-change flag fired on all of them. The change column now compares the two most recent complete months, and the part month is labelled MTD. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 16840b2)
Cost Explorer was only ever read live, by month, grouped by service. That says "S3 is up" three weeks after it went up, which is neither early nor a lead. Daily cost and usage are now stored per service, usage type and region, so the history can be walked without paying Cost Explorer for every look. Alongside it we count what the business was doing each day from our own records: active sites, running machines, volume and snapshot size, and the bytes written to the backup buckets. Remote File keeps its rows after deletion, so the upload series has real history rather than starting from today. Two detectors run over each series. A spike is one loud day against the same weekday's history. A level shift is the day a series settled at a new number and stayed there, which is the shape behind a bill that is a third higher than last month without any single day looking wrong. The shift runs first, because reporting "yesterday was high" when the answer is "it changed on the twelfth" is the failure being fixed. Growth its driver kept up with is the product working and is recorded quietly. Growth its driver did not follow is filed as unexplained, with the day it started, the money it costs, and the teams or machines behind it, and goes to Telegram. Cloud Cost Drilldown walks service to usage type to region to day, all from the stored table. Off by default; enable it and add an account on Cloud Cost Settings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 7852660)
The four providers answer the cost question in three different ways, so the adapters admit that instead of pretending otherwise. OCI meters like AWS does. Its Usage API gives daily cost and quantity per service and SKU, so it is read the same way and marked Billed. Hetzner has no billing API at all, and DigitalOcean issues invoices monthly, which is too late to warn anyone. For those two the daily figure is our own inventory priced at the provider's rate card, marked Accrued so nobody reads a model of the bill as the bill. Hetzner's rates come from its live pricing endpoint; DigitalOcean prices droplets from the sizes API, and block storage and snapshots from rates on Cloud Cost Settings, because it publishes none. Hetzner traffic is the reason this was worth doing. Every server reports its own outgoing bytes against its included allowance, so overage is priced the day it starts instead of turning up on an invoice a month later. Drivers and verdicts are now per provider. Hetzner volumes growing says nothing about whether AWS storage should have grown, so each series is judged against its own provider's count and only falls back to the fleet when that provider has none. A series with no driver mapped now reaches the queue and the alert too: unexplainable is not the same as fine, and staying quiet made an unmapped series the safest place for a leak to hide. Providers bill in their own currency and there is no exchange rate in Press worth trusting, so currency is stored per row and the report shows one at a time rather than summing euros into dollars. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 2cf42c6)
Three from review. Accrued providers were detected a day late. Hetzner and DigitalOcean have no cost API, so their rows are today's inventory priced now, but detection ended at yesterday because that is the last complete day a metered provider has billed. Each account now gets the window its own source can actually fill, so the two providers with a live reading are judged the day it is taken. An account that failed to ingest was quietly reported as fine. Ingest isolates each account so one bad token does not discard three healthy providers, which means one account can be left short of today while the rest are current. Its series then end in a gap, gaps are read as zeros, and a run of trailing zeros makes a real problem look like calm. Those accounts are now skipped and named in the error log, and the healthy ones are still judged. Making the whole pipeline atomic instead would trade a quiet failure for a louder one: a single provider outage would throw away every other provider's day, and four external APIs plus a full driver scan in one transaction is a long way to hold a lock. Also split the summary sentences into their own strings. They were implicitly concatenated inside a list literal, which reads like a missing comma. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 5ee4005)
Six doctypes for one feature was too many, and two of them held data nothing ever queried. Cloud Cost Account listed which accounts to pull from. Press already knows: AWS billing credentials live in Press Settings, and OCI, Hetzner and DigitalOcean each keep their token on the cluster. Accounts are worked out from that now, so there is no list to configure and none to keep in step with the clusters. AWS stays a single payer on purpose, because per-cluster keys run machines rather than read bills, and querying a member the payer already covers would count its spend twice. Cloud Usage Anomaly Contributor held rows that were only ever read. They render into one field on the finding, and the links still open the record. Settings went from thirteen fields to eight. The statistical ones, baseline length, the spike threshold, the level shift minimum and the organic tolerance, are the shape of detection rather than something operations tunes, so they are constants worth arguing about in a pull request. What is left is the two money floors that decide how much a change has to be worth before anyone hears about it, retention, and the DigitalOcean rates. Nothing about detection changed. Same 82 tests, four more of them covering how accounts are now derived. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 6a16953)
Contributor
Author
|
Tick the box to add this pull request to the merge queue (same as
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #7258 +/- ##
==========================================
+ Coverage 59.08% 59.55% +0.47%
==========================================
Files 1026 1046 +20
Lines 93867 95233 +1366
Branches 1210 1211 +1
==========================================
+ Hits 55462 56718 +1256
- Misses 38382 38492 +110
Partials 23 23
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is an automatic backport of pull request #7251 done by [Mergify](https://mergify.com).