Add resource consumption monitoring in the Buildomat agent

[emilyalbini]

In my Omicron CI performance work I developed an ad-hoc tool to gather CPU, memory and disk metrics for CI builds, writing them in a CSV file and uploading them as a Buildomat output. This worked well, and I could commit it to Omicron and call it a day, but having insights on the resource usage of a task is something that could prove useful for any Buildomat build.

We should polish the code I wrote in the ad-hoc tool and integrate it in the Buildomat agent, running it on every build. The agent would use the native system APIs to read metrics every second (reading /proc on Linux and calling kstat on illumos) without relying on subprocess, and upload the resulting CSV as the /buildomat/resource-usage.csv output.

We can then add a [rendered] link next to the output on the GitHub log viewer, linking to a page that renders the relevant graphs server side and displays them in the browser. I have other ideas on how to expose the graphs, but they'd rely on heavy JS which I'm not sure we want to add to the log viewer.

[jclulow]

As I said the other day, I'd really like to have this bake for a while in Omicron to get some experience with it prior to making it a first class feature.

[emilyalbini]

What are the things you'd like to specifically see tested in Omicron? So that I can make sure to gather the right data.

[jclulow]

I don't have a specific list of tests: I'm talking at an architectural level. Critically: this is functionality that doesn't need to be in the buildomat core right now in order to work as you've described it. It can run as a program in the job like any other, and can emit as outputs the CSV and even the rendered graph as an object that can just be viewed like any other output. Putting something in the core really means committing to not breaking or changing it later, like in the standard library of Rust. I'm not sure that (for example) having the core system create output artefacts like this is the right long term approach, so I would like to see it in use for a while and get a sense of how it pans out.

[emilyalbini]

I can partially implement this as a program that runs in the job, it's just really unfortunate: we then cannot have integration of resource usage in the log viewer, and I would need to go and manually add it to every single Buildomat job across the organization.

[...] and even the rendered graph as an object that can just be viewed like any other output.

Not really, because as far as I'm aware there is no way to run a command before outputs are uploaded, so we would not be able to generate the SVGs on the job right before uploading them.

Putting something in the core really means committing to not breaking or changing it later, like in the standard library of Rust.

I mean, that's a choice we are making. Buildomat is something we are developing for ourselves and as far as I know we are almost the only users of (with only two other repositories using it on GitHub, one of which by Rain). We have our own custom CI system, we should take advantage of that and expand it to suit our needs.

If we do want to stick to strict backwards compatibility, we can just label the feature as experimental.