Benchmark on venv

[jkowalleck]

Currently, the benchmark is done based on a requirements.txt, without hashes or anything.
This is a very poor(if not the worst possible) data source for an SBOM generator.
I understand the value in this.

I would like to ask if you could also benchmark based on a python virtual environment (venv).
This would be the other extreme: the best data source possible.

[vpetersson]

Yep, this was basically designed as the 'most basic example possible' for capturing a source SBOM. There's certainly value in a build SBOM (from venv), but the drawback is that you have less state tracked in the code base.

Maybe a better approach is to improve the state of the requirements.txt file to ensure that all transitive dependencies (with hashes) are fully tracked? That way the delta between build and source should be zero.

[jkowalleck]

possible solution:

1. rename .github/workflows/python.yml -> python-requirements.yml
2. create a .github/workflows/python-venv.yml
   for each tool:
   1. install the respective tool
   2. run python -m venv subject-under-test
   3. run subject-under-test/bin/python -m pip install -r requirements.txt
   4. run the respective tool with the subject-under-test as target

[vpetersson]

The problem is that some tools do not support venv, whereas most support lockfiles.

[jkowalleck]

a requirements.txt is a bad source, since it does not actually have any package meta data.
to fetch those, tools would need to download the packages, and process them.

some tools are not intended to do this, as they are designed to run "offline".
therefore, a venv setup is preferred as a subject-under-test.

[jkowalleck]

The problem is that some tools do not support venv, whereas most support lockfiles.

This is why i suggest having 2 benchmarks: one with requirements, and one with venv.

[vpetersson]

Agreed that requirements.txt is subpar in many ways. How about using Poetry instead perhaps? It generates far more complete metadata.

[jkowalleck]

poetry lock files are niche, and might be yet another target to runbenchmarks on.

the scope of my request is python (virtual) environment. My reasoning: venv is a pythonic built-in mechanism available in (almost) all python setups.
it is the basis for accurate SBOM, and with no doubts a truthful data source.

[vpetersson]

Let me ponder on this and see what I can do later in the week. The reason why I am trying to stick to lockfiles that they are 'standard' across languages, but there are good arguments for more language specific solutions where needed.

[vpetersson]

Spent a little bit of time on this today and there are some bad news: Basically only the official CycloneDX library supports this. None of the other tools support it. In theory, I think sbom4python should support it, but I would have to poke @anthonyharrison to get the syntax format right.

[anthonyharrison]

If you try sbom4python with the --system option in a python virtual environment it will report all of the installed modules and show their inter-relationships. Let me know if this is what you are looking for. An update to sbom4python is in the works (0.12) which will improve the speed of the tool and add some further enrichment.

[vpetersson]

Thanks, @anthonyharrison. But won't the side effect of --system be that we're including the sbom4python (along with its dependencies) in the SBOM or is this filtered out?

[anthonyharrison]

No. If you have sbom4python installed on your system outside of the virtual environment but run sbom4python inside the virtual environment it should work and sbom4python won't be seen in the SBOM.

…

On Fri, 29 Nov 2024, 17:12 Viktor Petersson, ***@***.***> wrote: Thanks, @anthonyharrison <https://github.com/anthonyharrison>. But won't the side effect of --system be that we're including the sbom4python (along with its dependencies) in the SBOM or is this filtered out? — Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACAID23KRQMMOAT7NUIUNXT2DCOBXAVCNFSM6AAAAABSOGJU3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMBYGE3TGMBRGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>