github CI: enable use of port tests

[catap]

Description

A lot of ports supports self testing. Here I add to GitHub CI a way to run tests by ports when it is supported.

Why? To increase quality of PRs :)

Depends on macports/mpbb#16

Type(s)

• enhancement

Tested on

Verification

Have you

• followed our Commit Message Guidelines?
• squashed and minimized your commits?
• checked that there aren't other open pull requests for the same change?

[pmetzger]

I think in general this is a good idea, but it might use much more CPU, so I think other people need to agree, not just me.

[reneeotten]

The conclusion back in the days was that this would take up too much CI time. As far as I'm concerned that hasn't changed so I don't think we should do this.

In the end, people submitting a PR are supposed to have run tests locally as listed in the checklist before submitting a PR. (Admittedly, the same holds true for building the port).

[catap]

@reneeotten here a two points.

1. CI builds it on clean environment and not with another ports, which is far less close to build bot, and allows to catch some issues linked to missed dependencies which might be missed when build it on non clean environment;
2. The same true for test.

Thus, I really doubt that a lot of people runs test for python port, special if we have 5-7 of different sub ports.

Regarding CPU time... well.. it affects only ports which has test.run yes inside, and I not sure that all heavy port likes LLVM has it. In fact majority of ports doesn't.

Plus we're talking of Github CI, and not build bot's time.

I don't see why we can't merge it to test and if slow things down to revert it. For example we may plan to make an experiment for September. What do you think about that?

[reneeotten]

These are not new arguments, and while I do understand them it doesn't change my opinion.

People submitting a PR should build the port and run the test if available. Both of them should be done in trace-mode to catch possible issue you are pointing out. The fact that trace-mode is still not working on newer OSes is unfortunate, but not something I am able to fix in macports-base.

If you feel strongly about it please start a discussion on the -devel mailinglist.

[pmetzger]

I think maybe it would be a good idea to get enough hardware that we can afford to run the tests as part of the CI. But yes, it should be a -devel discussion.

[catap]

I think maybe it would be a good idea to get enough hardware that we can afford to run the tests as part of the CI.

As far as I know CI utilizes Github actions which is provided by Github / Microsoft with some limits like 3 or 4 instances running at the same time per project.

=> I not sure which hardware you're referer here.

But yes, it should be a -devel discussion.

Ok, I'll draft email with my point in days.

[pmetzger]

My point was we could get our own hardware, and then we might have fewer limitations.

[ryandesign]

Many ports have test phases that can be completed quickly, but a few ports' test phases can take hours to run.

What is your intended outcome if a CI run shows that the test suite fails? Do you expect port maintainers or other contributors to engage with the developers to fix a test suite failure before you would accept a PR? What if the test suite had already been producing that error before that PR? How would you know if that were the case?

In my view, it's the developers' responsibility to run their test suites and fix any problems, not ours.

[pmetzger]

@ryandesign I think that, given enough hardware, we'd want to know what test suites were passing and aren't passing in an update. Without enough hardware, that's of course not possible. However, if the tests pass, it really does make it much easier to know that an update, especially for a critical package (say openssl or something) is not going to cause trouble.

[catap]

I understand that not all ports currently have tests, but a lot of them does.

My goal here is to enhance the quality of tests for the ports, which will allow me to proceed with a bot that runs port livecheck... and detects when ports are updated. Subsequently, it will open a PR to update a port with the updated version and checksums, but only if such a port has tests.

Why? To automate the process of updating ports. We have a vast number of ports, and many of them can be updated quite easily. Enabling the execution of port tests on GitHub will ensure that such automatically generated PRs are of sufficient quality to be merged.

What's the ultimate goal? To keep MacPorts dynamically updated.

P.S. Almost the same text was sent to macports-dev@.

[artkiver]

What's the ultimate goal? To keep MacPorts dynamically updated.

I can kind of imagine with what you are going for here, and the thought had crossed my mind on occasion to create some tools to attempt to simplify my process for updating some MacPorts as well, but to quote Han Solo, "I've got a bad feeling about this."

Over-automation is something which concerns me; I have seen such efforts "go wrong" in prod too many times, causing more problems than they professed to fix and really ruining this particular ops' day.

I don't want to speak for @neverpanic with regards to @pmetzger's comment about "especially for a critical package (say openssl or something)" but as someone who is active as a co-maintainer for the adjacent libressl and libressl-devel MacPorts, easing the checksums and file size check replacements in the Portfile is about the least of what is required. In particular, LibreSSL also utilizes signify (https://man.openbsd.org/signify.1) for signing releases which I check as part of the due diligence in updating those MacPorts.

To say nothing of various MacPorts which still aren't utilizing path:lib/libssl.dylib:openssl efficaciously (as a continued tangent, libssh2 and wget both seem to want to install openssl3 even if libressl or libressl-devel are installed and the upstream projects AFAIK, have been tested against LibreSSL. I have attempted, with limited success to improve some of these Portfiles locally but haven't submitted PRs to reflect such efforts yet as I still run into edge cases on occasion; unlike xar for example, where modifying that Portfile seemed to be a bit more seamless).

I realize I am relatively junior as a MacPorts contributor and still scratching my head a lot of the time and running into issues which other contributors have thankfully been immensely helpful with me in their guidance. Nonetheless, I think I echo @ryandesign's concerns here that it should really be up to the maintainers to run the tests.

I am still learning the nuances between CI and mpbb/build bots and have no idea how CI resources are allotted, yet @reneeotten citing previous concerns about taking up too much CI time, given that it's not uncommon for me to see CI tests queued after I submit a PR, doesn't seem like an unreasonable one unless something major has changed with GitHub's resource allocation to MacPorts to which I am unaware?

I still haven't gotten a local mpbb environment set up yet (it is on my list of things todo and I made a preliminary failed attempt or two) but I imagine that might be a better place for improvements to go rather than GitHub's CI frameworK?

Apologies if my "two cents" was a bit TL;DR but especially after I saw this hit the mailing list I felt a need to chime in, thanks!

[catap]

Over-automation is something which concerns me; I have seen such efforts "go wrong" in prod too many times, causing more problems than they professed to fix and really ruining this particular ops' day.

This isn't automatical update which simple bumps everything into master.

An idea to introduce semi-automatical update of ports which opens PR which is tested on Github CI and can be merged (manually).

For example Nix uses that for last couple of years: https://github.com/r-ryantm/nixpkgs-update

[cooljeanius]

Related: Trac ticket 42731: https://trac.macports.org/ticket/42731
(not sure if that should be seen as an alternative, or complementary?)

[reneeotten]

this is clearly not going to be merged in its current state. If there a are changes with the next release of base one can start a start on the -devel list. If there is consensus there it can be progressed into a PR.