Beta test on the bot

[francoto]

In order to make sure the bot is properly handling the different scenarios we discuss in the bot implementation, this issue will list the use cases we want to see in the tested repositories:

It should covers scenario described in the documentation

1. no previous analysis, without codemeta file and pitfalls
2. previous analysis, nothing changed
3. previous analysis, unsubscribe
4. previous analysis, analysis identical (new commit id) but no pitfalls fixed, previous issue closed
5. previous analysis, analysis identical (new commit id) but no pitfalls fixed, previous issue opened
6. previous analysis, analysis different with pitfalls, previous issue still opened
7. previous analysis, analysis different with pitfalls, previous issue closed
8. previous analysis, analysis different with no pitfalls, previous issue still opened
9. previous analysis, analysis different with no pitfalls, previous issue closed

[francoto]

Following up the list posted before, here is a suggestion list attributing the different scenarios to different repositories in the SoftwareUnderstadning group. @Anas-Elhounsri This is only a suggestion, if you wants to swap some repos between two scenarios, feel free to do so. It is only to make sure we cover all scenarios.

The format will be :
- [ ] scenario to create -> example repo

I only have 9 scenarios. so some repos will be left which is ok considering some of these are only a single readme, website or data.

Changes to make

• 2. previous analysis, no change -> somef server
• 3. previous analysis unsubscribe -> ScientificSoftwareKG
• 4. previous analysis, same analysis (new commit id but no change on pitfalls), previous issue closed -> Metadata-Adoption-Quantify
• 5. previous analysis, analysis identical (new commit id) but no pitfalls fixed, previous issue opened -> inspect4py
• 6. previous analysis, analysis different with pitfalls, previous issue still opened -> rolf
• 7. previous analysis, analysis different with pitfalls, previous issue closed -> completeR
• 8. previous analysis, analysis different with no pitfalls, previous issue still opened -> auto-papers-with-artifacts
• 9. previous analysis, analysis different with no pitfalls, previous issue closed -> c2t

[francoto]

I pushed the first analysis on dev branch https://github.com/SoftwareUnderstanding/sw-metadata-bot-monitor-web/tree/beta_evaluation_04_2026

[francoto]

I already adressed some issues :
3. commented unsubscribe
5. created a PR with codemeta.json. I don't know if it would change the actual analysis by RSMetacheck with the presence of the codemeta.json but it would be good to know. We could add another usecase where nothing has changed (small commit that changed nothing about the metadata quality).

[Anas-Elhounsri]

@francoto I followed the guide you did and pushed the fixes where it needed.

[francoto]

Analysis generated at : 2026-04-28T12:57:50Z

s id	repo	initial P|W	what we expect	what happened	notes
1
2	[somef server](https://github.com/SoftwareUnderstanding/somef_server)	P002|W002	same commit id -> replicate analysis	action: skipped, reason: repo_not_updated	🟢
3	[ScientificSoftwareKG](https://github.com/SoftwareUnderstanding/ScientificSoftwareKG)	W002|W004	repo url added to the opt-out list	action: skipped, reason: repo_not_updated	🟠 repo not updated > unsubscribe
4	[Metadata-Adoption-Quantify](https://github.com/SoftwareUnderstanding/Metadata-Adoption-Quantify)	P001|P002|P014|P016|W004	open new issue identical to previous one	"action": "skipped","reason_code": "identical_and_issue_open",	🔴 issue closed not detected correctly
5	[inspect4py](https://github.com/SoftwareUnderstanding/inspect4py)	W001	do nothing	"action": "failed", "reason_code": "missing_pitfall_file",	🔴 action failed ?
6	[rolf](https://github.com/SoftwareUnderstanding/rolf)	P001|P002|W001	comment existing issue with new analysis	"action": "skipped", "reason_code": "repo_not_updated",	🔴 rolf has not been updated ...
7	[completeR](https://github.com/SoftwareUnderstanding/completeR)	W001	open new issue	"action": "skipped", "reason_code": "repo_not_updated",	🔴 repo has not been updated ...
8	[auto-papers-with-artifacts](https://github.com/SoftwareUnderstanding/auto-papers-with-artifacts)	P002|W001|W002|W003|W004|W005	close existing issue with comment	"action": "skipped", "reason_code": "repo_not_updated",	🔴 repo not updated
9	[c2t](https://github.com/SoftwareUnderstanding/c2t)	P002|W002|W004	do nothing	"action": "updated_by_comment", "reason_code": "changed_and_issue_open",	🔃 repo has been updated and bot still found pitfalls (s 6) => 🟢

[francoto]

Some repos have not been actually updated but we will address this soon.

There is a potential issue with the repository inspect4py

There is a potential issue when using unsubscribe (s3):

• repo has not been updated is bypassing the decision tree and so the repo url has not been added to the opt-out. I think we should also close the issue if we detect the "unsubscribe" keyword.

[francoto]

I create an associated issue about the inspect4py on RSMetacheck. It seems the pitfalls.jsonld is not created correctly SoftwareUnderstanding/RsMetaCheck#75

[Anas-Elhounsri]

A quick update on the subject:

• completeR has been updated and merged (added codemeta.json)
• For rolf I suggest using another repository, because it seems that there is a problem with codemeta.json file detection (it reports as missing but there is codemeta.json file from 2 months agon, maybe the analysis is using an old commit?). We can study this case on RSMetaCheck.
• For auto-papers-with-artifacts, I noticed RSMetaCheck wrongly flags some pitfalls (I am working on a new release that will fix these issues). Should we study this one as it is or after I patch the issues from my side?

[francoto]

Thank you Anas for this review : I agree we can try to use other repos that the one not working correctly.

We should still keep these ones to make sure we fix the discovered issues.
But we indeed want to make sure the bot is working as expected

[francoto]

Do you have other candidates in mind ?
I think a lot of others repositories are website, single readme files, or data.

We can use RSMetacheck or sw-metadata-bot repos if we have no other candidates. As I say, we want to make sure that the bot is behaving correctly.

[Anas-Elhounsri]

I think we should use them both instead, we will also later test on the previous repos to see if we fixed the issues. I have also tried to look for other repositories but couldn't find a suitable one except for RSMetaCheck and the bot itself as well.

[francoto]

Ok I replace the repos not handled correctly by RSMetacheck like this:

• rofl -> RSMetacheck (scenario 6)
• auto-paper-with-artifcats -> sw-metadta-bot (scenario 8)

Then I rerun analysis (twice to replicate the scenario 6 and 8 for the new repos not part of the first set of repos)

[Anas-Elhounsri]

Update:

• RSMetaCheck follows scenario 6
• sw-metadata-bot follows scenario 8

I have merged the new commits on both repos.

[francoto]

All issues that has been spotted by this beta analysis will have a title containing "[Beta test SU 04/2026]"