Add sonar to project for quality checks (#452)

* Add sonar to project for quality checks

* Move pysonar to direct dependencies

* Add changelog entry

* Switch to nox task

* Continue nox task, add to templates, & update documentation

* Fix template .gitignore & inherit secrets for report

* Add new nox tasks to main tasks.py

* Fix to relative path from . for coverage.xml & session

* Add organization as required for pysonar

* Add statement to docs about adding to branch protections

* Rename variable to avoid code smell, as built-in

* Remove f from non-f-string

* Fix typo in documentation

* Move and rename to correct test folder

* fixup! Rename variable to avoid code smell, as built-in

* Fix class name to fit convention

* Group copy_artifact tests together

* Use variables to ensure names consistent throughout usually subsequent operations

* Simplify test and move into correct file

* Remove file check as already done before validating them

* Switch validations to booleans and simplify to all or fail

* Move over and simplify test for is_valid_lint_txt

* Move over and simplify test for is_valid_lint_json and mirror to is_valid_lint_txt

* Move over and simplify test for is_valid_security_json

* Convert to handle validation error

* Move over and simplify test for is_valid_coverage

* Switch to simple assert

* Add test for check_artifacts and switch prints to all bey stderr

* Fix warnings in sonar upload output

* Remove code smells where non-f-strings

* Switch python_files to be Iterable[str] as only used that way & restrict bandit to source files

* Add type ignore for unpacked list into session

* Project fix

* Fix comment

* Add inherit secrets for sonar to pr-merge.ymls

* Reduce scope of pylint to that of package

* Update documentation with summary and make clearer private vs public repos

* Re-lock dependencies

* Modify documentation per review

* Rename to sonar:check

Author: ArBridgeman

.github/workflows/ci.yml

@@ -1,26 +1,21 @@
 name: CI
 
 on:
-  push:
-    branches-ignore:
-      - "github-pages/*"
-      - "gh-pages/*"
-      - "main"
-      - "master"
+  pull_request:
+      types: [opened, synchronize, reopened]
   schedule:
-    # “At 00:00 on every 7th day-of-month from 1 through 31.” (https://crontab.guru)
+    # At 00:00 on every 7th day-of-month from 1 through 31. (https://crontab.guru)
     - cron: "0 0 1/7 * *"
 
 jobs:
-
   CI:
     uses: ./.github/workflows/merge-gate.yml
     secrets: inherit
     permissions:
       contents: read
-
   Metrics:
     needs: [ CI ]
     uses: ./.github/workflows/report.yml
+    secrets: inherit
     permissions:
       contents: read

.github/workflows/pr-merge.yml

@@ -25,5 +25,6 @@ jobs:
   metrics:
     needs: [ ci-job ]
     uses: ./.github/workflows/report.yml
+    secrets: inherit
     permissions:
       contents: read

.github/workflows/report.yml

@@ -35,6 +35,9 @@ jobs:
       - name: Generate Report
         run: poetry run -- nox -s project:report -- --format json | tee metrics.json
 
+      - name: Upload to sonar
+        run: poetry run -- nox -s sonar:check -- ${{ secrets.SONAR_TOKEN }}
+
       - name: Upload Artifacts
         uses: actions/upload-artifact@v4.6.2
         with:

.gitignore

@@ -7,6 +7,7 @@ odbcconfig/odbcinst.ini
 .html-documentation
 
 .coverage
+.sonar
 
 _build/
 

doc/changes/unreleased.md

@@ -1 +1,29 @@
 # Unreleased
+
+## Summary
+This version of the PTB adds nox task `sonar:check`, see #451. This allows us to
+use SonarQube Cloud to analyze, visualize, & track linting, security, & coverage. In
+order to properly set it up, you'll need to do the following instruction for each **public** project.
+At this time, PTB currently does not support setting up SonarQube for a **private** project.
+
+1. Specify in the `noxconfig.py` the relative path to the project's source code in `Config.source`
+    ```python
+        source: Path = Path("exasol/toolbox")
+    ```
+2. Add the 'SONAR_TOKEN' to the 'Organization secrets' in GitHub (this requires a person being a GitHub organization owner).
+3. Activate the SonarQubeCloud App
+4. Create a project on SonarCloud
+5. Add the following information to the project's file `pyproject.toml`
+    ```toml
+        [tool.sonar]
+        projectKey = "com.exasol:<project-key>"
+        hostUrl = "https://sonarcloud.io"
+        organization = "exasol"
+    ```
+6. Post-merge, update the branch protections to include SonarQube analysis
+
+## ✨ Features
+* #451: Added nox task to execute pysonar & added Sonar to the CI
+
+## ⚒️ Refactorings
+* #451: Reduced scope of nox tasks `lint:code` (pylint) and `lint:security` (bandit) to analyze only the package code

doc/user_guide/getting_started.rst

@@ -179,8 +179,8 @@ forward, and you just can use the example *noxfile.py* below.
 
 .. _toolbox tasks:
 
-7. Setup for deploying documentation (optional)
-+++++++++++++++++++++++++++++++++++++++++++++++
+7. Set up for deploying documentation (optional)
+++++++++++++++++++++++++++++++++++++++++++++++++
 Within the `gh-pages.yml`, we use the GitHub `upload-pages-artifact` and `deploy-pages`
 actions. In order to properly deploy your pages, you'll need to reconfigure the GitHub
 Pages settings for the repo:
@@ -199,8 +199,32 @@ We also need to configure settings for github-pages environment:
 5. In the 'Deployment branches and tags', click 'Add deployment branch or tag rule'
 6. Select 'Ref type' to be 'Tag' and set the 'Name pattern' to `[0-9]*.[0-9]*.[0-9]*` (or whatever matches that repo's tags)
 
+8. Set up for Sonar
++++++++++++++++++++
+PTB supports using SonarQube Cloud to analyze, visualize, & track linting, security, &
+coverage. In order to properly set it up, you'll need to do the following instructions
+for each **public** project. At this time, PTB currently does not support setting up
+SonarQube for a **private** project.
 
-8. Go 🥜
+1. Specify in the `noxconfig.py` the relative path to the project's source code in `Config.source`
+    .. code-block:: python
+
+        source: Path = Path("exasol/toolbox")
+2. Add the 'SONAR_TOKEN' to the 'Organization secrets' in GitHub (this requires a person being a GitHub organization owner).
+3. Activate the SonarQubeCloud App
+4. Create a project on SonarCloud
+5. Add the following information to the project's file `pyproject.toml`
+    .. code-block:: toml
+
+        [tool.sonar]
+        projectKey = "com.exasol:<project-key>"
+        hostUrl = "https://sonarcloud.io"
+        organization = "exasol"
+6. Post-merge, update the branch protections to include SonarQube analysis
+
+
+
+9. Go 🥜
 +++++++++++++
 You are ready to use the toolbox. With *nox -l* you can list all available tasks.