Include constant-time analysis framework by pablo-gf · Pull Request #2449 · open-quantum-safe/liboqs

pablo-gf · 2026-05-29T09:32:13Z

This PR includes the constant-time testing framework developed throughout the course of the LFX Mentorship project supervised by @bhess. These are the main features of this framework:

It is based on two dynamic analysis tools: Valgrind's memcheck with the Kyberslash patch (renamed to Valgrind-Varlat), and Clang's MemSanitizer (MemSan) to detect possible sources (warnigns) of non-constant-time behavior.
Provides unified scripts for running constant-time checks both in the CI pipeline and locally.
Includes a script that aggregates analysis outputs and generates visual graphs to identify and track constant-time warning trends.
Includes several documentation files outlining the installation, setup, and usage process.

I would be happy to walk through the details of any of these features either here or during a status call. You can also find detailed guides in the documentation files I’ve added under tests/ct_tooling and its subdirectories. Feel free to throw any feedback or comments so that we can move this from draft to "ready for review"!

Does this PR change the input/output behaviour of a cryptographic algorithm (i.e., does it change known answer test values)? (If so, a version bump will be required from x.y.z to x.(y+1).0.)
Does this PR change the list of algorithms available -- either adding, removing, or renaming? Does this PR otherwise change an API? (If so, PRs in fully supported downstream projects dependent on these, i.e., oqs-provider will also need to be ready for review and merge by the time this is merged. Also, make sure to update the list of algorithms in the continuous benchmarking files: .github/workflows/kem-bench.yml and sig-bench.yml)

AI assistance was used to generate documentation and minor editing. The resulting code was reviewed, edited and verified for accuracy.

Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>

coveralls · 2026-05-29T10:51:29Z

Coverage is 82.182% — pablo-gf:ct-tooling into open-quantum-safe:main. No base build found for open-quantum-safe:main.

bhess

Thank you @pablo-gf for this work on improving the constant‑time tooling in liboqs, this is an important step towards more exhaustive ct-testing in liboqs.

I’ve left a few initial comments inline.

I would be happy to walk through the details of any of these features either here or during a status call.

That would be excellent. @dstebila / @RodriM11: could we allocate some time in an upcoming status call for @pablo-gf to walk us through this contribution? I think a walkthrough would be very valuable.

bhess · 2026-05-29T15:02:41Z

+  workflow_dispatch:
+
+jobs:
+  interactive-inputs:


My understanding is that this job requires interactive input to select the algorithms. Is that correct? If so, what is the intended usage, and can it be configured to run non‑interactively (e.g., for inclusion in the weekly CI runs)?

Yes, that is correct. The original idea was to execute these tests selectively—like when introducing a new algorithm or modifying the implementation of one. In a previous discussion with @dstebila, we noted the importance of filtering out false-positive warnings to prevent spreading false alarms. I went for the interactive-input setup so that we can execute CT tests only on those algorithms that have gone through false-positive exclusion. If we want to include this testing in CI, we can make it non-interactive, but that would require hard-coding the algorithms that went through false positive exclusion. Another option is simply running the tests on all algorithms even if that means retrieving many warnings that may be false-positives. However, this may require too many resources due to the wide range of variables handled by the framework, which was another reason why I went for the interactive-input setup. I think it comes down to what we want the ultimate purpose of this framework to be in liboqs. Are we looking for a standard CI test that runs automatically, or a specialized tool for specific occasions?

bhess · 2026-05-29T15:05:05Z

+                  display: Select the algorithm(s) to execute valgrind-varlat constant-time testing on
+                  type: multiselect
+                  choices:
+                    - "BIKE-L1"


My concern with this list is maintainability: if the set of algorithms in liboqs changes, the list would need to be updated manually. Is there a way to generate this list dynamically based on the algorithms currently available?

Thank you for pointing this out. We could fix this by retriving all available algorithms using the available_kems_by_name() and available_sigs_by_name() from helpers.py. I have included it in the latest commit, and the tests are working on my fork.

Now that I think about it, I could also create a separate PR to fix this same issue for the kem-bench.yaml and sig-bench.yaml workflows with the same solution, if you find that appropiate.

bhess · 2026-05-29T15:12:23Z

@@ -0,0 +1,396 @@
+// SPDX-License-Identifier: MIT


The test_sig.c / test_kem.c files and the corresponding CMakeLists.txt appear to be largely copied from the versions in the /tests directory. Could we avoid duplicating these files and instead reuse the originals with additional macro definitions (e.g., to disable signature verification during ct-testing)? Or is there a specific reason why maintaining separate copies is necessary here?

Yes, I think that sounds like a better approach over the preliminary copy-pasting approach. I have updated it in the last commit and the tests are passing in my fork.

bhess · 2026-05-29T15:13:45Z

@@ -0,0 +1,15 @@
+fun:PQCP_MLKEM_NATIVE_MLKEM*_C_poly_decompress_d*


I really like this approach, having an allowlist mechanism similar to the Valgrind one makes a lot of sense.

bhess · 2026-05-29T15:27:27Z

+          INSTALL_PREFIX="$PWD/valgrind_varlat"
+
+          echo "Cloning Valgrind's source code"
+          git clone git://sourceware.org/git/valgrind.git valgrind_varlat_src> /dev/null 2>&1 || true


The statements with || true would silently fail.

Thanks! Forgot to remove these after testing. Should I also remove > /dev/null 2>&1? I initially kept them so that the output in GitHub Actions looks cleaner and it is easier to spot the pass/fail of tests, but we can keep the output if preferred.

RodriM11 · 2026-05-29T15:28:48Z

That would be excellent. @dstebila / @RodriM11: could we allocate some time in an upcoming status call for @pablo-gf to walk us through this contribution? I think a walkthrough would be very valuable.

For sure. Next week is TSC meeting, so we could do it the 9th of June, if @pablo-gf is available.

bhess · 2026-05-29T15:30:02Z

+            done
+
+            # Create backup files of the original tests files
+            mv "$LIBOQS_DIR/tests/CMakeLists.txt" "$LIBOQS_DIR/tests/CMakeLists.txt.bak"


This looks brittle since it makes assumptions about the liboqs file structure. IMO avoiding replacing the original files and instead work CT-specific macros in the original files would be more robust.

Fixed per the previous comment.

Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>

pablo-gf · 2026-06-01T17:11:49Z

I’ve left a few initial comments inline.

Thank you for your comments, @bhess. Let me know if you have any further ideas/suggestions.

For sure. Next week is TSC meeting, so we could do it the 9th of June, if @pablo-gf is available.

Sounds good @RodriM11. See you next week then!

Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>

+
+      - name: Example Interactive Inputs Step
+        id: interactive-inputs
+        uses: boasiHQ/interactive-inputs@v2


+
+      - name: Install Python deps
+        run: python -m pip install pytest requests
+


+
+      - name: Example Interactive Inputs Step
+        id: interactive-inputs
+        uses: boasiHQ/interactive-inputs@v2


pablo-gf · 2026-06-03T07:22:55Z

A couple of questions, @bhess:

With respect to the bot warnings: Is the "verified creator" an issue in this case? And would the pip command also require a pin by hash here?

Also, the failing job in CI seems to be a network issue during the docker pull. Any chance that could be re-executed?

bhess · 2026-06-03T08:20:02Z

With respect to the bot warnings: Is the "verified creator" an issue in this case? And would the pip command also require a pin by hash here?

I’m not entirely sure what to make of the “verified creator” warning, but it may have security implications. From what I can tell, this action is mainly meant for interactive use.
As you noted in the inline comment, this shouldn’t be necessary for regular CI runs if we run it non‑interactively, does that sound right?
For manual runs, could we instead trigger the action manually and pass the required parameters (the algorithm to test) explicitly?

Also, the failing job in CI seems to be a network issue during the docker pull. Any chance that could be re-executed?

Done

github-advanced-security AI found potential problems May 29, 2026

View reviewed changes

Comment thread .github/workflows/ct-tooling.yml Fixed

github-advanced-security AI found potential problems May 29, 2026

View reviewed changes

Comment thread .github/workflows/ct-tooling.yml Fixed

Add ct tooling logic

bd47e84

Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>

pablo-gf force-pushed the ct-tooling branch from 449b48e to bd47e84 Compare May 29, 2026 10:48

bhess reviewed May 29, 2026

View reviewed changes

fixed comments

e1874c3

Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>

pablo-gf added 4 commits June 2, 2026 10:57

fixed ci errors

017916a

Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>

more CI fixes

0284430

Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>

cleaned

3edf0de

Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>

more cleaning

e6c3a4c

Signed-off-by: Pablo Gutiérrez <pablogf@uma.es>

github-advanced-security AI found potential problems Jun 2, 2026

View reviewed changes

Comment thread .github/workflows/ct-tooling.yml

- name: Example Interactive Inputs Step

id: interactive-inputs

uses: boasiHQ/interactive-inputs@v2

github-advanced-security AI found potential problems Jun 2, 2026

View reviewed changes

Comment thread .github/workflows/ct-tooling.yml

- name: Install Python deps

run: python -m pip install pytest requests

Comment thread .github/workflows/ct-tooling.yml

- name: Example Interactive Inputs Step

id: interactive-inputs

uses: boasiHQ/interactive-inputs@v2

		@@ -0,0 +1,15 @@
		fun:PQCP_MLKEM_NATIVE_MLKEM_C_poly_decompress_d


		- name: Install Python deps
		run: python -m pip install pytest requests

Conversation

pablo-gf commented May 29, 2026

Uh oh!

Uh oh!

Uh oh!

coveralls commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bhess left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RodriM11 commented May 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pablo-gf commented Jun 1, 2026

Uh oh!

pablo-gf commented Jun 3, 2026

Uh oh!

bhess commented Jun 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

coveralls commented May 29, 2026 •

edited

Loading