Skip to content

Commit 628b6ed

Browse files
Gab-SanMartinaStarone
authored and
EmilioCorigliano
committed
feat(testing): add optional arguments to pytest
change conftest.py in order to be able to run different configuration files. add --tests-file <file1> <file2> ... (use different configuration files) add --suffix <version> (extends suport to versioned binaries) add multi_instruction tests for c: - add.c (simple addition) - call_less_two.c (calling a function with less than two instructions) - function.c - if_then_else.c (multiline if else bodies) - phi.c (testing phi nodes) Issue #9: to close (it seems to be working correctly) Issue #10 test multi_if_then_else has been added Issue #11: related to duplication algorithm since when enabling eddi it works correctly Co-authored-by: Martina Starone <martina.starone@mail.polimi.it>
1 parent 7ee73f8 commit 628b6ed

12 files changed

Lines changed: 409 additions & 22 deletions

File tree

‎testing/README.md‎

Lines changed: 33 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,12 +4,42 @@ This directory contains utilities and scripts for testing ASPIS.
44

55
## Local Testing
66

7-
For local testing, use the `test.py` script. Configure your tests using `test_config.json`.
8-
7+
For local testing, use the `test.py` script. Configure your tests using `tests.toml`. Configure the llvm_bin flag in `llvm_bin.toml`.
8+
Then run:
99
```bash
1010
pytest test.py
1111
```
1212

13+
> To run pytest the modules listed in requirements.txt must be installed.
14+
> To install the modules:
15+
> - directly install them globally with `pip install -r requirements.txt`
16+
> - use a tool like conda
17+
> - setup a python environment:
18+
> - `python -m venv env`
19+
> - launch environment: `source env/bin/activate`
20+
21+
### Writing a configuration file
22+
23+
Test config files must be `.toml` files with the following structure for each test:
24+
25+
```toml
26+
[[tests]]
27+
test_name = <name_for_test>
28+
source_file = <relative_path_to_src_file>
29+
expected_output = <output_expected>
30+
aspis_options = <compilation_flags>
31+
```
32+
33+
> `<relative_path_to_src_file>` is a relative path from `./tests/` folder
34+
35+
### Flags
36+
37+
It is possible to write different configuration test files.
38+
39+
Additional flags are:
40+
- `--suffix <version>` : Operates the same way as aspis, searching for binaries versions denoted by `<version>`.
41+
- `--tests-file <path_to_config_file_1> <path_to_config_file_2> ...` : Use the configuration files specified.
42+
1343
## Docker Testing
1444

1545
You can also test ASPIS using Docker with the `test_docker_pipeline.py` script. This Pytest script uses Docker Compose to manage the container and execute ASPIS.
@@ -23,4 +53,4 @@ To set up and run Docker tests:
2353
3. **Run Tests:** Execute the `test_docker_pipeline.py` script:
2454
```bash
2555
pytest test_docker_pipeline.py
26-
```
56+
```

‎testing/config/racfed+eddi.toml‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
[[tests]]
2+
test_name = "racfed_eddi_ml_add"
3+
source_file = "c/multi_instruction/add.c"
4+
expected_output = "30"
5+
aspis_options = "--eddi --racfed"
6+
7+
[[tests]]
8+
test_name = "racfed_eddi_ml_function_call"
9+
source_file = "c/multi_instruction/function.c"
10+
expected_output = "foo() 25"
11+
aspis_options = "--eddi --racfed"
12+
13+
[[tests]]
14+
test_name = "racfed_eddi_preserve_runtime_sig"
15+
source_file = "c/multi_instruction/call_less_two.c"
16+
expected_output = "0"
17+
aspis_options = "--eddi --racfed"
18+
19+
[[tests]]
20+
test_name = "racfed_eddi_ml_if_then_else"
21+
source_file = "c/multi_instruction/if_then_else.c"
22+
expected_output = "1001"
23+
aspis_options = "--eddi --racfed"
24+
25+
[[tests]]
26+
test_name = "racfed_eddi_ml_phi_instruction"
27+
source_file = "c/multi_instruction/phi.c"
28+
expected_output = "SUCCESS"
29+
aspis_options = "--eddi --racfed"
30+
31+
[[tests]]
32+
test_name = "racfed_eddi_function_pointer"
33+
source_file = "c/control_flow/function_pointer.c"
34+
expected_output = "42"
35+
aspis_options = "--eddi --racfed"
36+
37+
[[tests]]
38+
test_name = "racfed_eddi_loop_exit"
39+
source_file = "c/control_flow/loop_exit.c"
40+
expected_output = "2"
41+
aspis_options = "--eddi --racfed"
42+
43+
[[tests]]
44+
test_name = "racfed_eddi_n-branch_eddi"
45+
source_file = "c/control_flow/nested-branch.c"
46+
expected_output = "6"
47+
aspis_options = "--eddi --racfed"
48+
49+
[[tests]]
50+
test_name = "racfed_eddi_s-branch_eddi"
51+
source_file = "c/control_flow/simple-branch.c"
52+
expected_output = "OK"
53+
aspis_options = "--eddi --racfed"
54+
55+
[[tests]]
56+
test_name = "racfed_eddi_s-case_eddi"
57+
source_file = "c/control_flow/switch-case.c"
58+
expected_output = "300"
59+
aspis_options = "--eddi --racfed"
60+
61+
[[tests]]
62+
test_name = "racfed_eddi_data_dep_branches"
63+
source_file = "c/data_duplication_integrity/data_dep_branches.c"
64+
expected_output = "7"
65+
aspis_options = "--eddi --racfed"
66+
67+
[[tests]]
68+
test_name = "racfed_eddi_global_var_across_functions"
69+
source_file = "c/data_duplication_integrity/global_var_across_functions.c"
70+
expected_output = "2"
71+
aspis_options = "--eddi --racfed"
72+
73+
[[tests]]
74+
test_name = "racfed_eddi_misc_data_dup"
75+
source_file = "c/data_duplication_integrity/misc_data_dup.c"
76+
expected_output = "OK"
77+
aspis_options = "--eddi --racfed"
78+
79+
[[tests]]
80+
test_name = "racfed_eddi_volatile_io"
81+
source_file = "c/data_duplication_integrity/volatile_io.c"
82+
expected_output = "42"
83+
aspis_options = "--eddi --racfed"
84+
85+
[[tests]]
86+
test_name = "racfed_eddi_arit_pipeline"
87+
source_file = "c/misc_math/arit_pipeline.c"
88+
expected_output = "3"
89+
aspis_options = "--eddi --racfed"
90+
91+
[[tests]]
92+
test_name = "racfed_eddi_mixed_ops"
93+
source_file = "c/misc_math/mixed_ops.c"
94+
expected_output = "14.5"
95+
aspis_options = "--eddi --racfed"
96+
97+
[[tests]]
98+
test_name = "racfed_eddi_xor_cypher"
99+
source_file = "c/misc_math/xor_cypher.c"
100+
expected_output = "SUCCESS"
101+
aspis_options = "--eddi --racfed"
102+
103+
[[tests]]
104+
test_name = "racfed_ml_multi_if_else"
105+
source_file = "c/multi_instruction/multi_if_then_else.c"
106+
expected_output = "r > 200"
107+
aspis_options = "--eddi --racfed"
108+

‎testing/config/racfed.toml‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
[[tests]]
2+
test_name = "racfed_ml_add"
3+
source_file = "c/multi_instruction/add.c"
4+
expected_output = "30"
5+
aspis_options = "--no-dup --racfed"
6+
7+
[[tests]]
8+
test_name = "racfed_ml_function_call"
9+
source_file = "c/multi_instruction/function.c"
10+
expected_output = "foo() 25"
11+
aspis_options = "--no-dup --racfed"
12+
13+
[[tests]]
14+
test_name = "racfed_preserve_runtime_sig"
15+
source_file = "c/multi_instruction/call_less_two.c"
16+
expected_output = "0"
17+
aspis_options = "--no-dup --racfed"
18+
19+
[[tests]]
20+
test_name = "racfed_ml_if_then_else"
21+
source_file = "c/multi_instruction/if_then_else.c"
22+
expected_output = "1001"
23+
aspis_options = "--no-dup --racfed"
24+
25+
[[tests]]
26+
test_name = "racfed_ml_phi_instruction"
27+
source_file = "c/multi_instruction/phi.c"
28+
expected_output = "SUCCESS"
29+
aspis_options = "--no-dup --racfed"
30+
31+
[[tests]]
32+
test_name = "racfed_function_pointer"
33+
source_file = "c/control_flow/function_pointer.c"
34+
expected_output = "42"
35+
aspis_options = "--no-dup --racfed"
36+
37+
[[tests]]
38+
test_name = "racfed_loop_exit"
39+
source_file = "c/control_flow/loop_exit.c"
40+
expected_output = "2"
41+
aspis_options = "--no-dup --racfed"
42+
43+
[[tests]]
44+
test_name = "racfed_nested-branch"
45+
source_file = "c/control_flow/nested-branch.c"
46+
expected_output = "6"
47+
aspis_options = "--no-dup --racfed"
48+
49+
[[tests]]
50+
test_name = "racfed_simple-branch"
51+
source_file = "c/control_flow/simple-branch.c"
52+
expected_output = "OK"
53+
aspis_options = "--no-dup --racfed"
54+
55+
[[tests]]
56+
test_name = "racfed_switch-case"
57+
source_file = "c/control_flow/switch-case.c"
58+
expected_output = "300"
59+
aspis_options = "--no-dup --racfed"
60+
61+
[[tests]]
62+
test_name = "racfed_data_dep_branches"
63+
source_file = "c/data_duplication_integrity/data_dep_branches.c"
64+
expected_output = "7"
65+
aspis_options = "--no-dup --racfed"
66+
67+
[[tests]]
68+
test_name = "racfed_global_var_across_functions"
69+
source_file = "c/data_duplication_integrity/global_var_across_functions.c"
70+
expected_output = "2"
71+
aspis_options = "--no-dup --racfed"
72+
73+
[[tests]]
74+
test_name = "racfed_misc_data_dup"
75+
source_file = "c/data_duplication_integrity/misc_data_dup.c"
76+
expected_output = "OK"
77+
aspis_options = "--no-dup --racfed"
78+
79+
[[tests]]
80+
test_name = "racfed_volatile_io"
81+
source_file = "c/data_duplication_integrity/volatile_io.c"
82+
expected_output = "42"
83+
aspis_options = "--no-dup --racfed"
84+
85+
[[tests]]
86+
test_name = "racfed_arit_pipeline"
87+
source_file = "c/misc_math/arit_pipeline.c"
88+
expected_output = "3"
89+
aspis_options = "--no-dup --racfed"
90+
91+
[[tests]]
92+
test_name = "racfed_mixed_ops"
93+
source_file = "c/misc_math/mixed_ops.c"
94+
expected_output = "14.5"
95+
aspis_options = "--no-dup --racfed"
96+
97+
[[tests]]
98+
test_name = "racfed_xor_cypher"
99+
source_file = "c/misc_math/xor_cypher.c"
100+
expected_output = "SUCCESS"
101+
aspis_options = "--no-dup --racfed"
102+
103+
[[tests]]
104+
test_name = "racfed_ml_multi_if_else"
105+
source_file = "c/multi_instruction/multi_if_then_else.c"
106+
expected_output = "r > 200"
107+
aspis_options = "--no-dup --racfed"
108+

‎testing/conftest.py‎

Lines changed: 32 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import pytest
2+
import os
23

34
def pytest_addoption(parser):
45
parser.addoption(
@@ -7,7 +8,37 @@ def pytest_addoption(parser):
78
default=False,
89
help="Run tests inside a container",
910
)
11+
parser.addoption(
12+
"--tests-file",
13+
action="store",
14+
nargs="+",
15+
default=["config/tests.toml"],
16+
help="Path to the configuration file",
17+
)
18+
parser.addoption(
19+
"--suffix",
20+
action="store",
21+
help="LLVM version suffix",
22+
)
1023

1124
@pytest.fixture(scope="session")
1225
def use_container(pytestconfig):
13-
return pytestconfig.getoption("use_container")
26+
return pytestconfig.getoption("use_container")
27+
28+
# Optional: Add a check to ensure the file exists before any tests start
29+
def pytest_configure(config):
30+
tests_file_paths = config.getoption("--tests-file")
31+
for file_path in tests_file_paths:
32+
if not os.path.exists(file_path):
33+
pytest.exit(f"Config file not found: {file_path}")
34+
35+
@pytest.fixture(scope="session")
36+
def aspis_addopt(pytestconfig):
37+
addopt = {}
38+
suffix_opt = pytestconfig.getoption("--suffix")
39+
if suffix_opt is None:
40+
addopt["suffix"] = ""
41+
else:
42+
addopt["suffix"] = "--suffix " + suffix_opt
43+
44+
return addopt["suffix"]

0 commit comments

Comments
 (0)