Pattern repair - help for reviewing patterns

docs/How-to-run-CLI-Usage.md

@@ -22,6 +22,8 @@ The following main commands are currently implemented:
   - [`discovery`](./How-to-run-discover-measured-patterns.md): discover measured patterns within a project source code
   - [`manual-discovery`](./How-to-run-manual-discovery.md): execute discovery rules (normally associated to patterns) within a project source code
 - reporting: create reports about SAST measurement and/or pattern discovery (**TODO**)
+- [`checkdiscoveryrules`](./How-to-run-checkdiscoveryrules.md): Check/test the discovery rules of the pattern instances on the pattern instances themselves.
+- [`patternrepair`](./How-to-run-patternrepair.md): Helps you keeping your pattern catalogue nice and tidy.
 
 The following are under-investigation:
 

docs/How-to-run-Measure-SAST-tools-over-patterns.md

@@ -52,6 +52,10 @@ Instead of specifying certain pattern ids, you can use `-a`.
 ]
 ```
 
+The value in `result` is a boolean value, where 'true' signifies the tool's correct output matching the expected result, while 'false' indicates an incorrect outcome.
+For example, if it is expected, that the pattern does not contain a vulnerability, ('expectation': false) and the tool result is 'no vulnerability', than the value of `result` will be true.
+If it is expected that the pattern contains a vulnerability ('expectation': true) and the tool does not detect that, the `result` field will be false.
+
 ## Example
 
 Here a simple example that will measure patterns 1, 2, 4 and 7 from the PHP catalog with 3 workers:

docs/How-to-run-checkdiscoveryrules.md

@@ -0,0 +1,43 @@
+# How to run: Checkdiscoveryrules
+
+## Overview
+
+This commands allows to run the discovery rule on the pattern instance itself.
+
+## Command line
+
+To check discovery rules on your pattern run:
+
+```bash
+tpframework checkdiscoveryrules --help
+usage: tpframework [OPTIONS] COMMAND checkdiscoveryrules [-h] (--print | --export EXPORTFILE) -l LANGUAGE (-p PATTERN_ID [PATTERN_ID ...] | --pattern-range RANGE_START-RANGE_END | -a)
+                                                         [--tp-lib TP_LIB_DIR] [-s NUMBER] [--output-dir OUTPUT_DIR]
+
+options:
+  -h, --help            show this help message and exit
+  --print               Print measurements on stdout.
+  --export EXPORTFILE   Export measurements to the specified csv file.
+  -l LANGUAGE, --language LANGUAGE
+                        Programming language targeted
+  -p PATTERN_ID [PATTERN_ID ...], --patterns PATTERN_ID [PATTERN_ID ...]
+                        Specify pattern(s) ID(s) to test for discovery
+  --pattern-range RANGE_START-RANGE_END
+                        Specify pattern ID range separated by`-` (ex. 10-50)
+  -a, --all-patterns    Test discovery for all available patterns
+  --tp-lib TP_LIB_DIR   Absolute path to alternative pattern library, default resolves to `./testability_patterns`
+  -s NUMBER, --timeout NUMBER
+                        Timeout for CPG generation
+  --output-dir OUTPUT_DIR
+                        Absolute path to the folder where outcomes (e.g., log file, export file if any) will be stored, default resolves to `./out`
+```
+
+## Example
+
+Here a simple example that will run checkdiscoveryrules on the first PHP pattern and print the results to the cmd.
+`tpframework checkdiscoveryrules -p 1 -l php --print`
+
+Note: Minimum requirement for this command is a pattern, a language and either `--print` or `--export`.
+
+## Required fields in instance `json` metadata
+
+The explanation for the instance `json` metadata can be found [here](https://github.com/testable-eu/sast-testability-patterns/blob/master/docs/testability-patterns-structure.md)

docs/How-to-run-patternrepair.md

@@ -0,0 +1,62 @@
+# How to run: Patternrepair
+
+## Overview
+
+This commands should help you reviewing your testability catalogue and keep it nice and tidy.
+It might also help you, repair patterns, that are broken.
+
+## Command line
+
+To start repairing/reviewing your patterns runs:
+
+```bash
+tpframework patternrepair --help
+usage: tpframework [OPTIONS] COMMAND patternrepair [-h] -l LANGUAGE (-p PATTERN_ID [PATTERN_ID ...] | --pattern-range RANGE_START-RANGE_END | -a) [--tp-lib TP_LIB_DIR]
+                                                   [--output-dir OUTPUT_DIR] [--masking-file MASKING_FILE] [--measurement-results MEASUREMENT_DIR]
+                                                   [--checkdiscoveryrules-results CHECKDISCOVERYRULES_FILE] [--skip-readme]
+
+options:
+  -h, --help            show this help message and exit
+  -l LANGUAGE, --language LANGUAGE
+                        Programming language targeted
+  -p PATTERN_ID [PATTERN_ID ...], --patterns PATTERN_ID [PATTERN_ID ...]
+                        Specify pattern(s) ID(s) to test for discovery
+  --pattern-range RANGE_START-RANGE_END
+                        Specify pattern ID range separated by`-` (ex. 10-50)
+  -a, --all-patterns    Test discovery for all available patterns
+  --tp-lib TP_LIB_DIR   Absolute path to alternative pattern library, default resolves to `./testability_patterns`
+  --output-dir OUTPUT_DIR
+                        Absolute path to the folder where outcomes (e.g., log file, export file if any) will be stored, default resolves to `./out`
+  --masking-file MASKING_FILE
+                        Absolute path to a json file, that contains a mapping, if the name for some measurement tools should be kept secret, default is None
+  --measurement-results MEASUREMENT_DIR
+                        Absolute path to the folder where measurement results are stored, default resolves to `./measurements`
+  --checkdiscoveryrules-results CHECKDISCOVERYRULES_FILE
+                        Absolute path to the csv file, where the results of the `checkdiscoveryrules` command are stored, default resolves to `./checkdiscoveryrules.csv`
+  --skip-readme         If set, the README generation is skipped.
+```
+
+Note: At the moment only `patternrepair` for PHP is supported. Support your own language by writing an `InstanceRepair` class, that inherits from `InstanceRepair`.
+
+The `patternrepair` enforces the pattern structure as described [here](https://github.com/testable-eu/sast-testability-patterns/blob/master/docs/testability-patterns-structure.md).
+To do so, it is seperated into different steps:
+
+- `PatternRepair`: This will check the pattern JSON file, correct the references to the instance json files.
+- `InstanceRepair`: This will check and correct the instance json file for each instance. At the moment, only PHP patterns are supported.
+  - It generates opcode for every PHP file.
+  - It checks for the comments `// source` and `// sink` in the file in order to fill in the source and sink line in the correspoding instance json file.
+- `READMEGenerator`: This creates a README file for a pattern based on the JSON files. If you want to skip the generation of the README file, use the `--skip-readme` flag. As the README includes results of `measure` and `checkdiscoveryresults`, valid filepaths for these must be provided, when generating a README file.
+
+## Example
+
+Note: Minimum requirement for this command is a pattern and a language.
+
+### Example 1
+
+Here a simple example that will run patternrepair on the first PHP pattern without generating a new README file for that pattern.
+`tpframework patternrepair -p 1 -l php --skip-readme`
+
+### Example 2
+
+Here an example for a patternrepair, that repairs all php patterns and generates a new readme for each pattern.
+`tpframework patternrepair -a -l php  --measurement-results ./your_measurement_results --checkdiscoveryrules-results ./your_results.csv`

qualitytests/patternrepair/test_markdown.py

@@ -0,0 +1,80 @@
+import pytest
+
+from qualitytests.qualitytests_utils import join_resources_path
+
+from pattern_repair.README_markdown_elements import *
+from pattern_repair.README_generator import READMEGenerator
+
+
+class TestMarkdownPatternRepair:
+
+    def test_markdown_code(self):
+        code = MarkdownCode('<?php\necho "Hello World";\n', "PHP")
+        assert f'\n```php\n<?php\necho "Hello World";\n```\n' == code.to_markdown()
+
+    def test_markdown_comment(self):
+        comment = MarkdownComment("This is a comment")
+        assert "\n[//]: # (This is a comment)\n" == comment.to_markdown()
+
+    def test_markdown_heading(self):
+        heading = MarkdownHeading("Heading", 2)
+        assert "\n## Heading\n" == heading.to_markdown()
+
+    def test_markdown_collapsible(self):
+        coll = MarkdownCollapsible([MarkdownString("Hello")], MarkdownString("More"))
+        assert '\n<details markdown="1">\n<summary>\nMore</summary>\n\n\nHello\n\n</details>\n' == coll.to_markdown()
+
+    def test_markdown_string(self):
+        s = MarkdownString("Test")
+        assert "\nTest\n" == s.to_markdown()
+
+    def test_markdown_link(self):
+        link = MarkdownLink("Test", MarkdownHeading("Heading 1", 3))
+        assert "[Test](#heading-1)" == link.to_markdown()
+
+    def test_markdown_table(self):
+        test_content = {"0::column1": ["value1", "value1.1"], "column2": ["value2"]}
+        tab = MarkdownTable(test_content)
+        expected_tab = "\n| column1   | column2   |\n"
+        expected_tab += "|-----------|-----------|\n"
+        expected_tab += "| value1    | value2    |\n"
+        expected_tab += "| value1.1  |           |\n"
+        assert expected_tab == tab.to_markdown()
+
+    def test_markdown_document(self):
+        coll = MarkdownCollapsible([MarkdownString("Hello")], MarkdownString("More"))
+        doc = MarkdownDocument([coll])
+        assert '<details markdown="1">\n<summary>\nMore</summary>\n\nHello\n\n</details>\n' == doc.to_markdown()
+
+    def test_README_generation_one_instance(self):
+        path_to_test_pattern = join_resources_path("sample_patlib/PHP/2_global_variables")
+        path_to_tplib = join_resources_path("sample_patlib")
+        instance_jsons = [path_to_test_pattern / "1_instance_2_global_variables" / "1_instance_2_global_variables.json"]
+        md_doc = READMEGenerator(path_to_test_pattern, 'php', path_to_tplib, instance_jsons)._generate_README_elements()
+
+        assert 14 == len(md_doc.content)
+        assert isinstance(md_doc.content[0], MarkdownComment)
+        assert isinstance(md_doc.content[1], MarkdownHeading)       # Global Variables
+        assert isinstance(md_doc.content[2], MarkdownString)        # Tags: ...
+        assert isinstance(md_doc.content[3], MarkdownString)        # Version: ...
+        assert isinstance(md_doc.content[4], MarkdownHeading)       # Description
+        assert isinstance(md_doc.content[5], MarkdownString)        # <pattern_description>
+        assert isinstance(md_doc.content[6], MarkdownHeading)       # Overview
+        assert isinstance(md_doc.content[7], MarkdownTable)         # <overview_table>
+        assert isinstance(md_doc.content[8], MarkdownHeading)       # Instance 1
+        assert isinstance(md_doc.content[9], MarkdownHeading)       # Code
+        assert isinstance(md_doc.content[10], MarkdownCode)         # <instance_code>
+        assert isinstance(md_doc.content[11], MarkdownHeading)      # Instance Properties
+        assert isinstance(md_doc.content[12], MarkdownTable)        # <instance_properties table>
+        assert isinstance(md_doc.content[13], MarkdownCollapsible)  # More
+
+        assert 2 == len(md_doc.content[13].content)
+        assert isinstance(md_doc.content[13].content[0], MarkdownCollapsible)       # Compile
+        assert 1 == len(md_doc.content[13].content[0].content)
+        assert isinstance(md_doc.content[13].content[0].content[0], MarkdownCode)   # <bash_code>
+
+        assert isinstance(md_doc.content[13].content[1], MarkdownCollapsible)       # Discovery
+        assert 3 == len(md_doc.content[13].content[1].content)
+        assert isinstance(md_doc.content[13].content[1].content[0], MarkdownString) # <discovery_string>
+        assert isinstance(md_doc.content[13].content[1].content[1], MarkdownCode)   # <discovery_code>
+        assert isinstance(md_doc.content[13].content[1].content[2], MarkdownTable)  # <discovery_table>

qualitytests/patternrepair/test_pattern_repair.py

@@ -0,0 +1,61 @@
+import pytest
+import os
+import shutil
+from pathlib import Path
+
+from pattern_repair.pattern_repair import PatternRepair
+from pattern_repair.PHP.instance_repair_php import InstanceRepairPHP
+
+from qualitytests.qualitytests_utils import join_resources_path
+
+@pytest.fixture(autouse=True)
+def run_around_tests():
+    # Code that will run before the test
+    path_to_test_pattern = join_resources_path("sample_patlib/PHP/5_pattern_to_repair")
+    path_to_save = join_resources_path("sample_patlib/PHP/5_pattern_to_repair_copy")
+    # copy the directory, to save it
+    shutil.copytree(path_to_test_pattern, path_to_save)
+
+    # A test function will be run at this point
+    yield
+
+    # Code that will run after the test
+    # restore the saved pattern
+    shutil.rmtree(path_to_test_pattern)
+    os.rename(path_to_save, path_to_test_pattern)
+    assert os.path.exists(path_to_test_pattern)
+
+class TestPatternRepair:
+    def test_repair_test_pattern_assert_files_exist(self):
+        path_to_test_pattern = join_resources_path("sample_patlib/PHP/5_pattern_to_repair")
+        instance_path = path_to_test_pattern / "1_instance_5_pattern_to_repair"
+        assert os.path.exists(instance_path)
+
+        PatternRepair(path_to_test_pattern, "PHP", join_resources_path("sample_patlib")).repair(True)
+
+        expected_pattern_json = path_to_test_pattern / "5_pattern_to_repair.json"
+        assert expected_pattern_json.is_file()
+        expected_instance_json = instance_path / "1_instance_5_pattern_to_repair.json"
+        assert expected_instance_json.is_file()
+        expected_instance_php = instance_path / "1_instance_5_pattern_to_repair.php"
+        assert expected_instance_php.is_file()
+        expected_instance_bash = instance_path / "1_instance_5_pattern_to_repair.bash"
+        assert expected_instance_bash.is_file()
+        expected_instance_sc = instance_path / "1_instance_5_pattern_to_repair.sc"
+        assert expected_instance_sc.is_file()
+        expected_docs_dir = path_to_test_pattern / "docs"
+        assert expected_docs_dir.is_dir()
+        expected_description = expected_docs_dir / "description.md"
+        assert expected_description.is_file()
+        expected_README_file = path_to_test_pattern / "README.md"
+        assert expected_README_file.is_file()
+
+    def test_finding_source_and_sink_line(self):
+        path_to_test_pattern = join_resources_path("sample_patlib/PHP/5_pattern_to_repair")
+        instance_repair = InstanceRepairPHP("PHP", path_to_test_pattern, "", join_resources_path("sample_pathlib"))
+
+        path_to_php_file = path_to_test_pattern / "1_instance_5_pattern_to_repair" / "test.php"
+
+        source, sink = instance_repair._get_source_and_sink_for_file(path_to_php_file)
+        assert 2 == source
+        assert 3 == sink

qualitytests/patternrepair/test_pattern_repair_utils.py

@@ -0,0 +1,75 @@
+import pytest
+from unittest.mock import patch
+
+from qualitytests.qualitytests_utils import join_resources_path
+
+from core.exceptions import PatternDoesNotExists
+from pattern_repair.utils import (
+    assert_pattern_valid, compare_dicts,
+    get_dict_keys, get_instance_name,
+    get_files_with_ending, get_language_by_file_ending,
+    list_instances_jsons, repair_keys_of_json
+    )
+
+class TestPatternRepairUtils:
+    def test_assert_pattern_valid(self):
+        path_to_non_existing_pattern = join_resources_path("100_non_existing")
+        with pytest.raises(PatternDoesNotExists) as e_info:
+            assert_pattern_valid(path_to_non_existing_pattern)
+        assert "Specified Pattern `100_non_existing` does not exists." in str(e_info.value)
+
+    def test_compare_dicts(self):
+        o_dict = {"key1": 1, "key2": 3, "key3": 2}
+        n_dict = {"key1": 1, "key3": 3, "key4": 42}
+        assert {'key3': 2} == compare_dicts(o_dict, n_dict)
+
+    def test_get_dict_keys(self):
+        d = {
+            "key1": {
+                "key1.1": 0,
+                "key1.2": {"key1.2.1": 0}
+            },
+            "key2": 42
+        }
+        assert set(["key1:key1.1", "key1:key1.2:key1.2.1", "key2"]) == set(get_dict_keys(d))
+
+    def test_get_instance_name(self):
+        path_to_pattern = join_resources_path("sample_patlib/PHP/5_pattern_to_repair")
+        path_to_instance = path_to_pattern / "1_instance_5_pattern_to_repair"
+
+        assert "1 Instance", get_instance_name(path_to_instance)
+
+    def test_get_files_with_ending(self):
+        path_to_pattern = join_resources_path("sample_patlib/PHP/3_global_array")
+        assert [] == get_files_with_ending(path_to_pattern, ".php")
+        expected_instance_1_php_file = str(path_to_pattern / "1_instance_3_global_array" / "1_instance_3_global_array.php")
+        expected_instance_2_php_file = str(path_to_pattern / "2_instance_3_global_array" / "2_instance_3_global_array.php")
+        assert set([expected_instance_1_php_file, expected_instance_2_php_file]) == set(get_files_with_ending(path_to_pattern, ".php", True))
+
+    def test_get_language_by_file_ending(self):
+        assert "python" == get_language_by_file_ending("test.py")
+        assert "php" == get_language_by_file_ending("test.php")
+        assert "javascript" == get_language_by_file_ending("test.js")
+        assert "java" == get_language_by_file_ending("test.java")
+        assert "scala" == get_language_by_file_ending("test.sc")
+        assert "bash" == get_language_by_file_ending("test.bash")
+
+        with pytest.raises(NotImplementedError) as e_info:
+            get_language_by_file_ending("")
+        assert "The ending of the given filename  is not yet supported" in str(e_info.value)
+
+    def test_list_instance_jsons(self):
+        path_to_pattern = join_resources_path("sample_patlib/PHP/3_global_array")
+        expected_instance_1_json_file = str(path_to_pattern / "1_instance_3_global_array" / "1_instance_3_global_array.json")
+        expected_instance_2_json_file = str(path_to_pattern / "2_instance_3_global_array" / "2_instance_3_global_array.json")
+        assert set([expected_instance_1_json_file, expected_instance_2_json_file]) == set(list_instances_jsons(path_to_pattern))
+
+    def test_repair_keys_of_json(self):
+        json_dict_tested = {"a": 42, "b": {"b.0": 1}}
+        json_dict_ground_truth = {"a": 42, "b": {"b.0": 1, "b.1": 1}, "c": 42, "d": 36}
+        with patch("pattern_repair.utils.read_json") as read_json_mock, \
+            patch("pattern_repair.utils.write_json") as write_json_mock:
+            read_json_mock.side_effect = [json_dict_tested, json_dict_ground_truth]
+
+            repair_keys_of_json("", "", ["d"])
+            write_json_mock.assert_called_once_with("", {"a": 42, "b": {"b.0": 1, "b.1": ""}, "c": ""})

qualitytests/resources/sample_patlib/PHP/2_global_variables/2_global_variables.json

@@ -5,5 +5,6 @@
     "tags": ["sast", "php", "php_v7.4.9"],
     "instances": [
         "./1_instance_2_global_variables/1_instance_2_global_variables.json"
-    ]
+    ],
+    "version": "v0"
 }

qualitytests/resources/sample_patlib/PHP/3_global_array/3_global_array.json

@@ -1,9 +1,11 @@
 {
+    "description": "",
     "name": "Global Array",
     "family": "code_pattern_php",
     "tags": ["sast", "php", "php_v7.4.9"],
     "instances": [
         "./1_instance_3_global_array/1_instance_3_global_array.json",
         "./2_instance_3_global_array/2_instance_3_global_array.json"
-    ]
+    ],
+    "version": "v0"
 }

qualitytests/resources/sample_patlib/PHP/5_pattern_to_repair/1_instance_5_pattern_to_repair/test.php

@@ -0,0 +1,3 @@
+<?php
+$a = $_GET["_p1"]; // source
+echo $a; // sink

qualitytests/resources/sample_patlib/pattern_template/ID_pattern_name/ID_pattern_name.json

@@ -0,0 +1,10 @@
+{
+    "name": "Pattern Name",
+    "description": "",
+    "family": "code_pattern_LANG",
+    "tags": ["sast", "LANG"],
+    "instances": [
+        "./IID_instance_ID_pattern_name/IID_instance_ID_pattern_name.json"
+    ],
+    "version": "v0.draft"
+}