merge main to workflow branch

.github/ISSUE_TEMPLATE/01_bug_report.md

@@ -0,0 +1,21 @@
+---
+name: 🐜 Bug report
+about: If something isn't working 🔧
+---
+
+### Subject of the issue
+Describe your issue here.
+
+### Your environment
+* Version of ruby
+* Docker or manual installation?
+* Which browser and its version
+
+### Steps to reproduce
+Tell us how to reproduce this issue. 
+
+### Expected behaviour
+Tell us what should happen
+
+### Actual behaviour
+Tell us what happens instead

.github/ISSUE_TEMPLATE/02_feature_request.md

@@ -0,0 +1,21 @@
+---
+name: 🚀 Feature request
+about: If you have a feature request 💡
+---
+
+**Context**
+
+What are you trying to do and how would you want to do it differently? Is it something you currently you cannot do? Is this related to an issue/problem?
+
+**Alternatives**
+
+Can you achieve the same result doing it in an alternative way? Is the alternative considerable?
+
+**Has the feature been requested before?**
+
+Please provide a link to the issue.
+
+**If the feature request is approved, would you be willing to submit a PR?**
+
+Yes / No _(Help can be provided if you need assistance submitting a PR)_
+

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1 @@
+blank_issues_enabled: false

.github/pull_request_template.md

@@ -0,0 +1,19 @@
+# Task
+<!-- Please add link a relevant issue or task -->
+
+# Description
+<!-- Please include a summary of the change -->
+<!-- Any details that you think are important to review this PR? -->
+<!-- Are there other PRs related to this one? -->
+
+# How Has This Been Tested?
+<!-- Please describe how you tested your changes -->
+
+# Checklist
+<!-- Go over all the following points, and put an `x` in all the boxes that apply -->
+
+- [ ] This Pull-Request goes to the **development** branch.
+- [ ] I have successfully run prek locally.
+- [ ] I have added tests to cover my changes.
+- [ ] I have linked the issue-id to the task-description.
+- [ ] I have performed a self-review of my own code.

.github/workflows/python-app.yml

@@ -5,9 +5,9 @@ name: Python application
 
 on:
   push:
-    branches: [ "main" ]
+    branches: [ "main", "development" ]
   pull_request:
-    branches: [ "main" ]
+    branches: [ "main", "development" ]
 
 permissions:
   contents: read
@@ -28,11 +28,11 @@ jobs:
         activate-environment: true
         enable-cache: true
 
-    - name: Install project deps
-      run: uv pip install -e .[dev]
+    - name: Use specific detematelibrary version
+      run: chmod +x scripts/change_toml.sh && scripts/change_toml.sh $GITHUB_BASE_REF
 
     - name: Run pre-commit with prek
-      run: uv run prek run -a
-
+      run: uv run --dev prek run -a
+        
     - name: Test with pytest
-      run: uv run pytest
+      run: uv run --dev pytest --ignore tests/library_integration

.pre-commit-config.yaml

@@ -1,6 +1,6 @@
 # See https://pre-commit.com for more information
 # See https://pre-commit.com/hooks.html for more hooks
-exclude: '.*.log'
+exclude: '.*.log|.github/.*|docs/.*'
 repos:
 
 # Core cleanup
@@ -20,6 +20,11 @@ repos:
     - id: check-symlinks
     - id: check-toml
 
+- repo: https://github.com/myint/autoflake
+  rev: v2.3.1
+  hooks:
+    - id: autoflake
+      args: [--in-place, --remove-all-unused-imports, --remove-unused-variables]
 # Type checking
 - repo: https://github.com/pre-commit/mirrors-mypy
   rev: v1.18.2
@@ -31,15 +36,15 @@ repos:
         - pydantic
         - types-PyYAML
         - pydantic-settings
+        - types-requests
 
 # Code formatting
 - repo: https://github.com/hhatto/autopep8
   rev: v2.3.2
   hooks:
     - id: autopep8
-      args: [--max-line-length=110, --diff]
+      args: [--max-line-length=110, --in-place]
 
-# Linting
 - repo: https://github.com/PyCQA/flake8
   rev: 7.3.0
   hooks:
@@ -59,12 +64,6 @@ repos:
     - id: bandit
       exclude: ^tests/
 
-# Unused code detection
-- repo: https://github.com/jendrikseipp/vulture
-  rev: v2.14
-  hooks:
-    - id: vulture
-      args: [--min-confidence=70, src/]
 
 # Log and syntax pattern checks
 - repo: https://github.com/pre-commit/pygrep-hooks

README.md

@@ -24,19 +24,19 @@ If you plan to contribute to the development of this package, follow these steps
 
 ```bash
 uv sync --dev
-uv run prek install
+uv run --dev prek install
 ```
 
 Run the tests:
 
 ```bash
-uv run pytest -q
+uv run --dev pytest -q
 ```
 
 Run the tests with coverage (add `--cov-report=html` to generate an HTML report):
 
 ```bash
-uv run pytest --cov=. --cov-report=term-missing
+uv run --dev pytest --cov=. --cov-report=term-missing
 ```
 
 
@@ -73,19 +73,31 @@ Example configuration files can be found in the `tests/config` directory.
 Start the service:
 
 ```bash
-uv run detectmate start --settings examples/service_settings.yaml
+uv run detectmate --settings examples/service_settings.yaml
 ```
+To survey the state of your component and interact with the running service, use the detectmate-client tool.
 
 Get the service status:
 
 ```bash
-uv run detectmate status --settings examples/service_settings.yaml
+uv run detectmate-client status --url <http_host:http_port>
 ```
 
-Stop the service:
+Stop the engine:
 
 ```bash
-uv run detectmate stop --settings examples/service_settings.yaml
+uv run detectmate-client stop --url <http_host:http_port>
+```
+
+Start the engine:
+```bash
+uv run detectmate-client start --url <http_host:http_port>
+```
+
+Shutdown entire service:
+
+```bash
+uv run detectmate-client shutdown --url <http_host:http_port>
 ```
 
 
@@ -108,7 +120,15 @@ docker compose up demo
 
 ## Documentation
 
-- [Project Documentation](https://ait-detectmate.github.io/DetectMateService/)
+- [Project Documentation](https://ait-detectmate.github.io/DetectMateService/latest)
+
+## Contribution
+
+We're happily taking patches and other contributions. Please see the following links for how to get started:
+
+- [GitHub Workflow](https://ait-detectmate.github.io/DetectMateService/latest/Contribution)
+
+If you encounter any bugs, please create an issue on [Github](https://github.com/ait-detectmate/DetectMateService/issues).
 
 ## License
 

SECURITY.md

@@ -0,0 +1,32 @@
+# Security Policy
+
+## Supported Versions
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 1.x.x   | :white_check_mark: |
+| < 1.0.0   | :x:                |
+
+> [!IMPORTANT]
+> Currently DetectMateService is a work in progress and heavily under development. Possible vulnerabilities will not be treated any special and can be issued using [GitHub-Issues](https://github.com/ait-detectmate/DetectMateService/issues)
+
+## Reporting a Vulnerability
+
+Please email reports about any security related issues you find to aecid@ait.ac.at. This mail is delivered to a small developer team. Your email will be acknowledged within one business day, and you'll receive a more detailed response to your email within 7 days indicating the next steps in handling your report.
+
+Please use a descriptive subject line for your report email. After the initial reply to your report, our team will endeavor to keep you informed of the progress being made towards a fix and announcement.
+
+In addition, please include the following information along with your report:
+
+* Your name and affiliation (if any).
+* A description of the technical details of the vulnerabilities. It is very important to let us know how we can reproduce your findings.
+* An explanation who can exploit this vulnerability, and what they gain when doing so -- write an attack scenario. This will help us evaluate your report quickly, especially if the issue is complex.
+* Whether this vulnerability public or known to third parties. If it is, please provide details.
+* Whether we could mention your name in the changelogs.
+
+Once an issue is reported we use the following disclosure process:
+
+* When a report is received, we confirm the issue and determine its severity.
+* If we know of specific third-party services or software based on DetectMateService that require mitigation before publication, those projects will be notified.
+* Fixes are prepared for the last minor release of the latest major release.
+* Patch releases are published for all fixed released versions.

demo/config/detector_settings.yaml

@@ -1,7 +1,8 @@
 component_type: "detectors.new_value_detector.NewValueDetector"
 component_config_class: "detectors.new_value_detector.NewValueDetectorConfig"
 component_name: "test-nvd"
-manager_addr: "tcp://detector:8020"
+http_host: "127.0.0.1"
+http_port: 8020
 engine_addr: "tcp://detector:8021"
 log_level: "DEBUG"
 log_dir: "./logs"

demo/config/ipc/detector_settings.yaml

@@ -1,9 +1,9 @@
 component_type: "detectors.new_value_detector.NewValueDetector"
 component_config_class: "detectors.new_value_detector.NewValueDetectorConfig"
 component_name: "test-nvd"
-manager_addr: "ipc:///tmp/test_nvd_cmd.ipc"
+http_host: "127.0.0.1"
+http_port: 8020
 engine_addr: "ipc:///tmp/test_nvd_engine.ipc"
-
 log_level: "DEBUG"
 log_dir: "./logs"
 log_to_console: true

demo/config/ipc/parser_settings.yaml

@@ -1,7 +1,8 @@
 component_type: "parsers.template_matcher.MatcherParser"
 component_config_class: "parsers.template_matcher.MatcherParserConfig"
 component_name: "test-parser"
-manager_addr: "ipc:///tmp/test_parser_cmd.ipc"
+http_host: "127.0.0.1"
+http_port: 8010
 engine_addr: "ipc:///tmp/test_parser_engine.ipc"
 output_engine_addr: "ipc:///tmp/test_nvd_engine.ipc"
 log_level: "DEBUG"

demo/config/ipc/reader_settings.yaml

@@ -1,7 +1,8 @@
 component_type: "readers.log_file.LogFileReader"
 component_config_class: "readers.log_file.LogFileConfig"
 component_name: "test-reader"
-manager_addr: "ipc:///tmp/test_reader_cmd.ipc"
+http_host: "127.0.0.1"
+http_port: 8000
 engine_addr: "ipc:///tmp/test_reader_engine.ipc"
 log_level: "DEBUG"
 log_dir: "./logs"

demo/config/parser_settings.yaml

@@ -1,7 +1,8 @@
 component_type: "parsers.template_matcher.MatcherParser"
 component_config_class: "parsers.template_matcher.MatcherParserConfig"
 component_name: "test-parser"
-manager_addr: "tcp://parser:8010"
+http_host: "127.0.0.1"
+http_port: 8010
 engine_addr: "tcp://parser:8011"
 log_level: "DEBUG"
 log_dir: "./logs"

demo/config/reader_settings.yaml

@@ -1,7 +1,8 @@
 component_type: "readers.log_file.LogFileReader"
 component_config_class: "readers.log_file.LogFileConfig"
 component_name: "test-reader"
-manager_addr: "tcp://reader:8000"
+http_host: "127.0.0.1"
+http_port: 8000
 engine_addr: "tcp://reader:8001"
 log_level: "DEBUG"
 log_dir: "./logs"

docs/configuration.md

@@ -16,7 +16,8 @@ These settings control the service infrastructure.
 | `log_dir`                     | `DETECTMATE_LOG_DIR`                     | `./logs`                           | Directory for log files.                                                                                  |
 | `log_to_console`              | `DETECTMATE_LOG_TO_CONSOLE`              | `true`                             | Whether logs are written to stdout/stderr.                                                                |
 | `log_to_file`                 | `DETECTMATE_LOG_TO_FILE`                 | `true`                             | Whether logs are written to files in `log_dir`.                                                           |
-| `manager_addr`                | `DETECTMATE_MANAGER_ADDR`                | `ipc:///tmp/detectmate.cmd.ipc`    | Address for management commands (REQ/REP).                                                                |
+| `http_host`                | `DETECTMATE_HTTP_HOST`                | `127.0.0.1`    | Host address for the HTTP server.
+| `http_port`                | `DETECTMATE_HTTP_PORT`                | `8000`    | Port for the HTTP server.                                                            |
 | `manager_recv_timeout`        | `DETECTMATE_MANAGER_RECV_TIMEOUT`        | `100`                              | Receive timeout (ms) for the manager command channel.                                                     |
 | `manager_thread_join_timeout` | `DETECTMATE_MANAGER_THREAD_JOIN_TIMEOUT` | `1.0`                              | Timeout (s) when waiting for the manager thread to stop.                                                  |
 | `engine_addr`                 | `DETECTMATE_ENGINE_ADDR`                 | `ipc:///tmp/detectmate.engine.ipc` | Address for data processing (PAIR0/1).                                                                    |
@@ -35,8 +36,9 @@ component_name: "my-detector"
 log_level: "DEBUG"
 log_dir: "./logs"
 
-# Manager Interface (Command Channel)
-manager_addr: "ipc:///tmp/detectmate.cmd.ipc"
+# Manager Interface
+http_host: 127.0.0.1
+http_port: 8000
 
 # Engine Interface (Data Channel)
 engine_addr: "ipc:///tmp/detectmate.engine.ipc"
@@ -66,13 +68,40 @@ detectmate start
 
 In addition to the service settings (which configure the *runner*), you can also pass a separate configuration file for the specific component logic (e.g., detector parameters) using the `--config` flag in the CLI. This file is specific to the implementation of the component you are running.
 
+Component configuration controls the specific logic of the detector or parser. To support dynamic library loading, this file uses a nested structure.
+The configuration must be namespaced by the component category (detectors or parsers) and the specific class name to allow the library to correctly route parameters.
+Example detector_config.yaml
 
 
 ```yaml
-# detector-config.yaml
-threshold: 0.85
-sensitivity: high
-enabled: true
+detectors:                 # Category Level
+  NewValueDetector:        # Class Name Level
+    auto_config: false
+    method_type: new_value_detector
+    params:                # Implementation Specific Level
+      log_variables:
+        - id: test
+          template: dummy_template
+          variables:
+            - name: var1
+              pos: 0
+              params:
+                threshold: 0.0
 ```
 
 You can read more about Components in the [Using a Library Component](library.md) section.
+
+
+## HTTP Admin Interface
+
+The service provides a REST API for runtime management and monitoring.
+
+### Core Endpoints
+
+| Method | Endpoint | Description |
+| :--- | :--- | :--- |
+| `GET` | `/admin/status` | Returns the health, running state, and current effective configurations. |
+| `POST` | `/admin/start` | Starts the data processing engine thread. |
+| `POST` | `/admin/stop` | Stops the data processing engine thread. |
+| `POST` | `/admin/reconfigure` | Updates component parameters dynamically. |
+| `POST` | `/admin/shutdown` | Gracefully terminates the entire service process. |