Add support for partial results tracking on a scan #68

MFormenti · 2025-11-19T12:38:48Z

Adds infrastructure to support partial results by providing scan implementations access to their ScanJobDescription context during execution

Added:

JobStatus.RUNNING enum value for in-progress scans
getCurrentJobDescription() method in BulkScanWorker to access current job UUID via ThreadLocal
ThreadLocal context management in handle() method

- Add RUNNING status to JobStatus enum - Add ThreadLocal in BulkScanWorker to pass ScanJobDescription to scan implementations - Add upsertPartialScanResult() method to IPersistenceProvider interface - Implement upsertPartialScanResult() in MongoPersistenceProvider - Update DummyPersistenceProvider test implementation - Update BulkScanWorkerManager to pass ScanJobDescription to worker.handle() This enables real-time persistence of intermediate scan results to MongoDB as probes complete during TLS scans.

… input. scanTarget can be accessed from scanJobDescription, and the UUID in scanJobDescription is needed to write partial results to MongoDB during a scan

juaninf · 2025-11-20T15:38:48Z

Add one more test using DummyPersistenceProvider to check the partial results.

…ersistenceProvider

src/main/java/de/rub/nds/crawler/core/BulkScanWorkerManager.java

src/test/java/de/rub/nds/crawler/core/BulkScanWorkerTest.java

juaninf

Please check my comments

Replace all hostname usages with TEST-NET-1 IPs (RFC 5737)

ic0ns · 2025-11-26T13:15:51Z

looks good to me @XoMEX are you ok with this?

XoMEX · 2025-12-01T09:54:08Z

I don't get what getCurrentJobDescription is used for. Can you explain who calls this function and to do what.
Right now it feels weird to put a variable that is bound to the stack into a threadlocal. I feel like the current threading model is a bit hacky and do not feel comfortable making assumptions about it.

MFormenti · 2025-12-01T11:31:36Z

getCurrentJobDescription is used by a function ( TlsScanWorker.scan() ) that I modified in TLS-Crawler-Development (there is currently an open draft PR in TLS-Crawler-Development).
The reason why I modified this function is because we want to track partial results during a scan to updated our webapp's UI as the scan progresses, without needing to wait for the whole scan to finish. Partial results are stored in a MongoDB database, so to write partial results to MongoDB while the scan is running, the scan function needs access to the UUID of a scan.

XoMEX · 2025-12-01T12:03:27Z

Do I get it correctly, that you modify the mongodb also from the TLS-Crawler then?
To me this feels like a bad idea to do this in two places. I'd propose to modify the Future returned by handle to support returning partial results which are then persisted into the db by the crawler core.

…al results persistence. Rework Worker, BulkScanWorker and BulkScanWorkerManager handle method to use ScheduledScan instead of Future<Document> and removed use of ThreadLocal.

MFormenti · 2025-12-08T06:41:16Z

yes, that is correct, mongoDB was also modified in TLS-Crawler. I changed it so now mongoDB is only modified in Crawler Core.
I added a new class, called ScheduleScan, that acts as a wrapper for Future<Document> and implements partial results persistence.
Additionally, ThreadLocal is no longer needed in Crawler Core so I removed it.

XoMEX

Cool that you got it working that fast.
I am still a bit unhappy about how the partial results are passed back to the scanner. In my head I envision that the report is observable, and as soon as it is updated, the new data is persisted into the DB. But that comes with other issues (like, do we want to write all changes?) and a redesign of the scanner result class. I am not sure if it is worth the effort.

Other than that, I left some minor comments

XoMEX · 2025-12-08T13:28:59Z

src/main/java/de/rub/nds/crawler/core/ScheduledScan.java

+public class ScheduledScan {
+
+    private volatile Document currentResult;
+    private final CompletableFuture<Document> finalResult = new CompletableFuture<>();


What is the reasoning for wrapping the future instead of extending it?

I also find the name to not capture the functionality.
If this would extend the future, I'd call it something like ProgressableFuture<Document>.

XoMEX · 2025-12-08T13:29:52Z

src/main/java/de/rub/nds/crawler/core/BulkScanWorker.java

+    public void setPersistenceProvider(IPersistenceProvider persistenceProvider) {
+        this.persistenceProvider = persistenceProvider;
+    }


Can we instead pass this in the constructor? Is there a use-case to having this as a variable instead of a constant?

XoMEX · 2025-12-08T13:30:22Z

src/main/java/de/rub/nds/crawler/core/BulkScanWorker.java

+     * @param partialResult The partial result document to persist
+     */
+    protected void persistPartialResult(ScanJobDescription jobDescription, Document partialResult) {
+        if (persistenceProvider != null) {


Do we need this check?

XoMEX · 2025-12-08T13:35:08Z

src/main/java/de/rub/nds/crawler/core/BulkScanWorker.java

-    public abstract Document scan(ScanTarget scanTarget);
+    public abstract Document scan(ScanJobDescription jobDescription, ScheduledScan scheduledScan);


I dislike having an "out parameter". The only alternative I see would be a functional interface to feed the partial results into. But I guess that isn't that much of a difference.

XoMEX · 2025-12-08T13:36:20Z

src/main/java/de/rub/nds/crawler/core/BulkScanWorker.java

+     * @param jobDescription The job description for the scan
+     * @param partialResult The partial result document to persist
+     */
+    protected void persistPartialResult(ScanJobDescription jobDescription, Document partialResult) {


Why is this not somehow linked to ScheduledScan.updateResult?

I see two approaches:
Either this function updates the results in the ScheduledScan object
Or this function is registered as an observer for updates on the ScheduledScan object; i.e. if updateResult is called, it fires an even that causes persistPratialResult to be called.

Or are there cases where one function is called and not the other?

mattiaformenti added 3 commits November 14, 2025 10:39

Removed unused code

c5ee54f

Add BulkScanWorker unit tests

ca7fa87

MFormenti changed the title ~~Add support for scan partial results tracking~~ Add support for partial results tracking on a scan Nov 19, 2025

changed BulkScanWorker.handle() to take only scanJobDescription as an…

fccfe08

… input. scanTarget can be accessed from scanJobDescription, and the UUID in scanJobDescription is needed to write partial results to MongoDB during a scan

Modify BulkScanWorkerTest to simulate MongoDB operations using DummyP…

1f63b37

…ersistenceProvider

juaninf reviewed Nov 26, 2025

View reviewed changes

src/main/java/de/rub/nds/crawler/core/BulkScanWorkerManager.java Outdated Show resolved Hide resolved

src/test/java/de/rub/nds/crawler/core/BulkScanWorkerTest.java Outdated Show resolved Hide resolved

juaninf reviewed Nov 26, 2025

View reviewed changes

src/test/java/de/rub/nds/crawler/core/BulkScanWorkerTest.java Outdated Show resolved Hide resolved

juaninf reviewed Nov 26, 2025

View reviewed changes

src/test/java/de/rub/nds/crawler/core/BulkScanWorkerTest.java Outdated Show resolved Hide resolved

juaninf reviewed Nov 26, 2025

View reviewed changes

mattiaformenti added 3 commits November 26, 2025 11:29

Replace example.com with example.invalid

60b44eb

Fix BulkScanWorkerTest

2fc13dd

Update BulkScanWorkerTest to use IPs

4f437a8

Replace all hostname usages with TEST-NET-1 IPs (RFC 5737)

Add ScheduleScan, a wrapper for Future<Document> made to handle parti…

1a71f8d

…al results persistence. Rework Worker, BulkScanWorker and BulkScanWorkerManager handle method to use ScheduledScan instead of Future<Document> and removed use of ThreadLocal.

XoMEX reviewed Dec 8, 2025

View reviewed changes

		public abstract Document scan(ScanTarget scanTarget);
		public abstract Document scan(ScanJobDescription jobDescription, ScheduledScan scheduledScan);

Add support for partial results tracking on a scan #68

Are you sure you want to change the base?

Add support for partial results tracking on a scan #68

Uh oh!

Conversation

MFormenti commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

juaninf commented Nov 20, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

juaninf left a comment

Choose a reason for hiding this comment

Uh oh!

ic0ns commented Nov 26, 2025

Uh oh!

XoMEX commented Dec 1, 2025

Uh oh!

MFormenti commented Dec 1, 2025

Uh oh!

XoMEX commented Dec 1, 2025

Uh oh!

MFormenti commented Dec 8, 2025

Uh oh!

XoMEX left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

MFormenti commented Nov 19, 2025 •

edited

Loading