Migrate to Result Service and MinIO for Execution Results

[mengw15]

Feature Summary

Background

Currently, computing units are responsible for writing execution results and execution records directly to PostgreSQL / Iceberg, and execution result files are stored inside the computing units. This leads to several issues:

• Result persistence logic lives inside the computing units.
• Computing units need direct DB access and JDBC configuration.

To improve isolation, we want to move execution-result handling out of the computing units into a dedicated microservice.

Proposed Solution

Architecture Overview

Migrate computing units to use existing services instead of direct database access:

1. Result Service/REST Catalog Service (LakeKeeper) for Iceberg catalog operations

   • CUs use Result Service instead of direct JDBC connection.
   • Database credentials only known to Result Service.

2. MinIO (S3-compatible object storage) for storing result files

   • CUs write Parquet files directly to MinIO.
   • Global, shared storage accessible from all CUs.

High-level diagram of current architecture and the proposed one

Current:

New:

[chenlica]

@mengw15 If possible, please draw a diagram to show the current architecture and another diagram to show the new one.

[chenlica]

@mengw15 Please draw a diagram as suggested.

[bobbai00]

Please create multiple sub issues using the Task template under this issue. And for each PR you raised, closes JUST that sub issue.