Proposal: Add Separate Amber Worker Container in the Computing-Unit Pod and Route All UDFs There

[mengw15]

Summary

Currently, a computing unit is a single Pod with one Container running the Amber Coordinator (master). We propose adding a separate Container in the same Pod that runs an Amber worker. Operator scheduling will be updated so all UDF operators execute on the Worker container.

Motivation

This design leverages Pekko’s distribution features

Design (high level)

Separate Worker Container in the same Pod
Extend the Pod spec to start one new container for an Amber Worker.

Scheduling change
Update the scheduler so that all UDF operators are placed on the Worker container.
Non-UDF operators keep current placement (initially unchanged).

[chenlica]

@mengw15 Can you rotate the image to make it easy to read? :-)

[mengw15]

Sure

[chenlica]

Can you draw another one that's more accurate and clear?

[mengw15]

sure.

Hi @bobbai00, Could you verify this drawing? Thank you!

[chenlica]

Thanks. @SarahAsad23 You may also want to check this discussion and the diagram as they are related to your work of Python virtual environments.

[Yicong-Huang]

For the design, we would want UDFs to be placed at an isolated environment so that it does not touch driver environment. The current design puts all UDFs with driver, and I think it is not aligned with the goal?

[mengw15]

I think the goal here is to isolate UDFs execution from the Coordinator (master). The UDFs won’t run in the Coordinator’s container anymore.

[chenlica]

I think the container on the right is used to run all UDFs. @mengw15 Can you clarify the purpose of the "Amber Worker (Driver)"?

[Yicong-Huang]

lets avoid using "driver" name for workers. "driver" typically means the coordinator.

[mengw15]

Got it

[chenlica]

What's a good name for "Driver"?

[chenlica]

@mengw15 Let's finish this discussion.

[mengw15]

I think we can just use “Amber Worker”

[chenlica]

A minor comment: we recently changed Akka to Apache Pekko. @mengw15 Please revise the writing accordingly.

[chenlica]

@mengw15 Please lead this discussion.

[mengw15]

Got it. I’ve already made initial progress on splitting the Computing-Unit Pod into two containers (Coordinator + Worker). I’m currently working on another task. I’ll return to this task once it’s done.