[Design] Dynamic Resource Scaling for Performance-Based Reconcile

[Chulhee1Lee]

Background / 배경

Korean

App Performance 기반 Reconcile Use Case에서는 애플리케이션 성능 저하 시 Pullpiri가 Runtime에서 CPU 및 Memory 리소스를 동적으로 변경할 수 있어야 합니다.

현재 Pullpiri는 App 성능 정보 및 시스템 리소스 정보를 수집할 수 있지만, 실행 중인 Workload에 대한 Dynamic Resource Scaling 기능은 제공하지 않습니다.

본 이슈는 구현에 앞서 Dynamic Resource Scaling을 위한 SW Architecture 및 Component Interface를 설계하기 위한 목적입니다.

관련 Use Case

• App Performance-based Reconcile
• Strategy A: Dynamic Resource Scaling (CPU/Memory)

English

As part of the App Performance-based Reconcile use case, Pullpiri should support runtime CPU and memory updates for running workloads when application performance degrades.

Currently, Pullpiri can monitor application performance metrics and node resource usage, but it does not support dynamic resource scaling.

Before implementation, we need to define the software architecture, interfaces, workflows, and component responsibilities required for Dynamic Resource Scaling.

Related Use Case

• App Performance-based Reconcile
• Strategy A: Dynamic Resource Scaling (CPU/Memory)

---

Objective / 목표

Korean

실행 중인 Workload의 CPU 및 Memory를 Runtime에서 변경할 수 있도록 Pullpiri Software Architecture를 설계합니다.

설계 범위는 다음을 포함합니다.

• Architecture Design
• Component Responsibilities
• API / Proto Changes
• End-to-End Workflow
• Resource State Synchronization Strategy

English

Design a software architecture that enables runtime CPU and Memory updates for running workloads.

The design should cover:

• Architecture Design
• Component Responsibilities
• API / Proto Changes
• End-to-End Workflow
• Resource State Synchronization Strategy

---

Design Scope / 설계 범위

1. End-to-End Scaling Flow

Korean

Dynamic Resource Scaling을 위한 전체 Workflow를 정의합니다.

예시:

MonitoringServer
→ PolicyManager
→ ActionController
→ NodeAgent
→ Podman Update
→ Resource Allocation Updated

각 컴포넌트의 역할과 책임을 정의해 주세요.

English

Define the overall workflow for Dynamic Resource Scaling.

Example:

MonitoringServer
→ PolicyManager
→ ActionController
→ NodeAgent
→ Podman Update
→ Resource Allocation Updated

Clearly define component responsibilities.

---

2. API / Proto Design

Korean

필요한 인터페이스 및 Proto 변경 사항을 정의합니다.

검토 대상:

• PolicyManager
• ActionController
• NodeAgent
• Resource Manager

특히 Runtime Resource Update를 지원하기 위한 신규 Operation이 필요한지 검토해 주세요.

예시:

UpdateResources
UpdateCpuLimit
UpdateMemoryLimit

English

Identify required interfaces and protocol changes.

Potential targets:

• PolicyManager
• ActionController
• NodeAgent
• Resource Manager

Review whether additional operations are required to support runtime resource updates.

Examples:

UpdateResources
UpdateCpuLimit
UpdateMemoryLimit

---

3. Runtime Resource Update Mechanism

Korean

실행 중인 Workload의 CPU 및 Memory 변경 방법을 조사해 주세요.

검토 항목:

• Podman API
• podman update
• cgroup limitations
• Runtime constraints

산출물:

• 지원 가능한 기능
• 지원 불가능한 기능
• 제약사항
• 알려진 리스크

English

Investigate how runtime CPU and Memory updates can be applied to running workloads.

Topics:

• Podman API
• podman update
• cgroup limitations
• Runtime constraints

Deliverables:

• Supported operations
• Unsupported operations
• Limitations
• Known risks

---

4. Resource Availability Check

Korean

리소스 변경 수행 전 현재 노드의 Available Resource를 어떻게 검증할 것인지 정의합니다.

검토 항목:

• Available CPU
• Available Memory
• Resource 부족 상황 처리 방법
• 최대 할당 가능 범위

English

Define how resource availability should be validated before applying resource updates.

Consider:

• Available CPU
• Available Memory
• Handling insufficient resources
• Maximum allocation limits

---

5. Resource State Synchronization

Korean

다음 컴포넌트 간 Resource 상태 동기화 방안을 정의합니다.

• Resource Manager
• ActionController
• NodeAgent
• Monitoring Server

실제 할당된 리소스 정보와 Pullpiri 내부 관리 상태가 항상 일관성을 유지할 수 있어야 합니다.

English

Define synchronization strategies between:

• Resource Manager
• ActionController
• NodeAgent
• Monitoring Server

The actual allocated resources and the Pullpiri-managed state must remain consistent.

---

Deliverables / 결과물

Design Document

Korean

다음 내용을 포함한 설계 문서를 작성해 주세요.

• Architecture Diagram
• Sequence Diagram
• Component Responsibilities
• API / Proto Changes
• Resource Synchronization Strategy
• Known Limitations
• Risk Analysis

English

The design document should include:

• Architecture Diagram
• Sequence Diagram
• Component Responsibilities
• API / Proto Changes
• Resource Synchronization Strategy
• Known Limitations
• Risk Analysis

---

Out of Scope / 제외 범위

Korean

본 이슈에서는 아래 항목을 다루지 않습니다.

• Resource Scaling Policy
• Priority-based Application Termination
• Victim Selection Algorithm
• Stateful Offloading
• Storage Resource Scaling
• eBPF Monitoring

Resource Scaling Policy는 별도 Task로 진행 예정입니다.

English

The following items are out of scope for this issue.

• Resource Scaling Policy
• Priority-based Application Termination
• Victim Selection Algorithm
• Stateful Offloading
• Storage Resource Scaling
• eBPF Monitoring

Resource scaling policies will be handled separately.

[yonghyeon-kim]

flowchart TB

    %% =========================
    %% Monitoring Layer
    %% =========================
    subgraph Monitoring_Layer["Monitoring Layer"]
        MS["MonitoringServer<br/>- Collect App Performance Metrics<br/>- Collect CPU / Memory Usage<br/>- Detect Performance Degradation"]
    end

    %% =========================
    %% Decision Layer
    %% =========================
    subgraph Decision_Layer["Decision Layer"]
        PM["PolicyManager<br/>- Evaluate Scaling Condition<br/>- Decide Resource Adjustment<br/>- Generate Scaling Action"]
    end

    %% =========================
    %% Control Layer
    %% =========================
    subgraph Control_Layer["Control Layer"]
        AC["ActionController<br/>- Orchestrate Scaling Workflow<br/>- Request Resource Validation<br/>- Send Runtime Update Command<br/>- Track Scaling Result"]

        RM["ResourceManager<br/>- Check Available CPU<br/>- Check Available Memory<br/>- Maintain Desired Resource State<br/>- Prevent Resource Overcommit"]
    end

    %% =========================
    %% Node Execution Layer
    %% =========================
    subgraph Node_Execution_Layer["Node Execution Layer"]
        NA["NodeAgent<br/>- Receive UpdateResources Request<br/>- Apply Runtime Resource Update<br/>- Query Actual Runtime State<br/>- Report Update Result"]

        PODMAN["Podman Runtime<br/>- podman update<br/>- Apply CPU Limit<br/>- Apply Memory Limit"]
    end

    %% =========================
    %% Workload Layer
    %% =========================
    subgraph Workload_Layer["Workload Layer"]
        WL["Running Workload<br/>Containerized Application<br/>CPU / Memory Updated at Runtime"]
    end

    %% =========================
    %% Main Scaling Flow
    %% =========================
    MS -->|"Performance degradation detected"| PM
    PM -->|"Create scaling action"| AC
    AC -->|"Validate requested resources"| RM
    RM -->|"Validation result<br/>Available / Insufficient"| AC
    AC -->|"UpdateResources request"| NA
    NA -->|"podman update<br/>CPU / Memory"| PODMAN
    PODMAN -->|"Resource update applied"| WL

    %% =========================
    %% State Synchronization Flow
    %% =========================
    WL -->|"Actual resource usage / limit"| PODMAN
    PODMAN -->|"Actual runtime state"| NA
    NA -->|"Update result / actual state"| AC
    AC -->|"Update scaling status"| RM
    RM -->|"Synchronized resource state"| MS

    %% =========================
    %% Styling
    %% =========================
    classDef monitoring fill:#E8F5E9,stroke:#2E7D32,stroke-width:1.5px,color:#000;
    classDef decision fill:#E3F2FD,stroke:#1565C0,stroke-width:1.5px,color:#000;
    classDef control fill:#FFF8E1,stroke:#F9A825,stroke-width:1.5px,color:#000;
    classDef node fill:#F3E5F5,stroke:#7B1FA2,stroke-width:1.5px,color:#000;
    classDef workload fill:#ECEFF1,stroke:#455A64,stroke-width:1.5px,color:#000;

    class MS monitoring;
    class PM decision;
    class AC,RM control;
    class NA,PODMAN node;
    class WL workload;

Loading

[yonghyeon-kim]

flowchart LR

    MS["MonitoringServer<br/>App / Resource Metrics"]
    PM["PolicyManager<br/>Scaling Decision"]
    AC["ActionController<br/>Scaling Orchestration"]
    RM["ResourceManager<br/>Availability Check<br/>State Management"]
    NA["NodeAgent<br/>Runtime Executor"]
    PODMAN["Podman Runtime<br/>podman update"]
    WL["Running Workload<br/>Updated CPU / Memory"]

    MS --> PM
    PM --> AC
    AC --> RM
    RM --> AC
    AC --> NA
    NA --> PODMAN
    PODMAN --> WL

    NA --> AC
    AC --> RM
    RM --> MS

Loading

[yonghyeon-kim]

Korean

Dynamic Resource Scaling 아키텍처는 계층형 구조를 따른다.

1. NodeAgent에서 Resource 사용량 Monitoring하여 MonitoringServer로 전달.
2. MonitoringServer는 PolicyManager에게 JSON 형태로 전달.
3. PolicyManager는 수집된 Metric을 기반으로 Resource Scaling 필요 여부를 판단하고 Scaling Action을 생성한다. (핵심)

• Workload에 대한 Quota랑 limitation를 설정, 메트릭 받은거랑 비교해서 기준 % 이상이면 과부하로 판단
• 리소스를 Scaling의 범위를 설정 (Configuration에 기본은 2배 / 조정 가능하도록)
• ResourceManager는 역할 / 책임 범위를 어떻게 정의할지 별도 논의 필요
• ActionController에 Proto 추가하여 (어떤 노드, 어떤 워크로드, 어느 정도 리소스 조정)

4. ActionController는 Scaling Workflow를 오케스트레이션하며 ResourceManager와 NodeAgent를 제어한다.

• Podman 기반으로 CPU / Mem는 Podman REST API로 업데이트 가능
• 결과를 타겟 NodeAgent에게 전송
• NodeAgent가 REST API Call 하여 Runtime 중에 CPU / Mem 전환 가능

5. NodeAgent는 대상 노드에서 Podman Runtime을 통해 실행 중인 Workload의 CPU 및 Memory Limit을 Runtime에 변경한다.

English
The Dynamic Resource Scaling architecture follows a layered design.

MonitoringServer detects application performance degradation and provides runtime metrics.
PolicyManager evaluates whether resource scaling is required and generates a scaling action.
ActionController orchestrates the scaling workflow and coordinates with ResourceManager and NodeAgent.
ResourceManager validates available CPU and memory before applying the update and maintains the desired resource state.
NodeAgent executes the runtime resource update on the target node by using Podman runtime operations such as podman update.
After the update is applied, NodeAgent reports the actual runtime state back to ActionController and ResourceManager to keep Pullpiri-managed state synchronized with the actual container runtime state.

[yonghyeon-kim]

sequenceDiagram
    autonumber

    participant App as Running Workload
    participant MS as MonitoringServer
    participant PM as PolicyManager
    participant AC as ActionController
    participant RM as ResourceManager
    participant NA as NodeAgent
    participant PR as Podman Runtime

    %% =====================================================
    %% 1. Monitoring Phase
    %% =====================================================
    App->>MS: Report app performance metrics<br/>(latency, throughput, CPU, memory)
    MS->>MS: Detect performance degradation

    %% =====================================================
    %% 2. Policy Decision Phase
    %% =====================================================
    MS->>PM: Notify performance degradation<br/>with workload metrics
    PM->>PM: Evaluate scaling condition

    alt Scaling is required
        PM->>AC: Create scaling action<br/>(workload_id, requested CPU, requested memory)

        %% =====================================================
        %% 3. Resource Validation Phase
        %% =====================================================
        AC->>RM: Validate resource availability<br/>(requested CPU / memory)
        RM->>RM: Calculate available resources<br/>Total - Allocated - Reserved

        alt Resource is available
            RM-->>AC: Validation success<br/>resource can be allocated

            %% =====================================================
            %% 4. Desired State Update
            %% =====================================================
            AC->>RM: Update desired resource state<br/>(desired CPU / memory)
            RM-->>AC: Desired state recorded

            %% =====================================================
            %% 5. Runtime Resource Update Phase
            %% =====================================================
            AC->>NA: UpdateResources request<br/>(workload_id, CPU, memory)
            NA->>NA: Validate target workload<br/>and runtime capability

            NA->>PR: podman update<br/>(CPU / memory limits)
            PR->>App: Apply runtime resource limits

            alt Podman update success
                PR-->>NA: Update success
                NA->>PR: Query actual resource limits
                PR-->>NA: Return actual runtime state

                %% =====================================================
                %% 6. State Synchronization Phase
                %% =====================================================
                NA-->>AC: UpdateResources response<br/>(success, actual CPU, actual memory)
                AC->>RM: Update actual resource state
                RM->>RM: Compare desired state<br/>with actual state

                alt Desired state equals actual state
                    RM-->>AC: Resource state synchronized
                    AC-->>PM: Scaling action completed
                    PM-->>MS: Scaling result updated
                else Desired state differs from actual state
                    RM-->>AC: State mismatch detected
                    AC-->>PM: Scaling action partially failed<br/>or reconcile required
                    PM-->>MS: Report state inconsistency
                end

            else Podman update failed
                PR-->>NA: Update failed<br/>(runtime error / unsupported option)
                NA-->>AC: UpdateResources response<br/>(failed, reason)
                AC->>RM: Rollback or mark desired state failed
                RM-->>AC: Resource state marked as failed
                AC-->>PM: Scaling action failed
                PM-->>MS: Report scaling failure
            end

        else Resource is insufficient
            RM-->>AC: Validation failed<br/>insufficient CPU / memory
            AC-->>PM: Scaling action rejected
            PM-->>MS: Report resource shortage
        end

    else Scaling is not required
        PM-->>MS: No scaling action required
    end

Loading

[yonghyeon-kim]

sequenceDiagram
    autonumber

    participant MS as MonitoringServer
    participant PM as PolicyManager
    participant AC as ActionController
    participant RM as ResourceManager
    participant NA as NodeAgent
    participant PR as Podman Runtime
    participant WL as Running Workload

    MS->>PM: Performance degradation detected
    PM->>AC: Request dynamic resource scaling
    AC->>RM: Check available CPU / memory
    RM-->>AC: Resource validation result

    alt Resource available
        AC->>RM: Store desired resource state
        AC->>NA: UpdateResources(workload_id, cpu, memory)
        NA->>PR: podman update
        PR->>WL: Apply CPU / memory limits
        PR-->>NA: Update result
        NA-->>AC: Actual resource state
        AC->>RM: Synchronize actual resource state
        RM-->>AC: Desired / actual state synchronized
        AC-->>PM: Scaling completed
        PM-->>MS: Update scaling result
    else Resource insufficient
        AC-->>PM: Scaling rejected
        PM-->>MS: Report insufficient resource
    end

Loading

[yonghyeon-kim]

Korean

Dynamic Resource Scaling Sequence는 MonitoringServer가 애플리케이션 성능 저하를 감지하면서 시작된다.
PolicyManager는 수집된 Runtime Metric을 기반으로 Resource Scaling이 필요한지 판단하고, Scaling Action을 ActionController로 전달한다. ActionController는 ResourceManager를 통해 요청된 CPU 및 Memory가 현재 노드에서 할당 가능한지 먼저 검증한다.
리소스가 충분한 경우 ResourceManager는 Desired Resource State를 기록하고,
ActionController는 NodeAgent에 UpdateResources 요청을 전달한다.
NodeAgent는 Podman Runtime을 통해 실행 중인 Workload의 CPU 및 Memory Limit을 Runtime에 변경한다.
적용 이후 NodeAgent는 실제 Runtime Resource State를 조회하여 ActionController로 보고하고,
ResourceManager는 Desired State와 Actual State를 비교하여
Pullpiri 내부 관리 상태와 실제 Container Runtime 상태가 일관성을 유지하도록 동기화한다.

English

The Dynamic Resource Scaling sequence starts when MonitoringServer detects application performance degradation based on runtime metrics. PolicyManager evaluates whether a scaling action is required and forwards the scaling request to ActionController. ActionController first asks ResourceManager to validate the requested CPU and memory against the available node resources. If the requested resources are available, ResourceManager records the desired resource state, and ActionController sends an UpdateResources request to NodeAgent. NodeAgent applies the runtime resource update through Podman Runtime, for example by using podman update. After the update is applied, NodeAgent queries the actual runtime resource state and reports it back to ActionController. ResourceManager then synchronizes the actual resource state with the desired state to ensure that Pullpiri-managed resource information remains consistent with the actual container runtime state.

[yonghyeon-kim]

Component Responsibilities

Overview

Dynamic Resource Scaling은 여러 Pullpiri 컴포넌트가 협력하여 수행된다.

각 컴포넌트는 명확한 책임 범위를 가지며, 정책 결정(Decision), 실행(Execution), 상태 관리(State Management)를 분리하여 설계한다.

Component	Responsibility
MonitoringServer	Workload 및 Node Resource 모니터링
PolicyManager	Scaling 필요 여부 판단
ActionController	Scaling Workflow 실행
ResourceManager	Resource Validation 및 Resource State 관리
NodeAgent	Runtime Resource Update 수행
Podman Runtime	실제 Resource 적용

---

MonitoringServer

Responsibility

MonitoringServer는 Dynamic Resource Scaling의 Trigger Source 역할을 수행한다.

Functions

• CPU Usage Monitoring
• Memory Usage Monitoring
• Application Latency Monitoring
• Throughput Monitoring
• Runtime Resource Monitoring

Input

Container Metrics
System Metrics
Application Metrics

Output

Performance Event

Latency = 450ms
Threshold = 300ms

Status = Degraded

Not Responsible For

Scaling Decision
Resource Allocation
Runtime Resource Update

---

PolicyManager

Responsibility

PolicyManager는 Scaling 필요 여부를 판단하고 Scaling Action을 생성한다.

Functions

• Scaling Condition Evaluation
• Performance Threshold Comparison
• Scaling Action Generation

Input

Performance Metrics
Current Resource Information
Runtime Status

Output

ResourceScalingAction

Example

Workload A

Current CPU    = 2
Target CPU     = 4

Current Memory = 1GiB
Target Memory  = 2GiB

Not Responsible For

Resource Availability Validation
Container Runtime Update
Resource Reservation

---

ActionController

Responsibility

ActionController는 Dynamic Resource Scaling Workflow를 오케스트레이션한다.

Functions

• Scaling Action 수신
• Resource Validation 요청
• Desired State 적용 요청
• NodeAgent 제어
• Update Result 수집
• Error Handling

Input

ResourceScalingAction

Output

UpdateResourcesRequest

Managed State

Scaling Task State

Pending
Running
Succeeded
Failed

Example

CPU
2 → 4

Memory
1GiB → 2GiB

Not Responsible For

Scaling Policy 판단
Available Resource 계산
Container Runtime 직접 제어

---

ResourceManager

Responsibility

ResourceManager는 Resource Capacity 관리와 Resource State 관리를 담당한다.

본 설계에서 Desired Resource State의 Owner 역할을 수행한다.

Functions

Resource Validation

• Available CPU 계산
• Available Memory 계산
• Resource Reservation 확인
• Allocation 가능 여부 판단

Desired State Management

• Target CPU 관리
• Target Memory 관리
• Scaling State 저장

State Synchronization

• Desired State 관리
• Actual State 비교
• Reconcile 필요 여부 판단

Input

Scaling Request
Actual Runtime State

Output

Validation Result

Success
Rejected

Example

Node Total CPU = 8

Allocated CPU = 5
Reserved CPU  = 1

Available CPU = 2

Request

CPU +3

Result

Validation Failed

Managed State

Desired State

workload-a:
  cpu: 4
  memory: 2048Mi

Not Responsible For

Podman API 호출
Container Runtime Operation

---

NodeAgent

Responsibility

NodeAgent는 Runtime Resource Scaling Executor 역할을 수행한다.

Functions

• UpdateResources 요청 수신
• Runtime Capability Validation
• Podman Runtime 호출
• Actual Resource State 조회
• Scaling Result 반환

Input

UpdateResourcesRequest

Output

UpdateResourcesResponse

Example

podman update \
  --cpus 4 \
  --memory 2g

Managed State

Actual Runtime State

CPU Limit = 4
Memory Limit = 2GiB

Not Responsible For

Scaling Policy 결정
Resource Reservation
Available Resource 계산

---

Podman Runtime

Responsibility

Podman Runtime은 실제 Container Resource Limit을 적용한다.

Functions

• CPU Limit 변경
• Memory Limit 변경
• Runtime Resource State 제공

Input

podman update

Output

Updated Container Cgroup Configuration

Managed State

Actual CPU Limit
Actual Memory Limit

Not Responsible For

Scaling Decision
Resource Validation
Desired State Management

---

Responsibility Matrix (RACI)

Activity	MonitoringServer	PolicyManager	ActionController	ResourceManager	NodeAgent
Metric Collection	R	-	-	-

[yonghyeon-kim]

API / Proto Changes

Overview

Dynamic Resource Scaling을 지원하기 위해 Pullpiri 컴포넌트 간 신규 API 및 Proto Message 정의가 필요하다.

현재 Pullpiri는 Workload 배포 및 상태 관리 기능은 제공하지만, 실행 중인 Workload의 CPU 및 Memory Resource를 Runtime에 변경하는 인터페이스는 존재하지 않는다.

본 설계에서는 다음 기능을 지원하기 위한 API를 정의한다.

• Runtime CPU Update
• Runtime Memory Update
• Desired Resource State 관리
• Actual Resource State 조회
• Resource Update Result Reporting
• Resource State Synchronization

---

API Change Summary

Component	Change Type
PolicyManager	No Change
ActionController	New Resource Scaling Request
ResourceManager	New Resource State Management API
NodeAgent	New Runtime Resource Update API
Runtime Layer	Podman Update Integration

---

Design Principles

Unified Resource Update API

CPU와 Memory를 위한 개별 API를 만들지 않는다.

예)

UpdateCpuLimit()
UpdateMemoryLimit()

대신 하나의 API를 사용한다.

UpdateResources()

이를 통해 다음을 지원할 수 있다.

CPU Only Update

Memory Only Update

CPU + Memory Update

---

NodeAgent Service Changes

New RPC

rpc UpdateResources (
    UpdateResourcesRequest
)
returns (
    UpdateResourcesResponse
);

Purpose

실행 중인 Workload의 CPU 및 Memory Resource를 Runtime에 변경한다.

---

UpdateResourcesRequest

message UpdateResourcesRequest {

  string workload_id = 1;

  optional uint32 cpu_limit = 2;

  optional uint64 memory_limit = 3;

}

---

Fields

workload_id

대상 Workload 식별자

예

navigation-app

---

cpu_limit

변경할 CPU Limit

예

2 → 4

---

memory_limit

변경할 Memory Limit

단위

MiB

예

1024 → 2048

---

UpdateResourcesResponse

message UpdateResourcesResponse {

  bool success = 1;

  string message = 2;

  ResourceStatus status = 3;

}

---

Fields

success

true
false

---

message

실패 원인

예

Insufficient Memory

Podman Update Failure

Resource Validation Failed

---

status

적용 이후 Runtime Resource 상태

---

Resource Status Definition

Purpose

Actual Runtime State 저장

---

message ResourceStatus {

  uint32 actual_cpu_limit = 1;

  uint64 actual_memory_limit = 2;

  ResourceUpdateState update_state = 3;

}

---

Example

actual_cpu_limit = 4

actual_memory_limit = 2048

update_state = SUCCESS

---

Resource Update State

enum ResourceUpdateState {

  UNKNOWN = 0;

  PENDING = 1;

  RUNNING = 2;

  SUCCESS = 3;

  FAILED = 4;

}

---

Desired Resource State Model

Purpose

Pullpiri가 관리하는 목표 Resource State

---

message ResourceSpec {

  uint32 cpu_limit = 1;

  uint64 memory_limit = 2;

}

---

Example

Desired State

CPU = 4

Memory = 2048Mi

---

Actual Resource State Model

Purpose

Runtime에서 실제 적용된 상태

---

message ResourceRuntimeStatus {

  uint32 cpu_limit = 1;

  uint64 memory_limit = 2;

  string runtime_id = 3;

}

---

Example

Actual State

CPU = 4

Memory = 2048Mi

---

Resource Synchronization API

GetResourceStatus

Runtime 상태 조회

---

rpc GetResourceStatus (
    GetResourceStatusRequest
)
returns (
    GetResourceStatusResponse
);

---

Request

message GetResourceStatusRequest {

  string workload_id = 1;

}

---

Response

message GetResourceStatusResponse {

  bool success = 1;

  ResourceRuntimeStatus status = 2;

}

---

Resource Validation API

ValidateResourceUpdate

ResourceManager에서 Resource Availability 검증

---

rpc ValidateResourceUpdate (
    ValidateResourceUpdateRequest
)
returns (
    ValidateResourceUpdateResponse
);

---

Request

message ValidateResourceUpdateRequest {

  string workload_id = 1;

  uint32 requested_cpu = 2;

  uint64 requested_memory = 3;

}

---

Response

message ValidateResourceUpdateResponse {

  bool allowed = 1;

  string reason = 2;

}

---

Future Extension Compatibility

본 API 구조는 향후 다음 기능으로 확장 가능하다.

---

Scale Down

UpdateResources()

재사용

---

Offloading

UpdateResources()

+
RelocateWorkload()

---

Migration

MigrationRequest

추가 가능

---

Bandwidth Scaling

network_limit

필드 추가 가능

---

Storage Scaling

storage_limit

필드 추가 가능

---

API Ownership

API	Owner
UpdateResources	NodeAgent
GetResourceStatus	NodeAgent
ValidateResourceUpdate	ResourceManager

---

Summary

Dynamic Resource Scaling을 위해 다음 신규 인터페이스를 도입한다.

UpdateResources()
GetResourceStatus()
ValidateResourceUpdate()

또한 Resource 상태를 명확하게 관리하기 위해 다음 두 가지 State 모델을 구분한다.

Desired Resource State

(ResourceManager Owner)

Actual Resource State

(NodeAgent Owner)

이를 통해 Pullpiri 내부 상태와 실제 Runtime 상태 간의 불일치를 최소화하고 Kubernetes 스타일의 Reconcile Architecture를 구현할 수 있다.

[yonghyeon-kim]

Resource Synchronization Strategy

Overview

Dynamic Resource Scaling에서는 Pullpiri 내부에서 관리하는 Resource State와 실제 Container Runtime의 Resource State가 항상 일관성을 유지해야 한다.

Resource Synchronization Strategy의 목표는 다음과 같다.

• Desired State와 Actual State 간 불일치 방지
• Runtime Update 실패 감지
• Pullpiri Internal State 정합성 유지
• Reconcile 기반 상태 복구 지원

---

Synchronization Objectives

Consistency

Pullpiri 관리 상태와 Runtime 상태가 동일해야 한다.

Desired CPU    = 4
Actual CPU     = 4

---

Drift Detection

상태 불일치를 탐지할 수 있어야 한다.

Desired CPU    = 4
Actual CPU     = 2

결과

Resource Drift Detected

---

Recoverability

상태 불일치 발생 시 Reconcile 가능해야 한다.

Drift Detected
       ↓
Reconcile Trigger
       ↓
Apply Desired State

---

State Ownership Model

Desired Resource State

Owner

ResourceManager

---

Description

Pullpiri가 목표로 하는 Resource 상태

---

Example

workload-a:
  cpu_limit: 4
  memory_limit: 2048Mi

---

Actual Resource State

Owner

NodeAgent

---

Description

Container Runtime에 실제 적용된 Resource 상태

---

Example

workload-a:
  cpu_limit: 2
  memory_limit: 2048Mi

---

Synchronization Architecture

               Desired State
                     │
                     ▼

             ResourceManager
                     │
                     ▼

            ActionController
                     │
                     ▼

                NodeAgent
                     │
                     ▼

              Podman Runtime
                     │
                     ▼

              Actual State
                     │
                     ▼

                NodeAgent
                     │
                     ▼

             ResourceManager
                     │
                     ▼

         Desired vs Actual Compare

---

Resource Update Lifecycle

Step 1

Desired State 생성

---

ResourceManager 저장

cpu_limit: 4
memory_limit: 2048Mi

상태

PENDING

---

Step 2

NodeAgent Resource Update 수행

podman update \
    --cpus 4 \
    --memory 2g

상태

RUNNING

---

Step 3

Runtime Resource 조회

NodeAgent 수행

Query Container Resource Limit

---

Step 4

Actual State 수집

예

cpu_limit: 4
memory_limit: 2048Mi

---

Step 5

Desired State 비교

ResourceManager 수행

---

Success

Desired CPU = 4
Actual CPU = 4

Desired Memory = 2048
Actual Memory = 2048

결과

SYNCHRONIZED

---

Failure

Desired CPU = 4
Actual CPU = 2

결과

DRIFT_DETECTED

---

Synchronization States

enum ResourceSyncState {

    UNKNOWN = 0;

    PENDING = 1;

    APPLYING = 2;

    SYNCHRONIZED = 3;

    DRIFT_DETECTED = 4;

    FAILED = 5;
}

---

State Transition Model

          PENDING
              │
              ▼

          APPLYING
              │

      ┌───────┴────────┐
      │                │
      ▼                ▼

SYNCHRONIZED      FAILED
      │
      │
      ▼

DRIFT_DETECTED
      │
      ▼

RECONCILE
      │
      ▼

SYNCHRONIZED

---

Synchronization Trigger

Trigger 1

Resource Update Success

UpdateResources Completed

즉시 Actual State 확인

---

Trigger 2

Periodic Synchronization

주기적 상태 점검

예

30 sec
60 sec
300 sec

구현 정책에 따라 결정

---

Trigger 3

NodeAgent Restart

NodeAgent 재기동 시

Runtime State Rebuild

수행

---

Trigger 4

Container Restart

Container 상태 변경 발생 시

Resource State Refresh

수행

---

Drift Detection Strategy

Definition

Desired State
!=
Actual State

이면 Drift 발생

---

Example 1

CPU Drift

Desired CPU = 4

Actual CPU = 2

결과

DRIFT_DETECTED

---

Example 2

Memory Drift

Desired Memory = 4096Mi

Actual Memory = 2048Mi

결과

DRIFT_DETECTED

---

Reconciliation Strategy

Objective

Drift 발생 시 Desired State 기준으로 상태 복구

---

Flow

Drift Detected
        │
        ▼

ResourceManager

        │
        ▼

ActionController

        │
        ▼

NodeAgent

        │
        ▼

UpdateResources Retry

        │
        ▼

Actual State Refresh

        │
        ▼

State Compare

---

Retry Policy

예시

Retry Count = 3

Retry Interval = 5 sec

정책은 별도 정의

---

Failure Handling

Case 1

NodeAgent Unavailable

Actual State Unknown

상태

FAILED

---

Case 2

Podman Update Failure

Runtime Apply Error

상태

FAILED

---

Case 3

Runtime State Query Failure

Resource Verification Failed

상태

FAILED

---

Recovery Strategy

Startup Reconciliation

Pullpiri 재시작 시

Desired State Load

↓

NodeAgent Query Runtime

↓

Actual State Rebuild

↓

Compare

↓

Reconcile

---

Synchronization Responsibility Matrix

Operation	ResourceManager	ActionController	NodeAgent
Desired State Creation	R/A	C	-
Actual State Collection	-	-	R/A
State Comparison	R/A	C	-
Drift Detection	R/A	C	-
Resource Update Retry	C	R	C
Reconciliation Execution	C	R	R
Synchronization Status Update	R/A	C	C

Legend

R = Responsible
A = Accountable
C = Consulted

---

Summary

Dynamic Resource Scaling은 Kubernetes의 Desired State Reconciliation Pattern을 기반으로 설계한다.

ResourceManager
    ↓
Desired State
    ↓
ActionController
    ↓
NodeAgent
    ↓
Podman Runtime
    ↓
Actual State
    ↓
ResourceManager
    ↓
State Comparison
    ↓
Reconcile

이를 통해 Pullpiri가 관리하는 Resource 상태와 실제 Container Runtime 상태 간의 일관성을 유지하며, Runtime Update 실패나 상태 불일치 상황에서도 안정적으로 복구할 수 있다.

[yonghyeon-kim]

Known Limitations

Overview

Dynamic Resource Scaling은 실행 중인 Workload의 CPU 및 Memory Resource를 Runtime에 변경하는 기능을 제공하지만, Container Runtime, Linux Cgroup, Node Resource 상태, 그리고 Pullpiri Architecture 특성상 몇 가지 제약사항이 존재한다.

본 장에서는 현재 설계에서 예상되는 제한사항과 알려진 리스크를 정리한다.

---

Runtime Capability Dependency

Description

Dynamic Resource Scaling은 Container Runtime이 Runtime Resource Update를 지원한다는 전제에 의존한다.

현재 설계는 Podman Runtime의 Runtime Resource Update 기능을 활용하는 것을 가정한다.

---

Limitation

Runtime이 지원하지 않는 Resource 유형은 동적으로 변경할 수 없다.

예상 지원 가능 항목

CPU Limit
CPU Shares
CPU Affinity
Memory Limit

예상 제약 항목

HugePages
NUMA Binding
Kernel Parameters

---

Impact

일부 Resource는 Workload 재시작 없이 변경 불가

---

Resource Scaling is Best-Effort

Description

Resource Update 요청이 성공적으로 전달되더라도 실제 Resource Allocation이 항상 보장되는 것은 아니다.

---

Example

요청

CPU 2 → 4

실제 노드 상태

Available CPU = 1

결과

Scaling Rejected

---

Impact

Dynamic Resource Scaling은 Resource Availability에 따라 수행되는 Best-Effort Operation이다.

---

No Guarantee of Application Performance Improvement

Description

Resource를 증가시켜도 Application 성능이 반드시 개선된다는 보장은 없다.

---

Example

성능 저하 원인

External Service Latency

Network Congestion

Database Bottleneck

Application Logic Issue

---

결과

CPU Increase

Memory Increase

→ No Performance Improvement

---

Impact

Dynamic Resource Scaling은 App Performance Degradation의 모든 원인을 해결하지 못한다.

---

Temporary Resource Overcommit Risk

Description

여러 Workload가 동시에 Scaling을 요청하는 경우 Resource Race Condition이 발생할 수 있다.

---

Example

Workload A

CPU +2 Request

동시에

Workload B

CPU +2 Request

현재 Available CPU

CPU = 2

---

Impact

동시 요청 처리 방식에 따라 Resource Overcommit이 발생할 수 있다.

---

Mitigation

Resource Reservation

Resource Locking

Serialized Allocation

---

Desired State and Actual State Divergence

Description

Pullpiri 내부 상태와 실제 Runtime Resource 상태가 일시적으로 불일치할 수 있다.

---

Example

Desired State

CPU = 4

Actual State

CPU = 2

---

원인

Podman Update Failure

Runtime Error

NodeAgent Failure

---

Impact

State Drift

발생 가능

---

Mitigation

Periodic Synchronization

Reconciliation

---

Synchronization Delay

Description

Resource Update 이후 Pullpiri 내부 상태가 즉시 동기화되지 않을 수 있다.

---

Example

Resource Updated

↓

Synchronization Pending

↓

State Synchronized

---

Impact

짧은 시간 동안 Desired State와 Actual State가 다를 수 있다.

---

NodeAgent Availability Dependency

Description

Runtime Resource 변경은 NodeAgent를 통해 수행된다.

NodeAgent 장애 시 Resource Scaling 기능을 수행할 수 없다.

---

Example

NodeAgent Down

결과

Scaling Request Failed

---

Impact

NodeAgent는 Dynamic Resource Scaling의 Single Execution Point가 된다.

---

Runtime Query Inaccuracy

Description

Actual Resource State 조회 시점과 실제 Resource 반영 시점 간 차이가 존재할 수 있다.

---

Example

UpdateResources

직후

GetResourceStatus

호출

---

결과

Old State Returned

가능성 존재

---

Impact

일시적인 상태 불일치가 발생할 수 있다.

---

Scaling Granularity Limitation

Description

Resource Scaling은 Container 단위로 수행된다.

---

Not Supported

Thread-Level Scaling

Process-Level Scaling

Function-Level Scaling

---

Impact

애플리케이션 내부 특정 기능만 선택적으로 확장할 수 없다.

---

No Automatic Rollback

Description

Resource Update 실패 시 자동 Rollback은 본 설계 범위에 포함되지 않는다.

---

Example

CPU Update Success

Memory Update Failure

결과

Partial Update State

가능

---

Impact

실패 시 복구 정책은 별도 설계가 필요하다.

---

Multi-Node Consistency Not Covered

Description

본 설계는 단일 Workload에 대한 Runtime Resource Update를 목표로 한다.

---

Not Covered

Multi-Node Synchronization

Cluster Resource Balancing

Cross-Node Resource Migration

---

Impact

향후 Cluster-Level Resource Management 기능이 별도로 필요하다.

---

Scale-Down Risk

Description

Scale-Up보다 Scale-Down이 더 높은 위험을 가질 수 있다.

---

Example

CPU 4 → 1

또는

Memory 4096Mi → 512Mi

---

결과

Application Performance Degradation

OOM Events

Unexpected Behavior

---

Impact

Scale-Down 정책은 추가 검증이 필요하다.

---

Out of Scope Limitations

본 설계에서는 아래 항목을 다루지 않는다.

Resource Scaling Policy

Priority-based Application Termination

Victim Selection Algorithm

Stateful Offloading

Storage Resource Scaling

GPU Resource Scaling

eBPF Monitoring

Cluster-wide Scheduling

Automatic Rollback

---

Summary

Dynamic Resource Scaling은 Runtime CPU 및 Memory 변경을 가능하게 하지만, 실제 동작은 Container Runtime Capability, Node Resource Availability, Synchronization Delay, 그리고 Runtime Constraints에 영향을 받는다.

주요 제한사항은 다음과 같다.

Runtime Capability Dependency

Best-Effort Resource Allocation

State Divergence

Synchronization Delay

NodeAgent Dependency

Scale-Down Risk

Multi-Node Management Out of Scope

따라서 Pullpiri는 Kubernetes와 유사하게 Desired State 기반 Reconciliation 모델을 활용하여 Runtime 제약사항과 상태 불일치를 관리해야 한다.

[Chulhee1Lee]

@yonghyeon-kim, @youngtaekiim 해당 내용 7월 9일 오전에 리뷰 미팅 진행하시죠. 반영 결정되면 Architecture Document로 정리 작성하여 반영 부탁합니다.