Skip to content

Preserve leftover recordings on container restart#1249

Open
chalapathi444 wants to merge 1 commit into
livekit:mainfrom
chalapathi444:feat/recording-recovery
Open

Preserve leftover recordings on container restart#1249
chalapathi444 wants to merge 1 commit into
livekit:mainfrom
chalapathi444:feat/recording-recovery

Conversation

@chalapathi444

Copy link
Copy Markdown

What

The entrypoint wipes /home/egress/tmp on every start. If a container restarts mid-egress (crash, OOM, redeploy), the in-progress recording in tmp is deleted before anyone can recover it.

How

Before the cleanup, if tmp is non-empty, copy its contents to /home/egress/recovered so operators can retrieve partial output from the previous session:

if [ -d /home/egress/tmp ] && [ "$(ls -A /home/egress/tmp 2>/dev/null)" ]; then
    mkdir -p /home/egress/recovered
    cp -r /home/egress/tmp/. /home/egress/recovered/
fi
rm -rf /home/egress/tmp/*

Notes

  • When tmp is empty (the normal case), nothing is copied and startup is unchanged.
  • recovered/ is not auto-pruned — open to a retention/size cap if you'd prefer one (e.g. clearing it after N restarts or above a size threshold), happy to adjust.

The entrypoint wipes /home/egress/tmp on every start. If a container
restarted mid-egress (crash, OOM, redeploy), any in-progress recording
in tmp was lost before it could be recovered.

Copy a non-empty tmp into /home/egress/recovered before the cleanup, so
operators can retrieve partial output from the previous session. The
copy is skipped when tmp is empty, leaving normal startup unchanged.
@chalapathi444 chalapathi444 requested a review from a team as a code owner June 6, 2026 06:50
@CLAassistant

CLAassistant commented Jun 6, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@biglittlebigben biglittlebigben left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion. We do not think that copying files within the pod is a good idea: this could cause the pod disk usage to grow without limit and no process to cleanup. This also doesn't cover Amy of the cases where the pod is killed and restarted from a clean container.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants