Skip to content

Comments

fix: clean up temp images created for actions with --disable-cache#3986

Draft
dtrudg wants to merge 1 commit intosylabs:mainfrom
dtrudg:issue-3940
Draft

fix: clean up temp images created for actions with --disable-cache#3986
dtrudg wants to merge 1 commit intosylabs:mainfrom
dtrudg:issue-3940

Conversation

@dtrudg
Copy link
Member

@dtrudg dtrudg commented Feb 23, 2026

When an action (run/exec/shell...) is performed against a remote image URI then it is implicitly pulled before execution.

If the cache is active, the image is pulled into the cache and executed using the image in the cache. If the cache is disabled then the image is pulled to a temporary location, and executed from there.

Prior to this PR, a temporary image pulled when the cache is disabled was not being cleaned-up on exit. This means that TMPDIR space is consumed, and not released, each time singularity is run against a remote image URI with --disable-cache.

This PR corrects the issue by:

  1. Moving the implicit temporary image creation behaviour out of the client packages, into the CLI level.

  2. Adding a new 'PullTempDir` value to launcher and engine configuration, which tracks the created temporary directory that must be cleaned-up on exit.

  3. Adding code to the cleanup routines of the native runtime engine, and OCI launcher, to perform the actual deletion of the temp dir on container exit.

To test, run/shell/exec containers from remote (library, docker etc.) URIs with --disable-cache. During execution note that there is a /tmp/singularity-action-pull-xxxx directory containing the image, and that it is removed on container exit. The --debug output contains messages detailing this. e.g.:

$ singularity --debug run --disable-cache library://alpine
DEBUG   [U=1000,P=193493]  persistentPreRun()            Singularity version: 4.3.0+333-geb189737a
DEBUG   [U=1000,P=193493]  persistentPreRun()            Parsing configuration file /usr/local/etc/singularity/singularity.conf
DEBUG   [U=1000,P=193493]  maybeReExec()                 Checking whether to re-exec
DEBUG   [U=1000,P=193493]  handleConfDir()               /home/dtrudg-sylabs/.singularity already exists. Not creating.
DEBUG   [U=1000,P=193493]  handleRemoteConf()            Ensuring file permission of 0600 on /home/dtrudg-sylabs/.singularity/remote.yaml
DEBUG   [U=1000,P=193493]  uriToTempImage()              Cache disabled, pulling image to temporary file: /tmp/singularity-action-pull-4160879587/image
DEBUG   [U=1000,P=193493]  PullToFile()                  Cache disabled, pulling directly to: /tmp/singularity-action-pull-4160879587/image
...
Singularity> exit
...
INFO    [U=1000,P=193493]  CleanupContainer()            Cleaning up image temporary dir(s)...
VERBOSE [U=1000,P=193493]  CleanupContainer()            Removing /tmp/singularity-action-pull-4160879587
DEBUG   [U=1000,P=193493]  Master()                      Child exited with exit status 0

Fixes: #3940

@dtrudg dtrudg self-assigned this Feb 23, 2026
@dtrudg dtrudg force-pushed the issue-3940 branch 6 times, most recently from 376c234 to 1c0c98a Compare February 23, 2026 17:36
When an action (run/exec/shell...) is performed against a remote image
URI then it is implicitly pulled before execution.

If the cache is active, the image is pulled into the cache and
executed using the image in the cache. If the cache is disabled then
the image is pulled to a temporary location, and executed from there.

Prior to this PR, a temporary image pulled when the cache is disabled
was not being cleaned-up on exit. This means that TMPDIR space is
consumed, and not released, each time singularity is run against a
remote image URI with --disable-cache.

This PR corrects the issue by:

1. Moving the implicit temporary image creation behaviour out of the
client packages, into the CLI level.

2. Adding a new 'PullTempDir` value to launcher and engine
configuration, which tracks the created temporary directory that must
be cleaned-up on exit.

3. Adding code to the cleanup functions of the native runtime engine, and
OCI launcher, to perform the actual deletion of the temp dir on
container exit.

To test, run/shell/exec containers from remote (library, docker etc.)
URIs with `--disable-cache`. During execution note that there is a
`/tmp/singularity-action-pull-xxxx` directory containing the image, and
that it is removed on container exit. The `--debug` output contains
messages detailing this.

Fixes: sylabs#3940

e2e: add rudimentary action temp cleanup test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sbuild-tmp-cache-xxx files/dirs lack complete cleanup

1 participant