Skip to content

Image index worker can write to the database after shutdown, and stalls shutdown by 10s mid-encode #71

Description

@lstein

Summary

ImageIndexService.stop() (invokeai/app/services/image_index/image_index_default.py:189) joins the worker with a 10s timeout and, on expiry, logs a warning and abandons the daemon thread. Two consequences are not accounted for:

  1. Shutdown stalls for a full 10s whenever the worker is mid-encode. A cold load_model of CLIP-L can exceed that on its own.
  2. The abandoned thread keeps writing to the shared database after stop() has returned — and after the services it depends on have already been torn down.

Why the second one bites

self.image_index is the last attribute assigned in InvocationServices.__init__ (invocation_services.py:146), and Invoker.stop() iterates vars(self.services) in that order (invoker.py:36). So the image index is stopped last, and its abandoned thread then runs for up to 10 more seconds after events.stop(), model_manager.stop() (which calls cache.shutdown()), session_processor.stop(), and the object-serializer teardown have all completed. The worker calls into all of those.

Reproduction

Observed with a 13s encode:

WARNING --> Image index worker did not stop within 10s; abandoning daemon thread
stop() returned after 10.0s; worker alive = True
embedded at the moment stop() returned: 0
embedded 6s AFTER stop() returned  : 1     <- post-shutdown write on the shared connection

Suggested direction

Make the encode loop cancellable at batch granularity so the join succeeds promptly, and/or stop the image index earlier in the teardown order so it cannot outlive the services it uses. A guard that makes post-stop writes no-ops would limit the damage but leaves the 10s stall.

Context

Found during adversarial review of PR #35. Not user-reachable while image_index_enabled defaults to False, which is why it was deferred rather than fixed there.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions