Skip to content

storage-detaching hook fails on Juju 3.5 due to secrets creation for dying units #243

Description

@BON4

Steps to reproduce

  1. Clone the source code for cassandra-operator.
    If this issue is reviewed after a temporary bug fix on the operator side, remove the following lines from src/core/state.py:
        if (self.model.juju_version <= JujuVersion("3.5.7")) and  os.getenv("JUJU_HOOK_NAME", "") == f"{DATA_STORAGE}-storage-detaching":
            unit_additional_secret_fields = None
  1. Build and deploy 2 units:
juju deploy ./cassandra_ubuntu@24.04-amd64.charm cassandra --config profile=testing --num-units=2
  1. Remove a secondary unit:
juju remove-unit <secondary-unit>
  1. Wait until the storage-detaching hook completes on the departing unit.

  2. Observe that the hook fails on any remaining unit with the error:

cannot apply changes: creating secrets: cannot create secret for owner "<secondary-unit>" which is not alive

Expected behavior

  • After a unit begins removal (life = dying), no new secrets should be created on its behalf, neither by the departing unit nor by surviving peers.
  • DataPeerUnitData should skip secret creation for units whose life is dying or defer until the unit is alive again.
  • The unit should be removed cleanly after the storage-detaching hook completes successfully.

Actual behavior

  • DataPeerUnitData automatically attempts to create Juju secrets for all fields in additional_secret_fields.
  • Juju rejects requests for units that are not alive, causing the hook to fail and retry indefinitely, which blocks the application.

Versions

Operating system: Ubuntu 24.04.1 LTS

Juju CLI: 3.5.7-genericlinux-amd64

Juju agent: 3.5.7

Log output

unit-cassandra-1: 17:10:48 ERROR juju.worker.uniter.context cannot apply changes: creating secrets: cannot create secret for owner "unit-cassandra-1" which is not alive
unit-cassandra-1: 17:10:48 ERROR juju.worker.uniter.operation hook "data-storage-detaching" (via hook dispatching script: dispatch) failed: creating secrets: cannot create secret for owner "unit-cassandra-1" which is not alive

Additional context

  • This issue occurs only on Juju v3.5; Juju v3.6 does not have this problem.
  • The issue can be worked around by removing additional_secret_fields from DataPeerUnitData during unit removal:
        if (self.model.juju_version <= JujuVersion("3.5.7")) and  os.getenv("JUJU_HOOK_NAME", "") == f"{DATA_STORAGE}-storage-detaching":
            unit_additional_secret_fields = None

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions