Skip to content

fix(control): reconcile /v1/plugins/reload like the admin reload path#13745

Open
AlinsRan wants to merge 1 commit into
apache:masterfrom
AlinsRan:fix/control-plugins-reload-reconcile
Open

fix(control): reconcile /v1/plugins/reload like the admin reload path#13745
AlinsRan wants to merge 1 commit into
apache:masterfrom
AlinsRan:fix/control-plugins-reload-reconcile

Conversation

@AlinsRan

Copy link
Copy Markdown
Contributor

Description

Follow-up to #13714 (which fixed #13537 for the admin reload path).

#13714 made the admin plugins reload (PUT /apisix/admin/plugins/reload) resilient to the resty.events broker's lack of a delivery guarantee: it bumps a shared plugins_conf_version in ngx.shared["internal-status"] and a per-worker reconciliation timer reloads any worker whose applied version is behind. That way a worker that misses the broadcast (e.g. while it is reconnecting to the broker) still converges.

The control reload path (PUT /v1/plugins/reload) has the identical exposure but was left out of #13714post_reload_plugins() only calls events:post(). So a worker can miss the control broadcast and keep running the plugins/timers of a removed plugin (exactly the #13537 symptom) even though the API returned 200.

Note this is not a lua-resty-events bug to fix here — the broker has no replay for a reconnecting client by design; the reconciliation is the intended application-level compensation. This PR simply applies the existing compensation to the control path.

Fix

apisix/control/v1.lua post_reload_plugins() now bumps the same plugins_conf_version before broadcasting, so the reconciliation timer registered by the admin path converges any worker that missed the event. When the admin is disabled the timer is absent and the bump is a no-op, leaving the broadcast as the only path (unchanged).

Tests

t/control/plugins-reload-reconcile.t (mirrors t/admin/plugins-reload-reconcile.t):

  • TEST 1: PUT /v1/plugins/reload bumps plugins_conf_version.
  • TEST 2: reproduces a missed broadcast by bumping the shared version directly (without going through the events layer) and asserts the reconciliation timer reloads the worker — i.e. a worker that missed the event converges.

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change (no user-facing config/behavior change)
  • I have verified that the change follows the existing code style

@dosubot dosubot Bot added the size:M This PR changes 30-99 lines, ignoring generated files. label Jul 24, 2026
apache#13714 made the admin plugins reload converge missed workers by bumping a shared
plugins_conf_version and reconciling against it in a periodic timer, but the
control API /v1/plugins/reload was left out: it only broadcasts the event. The
resty.events broker gives no delivery guarantee while a worker is reconnecting,
so a worker can miss the broadcast and stay on stale plugin state even though the
API returned 200 -- the same failure apache#13537 described, via the control path.

Bump the same shared version from the control path so the existing reconciliation
timer converges missed workers too.
@AlinsRan
AlinsRan force-pushed the fix/control-plugins-reload-reconcile branch from c79504c to d06facd Compare July 24, 2026 09:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: plugins reload event may never reach the privileged agent, leaving timers of removed plugins running

1 participant