Skip to content

fix(limit-conn): use parent resource key for consumer isolation - #13600

Open
DanielWu-star wants to merge 4 commits into
apache:masterfrom
DanielWu-star:fix/limit-conn-consumer-isolation
Open

fix(limit-conn): use parent resource key for consumer isolation#13600
DanielWu-star wants to merge 4 commits into
apache:masterfrom
DanielWu-star:fix/limit-conn-consumer-isolation

Conversation

@DanielWu-star

Copy link
Copy Markdown

Previously, when limit-conn was configured at the consumer level, connection limits were applied per-route instead of globally per consumer. This occurred because the key generation concatenated ctx.conf_type and ctx.conf_version, which included route-specific information.

This commit fixes the issue by using conf._meta.parent.resource_key for key generation, consistent with the same fix already applied to limit-req (#13019) and limit-count plugins. Now consumer-level connection limits are properly shared across all routes accessed by the same consumer.

Changes:

  • Added gen_limit_key() function using parent.resource_key approach
  • Replaced key = key .. ctx.conf_type .. ctx.conf_version with gen_limit_key(conf, ctx, key) in run_limit_conn()
  • Updated test cases to match new key format
  • Added test cases (TEST 35-42) verifying consumer isolation across multiple routes

Description

Which issue(s) this PR fixes:

Fixes #13584

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

Previously, when limit-conn was configured at the consumer level,
connection limits were applied per-route instead of globally per
consumer. This occurred because the key generation concatenated
ctx.conf_type and ctx.conf_version, which included route-specific
information.

This commit fixes the issue by using conf._meta.parent.resource_key
for key generation, consistent with the same fix already applied to
limit-req (apache#13019) and limit-count plugins. Now consumer-level
connection limits are properly shared across all routes accessed by
the same consumer.

Fixes apache#13584

Changes:
- Added gen_limit_key() function using parent.resource_key approach
- Replaced key = key .. ctx.conf_type .. ctx.conf_version with
  gen_limit_key(conf, ctx, key) in run_limit_conn()
- Updated test cases to match new key format
- Added test cases (TEST 35-42) verifying consumer isolation across
  multiple routes
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. bug Something isn't working labels Jun 23, 2026
AlinsRan added 2 commits July 29, 2026 16:55
The expected keys were written as `routes/1`, but `resource_key` is the full
etcd key, so the assertions in limit-conn.t never matched; limit-conn-redis.t
still carried the pre-fix format. `error_log_like` is not a test-nginx section
either, so those two blocks were never evaluated.

The consumer-level blocks are moved to t/plugin/limit-conn-shared-counter.t and
rewritten to fire the requests in parallel: limit-conn counts concurrent
connections and releases the counter in the log phase, so the pipelined requests
copied from the limit-req test could never hit the limit.
@AlinsRan

Copy link
Copy Markdown
Contributor

I pushed two commits to this branch (merged master to clear the conflict, then fixed the tests). The plugin change itself is unchanged apart from the conflict resolution.

Conflict: master added _vid handling to the key right where this PR replaced the key construction, so that block would have been dropped. It now lives inside gen_limit_key(), matching limit-count.

Tests — none of the assertions this PR touched were actually checking the fix:

  • resource_key is the full etcd key, so the expected value is /apisix/routes/1:<conf_version>:<key>, not routes/1:. TEST 17 and TEST 19 were failing.
  • t/plugin/limit-conn-redis.t:591,683 still carried the pre-fix format and were failing too.
  • The two consumer blocks used --- error_log_like, which is not a test-nginx section — those assertions were never evaluated. Changed to --- error_log eval.
  • TEST 35-42 used --- pipelined_requests. That works for limit-req (rate over a time window) but not for limit-conn: pipelined requests are handled one after another and _M.decrease() runs in the log phase of each, so the counter is back to zero before the next request starts. TEST 39 returned [200, 200, 404] and TEST 41 [200, 200, 200, 200].

Those blocks are replaced by t/plugin/limit-conn-shared-counter.t, which fires the requests in parallel via ngx.thread.spawn against an upstream that sleeps, asserts the shared counter across two routes (200,503) and the isolation between two consumers (200,200), and asserts the key is /apisix/consumers/jack:<conf_version>:jack. Reverting the plugin change turns the first case into 200,200, so the test discriminates.

Locally t/plugin/limit-conn.t, t/plugin/limit-conn-shared-counter.t, t/stream-plugin/limit-conn.t and t/plugin/workflow3.t pass.

One thing worth adding to the PR description: the cross-instance half of #13584. ctx.consumer_ver is the local config-sync counter of core.config.new("/consumers"), not an etcd revision, so the old key differed per worker as well as per pod — with the redis policy the limit did not hold across replicas either.

Every other failure path in run_limit_conn() falls back to allow_degradation;
the new nil-key branch was the only one that always returned 500.

Also drop the now misleading `ver:` log line, since ctx.conf_version no longer
takes part in the key.

Tests:
- limit-conn-shared-counter.t keys both consumers on remote_addr, so they
  resolve to the same value and only the parent resource_key keeps them apart.
  TEST 5 previously passed with or without the fix; now it fails without it.
- The file deletes its routes and consumers again. /apisix/routes/2 (uri
  /limit_conn2) leaked into t/plugin/workflow3.t, which expects that uri to
  404 and got 401 from the leftover key-auth route.
- workflow3.t asserts both _vid suffixes: the two workflow rules carry
  identical limit-conn confs on one route, so _vid is all that separates them.
- stream-plugin/limit-conn.t asserts the stream_route key shape, pinning the
  fact that stream routes go through the same router.lua filter.
@AlinsRan

Copy link
Copy Markdown
Contributor

Second round, after a review pass. Pushed fd672b0b2.

One of these would have broken CI. t/plugin/limit-conn-shared-counter.t left /apisix/routes/2 (uri /limit_conn2, key-auth) in etcd. t/plugin/workflow3.t runs later in the same t/plugin/[l-z]*.t job and expects that uri to 404 — it got 401 from the leftover route. The file now deletes its routes and consumers again.

TEST 5 was not testing what it claimed. Both consumers were keyed on consumer_name, so jack and bob produced different key values regardless of the fix — the block passed with the plugin change reverted. Both are now keyed on remote_addr, so they resolve to the same value on the same route and only parent.resource_key separates them. Verified: reverting the plugin change now fails TEST 4 and TEST 5.

allow_degradation. Every other failure path in run_limit_conn() falls back to it; the new nil-key branch was the only one that always returned 500. Now consistent with the rest of the function.

Two invariants pinned.

  • t/plugin/workflow3.t TEST 4 asserts both _vid suffixes. The two workflow rules carry identical limit-conn confs on one route, so after this change _vid is the only thing separating their counters — worth an explicit assertion.
  • t/stream-plugin/limit-conn.t TEST 2 asserts /apisix/stream_routes/1:.... Stream routes get _meta.parent because apisix/router.lua's filter is passed to both router_http.init_worker() and router_stream.stream_init_worker(); that is easy to break without noticing.

Also dropped the core.log.info("ver: ", ctx.conf_version) line at the top of _M.increase()ctx.conf_version no longer takes part in the key, so it just misleads whoever is debugging.

One more thing for the description, beyond the consumer case: service-level limit-conn changes scope. The old key carried route.modifiedIndex .. "&" .. service.modifiedIndex (apisix/init.lua:903), i.e. one counter per route; it is now one counter per service shared by all its routes. That matches limit-count/limit-req and is the intended semantics, but it tightens the effective limit for anyone relying on the old behaviour and is worth calling out in the release notes.

Out of scope, but found while checking this — two pre-existing issues that also affect limit-req and limit-count, worth separate issues:

  1. conf_version() hashes the whole conf table, including _from_consumer, which merge_consumer_route() (apisix/plugin.lua:830-838) sets lazily and only when the route has no plugin of that name. Whether a worker caches conf._version before or after that flag is set depends on which route it served first, so the plugin conf version can still differ across workers for the same consumer.
  2. _M.decrease() uses break rather than goto CONTINUE when lim:leaving() fails, so one failing rule skips the release of every remaining rule.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: Is it expected that per-consumer limit-conn is isolated per-route and per-pod?

2 participants