Load testing - AuthZ Framework Model

[mariajgrimaldi]

Description

We need to run load tests on the new authorization model to verify it can handle a reasonable level of traffic and identify potential bottlenecks.

Details

• Use the latest installation. Main or Teak works.
• Document system resources of the target install.
• Run functional tests to catch behavior faults.
• Run load tests across user counts and operation mixes.
• Capture early performance notes for the library and our implementation.

Definition of Done

• We have a tested performance threshold that is reasonable for the MVP.
• We have results for multiple user levels and for different operations.
• We have a list of bottlenecks and performance issues.
• Critical issues are fixed.
• Non-critical issues have a work plan.
• Test results and environment details are documented.

[mariajgrimaldi]

Authorization Framework: Load Testing Summary (Teak Release)

Environment: Sandbox on Teak release
Tests executed with: K6
K6 setup: https://github.com/eduNEXT/openedx-k6-loadtesting/tree/MJG/rbac-testcases/authz

---

1. Context

We ran several load tests to understand how the authorization system behaves when the number of concurrent users increases. These tests were executed from my local machine in Spain, and the sandbox is hosted in Virginia (USA), so there is some network distance. We cannot fully measure how much of the latency comes from that distance, but we can observe clear behaviour patterns as concurrency grows.

The goal of these tests was not to find a performance bottleneck or to diagnose a specific issue. The goal was simply to document how the system behaves under different loads and confirm that the authorization checks continue to work correctly.

---

2. Infrastructure Setup

Below is the pod configuration used during the tests:

POD_AUTOSCALING_EXTRA_SERVICES:
  cms:
    avg_cpu: 75
    cpu_limit: 1
    cpu_request: 0.4
    enable_hpa: true
    max_replicas: 1
    memory_limit: 2500Mi
    memory_request: 1000Mi
    min_replicas: 1
  cms-worker:
    avg_cpu: 75
    cpu_limit: 2
    cpu_request: 1
    enable_hpa: true
    max_replicas: 1
    memory_limit: 2500Mi
    memory_request: 1000Mi
    min_replicas: 1
  lms:
    avg_cpu: 100
    cpu_limit: 2
    cpu_request: 1.5
    enable_hpa: true
    max_replicas: 1
    memory_limit: 2500Mi
    memory_request: 1500Mi
    min_replicas: 1
  lms-worker:
    avg_cpu: 100
    cpu_limit: 2
    cpu_request: 1.25
    enable_hpa: true
    max_replicas: 1
    memory_limit: 2500Mi
    memory_request: 1500Mi
    min_replicas: 1

OPENEDX_LMS_UWSGI_WORKERS: 4

Configured through: https://github.com/eduNEXT/tutor-contrib-pod-autoscaling

Notes:

• Only one LMS pod is running, using up to 2 CPUs.
• The LMS uses 4 uWSGI workers, each handling one request at a time.

---

3. Test Profiles

We executed four profiles:

Profile	Max VUs	Permissions per request
Small	10	10
Medium	30	15
Medium Plus	50	10
Large	100	5

Each profile performs:

• Token retrieval
• Permission validation across 6 scopes
• Checks on response structure

---

4. Functional Results

Across all tests:

• No permission validation errors.
• Authorization logic remained correct.
• A few structural check failures appeared in the Medium and Medium Plus profiles, but nothing indicating incorrect permission decisions.
• No 502s, crashes, or service interruptions occurred.

---

5. Performance Results

Small Profile (10 VUs)

• Avg: ~1.17 s
• p95: ~1.73 s
• p99: ~1.84 s
• Max: ~1.93 s

Grafana Dashboard

Medium Profile (30 VUs)

• Avg: ~6.06 s
• p95: ~14.15 s
• p99: ~15.50 s
• Max: ~16.36 s

Grafana Dashboard

Medium Plus Profile (50 VUs)

• Avg: ~6.37 s
• p95: ~11.70 s
• p99: ~11.77 s
• Max: ~12.75 s

Grafana Dashboard

Large Profile (100 VUs)

• Avg: ~5.86 s
• p95: ~11.82 s
• p99: ~11.91 s
• Max: ~12.02 s

Grafana Dashboard

---

6. Summary of Behaviour

• When concurrency increases, latency also increases.
• The biggest jump is between 10 → 30 VUs.
• From 30 to 100 VUs, latency stays in the 11–12 s p95 range.
• The system continues responding under all loads.

---

7. Possible Reasons for Latency Increase

• Infrastructure limits, including the 4 uWSGI workers, or the single pod per service (limited autoscaling).
• Casbin lock usage in the enforcer, which may affect read operations.
• Backend behaviour, such as synchronous operations - although no enforcer reloads were made.
• Queueing effects when concurrency exceeds what the pod can process.
• Local execution from Spain, adding some network latency (but this does not explain the jump between 10 and 30 VUs).

---

8. Final Notes

• This document summarizes how the system behaved during these tests.
• This is not a root-cause investigation.
• We will run the same tests again once the sandbox is on Ulmo to compare results.

[bmtcril]

I'm not familiar with k6, so apologies if I'm not reading this correctly. It looks to me like each user is logging in as admin and then making a single request to get permissions? Then there is a 1 second sleep before another user takes its place?

If so, the time here is split between the two requests. I'd be curious to see the same test run with just the login for comparison. I suspect that most of the time is spent in the request queue once you get over whatever the threshold is between 10-30 that saturates the req's per second that 4 processes can handle.

Another possibility is that it's using the sleep variable between requests, so there's a full second between the auth and the permission check?

[mariajgrimaldi]

@bmtcril: thanks Ty! I'm sorry I didn't explain how the test worked, my mistake. Here is what the test does:

Test Algorithm (test_authz.js)

https://github.com/eduNEXT/openedx-k6-loadtesting/blob/MJG/rbac-testcases/authz/test_authz.js

Setup Phase (runs once at the beginning):

1. Records the start time in UTC+1 (CET) for Grafana correlation
2. Creates test users (one per VU) - k6 spawns 100 goroutines, each running 1 VU
3. Gets an admin authentication token for role assignment operations
4. Assigns users to library roles (library_admin, library_author, library_user) across multiple library scopes
5. Returns the user list and start time - shared with all VUs for the test

Main Test Phase (all VUs run simultaneously in parallel):

Each VU executes independently in its own goroutine:

1. Picks a random user from the setup list
2. Gets an OAuth2 access token for that user
3. Creates a permission check request with multiple permissions to validate
4. Sends the request to /api/authz/v1/permissions/validate/me ~100 concurrent requests hit the API simultaneously
5. Retries on failure up to 3 times with exponential backoff (1s, 2s, 4s delays)
6. Validates the response contains permission results array
7. Records metrics (response time, success/failure)
8. Sleeps 1 second, then loops back to step 1

The test gradually increases concurrent users following the profile configuration (e.g., 10 → 25 → 50 → 75 → 100 users over time).

Teardown Phase (runs once at the end):

1. Records the end time in UTC+1 (CET)
2. Generates a Grafana dashboard URL with the test timeframe (±5 min buffer)
3. Exports a summary with metrics, Grafana URL, and profile configuration to JSON

[mariajgrimaldi]

@bmtcril: I ran with your suggestion of testing with only authentication (getting the access token) and for 100 concurrent users we got the following results:

Overhead = With Permission - Without Permission

p(95): 18,233ms - 818ms = 17,415ms (17.4 seconds)
p(99): 18,867ms - 861ms = 18,006ms (18.0 seconds)
Average: 8,707ms - 694ms = 8,013ms (8.0 seconds)
Median: 8,310ms - 712ms = 7,598ms (7.6 seconds)

I'm still battling about how much to trust these results. I've discussed internally with our infrastructure team, which usually conducts these tests on our installations, and they strongly recommend using a remote client instead of a local one for better stability. I'll try to get my hands on an EC2 instance to try this :)

[mariajgrimaldi]

I ran the large profile with 100 VUs to check the metrics in the Scout dashboard:

[bmtcril]

I think what we've found is that there are excessive queries to MySQL happening because get_user_by_username_or_email is being called once per policy and is taking up something like 75% of call time. We've discussed caching to handle this, which would be a big improvement, but even 31 cache hits is a fair amount of time. I'm not sure if we have any mechanism to force this evaluation further up the stack, or to limit the number of policies being checked before making the call, but we should look into that as a longer term fix. I think after Ulmo is fine if we can get a reasonable caching strategy.

[mariajgrimaldi]

Here's the discussion we had about the performance issues identified: https://openedx.slack.com/archives/C07PRCN6G0N/p1763983154101359?thread_ts=1763672523.867889&cid=C07PRCN6G0N

And after these changes #146: https://openedx.slack.com/archives/C07PRCN6G0N/p1764181886960179?thread_ts=1763672523.867889&cid=C07PRCN6G0N

I'll close this as done. Thank you all for the help!