WIP: HTTP parity regression

.gitattributes

@@ -0,0 +1,8 @@
+* text=auto eol=lf
+*.java text eol=lf
+*.xml text eol=lf
+*.properties text eol=lf
+*.md text eol=lf
+*.yaml text eol=lf
+*.yml text eol=lf
+*.sh text eol=lf

.github/workflows/ci-jmeter-compatibility.yaml

@@ -43,13 +43,23 @@ jobs:
         uses: actions/cache@v5
         with:
           path: ${{ github.workspace }}/.jmeter
-          key: jmeter-${{ runner.os }}-${{ hashFiles('pom.xml', 'src/test/resources/jmeter/**') }}
+          key: jmeter-${{ runner.os }}-${{ hashFiles('pom.xml', 'src/test/resources/jmeter/**', 'src/test/resources/jmeter-regression/**') }}
           restore-keys: |
             jmeter-${{ runner.os }}-
 
       - name: Build project and jmeter-test bundle
         run: xvfb-run -a mvn -U --batch-mode clean install
 
+      - name: Run JMeter HTTP parity regression (all tiers)
+        run: >-
+          xvfb-run -a mvn -Pjmeter-regression
+          -Dcheckstyle.skip=true
+          -DskipTests
+          -Djmeter.regression.tier=all
+          -Djmeter.regression.tolerateExternalServiceDrift=true
+          --batch-mode
+          verify
+
       - name: Run compatibility matrix on Java 17
         working-directory: target/jmeter-test
         env:

.gitignore

@@ -211,6 +211,7 @@ Temporary Items
 
 ### Maven ###
 target/
+cp.txt
 pom.xml.tag
 pom.xml.releaseBackup
 pom.xml.versionsBackup

README.md

@@ -427,6 +427,15 @@ Uses **Maven 3** from the repository root:
 
 Artifacts land under **`target/`**. Compilation uses the **`jmeter.version`** declared in **`pom.xml`**; at runtime install the packaged JAR against the JMeter build you intend to run and validate with a short smoke plan.
 
+### HTTP parity regression (JMeter 5.6.3 test plans)
+
+To compare **HttpClient4** with the migrated **BlazeMeter HTTP** sampler against Apache’s official `bin/testfiles` JMX plans, see **[docs/jmeter-regression.md](docs/jmeter-regression.md)**.
+
+```bash
+mvn -Dcheckstyle.skip=true package
+mvn -Pjmeter-regression verify
+```
+
 
 <a id="readme-license"></a>
 ## License

checkstyle.xml

@@ -10,6 +10,11 @@
   <module name="FileTabCharacter">
     <property name="eachLine" value="true"/>
   </module>
+  <module name="LineLength">
+    <property name="max" value="100"/>
+    <property name="ignorePattern"
+      value="^package.*|^import.*|a href|href|http://|https://|ftp://"/>
+  </module>
 
   <module name="TreeWalker">
     <module name="SuppressionCommentFilter"/>
@@ -104,12 +109,6 @@
     <module name="StaticVariableName"/>
     <module name="TypeName"/>
 
-    <module name="LineLength">
-      <property name="max" value="100"/>
-      <property name="ignorePattern"
-        value="^package.*|^import.*|a href|href|http://|https://|ftp://"/>
-    </module>
-
     <module name="EmptyForInitializerPad"/>
     <module name="EmptyForIteratorPad"/>
     <module name="EmptyLineSeparator">

docs/jmeter-regression.md

@@ -0,0 +1,230 @@
+# JMeter HTTP regression parity tests
+
+
+
+This suite compares Apache JMeter **HttpClient4** (reference) with the migrated **BlazeMeter HTTP** sampler (`HTTP2Sampler`) using official JMeter `bin/testfiles` plans from release **5.6.3**.
+
+
+
+## What it does
+
+
+
+1. Runs the stock JMX with `-Jjmeter.httpsampler=HttpClient4` (no plugin).
+
+2. Migrates HTTP Request samplers headlessly via `JmxBlazeMeterHttpMigrator`.
+
+3. Runs the migrated plan with the plugin installed under `lib/ext/`.
+
+4. Compares sample trees semantically (`HttpsampleResultComparator`): labels, success, response codes/messages, bodies, and normalized response headers. Timings and volatile headers are ignored.
+
+
+
+## Regression tiers
+
+
+
+| Tier | Maven | Plans | Notes |
+
+|------|-------|-------|-------|
+
+| **1** | `-Pjmeter-regression` | `TEST_HTTP`, `ResponseDecompression`, `TestHeaderManager`, `TestCookieManager` | Default CI; mirror + managers |
+
+| **4 (F4)** | `-Pjmeter-regression-f4` | `HTMLParserTestFile_2`, `TEST_HTTPS`, `Http4ImplDigestAuth`, `Http4ImplPreemptiveBasicAuth` | HTTPS + embedded HTML + auth |
+
+| **Flaky** | `-Pjmeter-regression-flaky` | `TestKeepAlive`, `TestRedirectionPolicies` | External services; opt-in only |
+
+| **5 (F5)** | `mvn test -Dtest=com.blazemeter.jmeter.http2.parity.*` | JUnit ports of Apache HTTP module tests | HttpClient4 vs plugin in-process; HTTP/1.1 |
+
+`BUG_62847` / `Bug54685` are vendored under `5.6.3/` but exercise JMeter controllers/Java samplers only — not HTTP plugin parity.
+
+`SlowCharsFeature` (optional): run with `-Djmeter.regression.tests=SlowCharsFeature -Djmeter.regression.enableSlowChars=true` once Jetty supports HttpClient4-style **CPS** throttling (`httpclient.socket.*.cps`). See [SlowCharsFeature](#slowcharsfeature) below.
+
+### HEAD method
+
+HttpClient4 supports `HEAD` for redirects and all standard samplers. The BlazeMeter client maps Jetty `Request.method("HEAD")` the same way; it was previously rejected by `SUPPORTED_METHODS` in `HTTP2JettyClient` (not a Jetty limitation). `HEAD` must not send a request body; response bodies may still be buffered internally but JMeter parity compares status, headers, and redirect semantics.
+
+### SlowCharsFeature
+
+Apache batch plan `SlowCharsFeature.jmx`:
+
+1. A setup thread sets `httpclient.socket.http.cps` and `httpclient.socket.https.cps` to **1500** (bytes per second).
+2. The main sampler requests `https://jmeter.apache.org/...` with `Range: bytes=0-7000` (partial content).
+3. Assertions expect **HTTP 206**, ~7001 bytes, and **elapsed time > 5s** (download throttled to ~1.5 KB/s).
+
+HttpClient4 implements CPS by wrapping socket streams with a rate limiter (`org.apache.http.impl.conn.SocketFactory` / connection socket config). **Jetty has no equivalent property**: you would need a custom `ClientConnector` / `EndPoint` wrapper that throttles `fill()`/`flush()` per connection, and decide how CPS applies across HTTP/2 multiplexing and HTTP/3 QUIC streams (HC4’s model is per-TCP-socket). Until that exists, the plan stays behind `-Djmeter.regression.enableSlowChars=true`.
+
+`TestKeepAlive` is compared on **http1-only** only (HttpClient4 never speaks HTTP/2; keep-alive / `Connection: close` are HTTP/1.1 semantics).
+
+
+
+Use `-Djmeter.regression.tier=4|flaky|all` instead of a profile, or override with `-Djmeter.regression.tests=PlanA,PlanB`.
+
+
+
+### SSL keystore / truststore ([PR #112](https://github.com/Blazemeter/jmeter-http-plugin/pull/112))
+
+
+
+F4 includes `TEST_HTTPS` (public TLS hosts) and auth JMX that may use custom stores. **Do not reimplement** keystore path resolution or trust-manager parity in this harness — that work belongs in [PR #112](https://github.com/Blazemeter/jmeter-http-plugin/pull/112) (`SslStorePathResolver`, `JMeterJettySslContextFactory` trust override).
+
+
+
+Until #112 is merged, F4 parity gaps involving **relative keystore paths**, **PKCS#11**, or **trust-all vs PKIX** should be fixed in that PR branch, not duplicated here.
+
+
+
+## Prerequisites
+
+
+
+- **Java 17+**
+
+- **Maven 3.6+**
+
+- Network access (first run downloads JMeter 5.6.3; tier 1/4 plans may call external hosts).
+
+
+
+## Build the plugin JAR
+
+
+
+```bash
+
+mvn -Dcheckstyle.skip=true package
+
+```
+
+
+
+## Run regression tests
+
+
+
+```bash
+
+mvn -Pjmeter-regression verify
+
+```
+
+
+
+### Options
+
+
+
+| Property | Default | Description |
+
+|----------|---------|-------------|
+
+| `jmeter.regression` | `false` (`true` with `-Pjmeter-regression*`) | Must be `true` to execute regression ITs |
+
+| `jmeter.regression.tests` | tier 1 list (see `pom.xml`) | Comma-separated JMX base names (no `.jmx`) |
+
+| `jmeter.regression.tier` | _(unset)_ | `1`, `4`, `flaky`, or `all` when `jmeter.regression.tests` is unset |
+
+| `jmeter.regression.version` | `5.6.3` | JMeter distribution version to download |
+
+| `jmeter.home` | _(auto)_ | Use an existing JMeter install instead of downloading |
+
+| `it.http3` | `false` | Also run each plan with HTTP/3 profile |
+
+| `jmeter.regression.protocol` | _(all profiles)_ | Filter: `http1-only`, `http2`, or `http3` (with `it.http3`) |
+
+| `jmeter.regression.timeoutMinutes` | `10` | Per-run timeout |
+
+| `jmeter.regression.tolerateExternalServiceDrift` | `false` (auto for F4 auth + flaky plans) | Skip strict compare when ref/plugin saw 5xx vs 2xx on the same sample (sequential runs vs flaky hosts) |
+
+
+
+### Examples
+
+
+
+Tier 1 only, HTTP/1.1 profile:
+
+
+
+```bash
+
+mvn -Pjmeter-regression -Djmeter.regression.protocol=http1-only verify
+
+```
+
+
+
+F4 — start with local HTML parser (no network):
+
+
+
+```bash
+
+mvn -Pjmeter-regression-f4 -Djmeter.regression.tests=HTMLParserTestFile_2 verify
+
+```
+
+
+
+Full F4 suite:
+
+
+
+```bash
+
+mvn -Pjmeter-regression-f4 verify
+
+```
+
+
+
+Flaky / optional batch plans:
+
+
+
+```bash
+
+mvn -Pjmeter-regression-flaky verify
+
+```
+
+
+
+### F5 — JUnit parity (Apache `src/protocol/http` tests)
+
+Runs in the normal unit-test phase (`mvn test`), not Failsafe. Each test executes the same sampler configuration through **HttpClient4** and **HTTP2JettyClient** (HTTP/1.1 forced).
+
+| Class | Apache source |
+|-------|----------------|
+| `HttpMirrorParityTest` | `TestHTTPSamplersAgainstHttpMirrorServer` (GET/PUT, POST urlencoded/multipart/raw, file upload) |
+| `HttpMirrorItemisedParityTest` | same source — `itemised_testPostRequest_UrlEncoded` (0–7), `itemised_testGetRequest_Parameters` (0–5) |
+| `HttpMirrorMultipartParityTest` | same source — `testPostRequest_FormMultipart` (0–6) |
+| `HttpMirrorRawBodyParityTest` | same source — `testPostRequest_BodyFromParameterValues` (0–9) |
+| `HttpMirrorFileUploadParityTest` | same source — `testPostRequest_FileUpload` (0–2) |
+| `HttpRedirectsParityTest` | `TestRedirects` (301–308, no follow; incl. HEAD/PUT/DELETE) |
+| `HttpRedirectsFollowParityTest` | redirect follow (`followRedirects` + `autoRedirects`) |
+| `HttpCookieManagerParityTest` | `TestHC4CookieManager` (set/echo cookies) |
+| `HttpCacheManagerParityTest` | `TestCacheManagerHC4` (embedded cache hit) |
+| `HttpDisableArgumentsParityTest` | skippable args (5.6.3); `enabled` checkbox → JMeter 5.7+ |
+
+```bash
+mvn test -Dcheckstyle.skip=true -Dtest=com.blazemeter.jmeter.http2.parity.*
+```
+
+
+
+## CI
+
+
+
+`ci-jmeter-compatibility.yaml` runs **all regression tiers** (`tier=all`: tier 1 + F4 + flaky) via `-Pjmeter-regression` after `mvn clean install` (which also runs F5 parity via Surefire). External-service drift tolerance is enabled in CI for auth/keep-alive plans.
+
+
+
+## Resources
+
+
+
+Vendored under `src/test/resources/jmeter-regression/5.6.3/` from [apache/jmeter rel/v5.6.3 bin/testfiles](https://github.com/apache/jmeter/tree/rel/v5.6.3/bin/testfiles).
+
+