Conformance test redesign

[sudo-bmitch]

This is a redesign of the conformance tests with the following goals:

• Run as a command instead of a go test. This will allow unit testing and avoid some of the Go testing wrappers.
• Make the API testing more granular.
• Design the tests to run against different types of input data, including the various updates to image-spec.
• Minimize dependencies to avoid upstream breaking changes.
• Where possible, note unsupported APIs and fallback to alternatives.

This is still in draft for now, with this open PR for better visibility. Various remaining tasks include:

• Sparse index tests.
• Edge cases on blob pushes (out of order, chunked push with data in the final put).
• Pushing content with the wrong digest or an invalid digest algorithm.
• Verify all allowed HTTP status codes are appropriately handled.
• Cache/reuse auth tokens between HTTP requests.
• Rename conformance2 to conformance and delete the old conformance tests.
• Update the GitHub actions to test against multiple registries, and consider if/when failing 3rd party registries should block a merge to the distribution-spec.
• Generate an image in GitHub actions to be used by a conformance action.
• Update the conformance action.
• Update the oci-conformance repo to handle the new input (results will likely be summarized with two columns, API and Data, with green or gray status to show passing tests or unsupported). (Implemented in Support for conformance results.yaml input oci-conformance#130)

Other wish list items:

• Add some unit tests.
• Add ability to run a single test.
• Add separate tests for foreign layers (URL defined) from non-distributable layers (specific media type).

Merging this will unblock #543. Closes #548. Closes #501. Closes #416.

[tianon]

These tests appear to be conflating two concepts 😇

For lack of an officially spec-defined term, blobs whose descriptors include the urls key are colloquially known as "foreign layers", and technically any blob could be foreign (with zero change to the mediaType), although most registries probably validate config differently from layers in that respect (I know Docker Hub certainly does 😇).

Non-distributable layers (those with mediaType of application/vnd.oci.image.layer.nondistributable.v1.tar*), on the other hand, may or may not be explicitly "foreign" (ie, they may or may not have urls set).

• https://github.com/opencontainers/image-spec/blob/v1.1.1/descriptor.md#:~:text=This%20OPTIONAL%20property%20specifies%20a%20list%20of%20URIs%20from%20which%20this%20object%20MAY%20be%20downloaded.

  This OPTIONAL property specifies a list of URIs from which this object MAY be downloaded.

• https://github.com/opencontainers/image-spec/blob/v1.1.1/media-types.md#:~:text=The%20following%20media%20types%20identify%20a%20%22Layer%22%20with%20distribution%20restrictions

• https://github.com/opencontainers/image-spec/blob/v1.1.1/layer.md#non-distributable-layers

  Descriptors referencing non-distributable layers MAY include urls for downloading these layers directly; however, the presence of the urls field SHOULD NOT be used to determine whether or not a layer is non-distributable.

(For historical curiosity, this evolved/matured/split in two parts from the application/vnd.docker.image.rootfs.foreign.diff.tar.gzip media type in the previous "Docker Image Manifest Version 2, Schema 2" spec: https://github.com/distribution/distribution/blob/10e4312e62ab3b0c53bc0e31ade82fbe38af1b43/docs/content/spec/manifest-v2-2.md#:~:text=application/vnd.docker.image.rootfs.foreign.diff.tar.gzip%3A%20%22Layer%22%2C%20as%20a%20gzipped%20tar%20that%20should%20never%20be%20pushed -- this is where I've cribbed the term "foreign layer" from, because "blob/descriptor with urls set" is a mouthful.)

[sudo-bmitch]

Are you suggesting that we test non-distributable layers without a URL defined, and the blob is still not pushed. Or are you suggesting that we define a foreign layer test, where the URL is defined, but the media type is not a non-distributable media type?

[tianon]

Yeah, as I clarified on the call yesterday (and writing here for posterity), I think I'm suggesting we should probably test all the permutations (urls and not urls, pushed blob and not pushed blob, and regular and nondistributable media types for both variances).

[sudo-bmitch]

@opencontainers/distribution-spec-maintainers while looking at the options for reusing auth tokens between requests, I'm seeing the following options:

1. Don't reuse tokens. This is the current design, matches the former conformance tests, is the most likely to work, but can be very slow and does not match how clients access registries today.
2. Generate the scope on the client side and reuse the token when the scope is unchanged. I think this matches some of the original docker implementations, but I'm not a fan of this client side logic since that goes against the upstream RFCs.
3. Cache the token response and always try to reuse it, but discard it and regenerate on any auth failure. I worry this could lead to bouncing tokens when things go from get to put to delete requests. And if a registry returns a 403 instead of a 401 when the old token is reused, we either fail or need to run 3 requests, one with a failing token to get a 403, one with no token to get the 401 + auth header, and a third with the valid token.
4. Cache the token for a "transaction" where the transaction may be multiple get requests, or a full push workflow for the layers and manifest. I think this most closely aligns with clients. However with a head followed by a push it can result in scope changes between the requests, and it assumes all clients have the same "transaction" concept.
5. Cache the token per repository + http method. This would allow the most reuse of tokens between tests, but I worry it doesn't match existing clients, and gets back into client side logic that doesn't match the upstream RFCs.

Is there a preference for which option we should use for the conformance test (at least pending the outcome of wg-auth)?

[jcarter3]

I like Option 5 the most, but I'm not sure if it's completely safe.

Given that these tests are for the dist spec (and there is no auth spec), I'm not terribly concerned about trying to match client behavior around auth, and so if Option 5 let's us reuse tokens the most and do these test faster, I'm all for it.

That said, since there is no auth spec... I know this approach works for Docker Hub and several other registries, but it's not guaranteed that every registry generates tokens the same way off of repository and http method, so it could introduce unrelated errors. Because of that, Option 4 is probably the safest from a compatibility standpoint, and should at least provide marginal re-use.