Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 22 additions & 18 deletions .github/workflows/fuzz.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,11 @@ name: fuzz
# across ALL seven targets.
#
# This runs UNATTENDED on a cron schedule, so every third-party action is
# SHA-pinned and cargo-fuzz is version-pinned — a fuzz smoke must not itself
# become a supply-chain hole. Read-only token; the only `run:` interpolation is
# the matrix target name, passed through `env:` (never inlined into the shell)
# so there is no template-injection surface (THREAT-CICD-001).
# SHA-pinned, cargo-fuzz is installed with its bundled lockfile on a pinned
# Rust toolchain, and fuzzing runs on a pinned nightly — a fuzz smoke must not
# itself become a supply-chain hole. Read-only token; the only `run:`
# interpolation is the matrix target name, passed through `env:` (never inlined
# into the shell) so there is no template-injection surface (THREAT-CICD-001).

on:
pull_request:
Expand All @@ -28,6 +29,11 @@ on:
permissions:
contents: read

env:
CARGO_FUZZ_VERSION: "0.13.1"
CARGO_FUZZ_INSTALL_TOOLCHAIN: "1.89.0"
FUZZ_NIGHTLY: "nightly-2025-08-15"

concurrency:
group: fuzz-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
Expand All @@ -52,19 +58,18 @@ jobs:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # 1.89.0; cargo-fuzz locked install
with:
toolchain: ${{ env.CARGO_FUZZ_INSTALL_TOOLCHAIN }}
- uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # nightly; channel via toolchain: below
with:
toolchain: nightly
toolchain: ${{ env.FUZZ_NIGHTLY }}
- name: Install cargo-fuzz (version-pinned)
# NB: no `--locked` — cargo-fuzz 0.13.1's bundled Cargo.lock pins an old
# rustix (0.36.x) that no longer compiles on current nightly. The tool
# itself stays version-pinned; its build deps resolve to compatible
# versions.
run: cargo install cargo-fuzz --version 0.13.1
run: cargo "+${CARGO_FUZZ_INSTALL_TOOLCHAIN}" install cargo-fuzz --version "${CARGO_FUZZ_VERSION}" --locked
- name: Smoke
env:
TARGET: ${{ matrix.target }}
run: cargo +nightly fuzz run "$TARGET" -- -max_total_time=60 -rss_limit_mb=4096
run: cargo "+${FUZZ_NIGHTLY}" fuzz run "$TARGET" -- -max_total_time=60 -rss_limit_mb=4096

# Weekly full sweep over all seven targets at a larger time budget.
weekly:
Expand All @@ -90,16 +95,15 @@ jobs:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # 1.89.0; cargo-fuzz locked install
with:
toolchain: ${{ env.CARGO_FUZZ_INSTALL_TOOLCHAIN }}
- uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # nightly; channel via toolchain: below
with:
toolchain: nightly
toolchain: ${{ env.FUZZ_NIGHTLY }}
- name: Install cargo-fuzz (version-pinned)
# NB: no `--locked` — cargo-fuzz 0.13.1's bundled Cargo.lock pins an old
# rustix (0.36.x) that no longer compiles on current nightly. The tool
# itself stays version-pinned; its build deps resolve to compatible
# versions.
run: cargo install cargo-fuzz --version 0.13.1
run: cargo "+${CARGO_FUZZ_INSTALL_TOOLCHAIN}" install cargo-fuzz --version "${CARGO_FUZZ_VERSION}" --locked
- name: Fuzz
env:
TARGET: ${{ matrix.target }}
run: cargo +nightly fuzz run "$TARGET" -- -max_total_time=300 -rss_limit_mb=4096
run: cargo "+${FUZZ_NIGHTLY}" fuzz run "$TARGET" -- -max_total_time=300 -rss_limit_mb=4096
121 changes: 115 additions & 6 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@
# `*.sigstore.json` bundle on the Release (`gh attestation verify`; also the
# Scorecard signing probe -> 8, a backup if the .intoto.jsonl ever regresses).
# * gh-action-pypi-publish -> PEP 740 attestations on PyPI (Integrity API).
# * post-publish PyPI JSON hash check -> every served wheel/sdist digest
# matches the staged dist files.
# * crates.io / PyPI publish via Trusted Publishing (OIDC) — NO stored tokens.
#
# Fail-closed: `release-assets-draft` and both publishes `needs:` attest +
Expand Down Expand Up @@ -270,10 +272,9 @@ jobs:
# could inject code into the shipped wheel (zizmor cache-poisoning,
# HIGH); the speedup isn't worth that risk on the release path. The CI
# path (`python.yml`) keeps sccache on for the PR/main cadence.
# Prove the freshly-built wheel runs before it can be published. The
# linux/aarch64 wheel is cross-built under QEMU and can't execute on the x86
# host, so it's skipped (covered by python.yml's native arm leg + the
# macos-arm64 leg here).
# Prove each native wheel runs before it can be published. The
# linux/aarch64 wheel is cross-built here and smoke-tested on a native
# ubuntu-24.04-arm runner by `smoke-linux-aarch64-wheel` below.
- name: Set up Python to test the built wheel
if: ${{ !(matrix.platform.runner == 'ubuntu-latest' && matrix.platform.target == 'aarch64') }}
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
Expand All @@ -284,15 +285,78 @@ jobs:
shell: bash
run: |
set -euo pipefail
WHEEL="$(python - <<'PY'
from pathlib import Path
wheels = sorted(Path("ordvec-python/dist").glob("*.whl"))
if len(wheels) != 1:
raise SystemExit(f"expected exactly one wheel, found {wheels}")
print(wheels[0])
PY
)"
python -m pip install --require-hashes -r ordvec-python/requirements-dev.txt
python -m pip install ordvec-python/dist/*.whl
python -m pip install --no-index "$WHEEL"
python -m pytest ordvec-python/tests -q
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: wheels-${{ matrix.platform.runner }}-${{ matrix.platform.target }}
path: ordvec-python/dist/*.whl
if-no-files-found: error

smoke-linux-aarch64-wheel:
name: smoke linux/aarch64 wheel
needs: [guard, build-wheels]
if: needs.guard.outputs.ok == 'true'
runs-on: ubuntu-24.04-arm
steps:
- name: Harden the runner
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Set up Python to test the built wheel
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: "3.13"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Download the exact linux/aarch64 wheel
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: wheels-ubuntu-latest-aarch64
path: wheelhouse
- name: Install exact wheel and run tiny RankQuant/Bitmap smoke
shell: bash
run: |
set -euo pipefail
WHEEL="$(python - <<'PY'
from pathlib import Path
wheels = sorted(Path("wheelhouse").glob("*.whl"))
if len(wheels) != 1:
raise SystemExit(f"expected exactly one linux/aarch64 wheel, found {wheels}")
print(wheels[0])
PY
)"
python -m pip install --require-hashes -r ordvec-python/requirements-dev.txt
python -m pip install --no-index "$WHEEL"
python - <<'PY'
import numpy as np
from ordvec import Bitmap, RankQuant

docs8 = np.arange(32, dtype=np.float32).reshape(4, 8)
rq = RankQuant(8, 2)
rq.add(docs8)
scores, ids = rq.search_asymmetric(docs8[:1], 2)
assert scores.shape == (1, 2)
assert ids.shape == (1, 2)

docs64 = np.arange(256, dtype=np.float32).reshape(4, 64)
bm = Bitmap(64, 16)
bm.add(docs64)
scores, ids = bm.search(docs64[:1], 2)
assert scores.shape == (1, 2)
assert ids.shape == (1, 2)
PY

build-sdist:
name: build sdist + SBOM
needs: guard
Expand Down Expand Up @@ -438,7 +502,7 @@ jobs:

release-assets-draft:
name: stage all assets on the DRAFT Release (does NOT un-draft)
needs: [guard, notes, attest, provenance, require-ci-green]
needs: [guard, notes, attest, provenance, require-ci-green, smoke-linux-aarch64-wheel]
if: needs.guard.outputs.ok == 'true'
runs-on: ubuntu-latest
permissions:
Expand Down Expand Up @@ -604,6 +668,51 @@ jobs:
uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # v1.14.0
with:
packages-dir: dist
- name: Post-publish PyPI hashes match staged dist
env:
VERSION: ${{ needs.guard.outputs.version }}
run: |
set -euo pipefail
python3 - <<'PY'
import hashlib
import json
import os
import sys
import time
import urllib.request
from pathlib import Path

version = os.environ["VERSION"]
dist = Path("dist")
local = {
path.name: hashlib.sha256(path.read_bytes()).hexdigest()
for path in sorted(dist.iterdir())
if path.is_file() and (path.name.endswith(".whl") or path.name.endswith(".tar.gz"))
}
if not local:
raise SystemExit("no local wheel/sdist files found in dist")

url = f"https://pypi.org/pypi/ordvec/{version}/json"
last_error = None
for attempt in range(1, 25):
try:
with urllib.request.urlopen(url, timeout=15) as response:
payload = json.load(response)
remote = {
item["filename"]: item["digests"]["sha256"]
for item in payload.get("urls", [])
}
if remote == local:
print(f"OK: PyPI-served hashes match staged dist for ordvec {version}")
break
last_error = f"local={local!r} remote={remote!r}"
except Exception as exc: # noqa: BLE001 - diagnostic for CI logs.
last_error = repr(exc)
print(f"waiting for PyPI JSON/hash propagation ({attempt}/24): {last_error}", file=sys.stderr)
time.sleep(5)
else:
raise SystemExit(f"PyPI post-publish hash verification failed for {url}: {last_error}")
PY

publish-github-release:
name: un-draft the GitHub Release (only after BOTH registry publishes succeed)
Expand Down
25 changes: 22 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@ All notable changes to this project are documented here.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]
## Unreleased

## 0.3.0 - 2026-05-29

### Added

Expand All @@ -18,6 +20,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Added optional typed calibration profile references to the v1 manifest
schema, with path/hash/identity/compatibility verification but no statistical
computation.
- Added the repo-local, publish=false `ordvec-ffi` crate with the base C ABI
for loading persisted `RankQuant` and `Bitmap` indexes and running
synchronous search through opaque handles.
- Added the repo-local `ordvec-go` cgo wrapper over the base C ABI.

### Documentation

Expand All @@ -27,6 +33,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Documented sidecar manifest verification as a pre-load provenance check that
does not sign, manage keys, call networks, or decide trust policy.

### Fixed

- Hardened Python `add()` input boundaries so attempts to grow an index beyond
`MAX_VECTORS` raise `ValueError` before crossing into Rust core asserts.
- Corrected Python package dependency wording to the published metadata:
CPython 3.10+ with `numpy>=2.2`.

### Security

- Hardened the tag-triggered release workflow with exact Linux/aarch64 wheel
smoke coverage, post-publish PyPI hash verification, reproducible
release-required fuzz installation, and stricter local release-order
invariants for OIDC and publish steps.

## [0.2.0] - 2026-05-26

First public release on crates.io / PyPI — the crate was not published before
Expand Down Expand Up @@ -64,7 +84,7 @@ internal history.
range, matching the fail-loud contract of `pack_buckets` / `bucket_centre`.
Valid rank vectors (a permutation of `[0, d)`) are unaffected.
- **Python bindings (`ordvec-python`):** raised the floor to **Python 3.10** and
**numpy 2.0**; the abi3 wheel target moves to `abi3-py310`. Python 3.9 reached
**numpy 2.2**; the abi3 wheel target moves to `abi3-py310`. Python 3.9 reached
end-of-life (October 2025) and pytest's CVE-2025-71176 fix dropped 3.9 support.

### Deprecated
Expand Down Expand Up @@ -140,6 +160,5 @@ system dependencies** — no BLAS, no `ndarray`, no `faer`.
AVX-512 intrinsics this crate relies on were stabilized.
- Dual-licensed under **MIT OR Apache-2.0**.

[Unreleased]: https://github.com/Fieldnote-Echo/ordvec/compare/v0.2.0...HEAD
[0.2.0]: https://github.com/Fieldnote-Echo/ordvec/compare/v0.1.0...v0.2.0
[0.1.0]: https://github.com/Fieldnote-Echo/ordvec/releases/tag/v0.1.0
4 changes: 2 additions & 2 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

27 changes: 26 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "ordvec"
version = "0.2.0"
version = "0.3.0"
edition = "2021"
rust-version = "1.89" # AVX-512 intrinsics stabilized in 1.89.0; also clears the 1.87 floor from u64::is_multiple_of
description = "Training-free ordinal & sign quantization for vector retrieval"
Expand All @@ -14,12 +14,37 @@ categories = ["algorithms", "science", "compression"]
# Keep dev/internal files out of the published crate — they stay in the repo
# but aren't useful to crate consumers.
exclude = [
".agents/",
".claude/",
".codex/",
".github/",
".gitignore",
".playwright-mcp/",
"CLAUDE.md",
"CODE_OF_CONDUCT.md",
"CONTRIBUTING.md",
"DCO",
"GOVERNANCE.md",
"RELEASING.md",
"ROADMAP.md",
"SECURITY.md",
"cliff.toml",
"THREAT_MODEL.md",
"codecov.yml",
"deny.toml",
"docs/ALTERNATIVES_CONSIDERED.md",
"docs/FOLLOWUP_BODY_KERNEL_TIE_BREAK.md",
"docs/INDEX_PROVENANCE.md",
"docs/c-api.md",
"fuzz/",
"ordvec-ffi/",
"ordvec-go/",
"ordvec-manifest/",
"ordvec-python/",
"tests/__pycache__/",
"tests/release_publish_invariants.py",
"tests/release_publish_invariants.sh",
"tests/release_signed_release_invariants.sh",
]

# docs.rs build configuration: build with default features only, so the
Expand Down
Loading
Loading