t7529: prevent a lost resume wakeup - #21
Open
ttaylorr-oai wants to merge 126 commits into
Open
Conversation
GitHub exposes manual dispatch only for workflows present on the default branch. The controller itself must remain on the orphan meta branch so master stays identical to upstream. Add the fixed caller as a dedicated active topic. It delegates to the meta-pinned reusable workflow without putting controller code on codex.
The reusable controller reads its publication key from the codex-publish environment. The default-branch trampoline therefore needs only read access to Actions and repository contents; it no longer forwards repository secrets. Keep this exact workflow in the automation topic so the Actions page can dispatch the controller pinned to meta.
Codex rebuilds already call a trusted workflow on meta, but their dispatch-only trampoline cannot run pull request or merge-queue checks. Topics could therefore reach codex without verifying their review, branch ownership, or published base. Run the pinned admission workflow for pull requests targeting codex and for merge-group checks. Keep rebuild preparation limited to explicit workflow dispatch, and grant the admission job only read access.
The default-branch trampoline only listened for pull requests against codex and ran one admission job for every event. A preview pull request would therefore have no required check, and merge groups could not distinguish the production and preview lanes. Listen for both generated outputs, keep the existing production job and check context unchanged, and add a target-specific preview job that calls the trusted meta admission workflow with read-only permissions.
The release workflow cross-compiles Linux arm64 on an x64 runner and skips the smoke test for arm64 POSIX bundles. That prevents the workflow from executing the Linux artifact it just produced. Run Linux arm64 on GitHub's arm64 runner and install native development packages rather than configuring a foreign dpkg architecture. All matrix entries can then run the existing distribution smoke test.
Codex consumes Git release artifacts built with the Makefile's default -O2 flags. The release job compiles each artifact without link-time optimization. Add a release-only config.mak.openai and copy it into Git's ignored config.mak slot before building. Use thin LTO for Clang targets and automatic LTO for GCC targets, then check GIT-CFLAGS records the selected flag in every distribution job. Keeping the setting in config.mak.openai avoids carrying release-only policy in the upstream Makefile.
LTO can optimize across translation units, but the release job has no execution profile for the status, diff, clone, fetch, and repack paths Codex invokes frequently. Git's built-in profile target runs the 1,048-script test suite serially. That is too expensive for every release target and weights test-harness paths more heavily than the local workload. Extend config.mak.openai with GCC and LLVM profile modes. Gate GIT-CFLAGS on an instrumented build, run a short offline trainer, merge LLVM raw profiles when needed, and rebuild with profile-use flags. Each matrix entry runs on its target architecture, so it can execute the instrumented binary. Check that final GIT-CFLAGS includes a profile-use flag and increase the timeout for the second compilation pass. The focused trainer took about 30 seconds locally; the full macOS build/install validation completed with thin LTO and LLVM profile-use enabled.
Integrate the current tb/codex/automation topic into the internally distributed codex branch. Codex-Integration: tb/codex/automation@17738e2cc87ba67ed36cd1ffde983d43e01a5f41
Integrate the current tb/codex/geometric-maintenance-promisor topic into the internally distributed codex branch. Codex-Integration: tb/codex/geometric-maintenance-promisor@dc2fffc37cead551f8036c9ecab5e52a4cbee37b
Integrate the current tb/codex/release topic into the internally distributed codex branch. Codex-Integration: tb/codex/release@ba107e0ae8c7142238bb612e530d51d42f0280d3
Integrate the current dr/codex/dugite topic into the internally distributed codex branch. Codex-Integration: dr/codex/dugite@988cecced01f69765d599a2d6c023406af98fa1b
Integrate the current tb/codex/lto-pgo topic into the internally distributed codex branch. Codex-Integration: tb/codex/lto-pgo@88fcb4ac12c583bedf010e97ebf83cec240e3120
valid_cached_dir() used match_stat_data_racy(), whose treatment of ctime and other fields follows core.trustCtime and core.checkStat. Tracked entries can correct a false stat match with a later content comparison, but cached directories have no equivalent check. Renaming a child and restoring its parent's mtime can therefore hide untracked paths under weak stat settings. Compare every field persisted in directory stat_data regardless of those tracked-file settings, and retain the existing racy-timestamp check. The untracked-cache status test renames a child, restores the directory mtime, and verifies that cached and uncached status agree. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Comparing only a pathname, device, and inode does not establish that two observations still describe the same unchanged filesystem object. Users that reopen a path need a reusable comparison covering the represented metadata fields. Represent an object's stat identity as a zero-initialized, fixed-width array containing device, inode, mode, link count, ownership, size, and modification and change timestamps. Include nanoseconds where available and add birth time and generation on Apple platforms. Provide comparison helpers and register both their library source and Clar unit suite in the Makefile and Meson builds. The unit tests check identity equality and reject changes to each represented stat field. No exclude-file validation or production caller is introduced here. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
valid_cached_dir() performs a synchronous lstat() whenever an untracked-cache directory cannot rely on fsmonitor. A separate validation pass cannot remove that duplicated work unless traversal knows whether a saved result was checked and matched. Add transient stat_checked and stat_matches bits to each cached directory and let traversal consume them only when its caller marks the cache preloaded. Clear that marker after traversal and on the symlink-leading-path exit. Existing callers leave the marker clear, so ordinary lstat() validation and the fsmonitor path remain unchanged. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Tracked-index refresh can invalidate the mutable untracked-cache tree. A concurrent directory-validation worker therefore cannot discover nodes or read their validation inputs directly from that live tree. Capture each node pointer, copied pathname, saved stat data, and prior validity before concurrent work begins. The opaque preload object also retains the cache, root, index timestamp, repository, and directory flags needed to recognize its original context. Expose construction and release as a complete ownership boundary. This preparatory change allocates one snapshot per cached directory but has no production caller and does not start workers or publish results. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A captured untracked-cache directory must not be marked reusable from a stale snapshot: tracked-index refresh may invalidate its live node, replace the cache, or change the traversal's directory flags. Run lstat() against each saved pathname and compare its immutable stat snapshot using the strict, racy-aware directory comparison. Publish the checked result only if the current cache, root, and directory flags still match. Preserve any invalidation that happened after capture; failed or changed stats leave ordinary traversal responsible for rescan. Validation remains synchronous, and no status caller invokes the new finish operation at this boundary. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Synchronous validation still places every cached-directory lstat() on one execution path. Independent, already-captured directory snapshots can instead be divided among bounded workers without reading the live cache from those workers. Partition the snapshots using approximately 1,000 directories per worker, cap the worker count at six and at three times the available CPU count, and permit a bounded test override. Workers retain results in their own snapshot ranges; the existing finish operation joins them before publishing anything. Record worker count, thread-creation failures, directory count, and elapsed worker time through the threads, thread_failure, dirs, and wall_us Trace2 keys. Publication validity and applied-result counters remain in the earlier finishing boundary. Run the work synchronously for a single worker or without pthreads. If thread creation stops partway through, join started workers and process every unstarted range synchronously. No status caller enables the preload at this boundary. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Directory validation and tracked-index refresh inspect different snapshots, but running them consecutively leaves both operations on the status command's critical path. Start the cached-directory preload after reading the index and before refresh_index(). Join its workers after configuring excludes and before collecting untracked paths, then pass their results into directory traversal. Release any unfinished preload when status buffers are freed. Keep activation behind GIT_TEST_UNTRACKED_CACHE_AUTO_PRELOAD until production eligibility is defined. The untracked-cache status test checks unchanged and modified directories, preserved output, and the worker count selected by the running build's pthread support. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Clearing CE_FSMONITOR_VALID is not enough to make a provider event authoritative. With core.trustctime disabled, core.checkStat set to minimal, and a restored modification time, stat matching can still accept changed file contents. The same stale match can affect diff, apply, checkout, and unpack-trees. Mark a reported entry with the in-memory CE_CONTENT_CHECK_REQUIRED flag, clear CE_UPTODATE, and discard its cached stat data. Route diff, apply, checkout, and unpack-trees comparisons through ie_match_stat_with_content_check(), which calls ie_modified() only for marked non-gitlinks. Other direct ie_match_stat() callers retain their existing paths. Marking an entry up to date clears the transient flag. Ordinary entries, gitlinks, and unmarked zero-stat entries retain their existing stat behavior. Add hook regressions for restored timestamps, diff and status, indexed apply, checkout, case-insensitive unpacking, unchanged reset, and ordinary zero-stat behavior in t/t7519-status-fsmonitor.sh. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Starting directory-validation workers for a small cache, restricted pathspec, incompatible traversal, or fsmonitor-managed cache adds work without providing a safe whole-worktree reuse opportunity. Enable automatic preload only when the existing untracked cache and its root are valid, fsmonitor is disabled, traversal flags agree, and a bounded count finds at least 2,000 cached directories. Reject pathspecs, disabled untracked output, ignored-output modes, and incompatible -uall cache settings. Keep the test override for focused small-cache coverage. Status tests exercise both sides of the directory threshold and verify that restricted pathspecs and incompatible -uall requests retain the ordinary traversal path. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A filesystem-monitor provider can know that its event history is incomplete without being able to identify every affected path. Treating such a response as an ordinary path leaves tracked entries, cached attributes, and untracked-cache state falsely valid. Reserve // as a provider-only global invalidation record. It cannot collide with a worktree-relative path. When the client receives it, discard cached attribute stacks and untracked-cache state, invalidate every tracked entry, and mark the fsmonitor extension changed. Recognize the existing trivial response only when a complete record consists of a single slash and NUL, newline, or carriage-return terminators. This prevents the new double-slash record from being discarded as a trivial response while preserving existing hook forms. Add a hook regression in t/t7519-status-fsmonitor.sh that changes a tracked file, restores its timestamp, emits the global marker, and requires status to report the change. Global invalidation intentionally scans the tracked index. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An implicitly started fsmonitor daemon inherits its caller's repository environment and current directory. In a linked worktree, inherited Git directory, worktree, common-directory, prefix, and index settings can make the child discover a different repository than the worktree whose status requested the daemon. Resolve the requested worktree to its canonical path, start the child from that directory, and remove repository-addressing variables from its environment. Keep the existing daemon start command and return an error if the worktree cannot be resolved. Add a macOS regression that implicitly starts fsmonitor from a linked worktree and checks the daemon child's working directory in Trace2. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
S06 and S10 need S01/P08 without S01/P01-P07. Merge the shared standalone commit after P07 and before P09-P11, so all three branches use the same P08 commit.
A pathname monitor cannot establish that every name for a multiply-linked regular file lies inside its watch cone. Persisting CE_FSMONITOR_VALID after checking the tracked name can therefore hide a later write through an unmonitored hardlink. Use fsmonitor_stat_can_be_valid() to exclude regular files with more than one link from persistent fsmonitor validity when the platform reports real link counts. Apply that decision where index refresh, threaded preload, and diff-files first consume an actual stat. Preserve CE_UPTODATE for the current process and retain existing persistent validity for single-link and nonregular entries. Windows and Cygwin synthesize their link counts, so preserve their existing fsmonitor behavior without claiming the hardlink guarantee there. Add a hardlink regression in t/t7519-status-fsmonitor.sh on platforms with trustworthy stat metadata. It keeps a tracked hardlink outside the fsmonitor-valid bitmap and checks that a write through an alias outside the worktree appears in status. The deliberate cost is another stat in a subsequent process. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Implicit fsmonitor startup resolves a Git command through the execution path and invokes its start subcommand. An overridden execution path can therefore select a different Git than the dispatcher that initiated the query, while adding another launcher between the client and daemon. Retain the absolute executable path during dispatcher initialization and expose it only for a real Git dispatcher. Start that executable directly with fsmonitor--daemon run --detach, then wait until its IPC socket is listening before accepting startup. Respect the configured startup timeout, defaulting to 60 seconds, and retain Git-command lookup when an authoritative dispatcher path is unavailable. The canonical worktree and sanitized environment established by S03/P01 remain in place. Update existing startup Trace2 checks for the direct invocation and add a macOS regression with a fake Git on the execution path to verify that the original executable is used. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Matching directory metadata alone cannot prove that its cached ignore rules are unchanged. A rewritten .gitignore with restored timestamps can otherwise leave preload results valid while changing which untracked paths should be visible. Snapshot each cached exclude object ID and validate its per-directory file on the existing preload workers. Open regular files with open_nofollow(), reject files larger than 1 MiB, and compare their raw or trailing-LF blob hash with the cached object ID. Verify the open file's stat identity before and after reading, then reopen its pathname and require the same identity through S01/P08. Publish directory results only when both stat and exclude checks match; otherwise invalidate the cached ignore state and fall back to ordinary traversal. A status test rewrites .gitignore, restores its mtime, and checks the result against uncached status. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Darwin FSEvents identifies the pathname associated with a hardlink event, not every name referring to the same inode. Invalidating only that pathname can leave another tracked hardlink trusted after its contents change. Classify the event's absolute path before handling its hardlink flags. For worktree events, enqueue the provider-wide marker introduced by S04/P02 so clients content-check the tracked set. Leave gitdir events in the existing cookie and gitdir handling; otherwise reads of hardlinked object files could repeatedly trigger global invalidation. Add a MACOS,HARDLINKS daemon regression that rejects a marker for a gitdir hardlink, then verifies the marker and correct status for a changed worktree hardlink with its timestamp restored. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The fsmonitor.startTimeout setting controls how long a client waits for daemon startup; the daemon's run subcommand does not consume it. Nevertheless, daemon configuration parsing validates that setting for every subcommand. A malformed value can consequently kill an implicitly started daemon before it opens its IPC socket. Pass a run-specific configuration flag into the callback and skip startup-timeout parsing only for run. Continue parsing other daemon settings normally, and preserve strict timeout validation for the explicit start subcommand. Add a macOS regression that verifies implicit status still starts the daemon with a malformed timeout while explicit daemon start rejects the same configuration. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An IPC provider cannot safely adopt missing semantic history merely because tracked entries are marked fsmonitor-valid. Minimal stat checks can conceal a content rewrite, and a clean token cannot retroactively certify workers started under different attributes. Capture the complete proof epoch before preparing semantic workers. Prime each worker's attribute frames and verify the starting token and complete epoch before hashing. After a clean closing query, apply the proof only if the pinned index, configuration, attribute content, manifest, worktree identity, and token remain consistent. Permit attribute-namespace bookkeeping to change only after its source bytes and initial namespace were verified. Accept tracked validity independently of untracked validity. Keep a query pending when the untracked scan has not run. Leave collapsed sparse indexes, pathspecs, ignored-file requests, unreliable file identity, non-IPC providers, and failed proofs on ordinary closure or complete refresh. Add scripted regressions for adopting missing tracked history without hiding a same-size rewrite and for preserving a collapsed sparse index on the ordinary closure path. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An untracked-cache preload can inspect cached excludes and directory state before tracked semantic adoption restores verified stat data. One provider response also cannot certify an untracked traversal performed after the tracked scan that response closes. Defer provider-backed untracked validation until the tracked proof has been applied and its first query has closed. Prime the untracked cache afterward, issue a second closing query, and recheck the full tracked proof before accepting either result. If the later query reports a change, invalidate both results, reprime during ordinary closure, and retry within the existing query bound. Factor the existing proof-current checks into the predicate used by proof application and deferred closure. Preserve automatic untracked preload when no provider is enabled or file identity is unreliable. Fall back to a complete scan if untracked validation or token closure fails. Add prerequisite-guarded scripted cases for successful deferred scans, failed untracked closure, and a change reported by the second closing query. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An as-is commit refreshes its index before running the pre-commit hook. If the hook rewrites a tracked path without changing its size or mtime, the later in-process status must not certify the earlier refresh as though it covered the hook. For a nonsplit index using an IPC provider, perform the initial refresh through status token closure. After an invoked hook, release the saved attribute snapshot and reopen the last accepted provider token before status runs again. Reject unavailable token state and invalidate the manifest, tracked semantics, and untracked cache before falling back to a complete refresh. Pin the post-hook named index before persisting strong invalidation. Write refreshed state only while its held descriptor, pathname, stored trailer checksum, and in-memory index still match. Preserve a hook-replaced index and the existing reread. Split indexes, platforms without reliable file identity, and non-IPC providers retain their original initial refresh. Add prerequisite-guarded scripted cases for successful post-hook closure without an untracked cache, failed closure with complete worktree refresh, and a hook that updates the index itself. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
S13 depends on S09, S11, and S12. Join those three histories once before applying S13. S09 and S11 add adjacent cleanup and test-list entries. Release both clean-status and bulk-preload state in read-cache.c, and keep t7529 before t7531 in t/meson.build. S12 requires no further resolution.
A bulk untracked scan cannot reuse its result merely because an ignore file has familiar stat data. A file or its parent may be replaced while the scan runs, an absent source may appear, and repeated reads of the same source may observe different patterns. Record each source beneath its nearest available anchored parent, along with its path, symlink policy, presence, size, and blob identity. Check descriptor and parent identities while capturing an observation, then resolve the current parent again and compare the actual source bytes at validation. Coalesce equivalent observations and invalidate the proof immediately when observations conflict. Validation uses nonblocking opens, so replacing a source with a FIFO cannot hang. Equal contents remain acceptable even if the source or its parent has a different identity. This also preserves an empty /dev/null and an equivalent empty FIFO; changed or missing contents, unavailable anchored primitives, and failed parent callbacks invalidate the proof. Register the implementation and focused unit suite in both Make and Meson. The tests cover source and parent replacement, stable absence, repeated and conflicting observations, missing buffers, no-follow policy, /dev/null, FIFO replacement, and parent-opener failure. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The ordinary exclude reader opens configured, repository, and per-directory ignore files by pathname. That is sufficient for a one-off walk, but a concurrent replacement or newly created ignore file makes a retained bulk result unsafe. Attach the optional source proof from S13/P01 to dir_struct and capture the exact bytes or stable absence observed by add_patterns(). Preserve symlink-following for standard excludes and the existing no-follow policy for per-directory .gitignore files. Visit configured and repository sources even when absent so their later appearance invalidates the proof. Mark failed, oversized, short, and index-backed reads unprovable rather than treating their results as stable filesystem observations. Existing callers without a proof retain their original opens, error handling, pattern parsing, and oversized-source guard. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A tracked-file bulk preload can already walk directories that ordinary status later scans for untracked files. Sharing those observations requires a complete, independently validated result; a partial list must never suppress the conventional untracked traversal. Add an explicit backend capability and optional borrowed destination for visible paths. Serialize the existing ignore matcher across scan workers, collapse an untracked directory after its first visible descendant, and sort the provisional results. Publish them only after the directory scan and the anchored ignore-source proof both complete. Reject duplicate paths and discard incomplete or conflicting untracked results without discarding independently valid tracked observations. Report completeness, visible-path count, and fallback reason through Trace2, and release all temporary path and proof state. No existing backend advertises the new capability and no status caller requests it at this boundary. Ordinary tracked and untracked behavior therefore remains unchanged. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The APFS tracked preload encounters untracked entries but ordinarily discards them. Repeating the entire directory walk to rediscover those entries costs work even when the existing scan can establish their visibility. Teach the APFS backend to advertise visible-path collection and supply its root-anchored exclude-parent opener. Classify regular files and symlinks with the normal exclusion machinery, and follow untracked directories only until their first visible descendant establishes the single directory entry that normal-mode status reports. Keep the top-level Git directory and tracked gitlinks out of the untracked result. Case aliases, embedded repositories, and foreign mounts invalidate provisional untracked observations while retaining separately valid tracked results. A replaced directory increments changed_dirs, so scan-wide validation discards the entire bulk result. Carry the collapsed-directory root through queued workers. Ordinary status remains unchanged until a caller explicitly requests the new backend capability. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Normal-mode status walks the worktree for untracked paths even after the opted-in APFS preloader has visited the same directories. Reusing that walk is incorrect if status requests different reporting semantics or the bulk scan cannot prove that its exclusion sources remain valid. Request visible paths only for an expanded index in normal untracked mode without a pathspec, ignored output, or an untracked cache. The bulk path requires core.preloadIndex, core.preloadIndexBulk, and a disabled fsmonitor. Transfer paths only after the bulk scan, worktree-namespace checks, and anchored exclusion-source proof finish successfully. Complete the ordinary tracked refresh, clear the borrowed index destination, and skip the second directory walk only for a complete untracked result. Retain ordinary traversal for unsupported backends, incomplete proofs, case aliases, embedded repositories, changed exclusion sources, and ineligible reporting modes. A failed untracked proof preserves independently valid tracked results; a changed directory instead invalidates the entire bulk scan. Extend the APFS tests to compare output with ordinary status. Cover collapsed directories, ignored-only and empty directories, special files, activation guards, case aliases, nested repositories, tracked submodules, separate Git directories, changed configured and repository exclusions, a newly appearing configured exclusion, bidirectional per-directory changes, hard-linked exclusions, and tracked-file replacement. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A Linux directory scan cannot substitute its results for lstat() when file identity, timestamps, or mount membership are missing. Depending on libc's statx declarations would also tie the implementation to the age of the installed Linux headers. Define the required statx syscall ABI locally and request complete basic statistics and a mount identifier. Reject invalid nanosecond fields, foreign mounts, and device, inode, link-count, owner, size, or timestamp values that cannot be represented in struct stat. Register the metadata module in the Make, CMake, and Meson Linux builds. The native Linux boundary build compiles it with DEVELOPER=1, but this patch does not select a backend or change the fallback. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A directory name observed during enumeration may resolve outside the original worktree after a rename, symlink replacement, magic-link traversal, or mount change. Path-based reopening would then inspect an unverified namespace. Introduce descriptor-relative Linux directory-open helpers. Prefer openat2() with beneath-root resolution and reject symlinks, magic links, and mount crossings when that syscall is available. Otherwise reject empty, absolute, dot-dot, and malformed paths. Open root-relative paths one component at a time and verify each mount. Keep direct child opens descriptor-relative; directory scanning verifies their mounts before enumeration. Register the opening module with Make, CMake, and Meson. The native Linux boundary build compiles it with DEVELOPER=1, but backend selection remains unchanged. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A getdents64 record supplies a type hint, not proof that a path is a regular file or directory. Trusting that hint can hide a tracked replacement, misapply ignore rules, or report the wrong visible untracked shape. Parse record lengths and names within a bounded 1 MiB worker buffer. Require statx metadata and matching mount identity for tracked paths. When collecting untracked paths, obtain authoritative metadata before classifying a wholly untracked file or directory. Use directory hints only to schedule paths with tracked descendants. Preserve per-entry fallback for special and multiply linked tracked files. Stop an untracked subtree once its normal-status witness is visible, and invalidate uncertain untracked results. Register the module in all three Linux builds; the native DEVELOPER=1 boundary build compiles it without selecting the backend. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Holding a directory descriptor establishes what workers read, but does not prove that the original directory stayed in the worktree. A child can also move under a different parent while queued. Publishing observations from either replacement could hide worktree changes. Capture the complete directory statx observation and converted stat identity before enumeration. Verify the descriptor mount and recheck both identities afterward. For queued children, resolve the parent through the held child descriptor and compare it with the recorded parent identity. Add the mount identifier to the shared directory identity and register the Linux scan module with Make, CMake, and Meson. The native DEVELOPER=1 boundary build compiles it, while recorded directory changes prevent the completed scan from being accepted. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Individually anchored descriptors do not establish that the mount namespace or named worktree root stayed unchanged throughout a scan. A mount replacement can invalidate otherwise consistent directory observations. Accept only ext-family and XFS filesystems with complete root statx and mount-identity data. Capture /proc/self/mountinfo before the scan, compare it at completion, and freshly reopen the named worktree root with O_NOFOLLOW to verify its original complete identity. Probe openat2() without requiring it. Register the topology module in all three Linux builds. The native DEVELOPER=1 boundary build compiles it. Missing namespace proof, unsupported filesystems, changed mount tables, or replaced roots reject the result; the retained mount snapshot adds memory and can reject unrelated namespace changes. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The separately registered Linux metadata, anchored-open, enumeration, directory-validation, and topology modules cannot safely publish a physical scan by themselves. They must share the existing bulk backend lifecycle so every closing check runs before results are accepted. Assemble those modules into the Linux backend and register the shared and platform objects with Make, CMake, and Meson. Retain the existing requirements that core.preloadIndex and core.preloadIndexBulk are enabled and fsmonitor is disabled. Preserve ordinary preload when a required syscall, filesystem, mount proof, or closing validation is unavailable. Cap Linux scans at 16 workers. Each worker can allocate a 1 MiB directory buffer; mount snapshots and retained scan results add further memory. Document ext-family and XFS support and keep directory-type injection confined to the documented test environment. Add and register t7532-preload-index-linux.sh with 12 Linux-only cases. Native Linux validation passes 12/12 in the threaded build and 12/12 in a separate NO_PTHREADS build; CMake and Meson link Git. The suite compares ordinary status for tracked changes, visible and ignored paths, false type hints, fallback shapes, and a synchronized child replacement. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The bulk preloader rejected an active fsmonitor provider, so status verified ambiguous tracked entries through a separate semantic scan. Publishing bulk observations or refreshed stat data before the closing provider query would permit a concurrent change to invalidate a clean result. Pass held parent descriptors, basenames, and observed metadata from both platform walkers to semantic_verify_file_at(). Borrow the captured provider proof epoch, hash eligible raw-safe files during the bulk walk, and retain clean states and stat updates provisionally. After the closing provider query confirms the same epoch, validate all pending positions before publishing clean states, refreshed stat data, and fsmonitor-valid bits. Clear provisional state on provider failure, epoch mismatch, or invalid updates, and retain the existing complete-refresh fallback. Choose the provider-backed bulk path from its actual safety conditions, not from whether semantic history is awaiting adoption. This lets a trivial daemon response or daemon restart rebuild and close an ordinary bulk proof, including for a skipHash index, while retaining the complete proof epoch and closing query. Require both preload settings, an expanded index, a pending built-in IPC token, and an eligible whole-worktree request. Keep APFS and Linux within their platform and filesystem limits. Allocate a bounded hash buffer and attribute check per content-verification worker, and retain tracked states and stat updates only until closure. Extend the APFS and Linux tests with same-size, restored-mtime content changes. Cover accepted closure, provider failure, dirty status, daemon token reset with a null-checksum index, and Trace2 evidence of hashing, deferred publication, and token acceptance or rejection. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A live exclude-source proof uses filesystem identity to keep one observation coherent. That identity cannot compare equivalent ignore sources captured by separate status processes: replacing a file with the same contents changes its identity without changing ignore semantics. Hash the existing, validated observations in first-observation order. Frame the digest with its version, source object format, unique source count, path, lookup policy, presence, and content identity. Exclude transient stat identity so an equivalent replacement retains the same semantic digest. Extend the existing exclude-proof unit tests to capture independent proofs across a same-content replacement and repeated observation. The digest is independently testable without issuing a sidecar or changing normal exclude-source validation. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A later status invocation cannot safely reuse an empty result unless its persistent record identifies the exact index and semantic inputs that the original scan proved. Accepting truncated, ambiguous, or forward-versioned records would turn a cache miss into a false clean result. Define the version-one CSTS encoding and serialize index identity in fixed-width network-byte-order fields. Bind the index format, entry count, checksum, HEAD tree, configuration and repository hashes, one exclude digest, and a bounded builtin-provider token. Protect the complete record with the repository's object-format checksum. Reject unknown flags, unsupported index formats, null required object IDs, invalid token bounds or prefixes, bad checksums, truncation, and trailing payload. Add fixed-width identity and sidecar unit coverage for both SHA-1 and SHA-256. Register the new source and unit suite in both Make and Meson. This patch defines and tests the format; it neither writes a sidecar nor changes status dispatch. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A valid sidecar encoding is not sufficient if its named index can be replaced between proof capture and publication. Publishing that record would let a later reader associate one clean result with another index. Expose the existing index-snapshot open and named-path revalidation helpers at their first store consumer. Require a durable index identity on local APFS, a matching index format, entry count, and checksum, and agreement between the held descriptor and the named index. Encode the sidecar under its own lockfile and repeat the index checks before committing that lock. Register the store unit suite with Make and Meson. Its local-APFS tests cover successful installation for SHA-1 and SHA-256 and rejection when the source index is replaced after pinning. Other filesystems fail closed. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A clean provider response does not establish that every index entry can be represented by an empty status result. Conflicted entries, submodules, sparse entries, intent-to-add entries, and independently trusted stat state can all require ordinary index processing. Introduce a single conservative certifiability check. Require a non-null index checksum and provider-valid ordinary entries. Reject gitlinks, nonzero stages, intent-to-add, skip-worktree, CE_VALID, and unrecognized entry flags while allowing the explicitly supported in-memory flags. Extend the existing index unit suite to exercise accepted ordinary entries and each unsupported entry shape. The classifier does not issue a proof or change status behavior by itself. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An empty status result cannot certify the next invocation unless its tracked and untracked observations, ignore sources, provider token, configuration, repository, HEAD, and named index belong to one completed scan. Publishing a digest before provider-token closure, or reusing cached replacement-ref state, could issue a false clean proof. Retain the standard-exclude digest produced by the complete bulk scan. Keep provider-originated digest state pending until token closure accepts it, and preserve the accepted digest when consuming single-use tracked results. Inspect a fresh, uncached ref store and reject effective replacement refs. Then fingerprint the held local-APFS index and worktree, repository paths, locale, and external attribute state. Issue a sidecar only for the literal, top-level, empty porcelain-v2 command after persistent semantic history, an eligible expanded index, a complete untracked scan, and the HEAD cache tree all agree. Install against the pinned index before rolling back its held index lock; otherwise retain ordinary index-update behavior. Add the focused sidecar integration suite and register its source and production code with the relevant Make and Meson builds. Cover prior semantic history, unchanged index contents, exact command shape, and rejection of external attributes, untracked-cache results, and alternate indexes. No early status answer is introduced here. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An installed sidecar cannot be inspected safely by opening an untrusted adjacent path without bounds. A symbolic link, named pipe, oversized record, or growing file could redirect the read, block status, or consume unbounded memory. Open the named sidecar without following symbolic links and request a nonblocking descriptor. Accept only a regular file of at most 8192 bytes, read exactly its recorded size, reject an additional byte, and parse its checksummed contents into caller-owned storage. Clear failed records and release storage explicitly. Platforms without nonblocking support fail closed. Extend the registered store unit suite to cover owned token storage under both object formats, symbolic links, FIFOs, and an oversized 8193-byte record. The loader is testable at this boundary; it does not yet bypass index deserialization. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Validating a sidecar must recapture standard excludes before status can trust an empty result. Opening an exclude source that has become a named pipe may otherwise block the supposedly cheap validation. Add an explicit nonblocking flag to exclude-source proof creation and carry it into the existing anchored source-open operation. Reject unknown flags, request nonblocking captures for sidecar issuance, and update the existing bulk-scan and unit-test callers to pass zero, preserving their current blocking and symbolic-link policies. Add a focused FIFO unit test showing that an opted-in proof captures and validates an empty pipe without waiting. The later early-status consumer can reuse nonblocking capture without changing ordinary exclude handling. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An issued clean-status sidecar has no latency benefit while status still deserializes the index before checking it. Moving the check earlier is safe only if the recorded proof is revalidated around an empty builtin-fsmonitor delta. Attempt the sidecar only for the literal top-level porcelain-v2 command on an eligible main worktree. Load the bounded record, pin the named local-APFS index, recapture excludes without blocking, and check configuration, attributes, repository identity, HEAD, and provider mode. Query the builtin provider directly from the stored token. Keep the attribute and exclude proofs alive across that query. Recheck configuration, HEAD, fresh replacement-ref and repository state, attribute contents and namespace, exclude-source identity, and both the held and named index before accepting an empty delta. Return without deserializing index entries only when every check succeeds; otherwise continue through ordinary status. Unsupported anchored-open platforms take that ordinary path. Register the fast-path source with Make and Meson. Extend the existing sidecar integration suite for read-only hits, dirty worktree shapes, loose, packed, and custom replacement refs, sidecar and exclude FIFOs, changed configuration, attributes, HEAD, null-checksum indexes, and post-query replacement or exclude races. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A clean-status sidecar is a narrowly scoped proof, not an alternate index or a general cache. Documenting only its serialized bytes would hide the full-scan issuance requirement and the revalidation needed before an empty provider response can answer status. Document the adjacent sidecar path, local-APFS and main-worktree eligibility, fixed-width version-one CSTS fields, a checksum using the repository object-format hash, the separate repository-identity hash, a bounded builtin-provider token, and the 8192-byte read limit. Explain why the source index, configuration, repository, HEAD, attributes, and standard excludes must remain coherent. Describe completed-scan issuance, persistent provider history, held index locks, the post-query race fence, nonblocking source opens, and read-only hits. State that every missing, unsupported, stale, malformed, or raced proof falls back to ordinary status. Register the technical document in both the documentation Makefile and Meson. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The complete stack has three maximal leaves: S17, S15, and S02. Join them once at the integration tip; S01 is now an ancestor of S05 so that paired FSUC state can safely reuse its recursive UNTR proof. Retain both independent status bitfields in wt-status.h and every test block at its validated position in t/t7519-status-fsmonitor.sh. The result contains all 105 patches.
The APFS bulk-preload race tests pause status until the test driver writes a byte to a resume FIFO. The child currently publishes its ready file before it opens the FIFO. If it is descheduled between those operations, the parent can observe readiness, write and close its descriptor, and discard the byte before a reader exists. Status then blocks forever in strbuf_read_file(), leaving a macOS CI job apparently hung. Open the resume FIFO first and read from that descriptor after publishing readiness. The parent opens the FIFO read/write before starting status, so the child open cannot block. Readiness now proves a reader is attached, and the resume byte cannot be lost. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
ttaylorr-oai
force-pushed
the
tb/codex/status-preview-unstable
branch
from
August 7, 2026 00:59
8358fcb to
949c04b
Compare
ttaylorr-oai
force-pushed
the
codex-unstable
branch
2 times, most recently
from
August 7, 2026 08:31
be8e9d4 to
2b48ff1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The APFS bulk-preload test barrier published its READY file before it
opened the resume FIFO. If the child was descheduled at that point, the
parent could observe READY, write and close the FIFO, and lose the byte
before any reader existed. The child then blocked forever while macOS CI
appeared hung.
Open the resume FIFO before publishing READY. The parent already holds
the FIFO open read/write before it starts status, so the child open does
not block; READY now proves that a reader owns the resume descriptor.
The previous topic tip is already published as
v2.55.0-openai.619.gb264c6f26648, so this correction is additive on topof that released history rather than rewriting it. Its resulting tree is
byte-identical to the separately audited owner-amended replay.
Validation:
preload_index_bulk_darwinunit suite (6/6)122/122 ordered parents