Skip to content

[flow] Store relative path suffixes in File_key.t to eliminate saved state denormalization#9375

Open
panagosg7 wants to merge 1 commit intomainfrom
export-D95327604
Open

[flow] Store relative path suffixes in File_key.t to eliminate saved state denormalization#9375
panagosg7 wants to merge 1 commit intomainfrom
export-D95327604

Conversation

@panagosg7
Copy link
Contributor

Summary:
Flow's saved state init spends ~13s single-threaded converting 1.87M relative paths back to absolute paths (the Denormalize phase). This is because File_key.t stores full absolute path strings internally, but the saved state stores relative paths for portability — so every load requires converting all paths back to absolute via string concatenation and allocation.

This diff applies Hack's Relative_path.t approach to Flow's File_key.t: paths are stored as relative suffixes internally, with the project root resolved on demand via to_string/to_absolute. The type shape is unchanged (LibFile of string | SourceFile of string | ...) to preserve OCaml Marshal compatibility with existing saved state files.

Performance (production build, fbcode//mode/opt):

  • Benchmark: Denormalize phase drops from 15.2s to 0.0s. Total benchmark time 64.9s → 40.0s (38% faster).
  • Real flow server on www: Total init 95.2s → 83.7s (11.5s faster, 12%). LoadSavedState 22.9s → 12.0s. RestoreHeaps, CommitModules, Indexing all within noise of baseline.
  • perf profile confirms zero denormalization functions appear at any threshold. The remaining 4.8s in the Denormalize timer is OCaml major GC triggered by the preceding Decompress phase.

Potentially helps memory as well. From 10 samples each:
{F1986414698}
So: shorter strings → less live data → smaller GC heap → 1.2 GB RSS reduction. The raw byte savings are ~240 MB; the GC amplifies it to ~1.2 GB.

Changelog: internal Improve saved state init performance.

Differential Revision: D95327604

…state denormalization

Summary:
Flow's saved state init spends ~13s single-threaded converting 1.87M relative paths back to absolute paths (the Denormalize phase). This is because `File_key.t` stores full absolute path strings internally, but the saved state stores relative paths for portability — so every load requires converting all paths back to absolute via string concatenation and allocation.

This diff applies Hack's `Relative_path.t` approach to Flow's `File_key.t`: paths are stored as relative suffixes internally, with the project root resolved on demand via `to_string`/`to_absolute`. The type shape is unchanged (`LibFile of string | SourceFile of string | ...`) to preserve OCaml Marshal compatibility with existing saved state files.

Performance (production build, `fbcode//mode/opt`):
- Benchmark: Denormalize phase drops from 15.2s to 0.0s. Total benchmark time 64.9s → 40.0s (38% faster).
- Real flow server on www: Total init 95.2s → 83.7s (11.5s faster, 12%). LoadSavedState 22.9s → 12.0s. RestoreHeaps, CommitModules, Indexing all within noise of baseline.
- perf profile confirms zero denormalization functions appear at any threshold. The remaining 4.8s in the Denormalize timer is OCaml major GC triggered by the preceding Decompress phase.

Potentially helps memory as well. From 10 samples each:
 {F1986414698}
So: shorter strings → less live data → smaller GC heap → 1.2 GB RSS reduction. The raw byte savings are ~240 MB; the GC amplifies it to ~1.2 GB.

Changelog: internal Improve saved state init performance.

Differential Revision: D95327604
@meta-cla meta-cla bot added the CLA Signed label Mar 6, 2026
@meta-codesync
Copy link

meta-codesync bot commented Mar 6, 2026

@panagosg7 has exported this pull request. If you are a Meta employee, you can view the originating Diff in D95327604.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants