Overhaul zsh completion: caching, global flags, --flag=value, test suite

[b-per]

Summary

• Caching overhaul: replace flat-file cache with zsh's native _retrieve_cache/_store_cache; fix cache-key collision bug where two projects compiled at the same second would share entries
• Global flags fix: dbt --profiles-dir /p run --<TAB> now correctly passes the full word sequence to Click instead of dropping everything after the global flag
• --flag=value support: --select=<TAB>, --exclude=<TAB>, etc. now list model selectors (using compset -P '*=' to strip the prefix)
• Multi-line description fix: Click's --warn-error-options emits a multi-line description that broke the old stride-3 parser; replaced with a forward-scan approach
• Test suite: new test.zsh with 72 unit tests covering all pure-logic functions without requiring a live binary

Caching approach

Completions are expensive — Click spawns a Python process (~1 s each call). The caching strategy has two layers.

1. Manifest cache (_dbt_list_models, _dbt_list_selectors)

Cache key: dbt_models_<path_hash>_<mtime> (same pattern for selectors).

• mtime — invalidate when the manifest changes on disk
• path_hash — cksum of the manifest path; added to fix a cross-project collision: two projects whose manifests have the same mtime (compiled in the same second) would previously share a cache entry

Storage: zsh's ~/.zcompcache/ via _retrieve_cache/_store_cache.

2. Click completion cache (_dbt_core_fetch_completions)

Cache key: dbt_click_<bin_mtime>_<ctx_hash>.

• bin_mtime — invalidate when the dbt binary changes (upgrade, reinstall)
• ctx_hash — cksum of the normalised COMP_WORDS string

Context normalisation is the key insight. Click returns the same flag list for dbt run regardless of which flags have already been typed. So instead of caching one entry per unique command line, the context is collapsed:

Situation	Normalised COMP_WORDS	Effect
dbt run --project-dir /foo --<TAB>	dbt run -	Same cache entry as…
dbt run --<TAB>	dbt run -	…this
dbt run --log-format te<TAB>	dbt run --log-format te	Choices for that specific flag
dbt ru<TAB>	dbt ru	dbt subcommands

The real partial is filtered locally after the cache is read, so the hit rate is high even while the user is still typing.

Exception — global flags: when dbt --profiles-dir /p run --<TAB> is typed, stripping global flags would shift the word positions Click sees (we can't know which flags take a value). In that case the full word sequence is passed verbatim. The current word is still normalised to - for flag-name completions so the same global-flag prefix shares one cache entry.

3. Fusion binary cache (_dbt_fusion_complete)

The clap_complete script from dbt completions zsh is stored to ~/.cache/dbt_fusion_completions.zsh and regenerated only when the binary mtime changes. Unchanged from before.

[joshuataylor]

Fantastic work on this, learnt a few more things about zsh completions -- something I've been meaning to look at.

Completions for -s seem fast, but typing:

dbt run -<TAB> is slow everytime, it seems it recalculates?

Found a couple of issues with dbt-core, zsh 5.9 and MacOS. I'm writing some comments now.

[b-per]

Thanks for giving it a spin!

So, normally dbt run -<TAB> should be slow the first time (running the completion script and saving it in the cache) ; but from the second time onwards it should be instantaneous. At least it is for me.

If it isn't for you there must be a bug in the caching mechanism somewhere.

[joshuataylor]

I've been wanting to delve further into zsh completions for a while, so I had a bit of a look, and commited my changes for my future reference.

Give https://github.com/joshuataylor/dbt-completion.bash/blob/feature/zsh-completion-overhaul/_dbt a go and let me know if it works for you. I haven't tested dbt fusion at all.

Take what you want :-).

Rambly notes below from a fun night of zshnes..

Cache issue (as you noted)

stat -f %m ... || stat -c %Y ... reads as BSD-stat syntax, but GNU stat on PATH (Homebrew coreutils) takes -f as --file-system and prints multi-line filesystem info -- exit 0, so the || stat -c %Y fallback never runs.

Cache keys ended up containing literal File: " substrings (after the embedded newline truncates the filename), so the cache effectively never hit. Every TAB respawned Click.

• 5b85fb6 Use zsh/stat module for cache-key mtime to avoid GNU stat shadowing. Switch to zmodload zsh/stat + a _dbt_mtime helper that uses zstat (no fork, no platform ambiguity), falling back to stat -c %Y then stat -f %m.

Bug fixes

• 1f29f4d Match dbtf in use-cache zstyle pattern. zstyle ':completion:*:complete:dbt:*' use-cache yes doesn't match the dbtf curcontext, so _retrieve_cache/_store_cache silently no-op'd for the fusion alias. Changed to (dbt|dbtf).
• f3e004b Include binary path hash in Click cache key. The Click cache was keyed only on bin_mtime + ctx_hash.
• caa0a12 Don't cache empty Click responses. If Click failed, the empty _dbt_response got stored and re-served until bin_mtime changed, leaving TAB silently dead.
• 64d15e5 Pre-render the renamed clap completion script. _dbt_fusion_complete was forking sed 's/^_dbt()/_dbt_clap()/' on every TAB. Do it once at regeneration time, write to dbt_fusion_completions.sourced.zsh, then plain source after that.
• db3531b Document DBT_MANIFEST_PATH stickiness across cd. Comment-only. If exported, the env var locks completions to one project across cd.

Bugs found while working on the above (all zsh footguns)

• deb1fb0 Use scalar REPLY in _dbt_mtime to avoid clobbering callers' reply array. _dbt_mtime wrote reply; to isolate that, callers declared local -a reply. But _dbt_bin_info itself returns via reply, so the shadow ate its return value. End result: $bin_path was empty, env ... "" invoked nothing, TAB silently dead. Fix: scalar REPLY for single-value helpers; reserve reply for multi-value.
• cb52d24 Avoid local 'path' -- it's tied to zsh's PATH and breaks command lookup. This was the symptom of "no completions at all" after the stat-module switch. local path="$1" in _dbt_mtime set local path to a file (not a directory), which silently destroys $PATH inside the function (lowercase path is the array form of PATH). Then stat -c %Y reported "command not found". Renamed to target. Also fixed the latent same-mistake in _dbt_manifest_path. Compound bug: ${+functions[zstat]} was always 0 because zstat is a builtin, not a function; switched to ${+builtins[zstat]}.
• da3ab6f Share cache entry when a flag follows another flag. dbt run --debug --s<TAB> was missing the cache. The build-comp-context helper checked prev_word == -* first, so any flag-after-flag matched the flag-VALUE branch and produced a unique cache key per partial typed. Reordered: check current_word first -- if the user is typing a flag, prev_word is irrelevant (a flag-value can't start with --). This was a fun one..

Cleanup: stop spawning sub-processes per TAB

• 2d7cb76 Scope variables in _parse_manifest and _parse_selectors. Both helpers leaked manifest_path, prefix, and a ~3KB prog scalar (the inlined Python) into the caller on every TAB. Hygiene; no perf.
• be8c104 Move parse-helper Python into module-level scalars; drop cat fork. Python source moved to _DBT_PARSE_MANIFEST_PY / _DBT_PARSE_SELECTORS_PY, declared once at file load (not rebuilt via cat <<EOF per call). Drop the gratuitous cat ... | python3 -c ... -- Python opens the file via sys.argv[1].
• 8406d0d Resolve dbt path via $commands[dbt] instead of $(command -v dbt). $commands is the zsh associative array populated by compinit.
• 0a9e130 Detect core-vs-fusion by binary shape, not by spawning dbt --help. I think you'll like this one @b-per ;-). Saves ~750ms on the very first TAB in a fresh shell.