diff --git a/.copilot-track/crawl/README.md b/.copilot-track/crawl/README.md new file mode 100644 index 0000000..6c39480 --- /dev/null +++ b/.copilot-track/crawl/README.md @@ -0,0 +1,44 @@ +# Crawl Phase — knife-ec-backup + +## Purpose + +This is the **Crawl** phase of the Crawl → Walk → Run AI-assisted development track. +The goal is to build foundational understanding of the codebase before making any +functional changes. + +## Chain-PR Workflow + +Each phase produces a single PR with evidence of learning: + +| Phase | Branch Pattern | Deliverables | +|-------|---------------|--------------| +| Crawl | `learn/crawl/-ex-` | SYSTEM-OVERVIEW, build-test, architecture diagram | +| Walk | `learn/walk/-ex-` | Low-risk code change + tests | +| Run | `learn/run/-ex-` | Production feature or bugfix | + +PRs chain: each phase's branch is based on the previous phase's merged branch. + +## Evidence in PRs + +Every PR must include: +1. **AI prompt transcript** — What prompts were used (summarized in PR description) +2. **Human review** — Developer confirms AI output was validated +3. **Test results** — `bundle exec rake spec` passes +4. **No blind merges** — Developer must understand every line before approving + +## Prompt Usage Guidelines + +- Use AI to **explore and understand** the codebase, not to blindly generate code +- Always **read the generated output** and verify against source files +- When AI suggests a fix, **trace the code path manually** to confirm correctness +- Record key prompts that produced useful insights in PR descriptions + +## Files in This Directory + +- `README.md` — This file (Crawl phase context) + +## Related Docs + +- `ai-track-docs/SYSTEM-OVERVIEW.md` — Full system overview +- `ai-track-docs/build-test.md` — Build and test instructions +- `ai-track-docs/architecture.mmd` — Mermaid architecture diagram diff --git a/ai-track-docs/SYSTEM-OVERVIEW.md b/ai-track-docs/SYSTEM-OVERVIEW.md new file mode 100644 index 0000000..746a736 --- /dev/null +++ b/ai-track-docs/SYSTEM-OVERVIEW.md @@ -0,0 +1,126 @@ +# knife-ec-backup System Overview + +## Summary + +**knife-ec-backup** (v3.0.8) is a Ruby gem extending the Chef `knife` CLI with subcommands +for full backup/restore of Chef Infra Server data. Packaged via Habitat for production +deployment on Chef Automate instances. + +--- + +## Language & Runtime + +| Aspect | Detail | +|--------|--------| +| Language | Ruby (100%) | +| Required Ruby | >= 3.1 | +| Framework | Chef Knife plugin (Thor-style CLI DSL) | +| Packaging | RubyGem + Habitat `.hart` package | +| Test Framework | RSpec + SimpleCov + FakeFS | +| CI/CD | Expeditor + Buildkite | + +--- + +## Entry Points + +| Entry | Path | Context | +|-------|------|---------| +| CLI (production) | `bin/knife` | Habitat: `hab pkg exec chef/knife-ec-backup knife ec ...` | +| Subcommand discovery | `lib/chef/knife/ec_*.rb` | Auto-registered by Chef Knife | +| Tests | `bundle exec rake spec` | RSpec, excludes smoke | +| Artifact test | `test/artifact/` | Smoke tests for Habitat package | + +--- + +## Module Inventory + +### Commands (lib/chef/knife/) +| File | Command | Role | +|------|---------|------| +| `ec_backup.rb` | `knife ec backup` | Download all server data to disk | +| `ec_restore.rb` | `knife ec restore` | Upload backup to a server | +| `ec_import.rb` | `knife ec import` | Import into pre-existing orgs (multi-tenant) | +| `ec_key_export.rb` | `knife ec key export` | Export users/keys via PostgreSQL | +| `ec_key_import.rb` | `knife ec key import` | Import users/keys via PostgreSQL | + +### Shared Modules (lib/chef/) +| File | Role | +|------|------| +| `knife/ec_base.rb` | Shared CLI options, config, auth, REST clients | +| `knife/ec_key_base.rb` | SQL connection, Sequel ORM, config loading | +| `knife/ec_error_handler.rb` | Thread-safe error logging to JSON | +| `server.rb` | Version detection, feature flags | +| `automate.rb` | Automate path detection | +| `tsorter.rb` | Topological sort for group dependencies | +| `org_id_cache.rb` | Org GUID lookup cache (SQL) | + +--- + +## Key Dependencies + +| Gem | Purpose | +|-----|---------| +| `chef ~> 18.0` | Knife framework, ChefFS parallel I/O, ServerAPI | +| `sequel ~> 5.9` | PostgreSQL access for key/user tables | +| `pg` | Native PostgreSQL driver | +| `veil` | Secrets management (webui_key, passwords) | +| `knife-tidy` | Backup data cleanup companion | +| `concurrent-ruby` | Thread pool (via chef-utils `parallel_map`) | + +--- + +## Concurrency Model + +ChefFS `copy_to` drives all parallel I/O: +- Uses `Concurrent::ThreadPoolExecutor` (default 10 threads) +- Configurable via `--concurrency N` → sets pool size to N-1 +- `fallback_policy: :caller_runs` prevents deadlock on recursive calls +- Even `--concurrency 1` still allows recursive parallelism within ChefFS + +--- + +## Data Flow (Backup) + +``` +knife ec backup + ├── Users: REST GET /users → users/*.json + ├── User ACLs: REST GET users//_acl → user_acls/*.json + ├── SQL Export (optional): Sequel → key_dump.json, key_table_dump.json + └── Per organization: + ├── org.json, members.json, invitations.json (REST) + └── ChefFS copy_to (parallel): + cookbooks, environments, roles, nodes, + data_bags, clients, groups, containers, acls +``` + +--- + +## 3 Low-Risk Modules (Safe to Modify) + +| Module | File | Why Safe | +|--------|------|----------| +| **EcErrorHandler** | `lib/chef/knife/ec_error_handler.rb` | Side-effect only (logging). Thread-safe. No data-path impact. Own spec. | +| **Tsorter** | `lib/chef/tsorter.rb` | 22-line TSort wrapper. Used only in restore/import group ordering. Own spec. | +| **Server** | `lib/chef/server.rb` | Read-only version detection. Clear version boundaries. Well-tested. | + +These are highlighted in green in `architecture.mmd`. + +--- + +## Production Usage + +```bash +# On Chef Automate server (Habitat) +sudo hab pkg exec chef/knife-ec-backup knife ec backup /backup/$(date +%Y%m%d) \ + --webui-key /hab/svc/automate-cs-oc-erchef/data/webui_priv.pem \ + -s https:/// \ + -c /hab/svc/automate-cs-nginx/config/knife_superuser.rb \ + --concurrency 5 + +# From remote workstation +knife ec backup ./backup \ + --webui-key webui_priv.pem \ + -s https:/// \ + -c knife_superuser.rb \ + --concurrency 1 +``` diff --git a/ai-track-docs/architecture.mmd b/ai-track-docs/architecture.mmd new file mode 100644 index 0000000..2b5a3f7 --- /dev/null +++ b/ai-track-docs/architecture.mmd @@ -0,0 +1,77 @@ +%% knife-ec-backup Architecture Diagram +%% Render with: https://mermaid.live or VS Code Mermaid extension + +graph TD + subgraph CLI["Knife CLI Framework (chef gem)"] + KNIFE[bin/knife] + end + + subgraph Commands["Knife Subcommands"] + BACKUP[EcBackup
ec_backup.rb] + RESTORE[EcRestore
ec_restore.rb] + IMPORT[EcImport
ec_import.rb] + KEYEXP[EcKeyExport
ec_key_export.rb] + KEYIMP[EcKeyImport
ec_key_import.rb] + end + + subgraph Shared["Shared Modules"] + BASE[EcBase
ec_base.rb] + KEYBASE[EcKeyBase
ec_key_base.rb] + ERRH[EcErrorHandler
ec_error_handler.rb] + SERVER[Server
server.rb] + AUTO[Automate
automate.rb] + TSORT[Tsorter
tsorter.rb] + ORGCACHE[OrgIdCache
org_id_cache.rb] + end + + subgraph External["External Dependencies"] + CHEFFS[ChefFS
parallel copy_to] + SEQUEL[Sequel
PostgreSQL] + VEIL[Veil
Secrets Store] + CONCURRENT[concurrent-ruby
ThreadPoolExecutor] + end + + subgraph Infra["Infrastructure"] + CHEFSERVER[Chef Infra Server
REST API] + POSTGRES[(PostgreSQL
opscode_chef DB)] + DISK[(Local Filesystem
Backup Directory)] + end + + KNIFE --> BACKUP + KNIFE --> RESTORE + KNIFE --> IMPORT + KNIFE --> KEYEXP + KNIFE --> KEYIMP + + BACKUP --> BASE + RESTORE --> BASE + IMPORT --> BASE + KEYEXP --> KEYBASE + KEYIMP --> KEYBASE + + BASE --> ERRH + BASE --> SERVER + BASE --> AUTO + BASE --> VEIL + IMPORT --> TSORT + RESTORE --> TSORT + + KEYBASE --> SEQUEL + KEYBASE --> AUTO + KEYBASE --> ORGCACHE + + BASE --> CHEFFS + CHEFFS --> CONCURRENT + + BACKUP --> CHEFSERVER + BACKUP --> DISK + RESTORE --> CHEFSERVER + RESTORE --> DISK + IMPORT --> CHEFSERVER + IMPORT --> DISK + KEYEXP --> POSTGRES + KEYIMP --> POSTGRES + + style ERRH fill:#90EE90,stroke:#333 + style TSORT fill:#90EE90,stroke:#333 + style SERVER fill:#90EE90,stroke:#333 diff --git a/ai-track-docs/build-test.md b/ai-track-docs/build-test.md new file mode 100644 index 0000000..7bcfbcb --- /dev/null +++ b/ai-track-docs/build-test.md @@ -0,0 +1,79 @@ +# Build & Test Guide — knife-ec-backup + +## Prerequisites + +- Ruby >= 3.1 (via rbenv/asdf) +- Bundler +- PostgreSQL client libraries (for `pg` gem compilation) +- Git + +## Setup + +```bash +# Install Ruby dependencies +bundle install + +# Verify installation +bundle exec knife ec backup --help +``` + +## Running Tests + +```bash +# Run all unit specs (excludes smoke tests) +bundle exec rake spec + +# Run a specific spec file +bundle exec rspec spec/chef/knife/ec_backup_spec.rb + +# Run with verbose output +bundle exec rspec --format documentation spec/ + +# Run only tests matching a pattern +bundle exec rspec -e "for_each_user" +``` + +## Test Coverage + +SimpleCov generates coverage reports automatically: +```bash +bundle exec rake spec +open coverage/index.html +``` + +## Linting + +No RuboCop config exists in-repo. Follow existing code style conventions. + +## Building the Habitat Package + +```bash +# Enter Habitat studio +hab studio enter + +# Build +build + +# Result: results/.hart +``` + +## Building the Gem + +```bash +gem build knife-ec-backup.gemspec +# Output: knife-ec-backup-3.0.8.gem +``` + +## Common Issues + +| Problem | Solution | +|---------|----------| +| `pg` gem fails to compile | Install PostgreSQL dev headers: `brew install libpq` or `apt install libpq-dev` | +| `veil` gem not found | It's a git dependency — ensure `bundle install` completes successfully | +| Specs fail with "ChefFS not found" | Ensure `chef ~> 18` gem is installed via bundle | + +## CI/CD + +- **Expeditor** manages automated version bumps and releases +- **Buildkite** runs specs and Habitat builds on PR +- Do NOT manually edit `VERSION` or `CHANGELOG.md` (auto-managed)