feat: Marketplace integration -- read marketplace.json for plugin discovery + governance

[Copilot]

Description

APM now reads existing marketplace.json files (both Copilot CLI and Claude Code formats) for plugin discovery, resolves plugins to Git URLs, then applies its full governance layer (lockfile, SHA pinning, audit trail). Collapses a two-tool workflow into one.

New module: src/apm_cli/marketplace/

• models.py -- Frozen dataclasses (MarketplaceSource, MarketplacePlugin, MarketplaceManifest) + JSON parser for both Copilot CLI and Claude Code formats
• errors.py -- Actionable error hierarchy with next-step commands in messages
• registry.py -- CRUD for ~/.apm/marketplaces.json with process-lifetime cache, atomic writes
• client.py -- GitHub Contents API fetch via AuthResolver.try_with_fallback(unauth_first=True), 1h TTL cache, stale-while-revalidate, auto-detect marketplace.json location
• resolver.py -- NAME@MARKETPLACE regex detection + resolution for 4 source types (github, url, git-subdir, relative; npm rejected with clear message)

Install hook (install.py)

• Pre-parse intercept before DependencyReference.parse() detects NAME@MARKETPLACE via ^[a-zA-Z0-9._-]+@[a-zA-Z0-9._-]+$ (no /, no :)
• Resolves to canonical owner/repo#ref, replaces package variable, stores provenance
• Backward-compatible grammar extension -- these inputs previously raised ValueError

CLI commands (commands/marketplace.py + cli.py)

• apm marketplace add/list/browse/update/remove
• apm search QUERY (top-level, across all registered marketplaces)

Lockfile provenance (lockfile.py)

• Two optional fields on LockedDependency: discovered_via, marketplace_plugin_name
• Backward compatible -- None by default, omitted from YAML when unset

# Register a marketplace
apm marketplace add acme-org/plugin-marketplace

# Browse and install
apm marketplace browse acme-plugins
apm install security-checks@acme-plugins
# -> Resolves to acme-org/security-plugin#v1.3.0, full governance applies

Type of change

• Bug fix
• New feature
• Documentation
• Maintenance / refactor

Testing

• Tested locally
• All existing tests pass
• Added tests for new functionality (if applicable)

114 unit tests across 8 files: models, resolver (20 regex positive/negative cases + 4 source types), client (cache TTL, stale-while-revalidate, auto-detect), registry (CRUD + persistence), CLI commands (CliRunner), lockfile provenance (round-trip + backward compat), install integration, error hierarchy. Full suite: 3295 passed.

[Copilot]

Pull request overview

This PR adds a first-class "marketplace" feature to APM: it can discover plugins from marketplace.json (Copilot CLI + Claude Code formats), resolve them into Git dependencies, and then apply the existing governance flow (locking, SHA pinning, provenance).

Changes:

• Introduces src/apm_cli/marketplace/ (models/parser, registry persistence, GitHub fetch + caching, and resolver for NAME@MARKETPLACE).
• Adds CLI surface area: apm marketplace add/list/browse/update/remove and top-level apm search.
• Extends install + lockfile to track marketplace provenance (discovered_via, marketplace_plugin_name) and documents the workflow.

Reviewed changes

Copilot reviewed 23 out of 24 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
tests/unit/marketplace/init.py	Adds marketplace unit test package.
tests/unit/marketplace/test_lockfile_provenance.py	Tests new lockfile provenance fields serialization/back-compat.
tests/unit/marketplace/test_marketplace_client.py	Tests marketplace fetch/caching behavior and auto-detection logic.
tests/unit/marketplace/test_marketplace_commands.py	Tests new CLI commands using Click CliRunner.
tests/unit/marketplace/test_marketplace_errors.py	Tests actionable marketplace error messages.
tests/unit/marketplace/test_marketplace_install_integration.py	Tests marketplace ref detection and _ValidationOutcome provenance field presence.
tests/unit/marketplace/test_marketplace_models.py	Tests dataclasses and parsing for both marketplace.json formats.
tests/unit/marketplace/test_marketplace_registry.py	Tests registry CRUD + persistence and corrupted file handling.
tests/unit/marketplace/test_marketplace_resolver.py	Tests NAME@MARKETPLACE regex and source-type resolution.
src/apm_cli/cli.py	Registers marketplace command group and top-level search.
src/apm_cli/commands/install.py	Adds marketplace pre-parse intercept; threads provenance into the install engine and lockfile generation.
src/apm_cli/commands/marketplace.py	Implements apm marketplace ... and apm search commands.
src/apm_cli/core/command_logger.py	Extends _ValidationOutcome to carry marketplace_provenance.
src/apm_cli/deps/lockfile.py	Adds provenance fields to LockedDependency + (de)serialization.
src/apm_cli/marketplace/init.py	Exposes marketplace public API symbols.
src/apm_cli/marketplace/client.py	Implements GitHub Contents API fetch, TTL cache, and path auto-detection.
src/apm_cli/marketplace/errors.py	Adds marketplace-specific exception hierarchy with next-step guidance.
src/apm_cli/marketplace/models.py	Adds frozen dataclasses + parser for Copilot/Claude marketplace.json.
src/apm_cli/marketplace/registry.py	Adds persistent registry in ~/.apm/marketplaces.json with caching + atomic writes.
src/apm_cli/marketplace/resolver.py	Adds NAME@MARKETPLACE parsing and plugin source-to-canonical resolution.
docs/src/content/docs/guides/marketplaces.md	New guide page describing marketplace concepts and workflows.
docs/src/content/docs/guides/plugins.md	Updates plugin discovery guidance to include marketplaces.
docs/src/content/docs/reference/cli-commands.md	Documents new commands and NAME@MARKETPLACE install syntax.
CHANGELOG.md	Adds Unreleased "Added" entries describing the feature.

[danielmeppiel]

Addressed all 12 review comments in commit e814a88:

Code fixes:

1. install.py L148 -- Narrowed except Exception to except ImportError so only missing marketplace module is silenced; real errors propagate.
2. install.py L2262 -- Fixed provenance key mismatch: provenance dict is now keyed by identity (owner/repo) instead of canonical (owner/repo#ref), matching how lockfile.dependencies is keyed.
3. resolver.py L90 -- _resolve_git_subdir_source() now includes subdir in the resolved string (e.g., owner/repo/packages/plugin-a#ref), with path traversal validation rejecting `.." segments.
4. resolver.py L98 -- _resolve_relative_source() now includes the relative path as a virtual subdir (e.g., owner/repo/plugins/my-plugin), with path traversal validation.
5. client.py L48 -- Added _sanitize_cache_name() that strips non-[a-zA-Z0-9._-] characters from marketplace names before use in file paths, preventing directory traversal.
6. marketplaces.md L128 -- Updated docs: "stale-while-revalidate" changed to "stale-if-error" to accurately describe the blocking fetch + expired-cache fallback behavior.
7. CHANGELOG.md L15 -- Consolidated 3 entries into a single line with (#503) per Keep a Changelog convention.
8. models.py L109 -- Removed unused _SUPPORTED_SOURCE_TYPES set.
9. client.py L173 -- Removed try/except from _auto_detect_path() entirely. Since _fetch_file() already returns None for 404s, any MarketplaceFetchError is a non-404 failure (auth, network) and should propagate.
10. marketplace.py L57 -- Added name validation in add command: display names must match [a-zA-Z0-9._-]+ for compatibility with plugin@marketplace syntax.
11. cli-commands.md L871 -- Changed example from --name "Acme Plugins" to --name acme-plugins.
12. marketplaces.md L65 -- Same fix as fix: pass env dict to setup_runtime_environment to preserve CI tokens #11.

Tests: Updated resolver test expectations to match corrected behavior, added path traversal rejection tests. All 3314 unit tests pass.

[danielmeppiel]

E2E Testing Against Real Marketplaces (commit ac1afbb)

Tested all marketplace commands against 3 real-world marketplaces:

Marketplace	Plugins	Format	Path
github/awesome-copilot	62	Copilot CLI	.github/plugin/marketplace.json
anthropics/skills	3	Claude Code	.claude-plugin/marketplace.json
microsoft/azure-skills	1	Claude Code	.claude-plugin/marketplace.json

Results: 25/25 PASS (after fixes)

Phase	Tests	Status
Registration (add/list/name validation)	6/6	All PASS
Browse & Search (cross-marketplace)	6/6	All PASS
Install (NAME@MARKETPLACE)	5/5	All PASS (after 2 bug fixes)
Update & Remove	4/4	All PASS
Error Cases	3/3	All PASS

Bugs Found & Fixed

Bug #1 (P1) — Copilot CLI format incompatibility: awesome-copilot uses "source": "github" (not "type": "github") and "path" (not "subdir") in its source dicts. Parser now handles both key names with normalization. Affects 8/62 plugins that use external github sources.

Bug #2 (P2) — Provenance never written: apm install NAME@MARKETPLACE passed raw marketplace refs as only_packages, which DependencyReference.parse() couldn't parse, causing identity filter to remove all deps (deps_to_install empty → "already installed"). Fix: use validated_packages (canonical strings) instead of raw click args. Lockfile now correctly shows discovered_via and marketplace_plugin_name.

Verified Post-Fix

# apm.lock.yaml now contains:
discovered_via: awesome-copilot
marketplace_plugin_name: azure

Full report: WIP/e2e-marketplace-test-report.md

[danielmeppiel]

E2E Marketplace Testing Report

Ran 25 test scenarios against 3 official marketplaces using the branch binary built from source.

Marketplaces Tested

Marketplace	Repo	Format	Plugins
Awesome Copilot	github/awesome-copilot	Copilot CLI ("source" key)	62 plugins
Anthropic Skills	anthropics/skills	Claude Code (relative sources)	3 plugins
Azure Skills	microsoft/azure-skills	Claude Code (relative sources)	1 plugin

Commands Tested & Results

Phase 1: Setup

#	Command	Result
1	apm init	PASS
2	apm marketplace list (empty state)	PASS

Phase 2: Registration (apm marketplace add)

#	Command	Result
3	apm marketplace add awesome-copilot github/awesome-copilot	PASS -- auto-detected .github/plugin/marketplace.json
4	apm marketplace add skills anthropics/skills	PASS -- auto-detected .claude-plugin/marketplace.json
5	apm marketplace add azure-skills microsoft/azure-skills	PASS -- auto-detected .claude-plugin/marketplace.json
6	apm marketplace list (3 registered)	PASS
7	apm marketplace add awesome-copilot github/awesome-copilot (duplicate)	PASS -- clear error
8	apm marketplace add bad ../traversal (invalid name)	PASS -- rejected

Phase 3: Browse & Search (apm marketplace browse/search)

#	Command	Result
9	apm marketplace browse awesome-copilot	PASS -- 62 plugins listed
10	apm marketplace browse skills	PASS -- 3 plugins listed
11	apm marketplace browse azure-skills	PASS -- 1 plugin listed
12	apm marketplace search azure (cross-marketplace)	PASS -- results from multiple marketplaces
13	apm marketplace search nonexistent-xyz	PASS -- empty results, no crash
14	apm marketplace browse unknown	PASS -- clear error message

Phase 4: Install (apm install NAME@MARKETPLACE)

#	Command	Result
15	apm install azure@azure-skills	PASS -- installed, provenance in lockfile
16	apm install sequential-thinking@skills	PASS -- installed from Anthropic
17	apm install azure@awesome-copilot	PASS -- Copilot CLI format resolved correctly
18	apm install unknown@skills	PASS -- clear "not found" error
19	apm install thing@nonexistent	PASS -- clear "marketplace not registered" error

Phase 5: Update & Remove (apm marketplace remove)

#	Command	Result
20	apm marketplace remove azure-skills	PASS
21	apm marketplace list (2 remaining)	PASS
22	apm marketplace remove azure-skills (already removed)	PASS -- clear error
23	apm marketplace remove skills && apm marketplace remove awesome-copilot	PASS

Phase 6: Lockfile & Provenance Verification

#	Command	Result
24	Verify apm.lock contains discovered_via and marketplace_plugin_name	PASS
25	Verify provenance keyed by dependency identity (not canonical)	PASS

Summary: 25/25 PASS

Two bugs were discovered during initial testing and fixed in commit ac1afbb:

1. Copilot CLI format compatibility -- awesome-copilot uses "source": "github" + "path" keys instead of "type" + "subdir". Parser now handles both formats.
2. Provenance not written to lockfile -- raw NAME@MARKETPLACE args could not be parsed by DependencyReference.parse(), causing identity mismatch. Fixed by using validated canonical package strings.

Both fixes have unit tests. Full suite: 3320 tests passing.