Skip to content

feat(design-data): audit generated Figma Variable names against real baseline (11k.4) - #1321

Merged
GarthDB merged 2 commits into
mainfrom
feat/figma-name-audit-11k.4
Jul 31, 2026
Merged

feat(design-data): audit generated Figma Variable names against real baseline (11k.4)#1321
GarthDB merged 2 commits into
mainfrom
feat/figma-name-audit-11k.4

Conversation

@GarthDB

@GarthDB GarthDB commented Jul 31, 2026

Copy link
Copy Markdown
Member

Description

Adds figma::audit::audit_names (sdk/core/src/figma/audit.rs) and a
design-data figma audit --snapshot <FILE> --token-dir <DIR> CLI subcommand.
It runs the existing generator (build_export_payload) offline against a
legacy token directory, then diffs the generated Figma Variable names against
a real Figma VariablesMeta snapshot, per collection:

  • matched — names that agree 1:1
  • figma_only — real variables the generator never emits (coverage gap
    within a covered collection)
  • generated_only — names the generator emits that Figma doesn't have
    (naming divergence)
  • whole collections the generator never targets at all (generator_covers: false)
  • the existing ExportSummary skip buckets (composite/alias/unknown-schema/
    unparseable-value tokens)
  • an overrides scaffold keyed by legacyKey, seeded from generated_only
    mismatches — the artifact the next step (11k.5) will consume

Regenerated the committed audit artifact
(sdk/core/tests/fixtures/figma/name-mapping.audit.json) against the 11k.3
baseline snapshot and verified two consecutive runs are byte-identical
(deterministic, no network).

Related Issue

spectrum-design-data-11k.4 (bd)

Motivation and Context

Feeds the July 31 director review pre-read with a measured status for the
Figma Variables authoring POC, replacing the previously qualitative "invert
the pipeline" framing with real coverage numbers: .Color theme 752/782
(96.2%) matched, .Platform scale 807/865 (93.3%) matched, and 4 collections
(Typography, Layout, S2.Color-theme, Iconography) — 1,304 variables —
entirely uncovered by the generator today.

How Has This Been Tested?

  • New unit test in audit.rs against a hand-built VariablesMeta mock,
    covering: an uncovered whole collection, a matched name, and a
    figma_only coverage gap within a covered collection.
  • All existing mapping.rs tests remain green, unchanged.
  • moon run sdk:codegen-check && moon run sdk:build && moon run sdk:test && moon run sdk:lint (clippy -D warnings) all pass.
  • Regenerated name-mapping.audit.json twice via the new CLI command and
    confirmed the two runs are byte-identical.
  • node tools/changeset-linter/src/cli.js check --fail-on-warnings passes.

Screenshots (if appropriate):

N/A — CLI/data change only.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • I have signed the Adobe Open Source CLA.
  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

…baseline (11k.4)

Adds `figma::audit::audit_names` and a `design-data figma audit` CLI
subcommand that diffs the generator's output against the S2-Web baseline
snapshot (11k.3), per collection, and commits the resulting audit artifact.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Jul 31, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 8f9f1c9

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 10 packages
Name Type
@adobe/spectrum-design-data Patch
@adobe/spectrum-component-api-schemas Patch
@adobe/design-system-registry Patch
@adobe/design-data-agent-mcp Patch
@adobe/design-data-mcp Patch
markdown-generator Patch
@adobe/token-corpus-migrate Patch
@adobe/token-naming-audit Patch
component-options-editor Patch
@adobe/spectrum-design-data-mcp Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

Copy link
Copy Markdown
Contributor

🎨 Token Changes Report

Tokens Changed (0)

Original Branch: main


This comment was automatically generated by the token diff tool. 🤖

@github-actions

Copy link
Copy Markdown
Contributor

🧩 Component Schema Changes Report

No component schema changes detected.

This comment was automatically generated by the component schema diff tool. 🤖

@github-actions

github-actions Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Run report for 8f9f1c99

Total time: 2m 14s | Comparison time: 2m 27s | Estimated savings: 12.7s (8.6% faster)

Action Time Status Info
⬛️ SetupProto(0.57.4) 0ms Skipped
🟩 SyncWorkspace 8.8ms Passed
🟩 SyncProject(sdk) 0.3ms Passed
⬛️ SetupToolchain(node:20.17.0) 684.8ms Skipped
⬛️ SetupToolchain(rust:1.88.0) 856.4ms Skipped
🟩 SetupEnvironment(rust, sdk) 92.9ms Passed
⬛️ InstallDependencies(rust, sdk) 2.2ms Skipped
⬛️ SetupToolchain(pnpm:10.17.1) 657.7ms Skipped
⬛️ SetupToolchain(javascript) 0ms Skipped
⬛️ SetupEnvironment(javascript, sdk) 0.6ms Skipped
🟩 RunTask(sdk:fmt-check) 484.2ms Passed
🟩 InstallDependencies(javascript, sdk) 5.7s Passed
🟩 RunTask(sdk:codegen-check) 180.5ms Passed
🟩 RunTask(sdk:lint) 35.9s Passed
🟩 RunTask(sdk:test) 2m 7s Passed SLOW
Changed files
.changeset/figma-name-audit.md
sdk/Cargo.lock
sdk/README.md
sdk/cli/src/main.rs
sdk/core/src/figma/audit.rs
sdk/core/src/figma/mod.rs
sdk/core/tests/fixtures/figma/name-mapping.audit.json

@GarthDB GarthDB left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overview

Adds an offline design-data figma audit --snapshot <FILE> --token-dir <DIR> CLI command and a new sdk/core/src/figma/audit.rs module (audit_names). It runs the existing generator (build_export_payload) against a legacy token directory and diffs the resulting variable names, per collection, against a real captured Figma VariablesMeta snapshot — reporting matched / figma_only (coverage gaps) / generated_only (naming divergences) counts, plus a legacyKey → override scaffold seeded from every generated_only name, feeding the follow-on 11k.5 work. Includes a committed ~3200-line JSON artifact from running the audit against the 11k.3 baseline fixture.

Code quality

  • Clean reuse of existing primitives (build_export_payload, summarize_variables, VariableAction naming convention) rather than reimplementing generation logic — no duplicated logic.
  • The real_by_collection union-by-name handling (comment at audit.rs:199-206) correctly guards against a real subtlety: two Figma collections can share a display name (local + remote-library stub), and HashMap iteration order is non-deterministic, so overwriting instead of unioning would make results flaky. Good catch, well-documented.
  • overrides scaffold keys off legacy_key via split_once('/') on the {prefix}/{legacyKey} convention — correctly matches make_variable_action's naming in mapping.rs.
  • Copyright header, module doc comment, and CLI help text all follow repo conventions.

Issues / things to verify

  1. Test coverage gap: the single unit test (uncovered_collection_and_coverage_gap_are_reported) never exercises the overrides field — the legacyKey scaffold that's the actual point of the 11k.5 handoff. Worth a small assertion that a generated_only name populates overrides with the correct stripped key.
  2. Cross-collection key collision (minor, likely theoretical): overrides is a single BTreeMap<String, String> keyed only by legacy_key, built across both .Color theme and .Platform scale collections. If the same legacy key ever produced a generated_only name in both collections, one would silently overwrite the other via .entry(...).or_default() (no-op on the second insert, but no signal either way). Not currently a bug given non-overlapping token namespaces, but worth a comment noting the assumption if it isn't already checked elsewhere.
  3. Documentation: sdk/README.md documents figma read and figma export (lines ~157-158) but the new figma audit subcommand isn't added there. Minor, but the CLI surface is now inconsistent with the README's command list.
  4. Fixture size: the ~3205-line name-mapping.audit.json is a large committed artifact for a PR; that seems intentional per the changeset (captures the baseline audit for 11k.5 to consume), so likely fine, but worth confirming it isn't meant to be regenerated per-run instead of hand-committed.

Correctness

  • Compared against mapping.rs and types.rs on disk — build_export_payload's error path (returns FigmaError::Api if .Color theme/.Platform scale aren't found) means audit_names will hard-fail rather than silently misreport if a snapshot lacks those collections — acceptable since a real snapshot always has them.
  • Changeset passes changeset-linter --fail-on-warnings (verified locally).
  • Recommend confirming cargo test -p design-data-core figma::audit is green in CI before merge (didn't finish compiling locally in review session; logic reads correct from inspection).

Risk

Low — additive, offline-only CLI command with no changes to existing export/read paths. No security concerns (no network calls, reads local files only).

- sdk/core/src/figma/audit.rs: cover the overrides scaffold in the existing
  unit test (a generated_only name now seeds its stripped legacyKey), and
  document the cross-collection key-collision assumption on the overrides
  map.
- sdk/README.md: document the figma audit subcommand alongside read/export.

Addresses review comments on PR #1321.
@GarthDB

GarthDB commented Jul 31, 2026

Copy link
Copy Markdown
Member Author

Pushed fixes for the review feedback (8f9f1c9):

  • Added test coverage for the overrides scaffold (a generated_only name now asserts it seeds overrides with the stripped legacyKey).
  • Documented the cross-collection key-collision assumption inline on the overrides map.
  • Added figma audit to sdk/README.md alongside read/export.
  • Confirmed cargo test -p design-data-core --lib --features figma and cargo clippy --features figma -p design-data-core -p design-data-cli -- -D warnings both pass clean.

Left issue 4 (fixture size) as-is per your own note — it's the intentional committed baseline artifact for 11k.5 to consume, not meant to be regenerated per-run.

PR #1322 (11k.5) has been rebased on top of this fix.

@GarthDB
GarthDB merged commit b879343 into main Jul 31, 2026
9 checks passed
@GarthDB
GarthDB deleted the feat/figma-name-audit-11k.4 branch July 31, 2026 05:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant