diff --git a/util/Seeder/Seeds/README.md b/util/Seeder/Seeds/README.md index e9c72eef2e55..9338281b18bf 100644 --- a/util/Seeder/Seeds/README.md +++ b/util/Seeder/Seeds/README.md @@ -1,13 +1,23 @@ # Seeds -Hand-crafted JSON fixtures for Bitwarden Seeder test data. +Hand-crafted JSON fixtures and preset configurations for Bitwarden Seeder test data. ## Quick Start -1. Create a JSON file in the right `fixtures/` subfolder -2. Add the `$schema` line — your editor picks up validation automatically +1. Pick a preset from the catalog below +2. Run: `dotnet run -- seed --preset {name} --mangle` 3. Build to verify: `dotnet build util/Seeder/Seeder.csproj` +## Presets + +Presets wire everything together: org + roster + ciphers. Organized by purpose: + +| Folder | Purpose | CLI prefix | Example | +|--------|---------|------------|---------| +| `features/` | Test specific Bitwarden features (SSO, TDE, policies) | `features.` | `--preset features.sso-enterprise` | +| `qa/` | Handcrafted fixture data for visual UI verification | `qa.` | `--preset qa.enterprise-basic` | +| `validation/` | Algorithm verification for seeder development | `validation.` | `--preset validation.density-modeling-power-law-test` | + ## Writing Fixtures ### Organizations @@ -43,18 +53,6 @@ Vault items. Each item needs a `type` and `name`. See: `fixtures/ciphers/enterprise-basic.json` -### Presets - -Presets **wire everything together**: org + roster + ciphers. You can reference fixtures by name or generate data with counts. - -Three styles: - -- **Fixture-based**: `enterprise-basic.json` — references org, roster, and cipher fixtures -- **Generated**: `wonka-teams-small.json` — uses `count` parameters to create users, groups, collections, ciphers -- **Feature-specific**: `tde-enterprise.json`, `policy-enterprise.json` — adds SSO config, policies - -Presets can also define inline orgs (name + domain right in the preset) instead of referencing a fixture — see `large-enterprise.json`. - ## Naming Conventions | Element | Pattern | Example | @@ -72,36 +70,20 @@ Your editor validates against `$schema` automatically — errors show up as red dotnet build util/Seeder/Seeder.csproj ``` -## QA Test Fixture Migration Matrix - -These Seeds consolidate test data previously found across the `bitwarden/test` repo. -The table below maps existing QA fixtures to their Seeder equivalents. - -| QA Source (`test/Bitwarden.Web.Tests/TestData/SetupData/`) | Used By | Seeder Preset | Org Fixture | Roster Fixture | Cipher Fixture | -| ---------------------------------------------------------- | --------------------------------- | ----------------------------------- | ------------------- | ------------------------ | ------------------------ | -| `CollectionPermissionsOrg.json` | Web, Extension | `collection-permissions-enterprise` | `cobalt-logistics` | `collection-permissions` | `collection-permissions` | -| `EnterpriseOrg.json` | Web, Extension, Android, iOS, CLI | `enterprise-basic` | `redwood-analytics` | `enterprise-basic` | `enterprise-basic` | -| `SsoOrg.json` | Web | `sso-enterprise` | `verdant-health` | `starter-team` | `sso-vault` | -| `TDEOrg.json` | Web, Extension, Android, iOS | `tde-enterprise` | `obsidian-labs` | `starter-team` | `tde-vault` | -| _(Confluence: Policy Org guide)_ | QA manual setup | `policy-enterprise` | `pinnacle-designs` | `starter-team` | — | -| `FamiliesOrg.json` | Web, Extension | `families-basic` | `adams-family` | `family` | — | - -### Not Yet Migrated +## QA Migration -| QA Source | Used By | Status | -| --------------------- | ---------------------------- | --------------------------------------------------------------------- | -| `FreeAccount.json` | All 7 platforms | Planned — `free-personal-vault` preset (separate PR due to file size) | -| `PremiumAccount.json` | Web, Extension, Android, iOS | Planned — `premium-personal-vault` preset | -| `SecretsManager.json` | Web | Planned — `secrets-manager-enterprise` preset | -| `FreeOrg.json` | Web | Planned — `free-org-basic` preset | +Mapping from legacy QA test fixtures to seeder presets: -### Additional Sources +| Legacy Source | Seeder Preset | +|--------------|---------------| +| `CollectionPermissionsOrg.json` | `qa.collection-permissions-enterprise` | +| `EnterpriseOrg.json` | `qa.enterprise-basic` | +| `SsoOrg.json` | `features.sso-enterprise` | +| `TDEOrg.json` | `features.tde-enterprise` | +| Policy Org (Confluence) | `features.policy-enterprise` | +| `FamiliesOrg.json` | `qa.families-basic` | -| Source | Location | Status | -| ---------------------------------- | ------------------------------- | ---------------------------------------------------------------------------------------------------------------- | -| `bw_importer.py` | `github.com/bitwarden/qa-tools` | Superseded by generation-based presets (`"ciphers": {"count": N}`) | -| `mass_org_manager.py` | `github.com/bitwarden/qa-tools` | Superseded by roster fixtures with groups/members/collections | -| Admin Console Testing Setup guides | Confluence QA space | Codified as `collection-permissions-enterprise`, `policy-enterprise`, `sso-enterprise`, `tde-enterprise` presets | +**Planned:** `qa.free-personal-vault`, `qa.premium-personal-vault`, `features.secrets-manager-enterprise`, `qa.free-org-basic` ## Security diff --git a/util/Seeder/Seeds/fixtures/presets/policy-enterprise.json b/util/Seeder/Seeds/fixtures/presets/features/policy-enterprise.json similarity index 82% rename from util/Seeder/Seeds/fixtures/presets/policy-enterprise.json rename to util/Seeder/Seeds/fixtures/presets/features/policy-enterprise.json index 7177d7312d6b..dc32fa637f65 100644 --- a/util/Seeder/Seeds/fixtures/presets/policy-enterprise.json +++ b/util/Seeder/Seeds/fixtures/presets/features/policy-enterprise.json @@ -1,5 +1,5 @@ { - "$schema": "../../schemas/preset.schema.json", + "$schema": "../../../schemas/preset.schema.json", "organization": { "fixture": "pinnacle-designs", "planType": "enterprise-annually" diff --git a/util/Seeder/Seeds/fixtures/presets/sso-enterprise.json b/util/Seeder/Seeds/fixtures/presets/features/sso-enterprise.json similarity index 87% rename from util/Seeder/Seeds/fixtures/presets/sso-enterprise.json rename to util/Seeder/Seeds/fixtures/presets/features/sso-enterprise.json index 7c8c041c2b65..ee64d55d19a8 100644 --- a/util/Seeder/Seeds/fixtures/presets/sso-enterprise.json +++ b/util/Seeder/Seeds/fixtures/presets/features/sso-enterprise.json @@ -1,5 +1,5 @@ { - "$schema": "../../schemas/preset.schema.json", + "$schema": "../../../schemas/preset.schema.json", "organization": { "fixture": "verdant-health", "planType": "enterprise-annually", diff --git a/util/Seeder/Seeds/fixtures/presets/tde-enterprise.json b/util/Seeder/Seeds/fixtures/presets/features/tde-enterprise.json similarity index 87% rename from util/Seeder/Seeds/fixtures/presets/tde-enterprise.json rename to util/Seeder/Seeds/fixtures/presets/features/tde-enterprise.json index 7fe9e75fbe7c..426f8c15c912 100644 --- a/util/Seeder/Seeds/fixtures/presets/tde-enterprise.json +++ b/util/Seeder/Seeds/fixtures/presets/features/tde-enterprise.json @@ -1,5 +1,5 @@ { - "$schema": "../../schemas/preset.schema.json", + "$schema": "../../../schemas/preset.schema.json", "organization": { "fixture": "obsidian-labs", "planType": "enterprise-annually", diff --git a/util/Seeder/Seeds/fixtures/presets/large-enterprise.json b/util/Seeder/Seeds/fixtures/presets/large-enterprise.json deleted file mode 100644 index 7adc49a0588b..000000000000 --- a/util/Seeder/Seeds/fixtures/presets/large-enterprise.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "$schema": "../../schemas/preset.schema.json", - "organization": { - "name": "Globex Corp", - "domain": "globex.example", - "seats": 10000 - }, - "users": { - "count": 5000, - "realisticStatusMix": true - }, - "groups": { - "count": 100 - }, - "collections": { - "count": 200 - }, - "ciphers": { - "count": 50000 - } -} diff --git a/util/Seeder/Seeds/fixtures/presets/collection-permissions-enterprise.json b/util/Seeder/Seeds/fixtures/presets/qa/collection-permissions-enterprise.json similarity index 82% rename from util/Seeder/Seeds/fixtures/presets/collection-permissions-enterprise.json rename to util/Seeder/Seeds/fixtures/presets/qa/collection-permissions-enterprise.json index d3866ea5cbb5..dbfd3a16e0d4 100644 --- a/util/Seeder/Seeds/fixtures/presets/collection-permissions-enterprise.json +++ b/util/Seeder/Seeds/fixtures/presets/qa/collection-permissions-enterprise.json @@ -1,5 +1,5 @@ { - "$schema": "../../schemas/preset.schema.json", + "$schema": "../../../schemas/preset.schema.json", "organization": { "fixture": "cobalt-logistics", "planType": "enterprise-annually", diff --git a/util/Seeder/Seeds/fixtures/presets/dunder-mifflin-enterprise-full.json b/util/Seeder/Seeds/fixtures/presets/qa/dunder-mifflin-enterprise-full.json similarity index 80% rename from util/Seeder/Seeds/fixtures/presets/dunder-mifflin-enterprise-full.json rename to util/Seeder/Seeds/fixtures/presets/qa/dunder-mifflin-enterprise-full.json index 7e0b8f93d70c..908dcec17e9d 100644 --- a/util/Seeder/Seeds/fixtures/presets/dunder-mifflin-enterprise-full.json +++ b/util/Seeder/Seeds/fixtures/presets/qa/dunder-mifflin-enterprise-full.json @@ -1,5 +1,5 @@ { - "$schema": "../../schemas/preset.schema.json", + "$schema": "../../../schemas/preset.schema.json", "organization": { "fixture": "dunder-mifflin", "planType": "enterprise-annually", diff --git a/util/Seeder/Seeds/fixtures/presets/enterprise-basic.json b/util/Seeder/Seeds/fixtures/presets/qa/enterprise-basic.json similarity index 81% rename from util/Seeder/Seeds/fixtures/presets/enterprise-basic.json rename to util/Seeder/Seeds/fixtures/presets/qa/enterprise-basic.json index d8102e7b83ca..62e8925b5bba 100644 --- a/util/Seeder/Seeds/fixtures/presets/enterprise-basic.json +++ b/util/Seeder/Seeds/fixtures/presets/qa/enterprise-basic.json @@ -1,5 +1,5 @@ { - "$schema": "../../schemas/preset.schema.json", + "$schema": "../../../schemas/preset.schema.json", "organization": { "fixture": "redwood-analytics", "planType": "enterprise-annually", diff --git a/util/Seeder/Seeds/fixtures/presets/families-basic.json b/util/Seeder/Seeds/fixtures/presets/qa/families-basic.json similarity index 84% rename from util/Seeder/Seeds/fixtures/presets/families-basic.json rename to util/Seeder/Seeds/fixtures/presets/qa/families-basic.json index 0e90d72990aa..35aa3c803a50 100644 --- a/util/Seeder/Seeds/fixtures/presets/families-basic.json +++ b/util/Seeder/Seeds/fixtures/presets/qa/families-basic.json @@ -1,5 +1,5 @@ { - "$schema": "../../schemas/preset.schema.json", + "$schema": "../../../schemas/preset.schema.json", "organization": { "fixture": "adams-family", "planType": "families-annually", diff --git a/util/Seeder/Seeds/fixtures/presets/stark-free-basic.json b/util/Seeder/Seeds/fixtures/presets/qa/stark-free-basic.json similarity index 86% rename from util/Seeder/Seeds/fixtures/presets/stark-free-basic.json rename to util/Seeder/Seeds/fixtures/presets/qa/stark-free-basic.json index dbbf9491931c..5456efe0330d 100644 --- a/util/Seeder/Seeds/fixtures/presets/stark-free-basic.json +++ b/util/Seeder/Seeds/fixtures/presets/qa/stark-free-basic.json @@ -1,5 +1,5 @@ { - "$schema": "../../schemas/preset.schema.json", + "$schema": "../../../schemas/preset.schema.json", "organization": { "fixture": "stark-industries", "planType": "free", diff --git a/util/Seeder/Seeds/fixtures/presets/validation/README.md b/util/Seeder/Seeds/fixtures/presets/validation/README.md deleted file mode 100644 index b4693febcc41..000000000000 --- a/util/Seeder/Seeds/fixtures/presets/validation/README.md +++ /dev/null @@ -1,153 +0,0 @@ -# Density Modeling Validation Presets - -These presets validate that the Seeder's density distribution algorithms produce correct relationship patterns. Run them, query the DB, and compare against the expected results below. - -Always use the `--mangle` flag to avoid collisions with existing data. - -## Verification Queries - -Run the first query to get the Organization ID, then paste it into the remaining queries. - -### Find the Organization ID - -```sql -SELECT Id, [Name] -FROM [dbo].[Organization] WITH (NOLOCK) -WHERE [Name] = 'PASTE_ORG_NAME_HERE'; -``` - -### Group Membership Distribution - -```sql -DECLARE @OrgId UNIQUEIDENTIFIER = 'PASTE_ORG_ID_HERE'; - -SELECT - G.[Name], - COUNT(GU.OrganizationUserId) AS Members -FROM [dbo].[Group] G WITH (NOLOCK) -LEFT JOIN [dbo].[GroupUser] GU WITH (NOLOCK) ON G.Id = GU.GroupId -WHERE G.OrganizationId = @OrgId -GROUP BY G.[Name] -ORDER BY Members DESC; -``` - -### CollectionGroup Count - -```sql -DECLARE @OrgId UNIQUEIDENTIFIER = 'PASTE_ORG_ID_HERE'; - -SELECT COUNT(*) AS CollectionGroupCount -FROM [dbo].[CollectionGroup] CG WITH (NOLOCK) -INNER JOIN [dbo].[Collection] C WITH (NOLOCK) ON CG.CollectionId = C.Id -WHERE C.OrganizationId = @OrgId; -``` - -### Permission Distribution - -```sql -DECLARE @OrgId UNIQUEIDENTIFIER = 'PASTE_ORG_ID_HERE'; - -SELECT - 'CollectionUser' AS [Source], - COUNT(*) AS Total, - SUM(CASE WHEN CU.ReadOnly = 1 THEN 1 ELSE 0 END) AS ReadOnly, - SUM(CASE WHEN CU.Manage = 1 THEN 1 ELSE 0 END) AS Manage, - SUM(CASE WHEN CU.HidePasswords = 1 THEN 1 ELSE 0 END) AS HidePasswords, - SUM(CASE WHEN CU.ReadOnly = 0 AND CU.Manage = 0 AND CU.HidePasswords = 0 THEN 1 ELSE 0 END) AS ReadWrite -FROM [dbo].[CollectionUser] CU WITH (NOLOCK) -INNER JOIN [dbo].[OrganizationUser] OU WITH (NOLOCK) ON CU.OrganizationUserId = OU.Id -WHERE OU.OrganizationId = @OrgId -UNION ALL -SELECT - 'CollectionGroup', - COUNT(*), - SUM(CASE WHEN CG.ReadOnly = 1 THEN 1 ELSE 0 END), - SUM(CASE WHEN CG.Manage = 1 THEN 1 ELSE 0 END), - SUM(CASE WHEN CG.HidePasswords = 1 THEN 1 ELSE 0 END), - SUM(CASE WHEN CG.ReadOnly = 0 AND CG.Manage = 0 AND CG.HidePasswords = 0 THEN 1 ELSE 0 END) -FROM [dbo].[CollectionGroup] CG WITH (NOLOCK) -INNER JOIN [dbo].[Collection] C WITH (NOLOCK) ON CG.CollectionId = C.Id -WHERE C.OrganizationId = @OrgId; -``` - -### Orphan Ciphers - -```sql -DECLARE @OrgId UNIQUEIDENTIFIER = 'PASTE_ORG_ID_HERE'; - -SELECT - COUNT(*) AS TotalCiphers, - SUM(CASE WHEN CC.CipherId IS NULL THEN 1 ELSE 0 END) AS Orphans -FROM [dbo].[Cipher] CI WITH (NOLOCK) -LEFT JOIN (SELECT DISTINCT CipherId FROM [dbo].[CollectionCipher] WITH (NOLOCK)) CC - ON CI.Id = CC.CipherId -WHERE CI.OrganizationId = @OrgId; -``` - ---- - -## Presets - -### 1. Power-Law Distribution - -Tests skewed group membership, CollectionGroup generation, permission distribution, and cipher orphans. - -```bash -cd util/SeederUtility -dotnet run -- seed --preset validation.density-modeling-power-law-test --mangle -``` - -| Check | Expected | -| ----------------- | -------------------------------------------------------------------------------------- | -| Groups | 10 groups. First has ~50 members, decays to 1. Last 2 have 0 members (20% empty rate). | -| CollectionGroups | > 0 records. First collections have more groups assigned (PowerLaw fan-out). | -| Permissions | ~50% ReadOnly, ~30% ReadWrite, ~15% Manage, ~5% HidePasswords. | -| Orphan ciphers | ~50 of 500 (10% orphan rate). | -| DirectAccessRatio | 0.6 — roughly 60% of access paths are direct CollectionUser. | - -### 2. MegaGroup Distribution - -Tests one dominant group with all-group access (no direct CollectionUser). - -```bash -cd util/SeederUtility -dotnet run -- seed --preset validation.density-modeling-mega-group-test --mangle -``` - -| Check | Expected | -| ---------------- | ------------------------------------------------------------------------ | -| Groups | 5 groups. Group 1 has ~90 members (90.5%). Groups 2-5 split ~10 members. | -| CollectionUsers | 0 records. DirectAccessRatio is 0.0 — all access via groups. | -| CollectionGroups | > 0. First 10 collections get 3 groups (FrontLoaded), rest get 1. | -| Permissions | 25% each for ReadOnly, ReadWrite, Manage, HidePasswords (even split). | - -### 3. Empty Groups - -Tests that EmptyGroupRate produces memberless groups excluded from CollectionGroup assignment. - -```bash -cd util/SeederUtility -dotnet run -- seed --preset validation.density-modeling-empty-groups-test --mangle -``` - -| Check | Expected | -| ----------------- | ---------------------------------------------------------------------------------- | -| Groups | 10 groups total. 5 with ~10 members each, 5 with 0 members (50% empty). | -| CollectionGroups | Only reference the 5 non-empty groups. Run `SELECT DISTINCT CG.GroupId` to verify. | -| DirectAccessRatio | 0.5 — roughly half of users get direct CollectionUser records. | - -### 4. No Density (Baseline) - -Confirms backward compatibility. No `density` block = original round-robin behavior. - -```bash -cd util/SeederUtility -dotnet run -- seed --preset validation.density-modeling-no-density-test --mangle -``` - -| Check | Expected | -| ---------------- | ---------------------------------------------------------------------------------------- | -| Groups | 5 groups with ~10 members each (uniform round-robin). | -| CollectionGroups | 0 records. No density = no CollectionGroup generation. | -| Permissions | First assignment per user is Manage, subsequent are ReadOnly (original cycling pattern). | -| Orphan ciphers | 0. Every cipher assigned to at least one collection. | diff --git a/util/Seeder/Seeds/fixtures/presets/wonka-teams-small.json b/util/Seeder/Seeds/fixtures/presets/wonka-teams-small.json deleted file mode 100644 index e805409eca20..000000000000 --- a/util/Seeder/Seeds/fixtures/presets/wonka-teams-small.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "$schema": "../../schemas/preset.schema.json", - "organization": { - "fixture": "wonka-confections", - "planType": "teams-annually", - "seats": 25 - }, - "users": { - "count": 25, - "realisticStatusMix": true - }, - "groups": { - "count": 8 - }, - "collections": { - "count": 15 - }, - "folders": true, - "ciphers": { - "count": 250, - "assignFolders": true - }, - "personalCiphers": { - "countPerUser": 25 - } -}