Export and AI Improvements, code reorganization

.claude/commands/clerk-migration.md

@@ -0,0 +1,60 @@
+---
+description: Unified entry point for the Clerk user migration tool. Routes to the right skill based on your task (export, migrate, create transformer, or Clerk-to-Clerk migration).
+---
+
+Help the user with their Clerk user migration task.
+
+## Workflow
+
+### Step 1: Verify Environment
+
+1. Check if dependencies are installed. If not, run `bun install`.
+2. Check if `.env` exists with `CLERK_SECRET_KEY`. If missing, ask for the key (https://dashboard.clerk.com/~/api-keys) and create/update `.env`.
+
+### Step 2: Identify task type
+
+Analyze $ARGUMENTS to determine the task:
+
+| User wants to...                                 | Skill to use           |
+| ------------------------------------------------ | ---------------------- |
+| Export users from a source platform              | `/export`              |
+| Import/migrate users into Clerk from a file      | `/migrate`             |
+| Create a custom transformer for unsupported data | `/transformer`         |
+| Move users between Clerk instances (dev → prod)  | `/clerk-migration`     |
+| Not sure / general help                          | Show the options below |
+
+If the task is unclear, present the options:
+
+> I can help with these migration tasks:
+>
+> 1. **Export users** — Export from Auth0, AuthJS, Better Auth, Clerk, Firebase, or Supabase (`/export`)
+> 2. **Migrate users** — Import users into Clerk from a JSON/CSV file (`/migrate`)
+> 3. **Create a transformer** — Generate a custom transformer for unsupported data formats (`/transformer`)
+> 4. **Clerk-to-Clerk migration** — Move users between Clerk instances, e.g. dev → prod (`/clerk-migration`)
+>
+> What would you like to do?
+
+### Step 3: Load the appropriate skill
+
+Based on the identified task, load the skill:
+
+- **Export**: `skill({ name: 'export' })`
+- **Migrate**: `skill({ name: 'migrate' })`
+- **Transformer**: `skill({ name: 'transformer' })`
+- **Clerk-to-Clerk**: `skill({ name: 'clerk-migration' })`
+
+### Step 4: Execute task
+
+Follow the loaded skill's instructions to complete the user's request.
+
+### Step 5: Summarize
+
+After the task completes, summarize what was done:
+
+- Number of users exported/migrated
+- Any errors or warnings from the logs
+- Suggested next steps (e.g., "run `bun migrate`" after an export)
+
+<user-request>
+$ARGUMENTS
+</user-request>

.claude/skills/clerk-migration/SKILL.md

@@ -0,0 +1,7 @@
+---
+name: clerk-migration
+description: Migrate users between Clerk instances (development to production or instance to instance). Use when user wants to move users from one Clerk instance to another.
+user-invocable: true
+---
+
+!`cat prompts/clerk-migration-prompt.md`

.claude/skills/export/SKILL.md

@@ -0,0 +1,7 @@
+---
+name: export
+description: Export users from authentication platforms (Auth0, AuthJS, Better Auth, Clerk, Firebase, Supabase) to JSON. Use when user wants to export users from their current auth provider.
+user-invocable: true
+---
+
+!`cat prompts/export-prompt.md`

.cursor/rules/agents.mdc

@@ -0,0 +1,7 @@
+---
+description: Project conventions and architecture for the Clerk user migration tool
+globs:
+alwaysApply: true
+---
+
+Read and follow the project instructions in `AGENTS.md` at the repository root. That file contains the project overview, structure, common commands, architecture, and implementation notes.

.cursor/rules/clerk-migration.mdc

@@ -0,0 +1,13 @@
+---
+description: Migrate users between Clerk instances (development to production or instance to instance). Use when user wants to move users from one Clerk instance to another.
+globs:
+alwaysApply: false
+---
+
+Read and follow the instructions in `prompts/clerk-migration-prompt.md` for Clerk-to-Clerk migration.
+
+Typical flow:
+1. Verify environment and detect instance type
+2. Export from source instance (`bun export:clerk`)
+3. Switch `.env` to destination key
+4. Import into destination (`bun migrate -y --transformer clerk --file exports/clerk-export.json`)

.cursor/rules/export.mdc

@@ -0,0 +1,11 @@
+---
+description: Export users from authentication platforms (Auth0, AuthJS, Better Auth, Clerk, Firebase, Supabase) to JSON. Use when user wants to export users from their current auth provider.
+globs:
+alwaysApply: false
+---
+
+Read and follow the instructions in `prompts/export-prompt.md` for the complete export workflow.
+
+Key commands:
+- `bun export` — Interactive platform picker
+- `bun export:auth0` / `bun export:authjs` / `bun export:betterauth` / `bun export:clerk` / `bun export:firebase` / `bun export:supabase` — Direct export

.cursor/rules/migrate.mdc

@@ -0,0 +1,11 @@
+---
+description: Run user migration to Clerk from various authentication platforms (Auth0, Supabase, Firebase, AuthJS, Clerk). Use when user wants to import, migrate, or load users from a data file (JSON/CSV).
+globs:
+alwaysApply: false
+---
+
+Read and follow the instructions in `prompts/migration-prompt.md` for the complete migration workflow.
+
+Key commands:
+- `bun migrate` — Interactive migration
+- `bun migrate -y -t <transformer> -f <file>` — Non-interactive migration

.cursor/rules/transformer.mdc

@@ -0,0 +1,13 @@
+---
+description: Generate custom Clerk user transformers from sample data. Use when user needs to create a new transformer for an unsupported platform or custom data format.
+globs:
+alwaysApply: false
+---
+
+Read and follow the instructions in `prompts/transformer-prompt.md` for transformer generation.
+
+Key steps:
+1. Analyze sample user data to identify field mappings
+2. Create transformer file in `src/transformers/`
+3. Register in `src/transformers/registry.ts`
+4. Run `bun run test` to verify

.env.example

@@ -1,10 +1,20 @@
 # ============================================================================
 # REQUIRED: Clerk Secret Key
 # ============================================================================
-# Get your secret key from the Clerk Dashboard: https://dashboard.clerk.com
+# Get your secret key from the Clerk Dashboard: https://dashboard.clerk.com/~/api-keys
 # Format: sk_test_... (development) or sk_live_... (production)
 CLERK_SECRET_KEY=sk_
 
+# ============================================================================
+# OPTIONAL: Clerk Publishable Key
+# ============================================================================
+# Enables automatic Dashboard configuration checking during migration.
+# The tool will cross-reference your Clerk settings (enabled identifiers,
+# password policy, etc.) against the import file and warn about mismatches.
+# Format: pk_test_... (development) or pk_live_... (production)
+# CLERK_PUBLISHABLE_KEY=
+# NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=
+
 # ============================================================================
 # OPTIONAL: Rate Limit Override
 # ============================================================================
@@ -14,7 +24,7 @@ CLERK_SECRET_KEY=sk_
 # - Production (sk_live_*): 100 requests/second (Clerk limit: 1000 req/10s)
 # - Development (sk_test_*): 10 requests/second (Clerk limit: 100 req/10s)
 #
-# Only set this if you need to reduce the rate for safety or testing, or if 
+# Only set this if you need to reduce the rate for safety or testing, or if
 # have a rate limit exception and can increase the speed
 # Example: RATE_LIMIT=50
 # RATE_LIMIT=
@@ -41,3 +51,59 @@ CLERK_SECRET_KEY=sk_
 # - CONCURRENCY_LIMIT=5   (slower, ~50 req/s, very safe)
 # CONCURRENCY_LIMIT=
 
+# ============================================================================
+# OPTIONAL: Supabase Export
+# ============================================================================
+# PostgreSQL connection string for exporting users from Supabase.
+# Find this in the Supabase Dashboard under Connect.
+# Format: postgresql://postgres:[PASSWORD]@db.[REF].supabase.co:5432/postgres
+# SUPABASE_DB_URL=
+
+# ============================================================================
+# OPTIONAL: Supabase OAuth Provider Detection
+# ============================================================================
+# Used during Supabase migration to cross-reference which OAuth providers
+# are enabled in Supabase and warn about unsupported providers in Clerk.
+# SUPABASE_URL=
+# NEXT_PUBLIC_SUPABASE_URL=
+# SUPABASE_ANON_KEY=
+# NEXT_PUBLIC_SUPABASE_ANON_KEY=
+# SUPABASE_SERVICE_ROLE_KEY=
+
+# ============================================================================
+# OPTIONAL: Auth0 Export
+# ============================================================================
+# Credentials for exporting users from Auth0 via the Management API.
+# Requires a Machine-to-Machine application with the read:users scope.
+# AUTH0_DOMAIN=my-tenant.us.auth0.com
+# AUTH0_CLIENT_ID=
+# AUTH0_CLIENT_SECRET=
+
+# ============================================================================
+# OPTIONAL: Firebase Export
+# ============================================================================
+# Path to Firebase service account JSON key file.
+# Download from Firebase Console → Project Settings → Service Accounts.
+# GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
+
+# ============================================================================
+# OPTIONAL: AuthJS Export
+# ============================================================================
+# Database connection string for exporting users from an AuthJS database.
+# Supports PostgreSQL, MySQL, and SQLite.
+# Examples:
+#   postgresql://user:password@host:5432/database
+#   mysql://user:password@host:3306/database
+#   /path/to/database.sqlite
+# AUTHJS_DB_URL=
+
+# ============================================================================
+# OPTIONAL: Better Auth Export
+# ============================================================================
+# Database connection string for exporting users from a Better Auth database.
+# Supports PostgreSQL, MySQL, and SQLite.
+# Examples:
+#   postgresql://user:password@host:5432/database
+#   mysql://user:password@host:3306/database
+#   /path/to/database.sqlite
+# BETTER_AUTH_DB_URL=

.gitignore

@@ -5,6 +5,7 @@ package-lock.json
 yarn.lock
 pnpm-lock.yaml
 logs
+exports
 test-convert-logs
 tmp/
 testing/

AGENTS.md

@@ -13,30 +13,55 @@ src/
 ├── clean-logs/          # Log cleanup utility
 ├── convert-logs/        # NDJSON to JSON converter
 ├── delete/              # User deletion functionality
+├── export/              # User export (Auth0, AuthJS, Better Auth, Clerk, Firebase, Supabase)
+│   ├── index.ts         # Entry point (platform dispatcher)
+│   ├── registry.ts      # Export registry (array-based, like transformers)
+│   ├── auth0.ts         # Auth0 export (Management API → JSON)
+│   ├── authjs.ts        # AuthJS export (DB query → JSON, supports PG/MySQL/SQLite)
+│   ├── betterauth.ts    # Better Auth export (DB query → JSON, supports PG/MySQL/SQLite)
+│   ├── clerk.ts         # Clerk export (API → JSON)
+│   ├── firebase.ts      # Firebase export (Admin SDK → JSON)
+│   └── supabase.ts      # Supabase export (Postgres → JSON)
+├── lib/                 # Shared utilities and helpers
+│   ├── index.ts         # General utils (file paths, tryCatch, transformKeys, etc.)
+│   ├── db.ts            # Database abstraction (PostgreSQL, MySQL, SQLite)
+│   ├── export.ts        # Shared export utils (coverage display, file writing, DB error hints)
+│   ├── settings.ts      # Settings persistence (loadSettings, saveSettings)
+│   ├── analysis.ts      # User data analysis (analyzeFields, validateUsers)
+│   ├── supabase.ts      # Supabase provider analysis (fetchSupabaseProviders, etc.)
+│   └── clerk.ts         # Clerk API helpers (detectInstanceType, fetchClerkConfig)
 ├── migrate/             # Main migration logic
 │   ├── cli.ts           # Interactive CLI
 │   ├── functions.ts     # Data loading and transformation
 │   ├── import-users.ts  # User creation with Clerk API
 │   ├── index.ts         # Entry point
 │   └── validator.ts     # Zod schema validation
 ├── transformers/        # Platform-specific transformers
+│   ├── index.ts         # Re-exports from registry
+│   ├── registry.ts      # Transformer registry (array-based, like exports)
 │   ├── auth0.ts
 │   ├── authjs.ts
+│   ├── betterauth.ts
 │   ├── clerk.ts
 │   ├── firebase.ts
-│   ├── supabase.ts
-│   └── index.ts
+│   └── supabase.ts
 ├── envs-constants.ts    # Environment configuration
 ├── logger.ts            # NDJSON logging
-├── types.ts             # TypeScript types
-└── utils.ts             # Shared utilities
+└── types.ts             # TypeScript types
 ```
 
 ## Common Commands
 
 ### Development Commands
 
 - `bun migrate` - Start the migration process (interactive CLI)
+- `bun export` - Export users (interactive platform picker)
+- `bun export:auth0` - Export users from Auth0 tenant
+- `bun export:authjs` - Export users from AuthJS database (PostgreSQL, MySQL, or SQLite)
+- `bun export:betterauth` - Export users from Better Auth database (PostgreSQL, MySQL, or SQLite)
+- `bun export:clerk` - Export users from Clerk instance
+- `bun export:firebase` - Export users from Firebase project
+- `bun export:supabase` - Export users from Supabase database
 - `bun delete` - Delete all migrated users (uses externalId to identify users)
 - `bun clean-logs` - Remove all log files from the `./logs` folder
 - `bun convert-logs` - Convert NDJSON log files to JSON array format for easier analysis
@@ -73,7 +98,7 @@ The migration tool uses a **transformer pattern** to support different source pl
 1. **Field Transformer**: Maps source platform fields to Clerk's schema
    - Example: Auth0's `_id.$oid` → Clerk's `userId`
    - Example: Supabase's `encrypted_password` → Clerk's `password`
-   - Handles nested field flattening (see `flattenObjectSelectively` in `src/migrate/functions.ts`)
+   - Handles nested field flattening (see `flattenObjectSelectively` in `src/lib/index.ts`)
 
 2. **Optional Default Fields**: Applied to all users from that platform
    - Example: Supabase defaults `passwordHasher` to `"bcrypt"`
@@ -88,10 +113,27 @@ The migration tool uses a **transformer pattern** to support different source pl
 
 **Adding a new transformer**:
 
-1. Create a new file in `src/transformers/` with transformer config
-2. Export it in `src/transformers/index.ts`
+1. Create a new file in `src/transformers/` with a transformer config satisfying `TransformerRegistryEntry`
+2. Import and register it in `src/transformers/registry.ts`
 3. The CLI will automatically include it in the platform selection
 
+### Export System
+
+The export tool (`src/export/`) exports users from various platforms to JSON files compatible with the migration tool.
+
+**Architecture**:
+
+- `registry.ts` — Registry of available exports (array-based, like transformers)
+- `index.ts` — CLI entry point / dispatcher (reads from registry)
+- `src/lib/export.ts` — Shared utilities (coverage display, file writing, DB error hints)
+- `[platform].ts` — Platform-specific export logic
+
+**Adding a new export**:
+
+1. Create `src/export/[platform].ts` with an export function, display summary, and CLI wrapper (`runXxxExport`)
+2. Register in `src/export/registry.ts`
+3. Add `"export:[platform]"` script to `package.json`
+
 ### Data Flow
 
 ```
@@ -148,7 +190,7 @@ The tool uses **p-limit for concurrency control** across all API calls.
 
 - If a 429 occurs, uses Retry-After value from API response
 - Falls back to 10 second default if Retry-After not available
-- Centralized in `getRetryDelay()` function in `src/utils.ts`
+- Centralized in `getRetryDelay()` function in `src/lib/index.ts`
 - Automatically retries up to 5 times (configurable via MAX_RETRIES)
 
 ### Logging System
@@ -157,19 +199,21 @@ All operations create timestamped logs in `./logs/` using NDJSON (Newline-Delimi
 
 - `{timestamp}-migration.log` - Combined log with all import entries
 - `{timestamp}-user-deletion.log` - Combined log with all deletion entries
+- `{timestamp}-export.log` - Combined log with all export entries
 
 **Log Entry Types** (defined in `src/types.ts`):
 
 - `ImportLogEntry` - Success/error for user imports
 - `DeleteLogEntry` - Success/error for user deletions
+- `ExportLogEntry` - Success/error for user exports
 - `ValidationErrorPayload` - Validation failures with path and row
 - `ErrorLog` - Additional identifier errors
 
 ### Error Handling
 
 The codebase uses a consistent error handling pattern:
 
-- `tryCatch()` utility (in `src/utils.ts`) - Returns `[result, error]` (error is null on success)
+- `tryCatch()` utility (in `src/lib/index.ts`) - Returns `[result, error]` (error is null on success)
 - Used extensively to make additional emails/phones non-fatal
 - Rate limit errors (429) trigger automatic retry with delay
 - Validation errors are logged but don't stop the migration
@@ -218,3 +262,4 @@ The tool auto-detects instance type from `CLERK_SECRET_KEY`:
 - [docs/creating-transformers.md](docs/creating-transformers.md) - Transformer development guide
 - [prompts/migration-prompt.md](prompts/migration-prompt.md) - AI prompt for running migrations
 - [prompts/transformer-prompt.md](prompts/transformer-prompt.md) - AI prompt for generating transformers
+- [prompts/export-prompt.md](prompts/export-prompt.md) - AI prompt for exporting users