Field LUTs + sync job + remove raw_data

[skyfallwastaken]

Should - in theory at least - lead to some good perf wins?

There's a migration doc we'd need to follow after too

(please please please make a backup before merging this)

[Copilot]

Pull request overview

This PR implements a database normalization migration for heartbeat data by introducing lookup tables (LUTs) for dimensions and removing the raw_data JSONB column to improve performance.

Changes:

• Creates 8 lookup tables for heartbeat dimensions (languages, categories, editors, operating systems, user agents, projects, branches, machines)
• Adds foreign key columns to the heartbeats table with corresponding indexes
• Implements a dual-write system via feature flags to populate both old string columns and new FK columns during migration
• Provides backfill job and rake tasks to populate FK columns for existing heartbeats

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
db/migrate/20260202194401_create_heartbeat_lookup_tables.rb	Creates the 8 lookup tables with unique indexes
db/migrate/20260202194406_add_lookup_foreign_keys_to_heartbeats.rb	Adds FK columns and indexes to heartbeats table
db/migrate/20260202194410_add_foreign_key_constraints_to_heartbeats.rb	Adds unvalidated foreign key constraints
db/schema.rb	Reflects all database schema changes including new tables, columns, and indexes
app/models/heartbeats/*.rb	Defines model classes for all lookup tables
app/models/concerns/heartbeat_dimension_resolver.rb	Implements dimension resolution logic for both single records and batch operations
app/models/heartbeat.rb	Integrates dimension resolver concern
app/jobs/backfill_heartbeat_dimensions_job.rb	Background job to backfill FK columns for existing heartbeats
app/jobs/migrate_user_from_hackatime_job.rb	Updated to conditionally skip raw_data and use batch dimension resolution
app/services/heartbeat_import_service.rb	Updated to conditionally skip raw_data and use batch dimension resolution
lib/tasks/backfill_heartbeat_dimensions.rake	Rake tasks to enqueue backfill jobs and check progress
docs/heartbeat_normalization_migration.md	Migration guide with step-by-step instructions

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.