🔘 Sift

Transform Passive Consumption into Active Mastery

An AI-powered active recall engine that turns any content into a precision-engineered learning experience.

Features • Architecture • Getting Started • Documentation

📖 What is Sift?

Sift is a minimalist active recall engine designed for students, professionals, and lifelong learners who want to truly master what they study—not just passively consume it. Built on the science of active recall and spaced repetition, Sift transforms your study materials into an engaging, flow-state learning experience.

The Philosophy

"Friction is the Enemy" — The time between "I have content" and "I'm being quizzed" should be less than 10 seconds.

Sift prioritizes:

🎯 Grounded Intelligence: AI generates questions strictly from your content—no hallucinations
⚡ Zero Friction: Upload → Generate → Learn in seconds
🧘 Zen Mode: Distraction-free, flow-state study sessions
📱 Mobile-First: Progressive Web App that works everywhere
🎨 Beautiful Design: Clean, dark-mode-first interface that reduces eye strain

✨ Features

Core Functionality

📤 Multi-Format Content Ingestion

Upload PDFs, Markdown, or text files
Paste text directly or URLs for instant processing
Future: Audio transcription and video content support

🤖 AI-Powered Quiz Generation

Automatically generates questions from your content using Google's Gemini AI
Multiple question formats:
- Flashcards for recall practice
- Multiple Choice for recognition testing
- Socratic Questions for deep understanding
Configurable depth: Shallow (key concepts) vs. Deep (detailed coverage)

🎓 Learning Paths

Create structured, long-term learning goals (e.g., "Master React")
AI generates progressive curriculum based on your progress
Tracks what you've learned to suggest next steps
Continuity across multiple study sessions

📊 Mastery Tracking (Echoes)

Track performance per topic across all sources
Visual "knowledge heatmap" showing strengths and weaknesses
Intelligent prioritization of weak areas in future quizzes
Spaced repetition system to prevent knowledge decay

🔐 Secure Authentication

Google OAuth integration via Better-Auth
Email/password authentication
Session management and protected routes

💾 The Vault

Save all your sources and study materials
View history of all completed "Sifts" (study sessions)
Track scores and progress over time
Archive old materials while keeping your knowledge graph

User Experience

Keyboard-Driven Interface: Fast shortcuts for power users
Gesture Support: Intuitive swipe controls on mobile
Instant Feedback: See source snippets for wrong answers
Source Anchoring: Every answer links back to your original material
Progressive Web App: Install on any device, works offline
Dark Mode First: Reduced eye strain for long study sessions

🏗️ Architecture

Tech Stack

Frontend

Next.js 16 - React 19, Server Components, Server Actions
TypeScript 5 - Full type safety
Tailwind CSS 4 - Utility-first styling
shadcn/ui - Beautiful, accessible components
Framer Motion - Smooth animations and transitions
Radix UI - Accessible component primitives

Backend & Data

Drizzle ORM - Type-safe database queries
PostgreSQL - Primary database
Better-Auth - Modern authentication
Vercel AI SDK - AI integration layer

AI & Content Processing

Google Gemini - Question generation
Firebase - File storage and real-time features
Cloudflare R2 - Object storage

Development Tools

Turborepo - Monorepo build system
Biome - Fast linter and formatter
pnpm - Fast, disk-efficient package manager

Project Structure

sift/
├── apps/
│   └── web/                    # Main Next.js application
│       ├── src/
│       │   ├── app/            # Next.js App Router
│       │   │   ├── api/        # API routes
│       │   │   ├── dashboard/  # User dashboard
│       │   │   ├── sift/       # Quiz interface
│       │   │   ├── learn/      # Learning paths
│       │   │   └── explore/    # Content exploration
│       │   ├── components/     # React components
│       │   │   ├── blocks/     # Landing page sections
│       │   │   ├── media/      # Media components
│       │   │   └── ui/         # shadcn components
│       │   ├── hooks/          # Custom React hooks
│       │   └── lib/            # Utilities and helpers
│       └── public/             # Static assets
│
├── packages/
│   ├── auth/                   # Authentication configuration
│   │   └── src/
│   │       ├── index.ts        # Better-Auth setup
│   │       └── types/          # Auth types
│   │
│   ├── db/                     # Database layer
│   │   └── src/
│   │       ├── schema/         # Drizzle schema definitions
│   │       │   ├── auth.ts     # User & session tables
│   │       │   ├── sources.ts  # Content sources
│   │       │   ├── sifts.ts    # Quiz sessions
│   │       │   ├── echoes.ts   # Knowledge tracking
│   │       │   ├── flashcards.ts
│   │       │   └── learning-paths.ts
│   │       ├── queries/        # Database query functions
│   │       └── types/          # Type definitions
│   │
│   ├── env/                    # Environment variable validation
│   └── config/                 # Shared TypeScript configs
│
├── docs/                       # Documentation
│   ├── soul.md                 # Product vision (v1)
│   ├── soul_v2.md              # Updated product vision
│   ├── learning_system_plan.md # Technical planning
│   └── future.md               # Future enhancements
│
├── turbo.json                  # Turborepo configuration
├── pnpm-workspace.yaml         # Workspace configuration
├── biome.json                  # Linting & formatting rules
└── package.json                # Root package configuration

Database Schema

Core Tables

sources - User-uploaded content

- id: string (primary key)
- userId: string (foreign key)
- title: string
- type: 'pdf' | 'text' | 'url' | 'audio' | 'markdown' | 'json'
- content: text (extracted content)
- metadata: json (page count, duration, etc.)
- timestamps: createdAt, updatedAt

sifts - Generated quiz sessions

- id: string (primary key)
- userId: string (foreign key)
- sourceId: string (foreign key)
- status: 'in_progress' | 'completed' | 'abandoned'
- config: json (depth, format settings)
- takeaways: json[] (key learnings)
- timestamps: createdAt, updatedAt

questions - Quiz questions

- id: string (primary key)
- siftId: string (foreign key)
- sectionId: string (optional foreign key)
- question: string
- answer: string (correct answer)
- options: json (for MCQs)
- explanation: text (source context)
- tags: string[] (topics covered)

echoes - Knowledge tracking

- id: string (primary key)
- userId: string (foreign key)
- sourceId: string (foreign key)
- topic: string
- masteryLevel: int (0-100)
- lastReviewedAt: timestamp
- Unique constraint: (userId, sourceId, topic)

learning_paths - Long-term learning goals

- id: string (primary key)
- userId: string (foreign key)
- title: string
- goal: string (user's original prompt)
- summary: text (AI progress summary)
- timestamps: createdAt, updatedAt

sift_sessions - Session tracking

- id: string (primary key)
- siftId: string (foreign key)
- userId: string (foreign key)
- status: 'in_progress' | 'completed'
- score: int
- startedAt, completedAt: timestamp

🚀 Getting Started

Prerequisites

Node.js 20+ (LTS recommended)
pnpm 10.28.0+
PostgreSQL 14+ database
Google Cloud Account (for Gemini AI API)

Installation

Clone the repository

git clone https://github.com/Vashishta-Mithra-Reddy/sift.git
cd sift

Install dependencies
```
pnpm install
```

Set up environment variables

Create apps/web/.env file:

# Database
DATABASE_URL="postgresql://user:password@localhost:5432/sift"
DATABASE_URL_UNPOOLED="postgresql://user:password@localhost:5432/sift"

# Authentication
BETTER_AUTH_SECRET="your-secret-key-min-32-chars"
BETTER_AUTH_URL="http://localhost:3001"
GOOGLE_OAUTH_CLIENT_ID="your-google-oauth-client-id"
GOOGLE_OAUTH_CLIENT_SECRET="your-google-oauth-secret"

# AI
GOOGLE_GENERATIVE_AI_API_KEY="your-gemini-api-key"

# Storage (Optional - for file uploads)
CLOUDFLARE_R2_ACCOUNT_ID="your-r2-account-id"
CLOUDFLARE_R2_ACCESS_KEY_ID="your-access-key"
CLOUDFLARE_R2_SECRET_ACCESS_KEY="your-secret-key"
CLOUDFLARE_R2_BUCKET_NAME="sift-uploads"
NEXT_PUBLIC_R2_URL="https://your-bucket-url.r2.dev"

Set up the database

# Push schema to database
pnpm run db:push

# Or run migrations
pnpm run db:migrate

Start the development server
```
pnpm run dev
```
Open the application

Navigate to http://localhost:3001

Quick Setup with Docker (Optional)

# Start PostgreSQL with Docker
docker run -d \
  --name sift-postgres \
  -e POSTGRES_PASSWORD=password \
  -e POSTGRES_DB=sift \
  -p 5432:5432 \
  postgres:16

📚 Documentation

Available Scripts

Command	Description
`pnpm run dev`	Start all applications in development mode
`pnpm run dev:web`	Start only the web application
`pnpm run build`	Build all applications for production
`pnpm run check-types`	Type-check all packages
`pnpm run check`	Run Biome linter and formatter
`pnpm run db:push`	Push schema changes to database
`pnpm run db:studio`	Open Drizzle Studio (database GUI)
`pnpm run db:generate`	Generate new migration files
`pnpm run db:migrate`	Run pending migrations

Key API Routes

Endpoint	Method	Description
`/api/ai/generate`	POST	Generate study plan or questions
`/api/sift/process`	POST	Process uploaded content
`/api/sift/[id]/status`	GET	Get sift status and results
`/api/notifications/action`	POST	Handle push notifications
`/api/auth/[...all]`	ALL	Better-Auth routes

Environment Variables Reference

Required Variables

DATABASE_URL - PostgreSQL connection string (pooled)
DATABASE_URL_UNPOOLED - PostgreSQL connection string (direct)
BETTER_AUTH_SECRET - Secret key for auth (min 32 chars)
BETTER_AUTH_URL - Base URL of your application
GOOGLE_GENERATIVE_AI_API_KEY - Gemini API key

OAuth Variables

GOOGLE_OAUTH_CLIENT_ID - Google OAuth client ID
GOOGLE_OAUTH_CLIENT_SECRET - Google OAuth client secret

Storage Variables (Optional)

CLOUDFLARE_R2_ACCOUNT_ID - Cloudflare R2 account ID
CLOUDFLARE_R2_ACCESS_KEY_ID - R2 access key
CLOUDFLARE_R2_SECRET_ACCESS_KEY - R2 secret key
CLOUDFLARE_R2_BUCKET_NAME - R2 bucket name
CLOUDFLARE_R2_ENDPOINT - R2 endpoint URL
NEXT_PUBLIC_R2_URL - Public R2 URL

🔧 Development

Code Style

This project uses Biome for linting and formatting:

# Check and auto-fix issues
pnpm run check

# The configuration follows these rules:
# - Tabs for indentation
# - Double quotes for strings
# - Organized imports
# - Sorted Tailwind classes

Database Workflow

# Make changes to schema files in packages/db/src/schema/

# Generate migration
pnpm run db:generate

# Apply migration
pnpm run db:migrate

# Or push directly (for development)
pnpm run db:push

# Explore data with GUI
pnpm run db:studio

Turborepo Caching

Turborepo caches build outputs for faster subsequent builds:

Build outputs: .next/, dist/
Cache location: .turbo/
Cache is local by default (can be configured for remote caching)

🚢 Deployment

Vercel (Recommended)

Connect your GitHub repository to Vercel
Configure environment variables in project settings
Set build command: pnpm run build
Set output directory: apps/web/.next
Deploy 🚀

Docker Deployment

# Coming soon - Dockerfile for containerized deployment

Database Hosting

Recommended PostgreSQL hosting providers:

Neon - Serverless Postgres
Supabase - Open source Firebase alternative
Vercel Postgres - Integrated with Vercel
Railway - Simple infrastructure

🗺️ Roadmap

v2.0 - The Foundation ✅ (Current)

Authentication system (Google OAuth + Email)
Responsive PWA layout
Content ingestion (text, files)
AI-powered quiz generation
Basic mastery tracking (Echoes)
Polished quiz interface

v2.1 - The Deep Dive 🔄 (In Progress)

Learning Paths implementation
Robust PDF/DOCX extraction
Detailed performance analytics
Spaced repetition scheduling
Knowledge heatmap visualization

v2.2 - The Community 📅 (Planned)

Public Sift Packs (shareable study materials)
Community-contributed content
Social learning features
Mobile app (React Native)

v2.3 - Advanced Features 🔮 (Future)

🤝 Contributing

Contributions are welcome! Please follow these guidelines:

Fork the repository
Create a feature branch: git checkout -b feature/amazing-feature
Make your changes and follow the code style
Run linting: pnpm run check
Test your changes thoroughly
Commit: git commit -m 'Add amazing feature'
Push: git push origin feature/amazing-feature
Open a Pull Request

Development Guidelines

Follow the existing code style (Biome enforces this)
Write meaningful commit messages
Update documentation for significant changes
Add comments for complex logic
Test on both desktop and mobile viewports

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

Built with Better-T-Stack
Inspired by the science of active recall and spaced repetition
UI components from shadcn/ui
Icons from HugeIcons

📞 Contact & Support

GitHub Issues: Report bugs or request features
Discussions: Join the conversation

⬆ back to top

Made with ❤️ by Vashishta Mithra Reddy

Name		Name	Last commit message	Last commit date
Latest commit History 45 Commits
.trae		.trae
.vscode		.vscode
apps/web		apps/web
docs		docs
packages		packages
.gitignore		.gitignore
README.md		README.md
biome.json		biome.json
bts.jsonc		bts.jsonc
package.json		package.json
pnpm-lock.yaml		pnpm-lock.yaml
pnpm-workspace.yaml		pnpm-workspace.yaml
tsconfig.json		tsconfig.json
turbo.json		turbo.json

Vashishta-Mithra-Reddy/sift

Folders and files

Latest commit

History

Repository files navigation