Hand Wave

Sign recognition for web and iOS, backed by a small Python inference service.

Apps

apps/web        TanStack Start frontend
apps/inference FastAPI inference service
apps/mobile    SwiftUI iOS client

Clients capture landmarks. The inference service loads the recognition model and decodes predictions.

Setup

Requirements: pnpm 10.33+, Node 22+, Python 3.11/3.12, uv, and moon.

cp .env.example .env
pnpm install
uv sync --project apps/inference

Set VITE_INFERENCE_URL in .env for local web builds.

Development

pnpm dev

Useful direct targets:

moon run web:dev
moon run inference:dev

Web runs on http://localhost:3000; inference docs run on http://localhost:8000/docs.

Inference

The API accepts client-owned landmark windows:

Method	Path	Use
POST	/v1/predict	Decode one landmark window
POST	/v1/recognize	Decode and smooth streaming output

The local checkpoint is apps/inference/models/best.ckpt. It must match the model class, config, vocab order, landmark layout, and preprocessing code.

Quality

pnpm check
pnpm check:affected
pnpm fix
pnpm test

License

MIT. See LICENSE.md.

Research Credits

• MiCT-RANet for ASL Fingerspelling, a prior ASL fingerspelling reference by Florent Mahoudeau.
• A Two-Stream Neural Network for Pose-Based Handshape Recognition in American Sign Language, the MiCT-RANet paper.
• Mixed 3D/2D Convolutional Tube for Human Action Recognition, the MiCT architecture paper.
• Fingerspelling Recognition in the Wild with Iterative Visual Attention, the ChicagoFSWild+ paper.
• FSBoard: Over 3 million characters of ASL fingerspelling collected via smartphones, a large mobile fingerspelling dataset paper.