Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions android/app/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,18 @@ android {
versionCode flutterVersionCode.toInteger() + 10000
versionNameSuffix "-wear"
}
xr {
dimension "device"
// Jetpack XR SDK / AI-glasses devices are Android 14+; pinning
// to 34 keeps the eventual androidx.xr:* artefacts resolvable.
// Refine once the SDK's minSdk is published.
minSdk 34
// Separate versionCode band so Play accepts phone, wear and xr
// AABs for the same applicationId in one release.
// Phone = base, wear = +10000, xr = +20000.
versionCode flutterVersionCode.toInteger() + 20000
versionNameSuffix "-xr"
}
}

signingConfigs {
Expand Down Expand Up @@ -110,6 +122,19 @@ dependencies {
phoneImplementation 'com.google.firebase:firebase-functions'
phoneImplementation 'com.google.firebase:firebase-firestore'
phoneImplementation 'com.google.android.gms:play-services-location:21.3.0'
// Android XR / AI-glasses surface — Kotlin + Compose Glimmer. Scoped to
// the xr flavour so the phone/wear AABs do not link these libraries.
// Uncomment and pin versions when the first XR activity lands; the SDK
// is still alpha and exact coordinates have not been published yet.
// xrImplementation 'androidx.xr.compose:compose:1.0.0-alpha01'
// xrImplementation 'androidx.xr.scenecore:scenecore:1.0.0-alpha01'
// xrImplementation 'androidx.xr.arcore:arcore:1.0.0-alpha01'
// xrImplementation 'org.jetbrains.kotlinx:kotlinx-coroutines-android:1.8.1'
// xrImplementation platform('com.google.firebase:firebase-bom:33.7.0')
// xrImplementation 'com.google.firebase:firebase-auth'
// xrImplementation 'com.google.firebase:firebase-functions'
// xrImplementation 'com.google.firebase:firebase-firestore'
// xrImplementation 'com.google.android.gms:play-services-location:21.3.0'
testImplementation 'junit:junit:4.13.2'
androidTestImplementation 'androidx.test:runner:1.4.0'
androidTestImplementation 'androidx.test.espresso:espresso-core:3.4.0'
Expand Down
11 changes: 11 additions & 0 deletions android/app/src/xr/AndroidManifest.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
<manifest xmlns:android="http://schemas.android.com/apk/res/android">

<!-- Postbox Game on Android XR AI-glasses. This source set is a
scaffold only; no activities are registered yet. A follow-up PR
will add the Compose Glimmer HUD activity, the Firestore-backed
StatsRepository (mirroring src/phone/.../car/), and the voice
controller. -->
<uses-feature
android:name="android.software.xr.immersive"
android:required="true" />
</manifest>
187 changes: 187 additions & 0 deletions docs/smart_glasses_integration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,187 @@
# Smart-glasses (Android XR) integration — feasibility & design

> **Status: exploration.** Nothing in this document ships behaviour. The
> companion changes in this PR (the `xr` product flavour and an empty
> `android/app/src/xr/AndroidManifest.xml`) reserve the build scaffolding
> so a follow-up PR can drop in the first real activity without reshaping
> Gradle first.

## Goal

Postbox Game is a "look around the real world, spot postboxes, claim
them" game. Smart glasses are a near-perfect form factor for the loop:
the player is already walking, already looking, and the **fuzzy
compass** (the deliberately-imprecise direction hint described in
`CLAUDE.md`) maps naturally onto a translucent HUD ring.

The smart-glasses experience we want to explore:

- Walk down a street wearing the glasses.
- A faint compass ring overlay shows red sectors where unclaimed
postboxes lie and grey sectors for already-claimed ones — **no pins,
no distances**, same fuzziness guarantee as the phone app.
- Postman James whispers a non-sequitur in the player's ear when the
player glances toward a sector: "Something Victorian-ish over that
way, I reckon."
- Saying "claim" while standing near a postbox runs the same
`startScoring` flow as the phone — quiz prompt is read aloud,
response captured by ASR.
- Standard stats ("what's my streak?", "who's leading today?") are
voice-answered against the same Firestore collections the phone and
Wear OS clients read.

## SDK summary

Reference: <https://developer.android.com/develop/xr/jetpack-xr-sdk/ai-glasses/build>

The Jetpack XR AI-glasses SDK is **Kotlin + Jetpack Compose Glimmer
only**. There is no Flutter engine on the glasses surface, so this
cannot be a Dart entrypoint like `lib/main_wear.dart`.

Relevant artefact families (exact coordinates not yet pinned by Google
— deps are commented out in `android/app/build.gradle` until the SDK
leaves alpha):

| Family | Purpose |
| --- | --- |
| `androidx.xr.compose` | Spatial UI with Compose Glimmer (HUD cards, chips, lists) |
| `androidx.xr.scenecore` | 3D entities, spatial audio cues |
| `androidx.xr.arcore` | Device pose (where the user is looking), depth, hand tracking |

What the XR SDK does **not** provide and we still need from stock
Android:

- `android.location.LocationManager` / `FusedLocationProviderClient` for
the user's GPS fix.
- `android.hardware.SensorManager` (magnetometer + accelerometer) for
the compass heading that drives the fuzzy-compass sectors. ARCore
device pose gives head orientation in the world frame, but we still
need true north.
- System TTS (`android.speech.tts.TextToSpeech`) for Postman James
lines.
- On-device ASR (`android.speech.SpeechRecognizer`) for "claim", "what's
nearby", "who's leading".

## Architecture

Mirror the existing **Android Auto** integration, which is the closest
pattern in this repo: a separate native-Kotlin surface that talks to
Firebase directly without ever loading the Flutter engine.

Reference files (do not modify these in this PR; they are the proven
template the XR surface will copy):

- `android/app/src/phone/kotlin/com/code418/postbox_game/car/PostboxCarAppService.kt`
- `android/app/src/phone/kotlin/com/code418/postbox_game/car/HomeCarScreen.kt`
- `android/app/src/phone/kotlin/com/code418/postbox_game/car/LeaderboardCarScreen.kt`
- `android/app/src/phone/kotlin/com/code418/postbox_game/car/StatsRepository.kt`
- `android/app/src/phone/kotlin/com/code418/postbox_game/car/ClaimAction.kt`

Proposed `android/app/src/xr/kotlin/com/code418/postbox_game/xr/`
layout (no files yet — follow-up PR):

| File | Responsibility | Lifted from |
| --- | --- | --- |
| `PostboxXrActivity.kt` | Compose Glimmer entry. Hosts the HUD scaffold and routes voice intents. | New |
| `FuzzyCompassHud.kt` | Compose Glimmer port of `lib/fuzzy_compass.dart` (`to8Sectors`, `vagueLabel`, sector painter). Draws claimed sectors grey and unclaimed sectors red around a North marker. | `lib/fuzzy_compass.dart` |
| `StatsRepository.kt` | One-shot reads of `users/{uid}` and `leaderboards/daily`. Mirrors the staleness check (discard if `periodKey` ≠ London-today). | `.../car/StatsRepository.kt` |
| `XrClaimAction.kt` | Auth check → fresh GPS fix → call `startScoring` callable → map result to `Claimed` / `Empty` / `AlreadyClaimedToday` / `TooFast` / `NotSignedIn` / `Error`. | `.../car/ClaimAction.kt` (almost verbatim) |
| `NearbyController.kt` | Calls the `nearbyPostboxes` callable, feeds `claimedCompassCounts` / `unclaimedCompassCounts` into the HUD. | New, but the callable contract is documented in `functions/src/index.ts` |
| `VoiceController.kt` | TTS + ASR wrapper. Maps "claim" / "what's nearby" / "who's leading" / "my streak" to actions. Reads Postman James lines from a Kotlin port of `lib/james_messages.dart`. | New |

## Server-side reuse

**No backend changes.** Every server-side need is already satisfied by
the existing callables documented in `CLAUDE.md` and exported from
`functions/src/index.ts`:

- `nearbyPostboxes` — returns `claimedCompassCounts` and
`unclaimedCompassCounts` (the exact shape the fuzzy compass HUD
needs).
- `startScoring` — handles the claim, quiz, streak update and
leaderboard write.
- `userClaimHistory` — stats for the voice-answered "how am I doing?"
query.

This means:

- No new Cloud Functions.
- No Firestore schema changes.
- No `firestore.rules` / `storage.rules` changes.
- No `pubspec.yaml` changes (Flutter side is untouched).

The fuzzy-compass contract from `CLAUDE.md` ("Avoid showing exact
bearings or distances that would allow pinpointing") still holds on
glasses — the HUD must only ever show 8-wind sectors, never a heading
needle pointing at a specific box.

## UX constraints

The glasses form factor reshapes UX in ways worth recording up front:

- **Voice-first.** Tap and pinch gestures exist but assume the player
is mid-walk with hands occupied (umbrella, dog lead, phone-camera
for photo evidence). Default every action to voice.
- **No full-screen UIs.** Compose Glimmer HUD chips and cards only.
Anything that occludes the player's view of an oncoming bus is a
bug.
- **Battery-conscious.** Don't poll ARCore device pose continuously —
only during an explicit "look around" gesture or voice command.
- **Privacy.** The glasses camera is **not** used for postbox claim
evidence in this design (UK street photography legality + bystander
privacy). The phone app remains authoritative for the
problem-reporting photo flow described in `CLAUDE.md`.
- **Auth.** Most AI-glasses pair with a phone and inherit its Google
session — we expect the existing Firebase Auth flow to "just work",
but it's an open question (see below).

## Risks and open questions

- **SDK maturity.** Jetpack XR is alpha at time of writing. Artefact
coordinates and minSdk are not pinned in the public docs; the
`xrImplementation` block in `android/app/build.gradle` is commented
out until that settles.
- **GPS accuracy on glasses.** The phone uses a 30 m claim radius. If
the glasses fix is noisier (no GNSS antenna, paired-phone tether),
we may need to relax `metresBounds` for XR-originated calls — but
doing so would weaken the travel-speed anti-spoof check in
`startScoring`. Worth measuring before we decide.
- **Shared-headset accounts.** Unclear whether a single pair of
glasses can switch between Google accounts mid-walk. If not, family
/ shared-device use cases get awkward.
- **OSM data freshness on glasses.** The fuzzy-compass HUD relies on
Firestore data ingested from OpenStreetMap. No change to that
pipeline — but if a player is in an area with sparse OSM coverage,
the HUD will look empty in a way that is harder to explain than on
the phone (no list view to fall back to).
- **Postman James voice.** Lines in `lib/james_messages.dart` were
written for the screen, not the ear. Some will need rewording for
TTS pacing.

## Phased rollout

| Phase | Scope | Branch shape |
| --- | --- | --- |
| 1 (this PR) | `xr` product flavour, empty `src/xr/` manifest, this doc. | `claude/smart-glasses-integration-jf0kW` |
| 2 | Read-only HUD prototype: `PostboxXrActivity` + `StatsRepository` showing today's points / streak / rank. Mirrors the Android Auto stat panel. | follow-up |
| 3 | Voice-driven claim. `XrClaimAction` + `VoiceController` wired to `startScoring`. | follow-up |
| 4 | Live fuzzy-compass overlay. `NearbyController` + `FuzzyCompassHud`, sensor-fusion heading. | follow-up |
| 5 | Public alpha behind a Firebase Remote Config flag, opt-in via the existing settings screen. | follow-up |

Each phase is small enough to ship behind feature-flag and back out if
the SDK shifts under us.

## What this PR actually changes

Just the build harness and this document. Specifically:

- `android/app/build.gradle` — add `xr` product flavour and a
commented `xrImplementation` dependency block.
- `android/app/src/xr/AndroidManifest.xml` — new, manifest-only
source set declaring `android.software.xr.immersive`. No activities,
no services.
- `docs/smart_glasses_integration.md` — this file.

Nothing under `lib/`, `functions/`, `firestore.rules`,
`storage.rules`, `pubspec.yaml`, `firebase.json`, the `wear` source
set, or the `phone` source set is touched.
Loading