Kiosk mode: camera motion detection for screen wake by nstefanelli · Pull Request #4497 · home-assistant/iOS

nstefanelli · 2026-04-10T03:33:47Z

Summary

Adds camera-based motion and presence detection to kiosk mode (PR2 of kiosk series, per discussion #2403).

Motion detection: CoreImage pixel-diff at 5 fps with configurable sensitivity (low/medium/high)
Presence detection: Vision framework person/face detection with hysteresis (2-frame confirm, 10s absence timeout)
Screen wake: Configurable wake-on-motion and wake-on-presence triggers
Settings UI: New "Camera Detection" section with toggles, sensitivity picker
macCatalyst: Camera detection excluded on Mac

Builds on #4422 (PR1: core kiosk infrastructure).

Architecture

KioskCameraDetectionManager (coordinator)
├── KioskCameraMotionDetector (CoreImage pixel-diff, 5 fps, .low preset)
└── KioskPresenceDetector (Vision person/face detection, 3-10 fps, .medium preset)
         │
         ▼
KioskModeManager.wakeScreen(source:)

Frame processing runs on dedicated serial queues, not the main thread
State updates dispatched to MainActor via Combine publishers
Presence activity timer keeps screen awake while person is present

New files

File	Lines	Purpose
`Camera/KioskCameraDetectionManager.swift`	~200	Coordinator singleton
`Camera/KioskCameraMotionDetector.swift`	~250	CoreImage motion detection
`Camera/KioskPresenceDetector.swift`	~300	Vision presence/face detection
`KioskCameraDetection.test.swift`	~115	MotionSensitivity + settings tests

Test plan

MotionSensitivity enum: threshold values, Codable roundtrip, ordering
Camera settings: defaults, roundtrip, backwards compatibility with pre-camera JSON
Enable motion detection → wave hand → screen wakes from screensaver
Enable presence detection → walk up → screen wakes; walk away → screensaver resumes
Deny camera permission → detection doesn't start, no crash
Both detectors simultaneously → no conflict
macCatalyst build → camera section not visible

…ings Adds MotionSensitivity enum (low/medium/high with threshold values) and 6 camera detection properties to KioskSettings: cameraMotionEnabled, cameraMotionSensitivity, wakeOnCameraMotion, cameraPresenceEnabled, cameraFaceDetectionEnabled, wakeOnCameraPresence. Replaces synthesized Codable with a custom init(from:) for backwards-compatible decoding so existing persisted settings without camera fields decode without error.

@published

- Make processFrame/calculateDifference nonisolated in motion detector so CIFilter and CIContext.render run on processingQueue, not MainActor - Make processFrame nonisolated in presence detector so Vision requests run on processingQueue; settings read on MainActor then dispatched - Mark previousFrame, ciContext, and Vision requests as nonisolated(unsafe) since they are only accessed from their respective processing queues - Remove @published from errorMessage (internal debugging state, not UI) - Replace PR-specific comments in tests with neutral wording

Custom init(from:) suppresses the synthesized default initializer. Add explicit public init() so KioskSettings() continues to work.

codecov · 2026-04-10T04:05:47Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@7ac3faf). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4497   +/-   ##
=======================================
  Coverage        ?   43.28%           
=======================================
  Files           ?      275           
  Lines           ?    16694           
  Branches        ?        0           
=======================================
  Hits            ?     7226           
  Misses          ?     9468           
  Partials        ?        0

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

bgoncal · 2026-04-13T18:56:45Z

Sorry the delay to pickup this review, I will still need a bit more time since I am working on something else right now, meanwhile I noticed that after releasing the foundation for kiosk mode, lots of questions about how it works and comparisons to the famous "Fully Kiosk browser" popped up in the community, so it would be nice if we created a documentation page for this in out companion app docs, could you propose it?

Reference: https://companion.home-assistant.io/docs/getting_started/

We need to make it clear things like... in iOS we cannot lock the user in a single app (from Home Assistant) and that the closest way possible to that is if the user uses iOS "guided access". So it needs to explain the difference of the "lock" that we mention inside Home Assistant app and the "lock" that the user may be expecting.

Macstar1601 · 2026-04-13T19:15:31Z

Sorry the delay to pickup this review, I will still need a bit more time since I am working on something else right now, meanwhile I noticed that after releasing the foundation for kiosk mode, lots of questions about how it works and comparisons to the famous "Fully Kiosk browser" popped up in the community, so it would be nice if we created a documentation page for this in out companion app docs, could you propose it?

Reference: https://companion.home-assistant.io/docs/getting_started/

We need to make it clear things like... in iOS we cannot lock the user in a single app (from Home Assistant) and that the closest way possible to that is if the user uses iOS "guided access". So it needs to explain the difference of the "lock" that we mention inside Home Assistant app and the "lock" that the user may be expecting.

The Kiosk Mode feature is really great. But for me, it’s not important to be locked into Kiosk Mode. It’s much more important that the Dashboard can be accessed more quickly and that you don’t have to unlock the iPad first. A true lock within an app also leads to other problems, such as not being able to use other apps at the same time, e.g., intercom apps. I opened an issue about this today: #4506

home-assistant · 2026-04-13T19:16:40Z

Please take a look at the requested changes, and use the Ready for review button when you are done, thanks 👍

Learn more about our pull request process.

bgoncal · 2026-04-13T19:19:59Z

@Macstar1601 thanks for sharing, we dont need to worry about that because the app is not able to lock itself as the only app, iOS doesn't allow that, the maximum we can do is to prevent the screen to turn off.

Copilot

Pull request overview

Adds camera-based motion and presence detection to kiosk mode, wiring new detection settings into the kiosk settings model/UI and integrating camera-driven screen wake triggers into KioskModeManager.

Changes:

Introduces camera motion (CoreImage pixel-diff) and presence/face detection (Vision) components plus a coordinator manager.
Extends KioskSettings + settings UI with a new “Camera Detection” section and persistence/backwards-compat decoding.
Regenerates SwiftGen string accessors to include explicit fallback values for Frontend/Core strings, and adds SwiftUI helpers to SwiftGen assets.

Reviewed changes

Copilot reviewed 11 out of 13 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
`Tests/App/Kiosk/KioskCameraDetection.test.swift`	Adds Swift Testing coverage for `MotionSensitivity` and camera-related `KioskSettings` fields (defaults/roundtrip/back-compat).
`Sources/Shared/Resources/Swiftgen/FrontendStrings.swift`	SwiftGen output updated to provide fallback strings for Frontend keys; `tr` implementation changed accordingly.
`Sources/Shared/Resources/Swiftgen/CoreStrings.swift`	SwiftGen output updated to provide fallback strings for Core keys; `tr` implementation changed accordingly.
`Sources/Shared/Assets/Assets.swift`	Adds SwiftUI `Image` convenience initializers and a `swiftUIImage` accessor for `ImageAsset`.
`Sources/App/Resources/en.lproj/Localizable.strings`	Adds new kiosk camera detection strings (section labels, sensitivity labels, footer).
`Sources/App/Kiosk/Settings/KioskSettingsView.swift`	Adds a camera detection settings section (excluded on macCatalyst).
`Sources/App/Kiosk/KioskSettings.swift`	Adds camera detection settings fields, `MotionSensitivity`, and backwards-compatible decoding.
`Sources/App/Kiosk/KioskModeManager.swift`	Starts/stops camera detection with kiosk mode and restarts detection on relevant settings changes.
`Sources/App/Kiosk/Camera/KioskPresenceDetector.swift`	New Vision-based presence + face detection implementation.
`Sources/App/Kiosk/Camera/KioskCameraMotionDetector.swift`	New CoreImage motion detection implementation.
`Sources/App/Kiosk/Camera/KioskCameraDetectionManager.swift`	New coordinator that binds detectors and exposes callbacks + state.
`HomeAssistant.xcodeproj/project.pbxproj`	Adds new kiosk camera source files and test file to the project.

Per bgoncal review on home-assistant#4497, ship motion detection first and defer presence/face detection to a follow-up PR for simpler review scope. Drops KioskPresenceDetector and the presence/face settings, UI toggles, manager wiring, and L10n keys.

Earlier SwiftGen run added a 'fallback:' arg to tr() calls that LocalizedManager doesn't accept (won't compile), reordered unrelated Strings entries, and added unused SwiftUI Image helpers to Assets.swift. Re-run with the project's existing template so the diff vs main is just the new kiosk.camera keys.

…tection

- KioskCameraDetectionManager.isActive now mirrors the underlying motion detector instead of being set from settings, so denied camera permission no longer leaves the manager stuck active. - Move previousFrame=nil into the processingQueue.async block to match its 'processingQueue-only' invariant. - Drop wakeOnCameraMotion from the settings-restart comparison; the closure already reads it at fire time, no need to recreate the capture session when only the wake flag toggles. - Toggling cameraMotionEnabled now requests camera permission and surfaces a denial alert that deep-links to iOS Settings, so the motion detector actually has something to start with.

nstefanelli · 2026-04-26T04:11:22Z

Stripped face/presence so this PR is motion-only. The inline items are addressed in the latest commits. Companion-app docs proposal coming in a separate issue.

bgoncal · 2026-04-28T07:27:28Z

Thanks for that, let me know whenever you have the docs PR so we can review that first before adding more features

bgoncal

Let's keep it in draft while we don't have the docs

bgoncal · 2026-05-04T09:28:53Z

@nstefanelli Do you need some help with the docs? Or if you have created already and I missed it please tag me and I'll review asap

nstefanelli · 2026-05-05T03:48:07Z

Thanks @bgoncal — opened the docs PR over at home-assistant/companion.home-assistant#1323. It covers kiosk mode end-to-end (PR1 + this PR), flagged as BETA, with screenshot placeholders left in for you to flag whichever captures you want included. Happy to mark this PR ready for review whenever the docs side looks good to you, or earlier if that's easier on your end.

nstefanelli added 12 commits April 9, 2026 19:54

test(kiosk): add unit tests for MotionSensitivity and camera settings

7b193c9

feat(kiosk): add camera motion detector

e2c3c3a

feat(kiosk): add presence detector with Vision framework

003bf22

feat(kiosk): add camera detection manager coordinator

1ea006d

feat(kiosk): integrate camera detection into KioskModeManager

a93d5b6

feat(kiosk): add camera detection settings section

7959b54

chore(kiosk): add camera detection files to Xcode project

cc655cb

fix(kiosk): address code review findings for camera detection

a63b9b1

fix(kiosk): add explicit init() to KioskSettings

8eba154

Custom init(from:) suppresses the synthesized default initializer. Add explicit public init() so KioskSettings() continues to work.

Copilot AI review requested due to automatic review settings April 10, 2026 03:33

home-assistant Bot added the cla-signed label Apr 10, 2026

Copilot started reviewing on behalf of nstefanelli April 10, 2026 03:34 View session

bgoncal requested review from Copilot and removed request for Copilot April 13, 2026 19:14

Copilot started reviewing on behalf of bgoncal April 13, 2026 19:14 View session

bgoncal requested changes Apr 13, 2026

View reviewed changes

Comment thread Sources/App/Kiosk/Camera/KioskCameraDetectionManager.swift Outdated

home-assistant Bot marked this pull request as draft April 13, 2026 19:16

Merge branch 'main' into kiosk-pr2-camera-detection

9922505

Copilot AI reviewed Apr 13, 2026

View reviewed changes

Macstar1601 mentioned this pull request Apr 16, 2026

Kiosk mode: iPad does not wake up when pressing on notification pop-up from other apps #4506

Closed

bgoncal mentioned this pull request Apr 24, 2026

iOS kiosk mode proposal OpenHomeFoundation/roadmap#51

Open

nstefanelli added 3 commits April 25, 2026 22:27

Merge remote-tracking branch 'upstream/main' into kiosk-pr2-camera-de…

d4d019a

…tection

nstefanelli marked this pull request as ready for review April 26, 2026 04:12

home-assistant Bot requested a review from bgoncal April 26, 2026 04:12

nstefanelli changed the title ~~Kiosk mode: camera motion/presence detection for screen wake~~ Kiosk mode: camera motion detection for screen wake Apr 26, 2026

bgoncal requested changes Apr 28, 2026

View reviewed changes

home-assistant Bot marked this pull request as draft April 28, 2026 11:29

CCOSTAN mentioned this pull request May 4, 2026

Joanna Idea Scout - 2026-05-04 CCOSTAN/Home-AssistantConfig#1733

Closed

nstefanelli mentioned this pull request May 5, 2026

Add iOS kiosk mode docs home-assistant/companion.home-assistant#1323

Draft

5 tasks

CCOSTAN mentioned this pull request May 5, 2026

Joanna Idea Scout - 2026-05-05 CCOSTAN/Home-AssistantConfig#1734

Closed

laupalombi mentioned this pull request May 11, 2026

Kiosk Mode for iOS - MVP #4621

Open

Conversation

nstefanelli commented Apr 10, 2026

Summary

Architecture

New files

Test plan

Related

Uh oh!

codecov Bot commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bgoncal commented Apr 13, 2026

Uh oh!

Macstar1601 commented Apr 13, 2026

Uh oh!

Uh oh!

home-assistant Bot commented Apr 13, 2026

Uh oh!

bgoncal commented Apr 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nstefanelli commented Apr 26, 2026

Uh oh!

bgoncal commented Apr 28, 2026

Uh oh!

bgoncal left a comment

Choose a reason for hiding this comment

Uh oh!

bgoncal commented May 4, 2026

Uh oh!

nstefanelli commented May 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov Bot commented Apr 10, 2026 •

edited

Loading