Put the headset on and you see your real room in colour passthrough. Point the controller at a light, pull the trigger, and a marker drops onto the real fixture — bound to that Home Assistant device. The pin stays where you put it: it survives an app restart and walking out of the room and back, because it's anchored to the room the Quest already knows. Walk up to any light and switch it on from exactly where it stands. A quick browser proof came first — but the things that make it actually work only exist in a native app, so Roomscribe is Unity + Meta XR, built and pushed to the headset without opening the editor once.
The idea was tested first as a WebXR page served straight from Home Assistant. You could point and pin and even toggle lights — but the pins wandered. The browser can't bind to the rooms the Quest has already mapped, its anchors drift, and it can't read the headset's furniture labels. So the moment you changed rooms, the markers lost their place. Everything that makes this trustworthy is native-only, so we rebuilt it in Unity with Meta's XR toolkit.
Meta's Scene & Mixed-Reality Utility Kit hand the app the rooms you set up on the headset — walls, floor and furniture — so a pin can belong to a real, recognised room, not floating coordinates.
Native spatial anchors re-locate to the physical room every session. That's the difference between "the pin is roughly there" and "the pin is on the bulb, still, tomorrow."
The Quest already labels tables, screens, storage and the rest. We draw those as clean outlines — a sense of the room with no camera and nothing private ever shown.
The controller casts a ray into the passthrough view; where it lands on the real room, a marker is dropped and tied to a Home Assistant device. From then on the marker is the control: point at it, pull the trigger, and Home Assistant toggles the device over the LAN — the marker even glows in the bulb's real colour.
The failure mode of the browser version was pins drifting when you moved between rooms. The fix is the heart of the app: the headset works out which mapped room you're physically inside (by your position within its bounds, not a guess), and every pin is stored as a pose relative to that room's floor anchor. Because the Quest re-locates the floor anchor to the real room each session, the pins ride back onto the real fixtures automatically — through restarts and room changes alike.
Devices and their live state come from Home Assistant's REST API; a tap on a pin is a single service call — reached on the Pi's own address over the home Wi-Fi, never the tunnel (which Cloudflare's bot check blocks for a non-browser client anyway).
// toggle the pinned device — one service call, over the LAN POST http://192.168.5.126:8123/api/services/homeassistant/toggle Authorization: Bearer <long-lived token> { "entity_id": "light.utility_lights" } // the pin's colour tracks the real state GET /api/states/light.utility_lights → { "state": "on", "attributes": { "rgb_color": [255,176,64] } }
The token lives only on the headset (pushed to the app's own data folder); nothing about the house leaves the network.
Everything is driven from one floating panel and the controller. It's built to be used stood in a room, one-handed, with the beam.
Two modes: Place drops pins, Test switches the real devices. The beam stops on the panel, the button under it lights up, and a cursor shows exactly where you're pointing.
Grab the panel by its top bar and drag it anywhere; press a button to summon it back in front of you. It spawns at eye level, wherever you're looking.
The room's furniture is drawn as faint coloured wireframes from the Quest's own scene labels — enough to read the space, with no camera feed and nothing photoreal.
The whole app was scaffolded, configured and built from the command line — Unity in batch mode, driven over SSH and the terminal, reading the build log and fixing compile errors in the loop, then side-loaded to the Quest over USB. No editor windows, no manual scene wiring.
# configure the project (XR, Android, Meta features) then build the APK — batch mode, no UI Unity.exe -batchmode -nographics -quit -projectPath C:\envy\roomscribe -buildTarget Android \ -executeMethod Roomscribe.EditorTools.ProjectConfigurator.Configure -logFile configure.log Unity.exe -batchmode -nographics -quit -projectPath C:\envy\roomscribe -buildTarget Android \ -executeMethod Roomscribe.EditorTools.BuildAndroid.Build -logFile build.log # push it to the headset + drop in the Home Assistant token adb install -r Builds\roomscribe.apk
The scene itself is assembled in C# at configure time — camera rig, passthrough, MRUK and the app components — so a build is fully reproducible from source with one command.
This one isn't a concept — it's live. The Quest exports the mapped rooms (floor, furniture and every pinned device) in one shared frame; a small three.js viewer renders them on the Home Assistant wall panel in their true relative positions. Three real spaces below — the utility, the back-door lobby and Kev's bedroom — walked, scanned and pinned in the headset, then stood up as a doll's-house you can spin. The pins carry live state: the red one is the back-door alarm light showing its real colour.
Right now a pinned device is a coloured blob — ideal for proving the thing works, plain to look at. Next it becomes a catalogue of real smart-home objects: choose what a device actually is from a category, drop it in, then refine it until it sits right. The same treatment turns the furniture outlines into real pieces and lets you add anything the scan missed. None of this is built yet — it's where Roomscribe goes next, and it all sits on the pin architecture that already works.
Placing stays one quick action — point, pull, done — but now you drop a chosen model and fine-tune it: slide it along the surface, spin it to face the right way, scale it to the real thing. Fittings that belong on a surface snap to it — a downlight to the ceiling, a switch to the wall. And the furniture the Quest outlined gets the same upgrade: a box becomes a sofa.
No scan catches everything — a chair tucked under a desk, a pot plant in the corner, a lamp the headset never saw. So any item from the library can be dropped in freely, bound to nothing, anchored to the room exactly like a device pin. The map becomes something you build, not just something the headset hands you.
Once a pin is a real object, it can show the real state. The lamp glows its actual colour and brightness; the TV screen lights when it's playing; the fan's blades turn when it's running; the thermostat shows its setpoint; the alarm bulbs pulse red when the house is armed. The map stops being a map and becomes a live dashboard you stand inside — cheap to add once the library exists, and the single biggest payoff of the whole idea.
Fittings orient themselves to the surface they belong on — a downlight flat to the ceiling, a switch upright on the wall, a TV against the wall plane — using the Quest's own wall, floor and ceiling planes. Placement becomes one correct click.
The app knows which devices in this room aren't on the map yet — "3 lights in Utility not placed" — and offers a best-guess spot. Confirm to place, instead of hunt-and-place.
The RTX box and Claude recognise objects from a walk-around and pre-drop the right model with a suggested device — you just confirm. The library is the manual layer; the AI fills it in for you.
Furniture as oriented boxes instead of axis-aligned, real door openings cut into the walls, and multiple floors stacked — so the dollhouse reads even closer to the real place.
Look at a device and just say it — "warm white" — hands-free, using mixed reality's unfair advantage: it already knows exactly what you're pointing at.
Select a few placed devices to save a scene, or draw a link in the air — "motion here → this light" — authoring automations by standing in the room. It ties straight into the Presence system.
The thing that maps your own house is also a surveying and quoting tool for the electrical work: scan a client's room, drop the proposed fittings, check spacing and coverage, and walk out with a positioned parts list. Roomscribe aimed outward — and a natural fit alongside the rest of the stack.
A mixed-reality layer over the same local Home Assistant that runs everything else — devices pinned to the real fixtures, holding their place through restarts and room changes, controlled from where they actually are. First proved in the browser, then rebuilt native for the parts that had to be. A working proof today; the foundation for a spatial way to run the whole house.