The question
Could one designer make a phone camera feel like a story?
The obvious version of this demo labels objects: "chair," "tree," "door." That's a novelty, and it would be embarrassing to show. I wanted characters with their own voices, who notice what you notice and talk to you about it. And I wanted to find out whether I could build it myself, with no backend experience, using AI as both the medium and the build tool.
The text of the first prompt, asking how to build a small app that reads my surroundings through my phone camera and responds with characters, as a quick demo of an immersive experience.
What I tried
Three choices shaped it, and testing outside changed two of them.
Agents with personas, not labels
I designed the agents before any code. I wrote a persona for each of four characters: personality rules, behavior patterns, constraints, and the triggers that decide which one appears based on what the camera sees. A poetic Storyteller, a wisecracking Conscience, a Fairy Godmother who sees what a room could become, and an adventurous Wayfinder. I chose and tuned each voice in ElevenLabs.
Two-way conversation. You can speak or type to the active character, about what it sees or anything else, and it answers in voice or text. Both directions stay open for privacy, accessibility and preference.
Initiative. Characters don't only wait to be asked. They speak up again after a quiet stretch, and when the camera's view changes significantly.
Turn-taking. Pressing look or talk while a character is thinking or speaking interrupts it and starts a fresh turn.
Early test, March 2026
Test where it would be used
I deployed to Vercel on day one and tested on my iPhone, indoors and then outside. Outdoors exposed what my desk never did:
The voice cut out when the free voice quota ran out. Now the app falls back to the phone's built-in voice instead of going silent.
Each look took several seconds across two AI calls. I designed a pulsing "thinking" state, and a rule that pressing again interrupts and starts fresh.
Too many visible controls invited taps that broke the flow. I cut the screen down to two big actions, look and talk, with small toggles beside them.
Field test, March 2026
Design it, then let AI build it
I redesigned the controls in Figma, with states and interruption rules annotated, then moved it into code with Claude and Cursor and kept refining it there by chat. One bug only appeared on Vercel: the server functions bundled differently than on my laptop. I traced it from the deploy logs and fixed it.
Live app, March 2026
What I learned
When you design by conversation, the reasons disappear
My Figma design was only a starting point. From there I kept changing the interface by chatting with the AI in Cursor, and the finished app looks different from the file. It has a frosted glass bar behind the buttons, for one. I believe I asked for it, but I can't prove it. The chats are gone, and with them the record of who decided what, and why.
Designing this way is fast, but the design file stops being the source of truth, and the rationale lives in a chat log nobody keeps. On a team, that's a real problem: no one can answer why the product looks the way it does. So now, before I ask an AI to change anything, I write a punch list of the changes I intend. I check its work against that list, and anything that changed without being on it gets a closer look, so I can decide whether it was intended and whether it stays.
Left: my starting design in Figma. Right: the app after more design changes made by chatting with the AI.
Where it stands
A working prototype, paused on purpose
It answered my question. The core concept works: one designer can get a phone camera to feel like a story, in less than a week, with no install. Speed and polish were never the point of this first step, so I stopped once the idea was proven.
Works
Live camera, scene-matched characters, voice narration, animated sprites, auto mode that looks every 12 seconds, talking or typing back to the character, and characters that speak up on their own when you go quiet or the scene changes. Runs in Safari on iPhone.
Still rough, on purpose
Responses are slow; making them fast is the first job before this becomes usable. The characters are simple placeholders, meant to be replaced with polished animated ones. A few small fixes remain.
Next
Faster responses. Polished, animated characters. A choice of stories, each with its own vibe and tone, that colors everything the characters say and how they sound.
It would take
Time on performance, an illustrator or animator for the characters, and a paid voice tier.