The idea
Assign behavior to a mug, spoon, or handful of coins, then use them as game controls, a physical sequencer, a classroom lesson, or a paper UI prototype.
How it works
- A camera recognizes objects and their movement on a marked tabletop.
- A person describes behavior such as “the mug is the throttle” or “coins power the engine.”
- A constrained program maps those recognized actions to predictable rules for a game, sequencer, classroom lesson, or paper UI prototype.
AI’s role: Translate spoken rules into a constrained program; ordinary code and measured camera inputs execute the rules and track game state.
First demonstration
Speak rules for a paper-controls game and use a phone on a stand to track coins, mug movement, and a shared game state.
What to solve next
Which object movements can the camera recognize reliably under ordinary room lighting and clutter?