Context-aware systems
Multimodal FSM for Adaptive Skill Guidance
Solo project, University of Michigan, SURE Project 9: Seeing Skill with States
Built for the University of Michigan SURE program (Project #9, Seeing Skill with States): a real-time system that watches and listens while someone washes their hands, tracks where they are in the procedure with a 12-state finite-state machine, and talks them through it.
Why hand washing. The task had to break into clear steps with distinct cues between them, run in real time, and be cheap to repeat. Hand washing fits all three. (Searing a steak was the runner-up: great sizzle and smoke cues, but hard to repeat.)
Cue design. Running water and a hand dryer are easy to hear, so YAMNet audio classification on the laptop CPU handles them cheaply. Questions that need eyes — are the hands under the faucet, is there soap, is a towel in use — go to a small VLM as binary (0/1) prompts, and a cue must hold for a moment before it moves the state machine, so noisy frames don’t make it flicker.
Latency. One Qwen3-VL-8B call through an SSH tunnel to the UMich HPC took about 1.1 s end to end. Three instances on three GPUs with round-robin dispatch give a fresh answer every ~0.36 s. The same config switches to ensemble voting, or to Gemini 2.5 Flash with a one-line change.
Not just a straight line. Stand idle too long and the machine falls back to Idle; soap can be re-applied mid-rinse; there are three ways to dry (towel, clothes, hand dryer); longer rinsing scores higher. When a step keeps failing, adaptive level-of-detail guidance makes the voice instructions progressively more specific. Each session ends with a score out of 100 and a log.
What I’d change next. The delays compound — every cue waits on a VLM query and then has to be sustained before a transition fires — so the system feels slower than its 0.36 s refresh. And hand washing is still fairly linear; a task with real branching and priorities, such as tidying a room, would test the state machine much harder.
What it involved
- Work
- Task and state-machine designMultimodal sensingModel serving on HPCReal-time GUIVoice feedbackScoring design
- Knowledge and tools
- Finite-state machinesQwen3-VL on vLLMGemini 2.5 FlashYAMNet audio classificationTensorFlow HubRound-robin GPU dispatchOpenCVText-to-speechPython
Video
Figures