Zuchen Li

All projects

Context-aware systems

Multimodal FSM for Adaptive Skill Guidance

Solo project, University of Michigan, SURE Project 9: Seeing Skill with States

Built for the University of Michigan SURE program (Project #9, Seeing Skill with States): a real-time system that watches and listens while someone washes their hands, tracks where they are in the procedure with a 12-state finite-state machine, and talks them through it.

Why hand washing. The task had to break into clear steps with distinct cues between them, run in real time, and be cheap to repeat. Hand washing fits all three. (Searing a steak was the runner-up: great sizzle and smoke cues, but hard to repeat.)

Cue design. Running water and a hand dryer are easy to hear, so YAMNet audio classification on the laptop CPU handles them cheaply. Questions that need eyes — are the hands under the faucet, is there soap, is a towel in use — go to a small VLM as binary (0/1) prompts, and a cue must hold for a moment before it moves the state machine, so noisy frames don’t make it flicker.

Latency. One Qwen3-VL-8B call through an SSH tunnel to the UMich HPC took about 1.1 s end to end. Three instances on three GPUs with round-robin dispatch give a fresh answer every ~0.36 s. The same config switches to ensemble voting, or to Gemini 2.5 Flash with a one-line change.

Not just a straight line. Stand idle too long and the machine falls back to Idle; soap can be re-applied mid-rinse; there are three ways to dry (towel, clothes, hand dryer); longer rinsing scores higher. When a step keeps failing, adaptive level-of-detail guidance makes the voice instructions progressively more specific. Each session ends with a score out of 100 and a log.

What I’d change next. The delays compound — every cue waits on a VLM query and then has to be sustained before a transition fires — so the system feels slower than its 0.36 s refresh. And hand washing is still fairly linear; a task with real branching and priorities, such as tidying a room, would test the state machine much harder.

What it involved

Work
Task and state-machine designMultimodal sensingModel serving on HPCReal-time GUIVoice feedbackScoring design
Knowledge and tools
Finite-state machinesQwen3-VL on vLLMGemini 2.5 FlashYAMNet audio classificationTensorFlow HubRound-robin GPU dispatchOpenCVText-to-speechPython

Video

Figures

Live view: vision and audio cue confidences on the left, the state machine on the right.
Figure 1. Live view: vision and audio cue confidences on the left, the state machine on the right.
The 12-state machine, with re-soap, skip-soap and skip-rinse paths and per-step scoring.
Figure 2. The 12-state machine, with re-soap, skip-soap and skip-rinse paths and per-step scoring.
Sensing, FSM and voice run on a laptop; three Qwen3-VL-8B instances run on the UMich HPC.
Figure 3. Sensing, FSM and voice run on a laptop; three Qwen3-VL-8B instances run on the UMich HPC.
Round-robin across three GPUs: a 1.1 s VLM call becomes a fresh answer every 0.36 s.
Figure 4. Round-robin across three GPUs: a 1.1 s VLM call becomes a fresh answer every 0.36 s.
One of three ways to dry: the hand dryer branch.
Figure 5. One of three ways to dry: the hand dryer branch.
A finished session, scored out of 100 from the quality of each step.
Figure 6. A finished session, scored out of 100 from the quality of each step.