Multimodal FSM for Adaptive Skill Guidance

A real-time skill-guidance system that watches and listens: VLM-based visual cues and YAMNet audio classification feed a 12-state finite-state machine that tracks where the user is in a hand-washing procedure — and adapts the level of voice-instruction detail when they stall or deviate.

Engineering highlights: switchable local Qwen3-VL and cloud Gemini backends, sustained-condition state transitions (no flicker on noisy frames), non-blocking TTS, a split-screen live state visualization, and post-session scoring and logging.