Human–AI interaction
How Fragile Are GUI Agents?
Project lead, University of Michigan, EECS 545: Machine Learning
GUI agents are usually tested on frozen copies of websites, but real interfaces keep changing: buttons get renamed, layouts reflow, forms turn into wizards. An agent that works today can quietly fail after the next redesign. A team project for EECS 545 with Opeyemi Akinniyi, Carol Kang, Xiangnong Wu and Hao Yin.
From archives to a controlled benchmark. We started with Wayback Machine snapshots of Reddit, IMDb and Craigslist from 2014 to 2023. They revealed five recurring kinds of change — layout, semantic, visual, structural and workflow — but archived pages also change their content and lose assets, so a failure can’t be pinned on the interface alone. We therefore generated our own websites (house renting, course registration, personal pages and a job-application site) in several styles and eras, each variant differing from its base by one typed, labelled change at one of three intensities.
What we found. We evaluated four VLM agents (Qwen3-VL-30B, UI-TARS-7B, Qwen2.5-VL-7B, InternVL2-8B) and a text-only baseline (GPT-oss-120B) in the VisualWebArena harness.
- Screenshots are the weak link. Text-only input averaged 92.4%; vision-only agents about 40%. On house renting, vision-only success fell from 57.9% on the classic layout to 16.5% once details moved behind tabs and toggles — content a static screenshot cannot see.
- Specialisation beats size, sometimes. UI-TARS-7B was more robust than the four-times-larger Qwen3-VL-30B on interaction-heavy pages, but every vision agent converged on the personal-website task, where cross-page navigation rather than perception is the bottleneck.
- Test-time fixes depend on the task. Experience memory recovered up to 52% of failures on one site and almost none on another, where it made up answers up to 94% of the time; chain-of-thought helped most when the agent had to reason about interactive state.
- Newer isn’t always harder. 2000s table-based HTML tripped agents more than modern semantic markup.
What it involved
- Work
- Benchmark designWeb archive studySynthetic website generationAgent evaluationFailure analysis
- Knowledge and tools
- VisualWebArenaVision-language modelsSet-of-Mark groundingWayback MachineLLM-driven web synthesisFlaskRobustness metricsMemory and chain-of-thought prompting
Figures