Zuchen Li

All projects

Human–AI interaction

How Fragile Are GUI Agents?

Project lead, University of Michigan, EECS 545: Machine Learning

GUI agents are usually tested on frozen copies of websites, but real interfaces keep changing: buttons get renamed, layouts reflow, forms turn into wizards. An agent that works today can quietly fail after the next redesign. A team project for EECS 545 with Opeyemi Akinniyi, Carol Kang, Xiangnong Wu and Hao Yin.

From archives to a controlled benchmark. We started with Wayback Machine snapshots of Reddit, IMDb and Craigslist from 2014 to 2023. They revealed five recurring kinds of change — layout, semantic, visual, structural and workflow — but archived pages also change their content and lose assets, so a failure can’t be pinned on the interface alone. We therefore generated our own websites (house renting, course registration, personal pages and a job-application site) in several styles and eras, each variant differing from its base by one typed, labelled change at one of three intensities.

What we found. We evaluated four VLM agents (Qwen3-VL-30B, UI-TARS-7B, Qwen2.5-VL-7B, InternVL2-8B) and a text-only baseline (GPT-oss-120B) in the VisualWebArena harness.

  • Screenshots are the weak link. Text-only input averaged 92.4%; vision-only agents about 40%. On house renting, vision-only success fell from 57.9% on the classic layout to 16.5% once details moved behind tabs and toggles — content a static screenshot cannot see.
  • Specialisation beats size, sometimes. UI-TARS-7B was more robust than the four-times-larger Qwen3-VL-30B on interaction-heavy pages, but every vision agent converged on the personal-website task, where cross-page navigation rather than perception is the bottleneck.
  • Test-time fixes depend on the task. Experience memory recovered up to 52% of failures on one site and almost none on another, where it made up answers up to 94% of the time; chain-of-thought helped most when the agent had to reason about interactive state.
  • Newer isn’t always harder. 2000s table-based HTML tripped agents more than modern semantic markup.

What it involved

Work
Benchmark designWeb archive studySynthetic website generationAgent evaluationFailure analysis
Knowledge and tools
VisualWebArenaVision-language modelsSet-of-Mark groundingWayback MachineLLM-driven web synthesisFlaskRobustness metricsMemory and chain-of-thought prompting

Figures

One listing in three interface eras; vision-only success falls from 57.9% to 16.5%.
Figure 1. One listing in three interface eras; vision-only success falls from 57.9% to 16.5%.
Vision-only success drops to about 0% when the answer sits behind a tab or a click.
Figure 2. Vision-only success drops to about 0% when the answer sits behind a tab or a click.
Robustness per agent: a 7B GUI specialist beats a 30B generalist on house renting.
Figure 3. Robustness per agent: a 7B GUI specialist beats a 30B generalist on house renting.
Memory recovers up to 52% of failures on one site and almost none on another.
Figure 4. Memory recovers up to 52% of failures on one site and almost none on another.
One job-application site, generated in three visual styles.
Figure 5. One job-application site, generated in three visual styles.
Five typed perturbations, each at three intensity levels.
Figure 6. Five typed perturbations, each at three intensity levels.
Preliminary study: Reddit snapshots from 2014 to 2023.
Figure 7. Preliminary study: Reddit snapshots from 2014 to 2023.
The EECS 545 poster.
Figure 8. The EECS 545 poster.