Under review Ubiquitous sensing Context-aware systems
ThermalKitchen
An Egocentric Video QA Benchmark for Thermal-State Understanding
Co-author, University of Michigan
Everyday physical tasks depend on judging heat — whether a pan is still hot after the stove is off, whether food out of the microwave is actually warm. Appearance often doesn’t say, and a thermal camera can contradict what the scene suggests. ThermalKitchen asks whether vision-language models can use thermal observations to judge the thermal state of the right object at the right moment.
The benchmark. Participants wore a head-mounted rig with an RGB camera and a radiometric thermal camera while cooking at home: 49 unscripted episodes by eight people in eight kitchens, plus four scripted scenario collections, 28.37 hours in all. Every frame comes with per-pixel temperatures. On top of it sit 395 human-written multiple-choice questions about physical states and how they change, each labelled by the evidence it needs: RGB (61), thermal (106) or both (228).
What models do. We evaluated six VLMs with the question alone and with RGB-only, thermal-only and side-by-side RGB–thermal video. Question-only accuracy stays near chance; thermal-only beats RGB-only for every model by 9.1–15.4 points, and joint input is best, peaking at 70.1%.
Where they fail. A diagnostic set, Hard100, audits each explanation against reference observations. With joint input, over 90% of the claims about objects and events are right, but only about 30% of local thermal-state readings — models find the right pan and the right moment, then misread how hot it is. Stating the relevant observations in text lifts one model from 30.9% to 83.0%, without giving away the rule or the answer.
Figures
