Web agents can navigate changing websites by looking at the screen

Dhruv Batra explains how web agents can use visual information from a browser instead of relying on fixed APIs or hard-coded website structures.

The agents look at the pixels on the screen, take actions, and keep learning from interactions with live websites. When layouts change, they update based on new data instead of needing every change programmed manually.

He says that on poorly designed websites, the agents can sometimes navigate the interface better than he can.