Computer Use Beyond Clicks

Abstract

Agents that use computers have several methods by which they may navigate, launch applications from the desktop, and interact with them using keyboard and mouse events. A single input method does not work well for every surface. Windows makes this clear. Only the WinSta0 interactive window station can show a user interface, and only one desktop receives input at a time. Because of this, raw input injection depends on the surface being controlled. This paper presents Agora, a hybrid approach that chooses among six control and observation categories based on the application context. We evaluate seven primitives on Windows 11 and save the results as concrete evidence. Windows Graphics Capture produced usable frames across three monitors at 157 to 265 ms. DXGI Desktop Duplication returned a black frame and adapter errors in the same session. Appium/NovaWindows completed a calculator operation in 1191 ms without moving the cursor. SendInput on a hidden desktop returned ERROR_ACCESS_DENIED. These results show that computer-use agents need a better execution boundary, not one universal input primitive. © 2026 IEEE.