Driving the Windows desktop as an agent: UI Automation beats screenshots, plus the silent-failure traps

Driving the Windows desktop as an agent

All numbers were measured in August 2026 on Windows 11 with a mixed-DPI dual-monitor setup.

The screenshot is almost always the wrong tool

| Approach | Cost | |---|---| | Full-desktop screenshot sent to a vision model | about 8,600 image tokens, and data leaves the machine | | UIA observation of a dense window | about 2,100 tokens, local, exact | | Same window, diff after acting | about 78 tokens | | Enumerate all open windows | about 496 tokens, 25 ms | | Read one control's value | about 1 token, 77 ms |

Escalation ladder, best first: a connected tool (MCP), the app's CLI, a PowerShell module or OS CLI, COM, URI or cloud REST API, browser DOM refs, UIA control patterns, MSAA, UIA geometry with a guarded click, and finally screenshot plus vision.

A curated app-to-route table matched 58% of apps; generic probing matched 11%. For Teams, To Do and Outlook the best route is Microsoft Graph, a cloud API that no local probe can discover.

UIA control patterns are the modal answer (6 of 12 live apps)

InvokePattern, TogglePattern, ValuePattern.SetValue, RangeValuePattern, SelectionItem and ExpandCollapse actuate a control without moving the mouse. They are immune to occlusion, DPI, z-order and window position. Paint alone exposed 50 Invoke, 30 Toggle, 12 ExpandCollapse and 2 RangeValue patterns. Notepad's File menu went from Collapsed to Expanded in 17 ms with the cursor provably unmoved.

With Paint covering Notepad, a guarded click refused while Expand on the same covered element succeeded.

Caveats:

• Pattern invocation can silently no-op. ExpandCollapse.Expand() on Notepad's File menu returned success and opened nothing, so verify after about 300 ms. An immediate re-read gives the old state.
• ScrollItem and VirtualizedItem are not actuation. Counting "any pattern" scored Chrome as 100% actuatable against a true 46%.

The occlusion trap

IsOffscreen means "scrolled out of its own container", not "visible". It caught 0 of 225 occluded controls. With Paint over Settings, 0 of 64 click points hit the target; over Explorer 40 of 201 did, which is the dangerous case because a 20% success rate looks like flakiness. UIA once reported Paint's "Zoom in" button as clickable while those pixels showed a web page under a Chrome toast.

Guard with WindowFromPoint plus GetAncestor(GA_ROOT) inside the click primitive, not at observe time, to avoid a time-of-check/time-of-use gap. isForeground is not the test. An open light-dismiss XAML popup also swallows the next click anywhere on screen, even with no overlap.

DPI: use per-monitor-v2, not SetProcessDPIAware

On 2560x1440 at 150% plus 1920x1080 at 100%:

• DPI-unaware is internally inconsistent (scaled size, physical origin) and unrecoverable.
• SetProcessDPIAware() inflates the 100% monitor to 2880x1620 at x=3840, so clicks land about 1280 px off.
• SetProcessDpiAwarenessContext((IntPtr)-4) is correct.

Call it before System.Windows.Forms and UIAutomationClient load. Declare the argument as IntPtr, not int, because the -4 sentinel can arrive as 0x00000000FFFFFFFC. Assert that the sum of monitor widths equals the virtual-screen width and abort rather than emit coordinates. The bug is exactly zero on the 100% monitor, so testing there concludes it works.

Silent wrong-answer classes

• Elevated windows: UIPI blocks a non-elevated read but does not raise an error. Task Manager returned 18 named, enabled, on-screen nodes with 1 pattern in 28 ms, indistinguishable from a sparse app. Detect it with OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION) plus GetTokenInformation(TokenElevation) before concluding an app is opaque.
• MSAA proxies: the managed client does not load them by default. msinfo32 gave 9 pattern-less nodes without them and 165 elements, including a full menu bar, with them registered.
• UWP: MainWindowHandle is 0 and the window belongs to ApplicationFrameHost. Resolve by scanning descendants for a foreign PID. Looking for a Windows.UI.Core.CoreWindow child fails while the app is suspended, which is exactly when it is about to be activated.
• Virtualisation: System32 with 4,867 files exposed 26 ListItems (the viewport). "Not found" from a UIA search is meaningless without ScrollIntoView.
• Minimized Win32 and Electron windows return full actionable trees at rects near -32000,-32000. SetCursorPos never fails; it clamps, so those clicks land on a real window elsewhere.
• Virtual-screen dead zone: with unequal monitor heights, about 10% of the virtual rect maps to no monitor. Validate against the union of monitor rects, not the bounding box.
• App identity: ChatGPT.exe has ProductName "Codex"; msinfo32 carries the generic Windows string shared by every in-box tool (a cache-key collision); msedgewebview2 is Teams. Key on package id or exe basename, never ProductName.
• Timing: a fresh VS Code had 13 descendants at 61 ms and 163 at 1.6 s. One FindAll against a frozen UI thread ran 39.6 s without an exception, so capture out of process with a hard kill.

Keyboard shortcuts: UIA covers about a third

Only 9 of 29 live windows exposed any AcceleratorKey; VS Code and ChatGPT exposed none. Merge three sources: RT_ACCELERATOR (PE resource type 9; 22% of msinfo32's bindings exist nowhere else), GetMenu plus GetMenuStringW (command ids join to RT_ACCELERATOR exactly), then UIA. Pressing Tab to discover tab order mutates focus and can trigger validation.

Privacy

Any UIA plus screenshot tool is a whole-desktop egress path. ValuePattern.Value returns live edit-control text, and element names leak too: Notepad names tabs after the first line of the buffer, and one window emitted 877 characters of document text. Redaction by default is sensible.