What a GUI Agent Is

A GUI agent does its work through the same interface a person would use. It inherits that interface’s constraints and none of the guarantees a private API would give it. Once that is true, the honest unit of measurement is the run, recorded as a trace, judged against the terminal state, not against the agent’s own report of success. Scoring dimensions, rubrics, and judges live in Evaluation. This page only names the objects those instruments read.

Interface constraint

The agent has no private channel into the application. It sees what is on the screen, chooses an action a person could take (tap, type, scroll, press a key), and waits for the interface to respond. Rendering, focus, timing, and layout are part of the problem.

Figure 1.The loop. Only a slice of environment state is visible. Observation enters the agent; action and check return to the environment. Animated edges are emphasis only; the diagram reads without motion.

Figure 2.Illustrative plate. Most of the interface stays outside the agent's sharp window. Atmosphere only; not a measurement.

A recurring failure in this class of system is a confident success message over a screen that still shows the task unfinished. Score what the screen shows at the end, not a moment mid-run that briefly looked right. If success cannot be pointed at in a recorded frame, it has not been shown.

Figure 3.Illustrative plate. The screen is the evidence; a Done claim beside an unfinished screen is not. Atmosphere only; not a measurement.

Two runs can reach the same end state in seven steps or in fifty. One of them may have fired a purchase, a send, a delete, or a booking without confirmation. The end screen alone cannot tell those runs apart; the ordered steps can.

Definition

A run is one attempt at one task. A trace is its record (task text, ordered steps, captured frames, end state). A step is one observe, decide, act cycle with the screen before and after. The terminal state is the end screen, the evidence for any completion claim. A consequential action is a step that needs authorization before it fires.

Reading. After the fact, the trace is the only object available to audit or score. The ordered steps are the route. Collapse the run to a single pass or fail and that route averages away.

Object	Meaning
Run	One attempt at one task, from the opening screen until the agent stops
Trace	The record of a run
Step	One observe, decide, act cycle
Terminal state	End screen; evidence for completion
Consequential action	Step that needs authorization before it fires

What measurement inherits

Whether one run succeeded is a Bernoulli outcome. A set of $n$ independent runs with $k$ successes yields a binomial proportion $\hat{p} = k/n$. A proportion without its $n$ and an interval is not yet a result (Brown, Cai, and DasGupta 2001).

Per-step success and per-run success are different quantities. Under an independence assumption, if each step succeeds with probability $p$, then a run of $n$ steps succeeds with probability

$$P(\text{run succeeds}) = p^{n}.$$

Formula

$p$: per-step success under the independence sketch. $n$: steps in the run. $P(\text{run succeeds}) = p^{n}$.

Reading. A high per-step number and a low per-run number can describe the same agent, because length compounds. Reporting only step accuracy flatters systems that fail on long tasks.

Limit. Steps in a live interface are not independent: an error changes the state the next step observes. Treat $p^{n}$ as an intuition pump for why length matters, not as a deployed rate (Ross and Bagnell 2010).

Figure 4.Illustrative compounding at p = 0.95 under independence. Bar heights are computed in the figure from that p and the step counts shown. Not a measured rate; independence is the wrong assumption in practice.

If one number cannot carry the run, the output has to be several, and the unit of report is the run with its trace. How those numbers are named, scored, and gated is Evaluation’s job. What the agent can see and do, and how a long run holds together, are the next two pages in this section.

References

Work	Citation
Brown, L. D., Cai, T. T., and DasGupta, A. (2001). Interval Estimation for a Binomial Proportion. Statistical Science 16(2), 101-133.
Ross, S. and Bagnell, J. A. (2010). Efficient Reductions for Imitation Learning. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 9:661-668.