In brief
- The deciding factor is not the newest technology, but the most stable controllable interface for the specific step.
- RPA follows predefined UI steps and selectors; computer use interprets the interface case by case with a model.
- A good process may combine APIs, RPA and computer use, while keeping their boundaries visible.
- Success rate alone is insufficient: incorrect actions, human rework, UI drift, runtime, cost and safe stopping belong in the assessment.
What distinguishes APIs, RPA and computer use?
An API is designed for software and uses explicit endpoints, fields and error codes. RPA targets an existing user interface and executes predefined steps, often through UI elements or selectors. Computer use sees the interface through screenshots or other visual signals, interprets its state and generates mouse and keyboard actions dynamically.
The most visually flexible option is not automatically best. The more a model chooses the next UI step itself, the harder reproducible testing becomes and the more important execution boundaries are. Where a stable API exists, it generally remains the more dependable system boundary for writing, posting or approving.
| Access route | Strength | Typical weakness | Suitable use |
|---|---|---|---|
| API | Structured data, explicit contract, targeted permissions and reproducible error handling | Not every legacy application exposes the required function; integration development is needed | Stable queries and transactions with defined domain semantics |
| RPA | Automates existing, repeatable interfaces without changing the target system | Selectors, windows or sequences break when the UI or exceptions change | Rule-based legacy processes with a stable interface and manageable variation |
| Computer use | Can interpret visually variable interfaces and context-dependent navigation | Non-deterministic; screenshots may contain sensitive data and untrusted instructions | Bounded, hard-to-integrate cases where flexible UI interpretation brings measurable benefit |
| Human handling | Can address new exceptions, ambiguity and accountability directly | Time-consuming and not infinitely scalable | Rare, high-risk or insufficiently understood cases and escalations |
How should the right access route be selected?
Selection should happen per process step, not once for an entire application. A system might reliably read data through an API, open a rare legacy form through RPA and use AI to interpret an unstructured attachment. Critical write actions still remain at the most stable available boundary.
- Step 1
Define the domain action
Clarify which information is read or which exact change is executed, and what consequence an error would have.
- Step 2
Check the API first
Prefer a documented endpoint when it offers the required semantics, authentication, authorisation and error handling.
- Step 3
Assess UI stability and rule structure
If the sequence is known and the interface exposes robust elements, RPA can bridge the missing interface.
- Step 4
Demonstrate the need for flexibility
Consider computer use only when visual variation or inconsistent navigation cannot be handled economically with fixed rules.
- Step 5
Define the fallback before production
Timeout, an unexpected screen, an authorisation error or low confidence must lead to stopping, a queue or human takeover.
What does a dependable hybrid architecture look like?
AI should not click unchecked from visible information all the way to a final posting. A robust design separates interpretation, validated domain data and execution. Computer use or document AI may create a draft; a rules component checks required fields and values; the final action uses an API or a tightly defined RPA step.
- Flexible perception
- AI reads an attachment or finds a relevant location in a variable interface. Its result remains a proposal with provenance and uncertainty.
- Deterministic validation
- Schemas, business rules, duplicate checks and permitted values are enforced in software before any action is possible.
- Bounded execution
- An API or RPA receives only validated fields and may execute one named transaction rather than navigating freely.
- Evidence and recovery
- Input, decision, action, result and exception remain correlated; failure enters an owned queue.
Which permissions and safeguards does UI automation need?
A desktop or browser often exposes more data than the current task needs. Screenshots can capture names, messages, passwords, health or financial information. Computer use should therefore run in a dedicated virtual environment with minimal permissions, limited applications and restricted network access where possible.
Content on webpages, in documents or images is not automatically a trusted instruction. A model may confuse such text with the real assignment. Critical decisions such as payment, contract acceptance, data export or deletion need a separate, understandable approval based on the actual action and values.
- Use a dedicated service identity and minimum roles rather than personal administrator accounts
- Technically allow or deny target applications, domains, actions, file paths, amounts and recipients
- Do not have secrets typed visibly; provide short-lived credentials through controlled mechanisms
- Classify screenshots before retention, minimise sensitive regions and limit storage duration
- Treat external screen content as potential prompt injection and never as authorisation
- Require context-rich human approval before irreversible or legally significant actions
How should stability and operations be measured?
UI automation fails not only on domain logic, but also on changed windows, selectors, loading times, dialogs or screen resolutions. Microsoft notes, for example, that changed window names or element structures can invalidate existing selectors. Such UI drift belongs in monitoring and maintenance as an expected operational reality.
Tests need a representative case library with standard cases, exceptions, authorisation failures, pop-ups, slow responses and intentionally changed interfaces. Computer use additionally needs visual ambiguity, manipulated screen content and safe non-execution. Relevant tests run again after every model, prompt, application or operating-system update.
| Measure | What it reveals | Warning sign |
|---|---|---|
| End-to-end success rate | Share of correctly completed cases including the domain outcome | Technically completed but incorrectly posted cases are not tracked separately |
| Incorrect-action and stop rate | Wrong clicks, disallowed actions and safe stops by cause | High overall success hides rare but severe incorrect actions |
| Human rework | Time spent reviewing, correcting, restarting and handling exceptions | Automation merely moves work into a less visible queue |
| Drift and maintenance effort | Failures after UI, model or system changes and time to repair | Every small interface update causes unpredictable production outages |
| Runtime and cost | Compute, licence and infrastructure cost per successful case | A flexible computer-use route is slower and more expensive than an available API integration |
Example: order entry in a legacy ERP
An organisation receives orders as email attachments. Document AI extracts a draft. Customer number, items, price and delivery address are checked against master data and business rules. The legacy ERP has a read API but no posting API: master data therefore comes through the API, while an RPA flow opens a stable order form and enters validated fields. Computer use is limited to two rare variants where the screen layout differs by customer type; it runs in an isolated session and may save only a draft. A person approves price deviations. Unexpected dialogs, unknown customers or UI changes enter a queue rather than triggering further clicks.
What to remember
Prefer the most structured and testable boundary for each action: API before stable RPA, and RPA before open-ended computer use. Apply visual flexibility selectively and contain it with validation, minimal permissions, approvals and a dependable fallback.
Sources and further reading
These primary sources provide further detail on definitions, technical foundations or responsible use.
Content reviewed