From Manual Clicks to Full GUI Bots: A Step‑by‑Step PyAutoGUI Tutorial to Automate a Daily Desktop Workflow

A reliable GUI step verifies the expected screen state before and after a click.

A daily desktop bot becomes useful when it can answer two questions before every click: “Is the expected control visible?” and “Did the click produce the expected next screen?” If either answer is no, the workflow should stop instead of guessing.

This tutorial builds that narrower, safer bot. It does not replay recorded coordinates. If coordinate recording is what you need, the separate mouse record-and-replay tutorial explains that workflow. Here, each step is guided by a reference image and followed by a visual check.

📌 The operating rule is simple: wait for a known screen state, click once, verify the next state, then continue.

Why fixed coordinates are not enough

A coordinate such as (720, 410) identifies a place on one screen. It does not identify the button occupying that place. Window position, display scaling, an unexpected dialog, or a slow-loading application can all change what receives the click.

PyAutoGUI’s official screenshot documentation provides locateOnScreen() and locateCenterOnScreen() for finding a reference image. Current documentation also notes that a missing image raises ImageNotFoundException. The optional confidence setting requires OpenCV.

Image matching is still not an understanding of the interface. A similar-looking control can be matched, and a redesigned button can disappear. The important improvement is therefore not “use images and trust them”; it is use bounded waits, verify transitions, and stop on uncertainty.

Design the workflow before writing code

Write each manual step as three fields:

Step	Click only when	Continue only when
Open report	open-report.png is visible	report-ready.png appears
Export CSV	export-csv.png is visible	export-complete.png appears

Capture each image from the exact application and display setup you will use. Crop tightly enough to avoid unrelated text, but include enough visual detail to distinguish the control. Do not use screenshots containing account names, customer data, or other private information.

💡 A verification image should represent the result of an action, not the button that was just clicked. For example, use a “Report ready” heading or an “Export complete” message.

Install the dependencies

py -m pip install "PyAutoGUI>=0.9.54,<1" "opencv-python>=4.10,<5"

OpenCV is included because this example uses confidence=0.9. Without the confidence argument, PyAutoGUI can perform exact image matching without OpenCV.

Describe the daily workflow in JSON

Save this as workflow.json and replace the image paths with your own captures:

{
  "steps": [
    {
      "name": "Open report",
      "target_image": "images/open-report.png",
      "verify_image": "images/report-ready.png",
      "timeout_seconds": 10
    },
    {
      "name": "Export CSV",
      "target_image": "images/export-csv.png",
      "verify_image": "images/export-complete.png",
      "timeout_seconds": 15
    }
  ]
}

The timeout keeps a missing control from turning into an endless search. The script accepts values from 0.5 to 60 seconds and stops when a required state does not appear.

Save the runner as state_aware_gui_bot.py

import argparse
import json
import time
from dataclasses import dataclass
from pathlib import Path


class WorkflowStopped(RuntimeError):
    pass


@dataclass(frozen=True)
class Step:
    name: str
    target_image: Path
    verify_image: Path | None
    timeout_seconds: float


def load_plan(path):
    raw = json.loads(path.read_text(encoding="utf-8"))
    items = raw.get("steps") if isinstance(raw, dict) else None
    if not isinstance(items, list) or not items:
        raise ValueError("Plan must contain a non-empty steps list")

    steps = []
    for index, item in enumerate(items):
        name = str(item.get("name") or "").strip()
        target = str(item.get("target_image") or "").strip()
        verify = str(item.get("verify_image") or "").strip()
        timeout = item.get("timeout_seconds", 10)
        if not name or not target:
            raise ValueError(f"Step {index} needs name and target_image")
        if not isinstance(timeout, (int, float)) or not 0.5 <= timeout <= 60:
            raise ValueError(f"Step {index} has an invalid timeout")
        steps.append(Step(
            name=name,
            target_image=(path.parent / target).resolve(),
            verify_image=(path.parent / verify).resolve() if verify else None,
            timeout_seconds=float(timeout),
        ))
    return steps


def validate_assets(steps):
    missing = []
    for step in steps:
        for image in (step.target_image, step.verify_image):
            if image is not None and not image.is_file():
                missing.append(str(image))
    if missing:
        raise ValueError("Missing reference images: " + ", ".join(missing))


class PyAutoGuiAdapter:
    def __init__(self, confidence=0.9, region=None):
        import pyautogui

        self.gui = pyautogui
        self.confidence = confidence
        self.region = region
        self.gui.FAILSAFE = True
        self.gui.PAUSE = 0.2

    def locate(self, image):
        try:
            point = self.gui.locateCenterOnScreen(
                str(image),
                confidence=self.confidence,
                region=self.region,
            )
        except self.gui.ImageNotFoundException:
            return None
        return None if point is None else (int(point.x), int(point.y))

    def click(self, point):
        self.gui.click(*point)


def wait_for_image(adapter, image, timeout, poll=0.25):
    deadline = time.monotonic() + timeout
    while time.monotonic() < deadline:
        point = adapter.locate(image)
        if point is not None:
            return point
        time.sleep(poll)
    return None


def run(steps, adapter, execute=False):
    if not execute:
        print("Dry run: " + " -> ".join(step.name for step in steps))
        print("No mouse input was sent.")
        return

    for step in steps:
        point = wait_for_image(adapter, step.target_image, step.timeout_seconds)
        if point is None:
            raise WorkflowStopped(f"Stopped before {step.name}: target not found")
        adapter.click(point)

        if step.verify_image is not None:
            verified = wait_for_image(adapter, step.verify_image, step.timeout_seconds)
            if verified is None:
                raise WorkflowStopped(f"Stopped after {step.name}: result not verified")
        print(f"Completed: {step.name}")


parser = argparse.ArgumentParser()
parser.add_argument("plan", type=Path)
parser.add_argument("--execute", action="store_true")
args = parser.parse_args()

steps = load_plan(args.plan.resolve())
if args.execute:
    validate_assets(steps)
    print("Starting in 3 seconds. Move the pointer to a screen corner to stop.")
    time.sleep(3)
    adapter = PyAutoGuiAdapter()
else:
    adapter = None

run(steps, adapter, execute=args.execute)

Run the plan without clicking

py state_aware_gui_bot.py workflow.json

The default command only prints the step order. It does not import PyAutoGUI or move the pointer. This is the moment to check that every step has a target image, a verification image, and a sensible timeout.

Test one harmless copy before execution

Open a disposable test application containing no account, payment, deletion, or customer data. Arrange the windows exactly as they were when the reference images were captured, then run:

py state_aware_gui_bot.py workflow.json --execute

PyAutoGUI’s fail-safe remains enabled. Its documentation states that moving the pointer to a corner raises FailSafeException, giving you an emergency stop while PyAutoGUI calls are running.

⚠️ A successful test does not make a coordinate-and-image bot safe for consequential unattended work. Do not use this pattern for payments, destructive actions, account permissions, or workflows where a mistaken click cannot be reversed. Prefer an application API or element-aware browser automation when one is available.

Reduce false matches and slow searches

Use these adjustments only after the basic workflow stops correctly:

• Crop reference images around distinctive controls rather than entire windows.
• Restrict the search to a stable screen region when possible.
• Keep a timeout; never replace it with an infinite loop.
• Use a result image that is visually different from the clicked button.
• Re-capture references after display scaling, theme, language, or application design changes.
• Start with a higher confidence and lower it only after inspecting false matches.

The official PyAutoGUI documentation notes that locate calls can take roughly one or two seconds on a 1920×1080 screen and suggests a smaller region to reduce the search area. Treat that timing as documentation guidance, not a guarantee for your machine.

What was tested for this revision

The accompanying implementation was checked with five local unit tests. They cover successful transition verification, a missing target, a missing result state, dry-run behavior, and invalid timeout configuration. The tests use a fake GUI adapter, so they verify stop/continue logic without moving the real pointer. Live matching still depends on the user’s screenshots, application, operating system, and display settings.

Practical stopping rule

The workflow is ready for a limited test only when all of these are true:

• Dry run shows the intended step order.
• Every reference image contains no private data.
• Every click has a separate result-state image.
• A missing image stops the workflow before the next click.
• The corner fail-safe is still enabled.
• The first execution happens in a reversible test target.

Official references

• PyAutoGUI screenshot and locate functions
• PyAutoGUI fail-safes and global pause
• PyAutoGUI mouse control functions
• Python time.monotonic() documentation
• Python dataclasses documentation