Step-by-Step PyAutoGUI Tutorial: Record and Replay Mouse Actions for GUI Task Automation
The useful version of “record and replay” is not a script that starts clicking as soon as it opens. It should separate four steps: record, inspect, validate, then execute. The example below follows that order and leaves PyAutoGUI’s emergency stop enabled.
It records mouse movement, clicks, and scrolling. It deliberately does not record keyboard text, so passwords and messages are not written to the macro file.
What this tutorial builds
The recorder uses pynput because PyAutoGUI controls the mouse but does not provide a mouse-event recorder. Each event is stored in JSON with a relative timestamp from time.monotonic(). The replay command first checks the file and current screen size.
The two commands look similar, but they do different things:
# Validate and summarize only. This does not move the mouse.
py safe_mouse_macro.py replay macro.json
# Move the mouse only after you have reviewed macro.json.
py safe_mouse_macro.py replay macro.json --execute
📌 The safe default is a dry run. The script cannot control the pointer unless --execute is present.
Install the two packages
Create an empty folder, open a terminal in that folder, and run:
py -m pip install "PyAutoGUI>=0.9.54,<1" "pynput>=1.7.6,<2"
PyAutoGUI’s installation page documents the pip install pyautogui command. Depending on the operating system, pynput may also need accessibility or input-monitoring permission before it can observe mouse events.
Save this as safe_mouse_macro.py
This compact version keeps the important controls visible. It rejects unknown event types, off-screen coordinates, more than 10,000 events, and recordings longer than 15 minutes.
import argparse
import json
import time
from pathlib import Path
MAX_EVENTS = 10_000
MAX_SECONDS = 15 * 60
def save(events, path):
path.write_text(
json.dumps({"format_version": 1, "events": events}, indent=2),
encoding="utf-8",
)
def load_and_validate(path, screen_size):
data = json.loads(path.read_text(encoding="utf-8"))
if data.get("format_version") != 1:
raise ValueError("Unsupported macro format")
events = data.get("events")
if not isinstance(events, list) or not events:
raise ValueError("Macro contains no events")
if len(events) > MAX_EVENTS:
raise ValueError("Too many events")
width, height = screen_size
previous_time = 0.0
for index, event in enumerate(events):
if event.get("type") not in {"move", "button", "scroll"}:
raise ValueError(f"Unsupported event at index {index}")
event_time = event.get("time")
if not isinstance(event_time, (int, float)) or event_time < previous_time:
raise ValueError(f"Invalid time at index {index}")
if event_time > MAX_SECONDS:
raise ValueError("Macro is longer than 15 minutes")
previous_time = event_time
if event["type"] in {"move", "button"}:
x, y = event.get("x"), event.get("y")
if not isinstance(x, int) or not isinstance(y, int):
raise ValueError(f"Invalid coordinates at index {index}")
if not (0 <= x < width and 0 <= y < height):
raise ValueError(f"Off-screen coordinates at index {index}")
if event["type"] == "button":
if event.get("button") not in {"left", "middle", "right"}:
raise ValueError(f"Invalid button at index {index}")
if not isinstance(event.get("pressed"), bool):
raise ValueError(f"Invalid button state at index {index}")
return events
def record(path):
from pynput import keyboard, mouse
events = []
started = time.monotonic()
last_move_time = -0.02
def elapsed():
return round(time.monotonic() - started, 4)
def append(event):
if len(events) >= MAX_EVENTS or elapsed() > MAX_SECONDS:
raise RuntimeError("Recording safety limit reached")
events.append(event)
def on_move(x, y):
nonlocal last_move_time
now = elapsed()
if now - last_move_time >= 0.02:
append({"type": "move", "time": now, "x": int(x), "y": int(y)})
last_move_time = now
def on_click(x, y, button, pressed):
append({
"type": "button",
"time": elapsed(),
"x": int(x),
"y": int(y),
"button": str(button).split(".")[-1],
"pressed": bool(pressed),
})
def on_scroll(_x, _y, dx, dy):
append({"type": "scroll", "time": elapsed(), "dx": int(dx), "dy": int(dy)})
mouse_listener = mouse.Listener(
on_move=on_move,
on_click=on_click,
on_scroll=on_scroll,
)
def on_key(key):
if key == keyboard.Key.esc:
mouse_listener.stop()
return False
print("Recording mouse actions only. Press Esc to stop.")
mouse_listener.start()
with keyboard.Listener(on_press=on_key) as keyboard_listener:
mouse_listener.join()
keyboard_listener.stop()
if not events:
raise ValueError("No mouse events were recorded")
save(events, path)
print(f"Saved {len(events)} events to {path}")
def replay(path, execute):
import pyautogui
events = load_and_validate(path, tuple(pyautogui.size()))
duration = events[-1]["time"]
print(f"Validated {len(events)} events over {duration:.2f} seconds")
if not execute:
print("Dry run only. Review the JSON, then add --execute if it is safe.")
return
pyautogui.FAILSAFE = True
pyautogui.PAUSE = 0.1
print("Starting in 3 seconds. Move the pointer to a screen corner to stop.")
time.sleep(3)
started = time.monotonic()
try:
for event in events:
wait = event["time"] - (time.monotonic() - started)
if wait > 0:
time.sleep(wait)
if event["type"] == "move":
pyautogui.moveTo(event["x"], event["y"])
elif event["type"] == "button":
pyautogui.moveTo(event["x"], event["y"])
action = pyautogui.mouseDown if event["pressed"] else pyautogui.mouseUp
action(button=event["button"])
else:
pyautogui.hscroll(event["dx"])
pyautogui.scroll(event["dy"])
except pyautogui.FailSafeException:
print("Replay stopped by the PyAutoGUI fail-safe.")
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(dest="command", required=True)
record_parser = subparsers.add_parser("record")
record_parser.add_argument("file", type=Path)
replay_parser = subparsers.add_parser("replay")
replay_parser.add_argument("file", type=Path)
replay_parser.add_argument("--execute", action="store_true")
args = parser.parse_args()
if args.command == "record":
record(args.file)
else:
replay(args.file, args.execute)
Record one short task
Close private chats, password managers, and any window that could trigger a purchase or deletion. Then start a short recording:
py safe_mouse_macro.py record macro.json
Perform the mouse actions once and press Esc. Open macro.json in a text editor. You should see events such as move, button, and scroll, each with a relative time. There should be no typed text in the file.
Validate before anything moves
Run the replay command without the execution flag:
py safe_mouse_macro.py replay macro.json
This checks the file structure, duration, event count, and coordinates against the current screen. It then prints a summary and exits.
💡 Why check the screen size? Coordinate macros are fragile. A changed resolution, display scaling, window position, or web page layout can make a previously correct click land on the wrong control. Validation catches off-screen coordinates, but it cannot prove that the button at a coordinate is still the same button.
Execute only in a disposable test target
Use a harmless application first—an empty text editor window or a test page with no account logged in. Keep one hand ready to move the pointer to a screen corner.
py safe_mouse_macro.py replay macro.json --execute
PyAutoGUI documents that its fail-safe raises FailSafeException when the pointer reaches a corner of the primary monitor. The script keeps that protection on and also inserts a short pause after each PyAutoGUI call.
⚠️ Do not use raw coordinate replay for payments, deletion, account settings, or unattended production work. It cannot understand what is on screen. For a durable workflow, prefer an API or a browser automation tool that can identify elements and verify state before acting.
What was tested for this revision
The accompanying implementation was checked with five local unit tests covering JSON round-trip, event summary, off-screen coordinates, non-monotonic timestamps, unknown event types, and unsupported file versions. Those tests do not move the real pointer. Live mouse control was intentionally left for a disposable user-run test because GUI coordinates depend on the user’s current display.
Quick checklist
- Run the validation-only command first.
- Read the JSON before adding
--execute. - Keep
FAILSAFEenabled. - Test in a harmless window with the same display layout.
- Re-record when resolution, scaling, window position, or page layout changes.
- Use APIs or element-aware browser automation for consequential tasks.
Comments
Post a Comment