g3/docs/macax-tools.md

macOS Accessibility Tools Guide

Last updated: January 2025
Source of truth: crates/g3-computer-control/src/macax/

Purpose

G3 includes tools for controlling macOS applications via the Accessibility API. This enables automation of native macOS apps, including those you're building with G3.

Overview

The macOS Accessibility API provides programmatic access to UI elements in any application. G3 exposes this through the macax_* tools, allowing you to:

• List and activate applications
• Inspect UI element hierarchies
• Find elements by role, title, or identifier
• Click buttons and interact with controls
• Read and set values in text fields
• Simulate keyboard input

Setup

1. Enable in Configuration

# ~/.config/g3/config.toml
[macax]
enabled = true

Or use the CLI flag:

g3 --macax

2. Grant Accessibility Permissions

1. Open System Preferences → Security & Privacy → Privacy
2. Select Accessibility in the left sidebar
3. Click the lock icon and authenticate
4. Add your terminal application (Terminal, iTerm2, etc.)
5. Restart your terminal

Note: If using VS Code's integrated terminal, add VS Code to the list.

3. Verify Setup

{"tool": "macax_list_apps", "args": {}}

This should return a list of running applications.

Available Tools

macax_list_apps

List all running applications.

Parameters: None

Example:

{"tool": "macax_list_apps", "args": {}}

Returns:

Running Applications:
- Safari (com.apple.Safari)
- Finder (com.apple.finder)
- Terminal (com.apple.Terminal)
- MyApp (com.example.myapp)

---

macax_get_frontmost_app

Get the currently active (frontmost) application.

Parameters: None

Example:

{"tool": "macax_get_frontmost_app", "args": {}}

Returns:

Frontmost Application: Safari (com.apple.Safari)

---

macax_activate_app

Bring an application to the front.

Parameters:

• app_name (string, required): Application name

Example:

{"tool": "macax_activate_app", "args": {"app_name": "Safari"}}

---

macax_get_ui_tree

Get the UI element hierarchy of an application.

Parameters:

• app_name (string, required): Application name
• max_depth (integer, optional): Maximum tree depth (default: 5)

Example:

{"tool": "macax_get_ui_tree", "args": {"app_name": "Calculator", "max_depth": 3}}

Returns:

UI Tree for Calculator:
└── AXApplication "Calculator"
    └── AXWindow "Calculator"
        ├── AXGroup
        │   ├── AXButton "1" [id: digit_1]
        │   ├── AXButton "2" [id: digit_2]
        │   ├── AXButton "+" [id: add]
        │   └── AXButton "=" [id: equals]
        └── AXStaticText "0" [id: display]

Notes:

• Use lower max_depth for complex apps to avoid overwhelming output
• Elements show role, title, and accessibility identifier (if set)

---

macax_find_elements

Find UI elements matching criteria.

Parameters:

• app_name (string, required): Application name
• role (string, optional): Element role (e.g., "button", "textField")
• title (string, optional): Element title/label
• identifier (string, optional): Accessibility identifier

Example:

{"tool": "macax_find_elements", "args": {
  "app_name": "Safari",
  "role": "button"
}}

Returns:

Found 5 elements:
1. AXButton "Back" [id: BackButton]
2. AXButton "Forward" [id: ForwardButton]
3. AXButton "Reload" [id: ReloadButton]
4. AXButton "Share" [id: ShareButton]
5. AXButton "New Tab" [id: NewTabButton]

---

macax_click

Click a UI element.

Parameters:

• app_name (string, required): Application name
• identifier (string, optional): Accessibility identifier
• title (string, optional): Element title
• role (string, optional): Element role

At least one of identifier, title, or role must be provided.

Examples:

// Click by identifier (most reliable)
{"tool": "macax_click", "args": {
  "app_name": "Calculator",
  "identifier": "digit_5"
}}

// Click by title
{"tool": "macax_click", "args": {
  "app_name": "Calculator",
  "title": "5"
}}

// Click by role and title
{"tool": "macax_click", "args": {
  "app_name": "Safari",
  "role": "button",
  "title": "Reload"
}}

---

macax_set_value

Set the value of a UI element (text fields, sliders, etc.).

Parameters:

• app_name (string, required): Application name
• identifier (string, optional): Accessibility identifier
• title (string, optional): Element title
• value (string, required): Value to set

Example:

{"tool": "macax_set_value", "args": {
  "app_name": "TextEdit",
  "role": "textArea",
  "value": "Hello, World!"
}}

---

macax_get_value

Get the current value of a UI element.

Parameters:

• app_name (string, required): Application name
• identifier (string, optional): Accessibility identifier
• title (string, optional): Element title

Example:

{"tool": "macax_get_value", "args": {
  "app_name": "Calculator",
  "identifier": "display"
}}

Returns:

Value: 42

---

macax_press_key

Simulate a key press.

Parameters:

• key (string, required): Key to press
• modifiers (array, optional): Modifier keys

Supported modifiers: command, shift, option, control

Examples:

// Simple key press
{"tool": "macax_press_key", "args": {"key": "a"}}

// With modifiers (Cmd+S)
{"tool": "macax_press_key", "args": {
  "key": "s",
  "modifiers": ["command"]
}}

// Multiple modifiers (Cmd+Shift+N)
{"tool": "macax_press_key", "args": {
  "key": "n",
  "modifiers": ["command", "shift"]
}}

// Special keys
{"tool": "macax_press_key", "args": {"key": "return"}}
{"tool": "macax_press_key", "args": {"key": "escape"}}
{"tool": "macax_press_key", "args": {"key": "tab"}}
{"tool": "macax_press_key", "args": {"key": "delete"}}

Special key names:

• return, enter
• escape, esc
• tab
• delete, backspace
• space
• up, down, left, right
• home, end, pageup, pagedown
• f1 through f12

Common Roles

Role	Description
button	Clickable button
textField	Single-line text input
textArea	Multi-line text input
checkbox	Checkbox control
radioButton	Radio button
popUpButton	Dropdown/popup menu
slider	Slider control
table	Table view
list	List view
outline	Outline/tree view
group	Container group
window	Application window
sheet	Modal sheet
dialog	Dialog window
staticText	Non-editable text
image	Image element
scrollArea	Scrollable container
toolbar	Toolbar
menuBar	Menu bar
menu	Menu
menuItem	Menu item

Best Practices

1. Use Accessibility Identifiers

When building apps you'll automate with G3, add accessibility identifiers:

SwiftUI:

Button("Submit") { ... }
    .accessibilityIdentifier("submit_button")

UIKit:

button.accessibilityIdentifier = "submit_button"

AppKit:

button.setAccessibilityIdentifier("submit_button")

Identifiers are more reliable than titles (which may be localized).

2. Inspect Before Automating

Always inspect the UI tree first:

{"tool": "macax_get_ui_tree", "args": {"app_name": "MyApp", "max_depth": 4}}

This helps you understand:

• Element hierarchy
• Available identifiers
• Correct role names

3. Activate App First

Some actions require the app to be frontmost:

{"tool": "macax_activate_app", "args": {"app_name": "MyApp"}}
{"tool": "macax_click", "args": {"app_name": "MyApp", "identifier": "button1"}}

4. Handle Timing

UI updates may take time. If an element isn't found:

1. Wait briefly
2. Retry the operation
3. Check if the app state changed

5. Prefer Identifiers Over Titles

// Good: Uses identifier
{"tool": "macax_click", "args": {"app_name": "MyApp", "identifier": "save_btn"}}

// Less reliable: Uses title (may be localized)
{"tool": "macax_click", "args": {"app_name": "MyApp", "title": "Save"}}

Example: Automating Calculator

// 1. Activate Calculator
{"tool": "macax_activate_app", "args": {"app_name": "Calculator"}}

// 2. Inspect UI
{"tool": "macax_get_ui_tree", "args": {"app_name": "Calculator", "max_depth": 3}}

// 3. Click "5"
{"tool": "macax_click", "args": {"app_name": "Calculator", "title": "5"}}

// 4. Click "+"
{"tool": "macax_click", "args": {"app_name": "Calculator", "title": "+"}}

// 5. Click "3"
{"tool": "macax_click", "args": {"app_name": "Calculator", "title": "3"}}

// 6. Click "="
{"tool": "macax_click", "args": {"app_name": "Calculator", "title": "="}}

// 7. Read result
{"tool": "macax_get_value", "args": {"app_name": "Calculator", "role": "staticText"}}

Troubleshooting

"Accessibility permission denied"

1. Check System Preferences → Security & Privacy → Accessibility
2. Ensure your terminal app is listed and checked
3. Restart the terminal after granting permission

"Application not found"

1. Use exact app name (case-sensitive)
2. Run macax_list_apps to see available apps
3. App must be running

"Element not found"

1. Inspect UI tree to verify element exists
2. Check identifier/title spelling
3. Element may be in a different window or sheet
4. App state may have changed

"Cannot perform action"

1. Element may be disabled
2. App may need to be frontmost
3. Element may not support the action
4. Check element role supports the operation

Slow Performance

1. Reduce max_depth in macax_get_ui_tree
2. Use specific identifiers instead of searching
3. Complex apps have large UI trees

Comparison with Other Tools

Feature	macax	Vision Tools	WebDriver
Native apps	✅	✅ (via OCR)	❌
Web browsers	✅	✅	✅
Electron apps	✅	✅	Partial
Reliability	High	Medium	High
Setup	Permissions	None	Driver
Speed	Fast	Slower	Medium

Use macax when:

• Automating native macOS apps
• You control the app and can add identifiers
• Need reliable, fast automation

Use Vision tools when:

• App doesn't expose accessibility
• Need to find text visually
• Cross-platform approach needed

Use WebDriver when:

• Automating web content
• Need JavaScript execution
• Testing web applications