> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tazolabs.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Personal Assistant for Mac

> Turn spoken desktop commands into bounded action plans with hierarchical Tazo decisions over the original audio.

A desktop voice assistant needs to determine more than which app to open. "Open Finder
and show Downloads" contains an app, an operation and an exact target. Putting every
possible combination into one question would create a large, repetitive menu.

This cookbook builds a small hierarchical router with the [Tazo Python SDK](https://docs.tazolabs.com/sdk/python/overview.md).
Tazo Large hears the **same original audio** at every stage, with earlier selections as
context. Each answer chooses from a small local menu. The application assembles an action
plan and releases it only when the required decisions pass.

The [runnable backend and recordings](https://github.com/tazolabs/build-with-tazo-cookbooks/tree/main/tazo-assistant)
print plans and traces. Device execution is an adapter you can add after reviewing a plan.

```mermaid theme={null}
flowchart TD
    wav["spoken command WAV"] --> shape["Tazo: whole command structure"]
    shape -->|app command| app["Tazo: app in spoken order"]
    app -->|app plus action| op["Tazo: operation"]
    op --> target["Tazo: exact target"]
    app -->|open only| check["Tazo: all and only requested effects?"]
    target --> check
    shape -->|timer| duration["Tazo: 30 / 60 / 120 minutes"]
    shape -->|Coding mode| fixed["fixed local plan"]
    check -->|complete| plan["release typed plan"]
    duration --> plan
    fixed --> plan
    check -->|missing| stop["discard plan; ask for clarification"]
```

Every model stage also has `NO_TOOL` and confidence checks. An unsupported or uncertain
choice stops routing before a plan is released.

## Setup

Use Python 3.11 or newer:

```bash theme={null}
python -m pip install 'tazo-sdk>=0.1.2'
export TAZO_API_KEY='your-key'
```

Commands must be nonempty mono 16 kHz PCM16 WAVs of at most 30 seconds. This backend runs
without macOS. The source app's hotkey recorder, SwiftUI interface and native device
adapters are outside this recipe; its hierarchical audio-routing pattern is preserved.

## Define the command tree

This example supports a small subset of the source assistant:

| Structure | Choices / result |
| - | - |
| Open one app | Safari, Notes, Finder |
| One app plus one action | Safari: saved site or window position; Notes: window position; Finder: standard folder or window position |
| Open two apps | Two different supported apps, in spoken order, with no extra actions |
| Standalone timer | Exactly 30, 60 or 120 minutes |
| Coding mode | Fixed sequence: Cursor, Linear, Codex, Slack |

General app commands select exact targets from the local maps below. The model returns an
ID such as `downloads`; local code supplies the actual target. It cannot invent a URL,
app name, shell command or executable argument. Coding mode's extra apps are available
only through that named preset.

```python theme={null}
import argparse
import asyncio
import json
from dataclasses import dataclass, field
from pathlib import Path

from tazo import AsyncTazo, wav_chunks
from tazo.types import ChoiceAnswer

MODEL = "tazo-large"
MIN_CONFIDENCE = 0.65
MIN_MARGIN = 0.12
MAX_DECISIONS = 10
TEXT_BUDGET = 1000
EXAMPLES = Path(__file__).resolve().parent / "examples"

SHAPES = {
    "open_app": "Open or switch to one app; no further action.",
    "app_action": "Open one app, then do one action such as show a folder or move its window.",
    "two_apps": "Open two distinct apps in spoken order, with no other actions.",
    "timer": "Start a standalone timer for 30, 60 or 120 minutes.",
    "coding_mode": "Explicitly enter Coding mode: open Cursor, Linear, Codex and Slack.",
}
APPS = {"safari": "Safari browser.", "notes": "Notes app.", "finder": "Finder file browser."}
OPERATIONS = {
    "safari": {"open_site": "Open a saved website.", "layout": "Move or resize its window."},
    "notes": {"layout": "Move or resize its window."},
    "finder": {"open_folder": "Show a standard folder.", "layout": "Move or resize its window."},
}
TARGETS = {
    "open_site": {"youtube": "YouTube.", "tazo": "Tazo Labs website."},
    "open_folder": {"desktop": "Desktop.", "documents": "Documents.", "downloads": "Downloads."},
    "layout": {"left": "Left half.", "right": "Right half.", "fill": "Fill the usable screen."},
}
# The model chooses IDs. Only local code supplies concrete URLs and native targets.
VALUES = {
    "open_site": {"youtube": "https://www.youtube.com", "tazo": "https://tazolabs.com"},
    "open_folder": {"desktop": "Desktop", "documents": "Documents", "downloads": "Downloads"},
    "layout": {"left": "left", "right": "right", "fill": "fill"},
}
CODING_PLAN = [{"tool": "open_app", "app": app} for app in ("cursor", "linear", "codex", "slack")]


class UnresolvedCommand(Exception):
    pass
```

## Ask one question at a time

`Router.ask` sends the original WAV and a fresh question for the current stage. Earlier
selections tell Tazo what has already been chosen. A stage needs at least 65% confidence
and a 12-point lead over the runner-up. `NO_TOOL` or an unrecognized choice never passes.

The helper records a trace and enforces a ten-decision ceiling and a 1,000-character text
budget per stage. If a menu grows too large, shorten descriptions or split the stage into
smaller choices rather than silently dropping options.

```python theme={null}
@dataclass
class Router:
    audio: Path
    trace: list[dict] = field(default_factory=list)

    async def ask(self, client: AsyncTazo, stage: str, instructions: str, options: dict, prior: list) -> str:
        if len(self.trace) >= MAX_DECISIONS:
            raise UnresolvedCommand("Decision limit reached; no plan was released.")
        state = (
            "Desktop voice command. Listen to the whole recording at every stage. "
            "Only listed commands are supported. Earlier selections: "
            + json.dumps(prior, separators=(",", ":"))
        )
        criteria = {**options, "NO_TOOL": "Unsupported, unclear, silent, or no exact offered match."}
        size = len(state) + len(stage) + len(instructions) + sum(len(k) + len(v) for k, v in criteria.items())
        if size > TEXT_BUDGET:
            raise ValueError("Shorten this stage's state, instructions or choices before sending it.")
        result = await client.decisions.create(
            audio=self.audio,
            model=MODEL,
            state=state,
            questions={"route": {"type": "choice", "instructions": instructions, "criteria": criteria}},
        )
        answer = result.answers["route"]
        assert isinstance(answer, ChoiceAnswer)
        ranked = sorted(answer.probabilities.values(), reverse=True)
        runner_up = ranked[1] if len(ranked) > 1 else 0.0
        trusted = (
            answer.choice in options
            and answer.confidence >= MIN_CONFIDENCE
            and answer.confidence - runner_up >= MIN_MARGIN
        )
        self.trace.append({"stage": stage, "choice": answer.choice, "confidence": round(answer.confidence, 2)})
        if not trusted:
            raise UnresolvedCommand(f"Unresolved {stage}; ask the speaker to clarify.")
        return answer.choice
```

## Assemble the plan

Add this method to `Router`. First choose a structure for the **whole** command. Open-only
commands select one or two distinct apps. An app-action command then selects an operation
from that app's menu and a target from that operation's menu.

A final app-command question compares the entire recording with the plan: opening Finder
does not also fulfill "show Downloads." A confident `missing` answer discards the plan,
just as an uncertain stage does. This recipe asks the speaker to clarify rather than trying
to repair a partial plan automatically.

Timers select one exact duration, and Coding mode expands a fixed local preset; these two
branches return directly after their bounded decisions.

```python theme={null}
    async def build_plan(self, client: AsyncTazo) -> list[dict]:
        shape = await self.ask(
            client, "shape", "Which structure matches ALL spoken effects? Unsupported combinations use NO_TOOL.", SHAPES, []
        )
        if shape == "coding_mode":
            return [step.copy() for step in CODING_PLAN]
        if shape == "timer":
            duration = await self.ask(
                client, "duration", "Which exact timer duration was spoken?",
                {str(m): f"Exactly {m} minutes." for m in (30, 60, 120)}, [shape]
            )
            return [{"tool": "start_timer", "minutes": int(duration)}]

        plan, chosen = [], []
        for slot in range(2 if shape == "two_apps" else 1):
            app = await self.ask(
                client, f"app_{slot + 1}", f"Choose requested app {slot + 1} in spoken order; do not repeat earlier apps.",
                {k: v for k, v in APPS.items() if k not in chosen}, [shape, *plan]
            )
            chosen.append(app)
            plan.append({"tool": "open_app", "app": app})
        if shape == "app_action":
            app = chosen[0]
            operation = await self.ask(
                client, "operation", f"Which additional operation is requested for {app}?", OPERATIONS[app], plan
            )
            target = await self.ask(
                client, "target", f"Which exact target is requested for {operation}?", TARGETS[operation], [*plan, operation]
            )
            plan.append({"tool": operation, "app": app, "target": VALUES[operation][target]})
        await self.ask(
            client, "complete", "Does this plan cover EVERY requested effect, with no unspoken effects? Opening an app does not fulfill another action.",
            {"complete": "All and only the requested effects are present.", "missing": "The plan omits a spoken effect or adds an unrequested one."}, plan
        )
        if self.trace[-1]["choice"] != "complete":
            raise UnresolvedCommand("The plan does not match the whole command; no plan was released.")
        return plan
```

## Route recordings and inspect the result

`route_command` validates the recording before sending it. It returns an empty plan when
routing is unresolved. API and SDK errors propagate, so a failed request cannot return a
partly built plan as though it were complete.

```python theme={null}
async def route_command(audio: Path) -> dict:
    frames = list(wav_chunks(audio))
    if not frames or sum(map(len, frames)) > 30 * 32000:
        raise ValueError("Provide a nonempty mono 16 kHz PCM16 WAV of at most 30 seconds.")
    router = Router(audio)
    async with AsyncTazo() as client:
        try:
            plan = await router.build_plan(client)
        except UnresolvedCommand as error:
            return {"status": "needs_clarification", "reason": str(error), "plan": [], "trace": router.trace}
    # Connect a reviewed plan to your own device adapters here.
    return {"status": "planned", "plan": plan, "trace": router.trace}
```

From the cookbook repository root:

```bash theme={null}
python tazo-assistant/tazo_assistant.py tazo-assistant/examples/open-finder-downloads.wav
```

A successful result has this shape; confidence values are illustrative:

```json theme={null}
{
  "clip": "open-finder-downloads.wav",
  "status": "planned",
  "plan": [
    {"tool": "open_app", "app": "finder"},
    {"tool": "open_folder", "app": "finder", "target": "Downloads"}
  ],
  "trace": [
    {"stage": "shape", "choice": "app_action", "confidence": 0.91},
    {"stage": "app_1", "choice": "finder", "confidence": 0.95},
    {"stage": "operation", "choice": "open_folder", "confidence": 0.68},
    {"stage": "target", "choice": "downloads", "confidence": 0.97},
    {"stage": "complete", "choice": "complete", "confidence": 0.95}
  ]
}
```

The sample recordings are original commands synthesized with macOS `say`, voice Samantha.
A live SDK 0.1.2 check routed the six supported fixtures to their intended plans and
rejected both unsupported fixtures. Outputs remain probabilistic between runs.

| Recording / spoken command | Plan when all stages pass | Requests |
| - | - | - |
| `open-notes.wav`: "Open Notes." | Open Notes | 3 |
| `open-safari-and-notes.wav`: "Open Safari and Notes." | Open Safari, then Notes | 4 |
| `open-finder-downloads.wav`: "Open Finder and show my Downloads folder." | Open Finder, then show Downloads | 5 |
| `open-notes-left.wav`: "Open Notes and put its window on the left." | Open Notes, then position it left | 5 |
| `set-timer-thirty.wav`: "Set a timer for thirty minutes." | Start a 30-minute timer | 2 |
| `enter-coding-mode.wav`: "Enter Coding mode." | Open Cursor, Linear, Codex, Slack | 1 |
| `chatter.wav`: conversation about the weekend | No plan; clarification status | 1 |
| `dictation.wav`: "Type this sentence: let's meet tomorrow at noon." | Unsupported; no plan | 1 |

Multiple input files are routed independently. A result with `status="needs_clarification"`
includes a reason and trace, with `plan=[]`. No desktop side effect occurs in this example.

## Build on it

* **Add an app or action.** Extend the app menu, its operation map, and its exact target
  maps together. Keep concrete URLs and device identifiers in application code.
* **Filter by availability.** Before asking, remove choices whose apps, saved targets or
  permissions are unavailable on the current device.
* **Execute a reviewed plan.** Preflight every step, show the entire plan for review, then
  call typed adapters in order. Keep a receipt if a later step fails after earlier effects.
* **Support longer combinations.** Add another bounded structure and corresponding stages.
  Keep the completion check and decision ceiling, and account for extra serial latency.

## Limitations

* This is a planner. Timers, app launches and window placement are output steps; real desktop
  adapters, a countdown and a review interface belong in your application.
* Only the structures and targets above are supported. A request for an arbitrary URL,
  dictation or a third general-command app must stop without a plan.
* Confidence and completeness checks are probabilistic, not guarantees of correct routing.
  Test your actual spoken commands before connecting side effects.
* Every stage resends the same recording and consumes another decision. More stages add
  inference usage and serial network time. Failed routes are not automatically replayed.

The full [Personal Assistant for Mac source](https://github.com/tazolabs/tazo-cookbooks/tree/main/projects/tazo-assistant)
adds a native macOS interface, more app operations and bounded command compositions.
For a smaller state-machine routing example, see [Graph agents](https://docs.tazolabs.com/build-with-tazo/cookbooks/graph-agents.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.