How it works

The user asks for real things: a reminder, a calendar event, a contact, a text, an answer from their mail, a refusal. The assistant acts on a real iOS device. Afterwards, PAVE reads the device. A second evaluator reads only the conversation. Every task gets both verdicts.

PAVE treats the assistant as a black box. It takes audio or text in and returns tool calls. Behind that boundary it can be a text model, a cascaded pipeline (speech-to-text → LLM), a speech-to-speech model, or a speech-to-speech model that also voices its replies. The ear, the brain or the voice can be changed independently while the 30 tasks, the phone, the grader and the simulated user stay exactly the same.

1. Seed. Contacts, calendar, reminders and mail are placed on the device, so the “before” state is known.
2. Speak. The request is synthesized and played as audio. Under-specified tasks get a simulated user that answers only when the assistant genuinely stops to ask.
3. Act. Whatever the design, it returns tool calls; the harness runs them for real in Reminders, Calendar, Contacts, Messages.
4. Read. The device is re-read and compared with what the task required. The reply is not trusted.
5. Keep. Every run leaves a full trace and can be recorded as a captioned video.