xAI patient intake

Overview

In this recipe, build the patient intake coordinator for Maplewood Family Medicine. It's a front-desk agent that answers policy questions, manages appointments and recorded data, and escalates a possible emergency. You can talk to the agent on this page, or call it on its own deployment at xai.livekit.space .

The design is one agent, one conversation, and one fixed set of typed tools. The model keeps track of what the caller says, then passes those facts to a tool when it must read or change practice state. This recipe covers each job the agent does on a call:

• Booking, including registration of a first-time patient in the same step.
• Rescheduling and cancelling an existing appointment.
• Answering a policy question in the middle of another task.
• Routing a message to the refill, results, billing, referral, nurse, or records queue.
• Collecting pre-visit intake one question at a time.
• Escalating an emergency before all other front-desk work.

Speech-to-text, the LLM, and text-to-speech all run on xAI models through LiveKit Inference: xai/stt-1, xai/grok-4.3, and xai/tts-1.

Prerequisites

To complete this guide, you need the following prerequisites:

• Complete the Voice AI quickstart. This gives you a working project, the uv toolchain, and every package this recipe needs.
• Create a LiveKit Cloud project. LiveKit Inference gives you access to the xAI models.
• Use livekit-agents>=1.7.0. Expressive mode requires this version.

The agent needs three environment variables in .env.local:

LIVEKIT_URL=<your-livekit-url>
LIVEKIT_API_KEY=<your-api-key>
LIVEKIT_API_SECRET=<your-api-secret>

Project structure

The agent is a small set of modules. agent.py composes the session:

How a call flows

Every call runs the same loop. The agent greets the caller and listens for the request. It checks for an emergency first. It then collects the identity that practice records require, calls the tool that does the work, states the outcome, and gives the turn back to the caller:

Four rules apply for the whole call and keep the flow predictable:

• Emergencies interrupt everything. A described emergency is escalated before identity or scheduling work, and the ordinary call is over afterward.
• Identity comes before records. Established-patient work needs a last name and a full date of birth. If they don't match, the agent asks the caller to re-check those two details rather than guessing or creating a chart.
• No claim before the tool succeeds. The agent never says something is booked, moved, cancelled, saved, or sent until the tool that does it has returned.
• One outcome, then silence. A completed action gets one declarative sentence and the turn ends. The agent doesn't ask whether the caller needs anything else.

Booking an appointment

Booking takes two tool calls. When the caller picks one of the times the search returns, that choice is the confirmation. The agent doesn't ask the caller to confirm the same time twice.

For a caller who has never visited the practice, registration is part of the booking. The agent treats "I've never been there" as a request to register. It asks for a full name and date of birth, and doesn't mention charts, registration, or a failed lookup:

1. Caller“I need to see someone about my knee. I've never been there before.”
2. Agent“Of course. What's your full name and date of birth?”
3. Caller“John Doe, August first, nineteen eighty-nine.”
4. Toolfind_open_times(patient_status="new", last_name="Doe", date_of_birth="1989-08-01")
5. Result←Open appointments — SLOT-0818-1330: Tuesday, August 18 at 1:30 PM with Doctor Elena Alvarez
6. Agent“I have Tuesday the 18th at 1:30 with Doctor Alvarez.”
7. Caller“Tuesday at 1:30 works.”
8. NoteThe caller picked a returned time, which is the confirmation. The agent books it immediately.
9. Toolbook_appointment(patient_status="new", first_name="John", last_name="Doe", date_of_birth="1989-08-01", slot_id="SLOT-0818-1330", visit_type="sick_visit", reason="knee pain")
10. Result←Booked Tuesday, August 18 at 1:30 PM with Doctor Elena Alvarez.
11. Agent“You're booked Tuesday, August 18 at 1:30 with Doctor Alvarez.”

The practice's scheduling rules live in the search, so the agent can only offer an opening the caller can take:

• Age routing. Patients under 18 see Doctor Priya Raman. If the caller asks for an adult doctor for a child, the search says so and returns pediatric openings instead.
• Panel status. Doctor Marcus Chen isn't accepting new patients. A new patient who asks for him gets that answer and a list of alternatives.
• Weekends. The practice is closed on Saturday and Sunday. The agent answers from the published guide instead of relabeling a weekday opening.
• Exact times. If the caller asks for 7:00 AM, the reply names the unavailable time and lists the next real openings. The agent never has to invent an alternative.
• Changed preferences. A change to the day, time, provider, or time of day starts another search. Every other preference carries over, so a switch from morning to afternoon doesn't reset the date.

The visit type comes from the request: sick_visit for a new problem, annual_physical for an adult preventive exam, well_child for a child's routine exam, follow_up for a recheck, and telehealth only for a problem that needs no physical examination. A problem that needs an examination, such as an ear, a rash, or an injury, gets an in-person visit. The agent says the reason out loud before it searches.

Rescheduling and cancelling

One tool handles existing appointments with three actions, so the workflow is list first, then act:

1. Caller“I need to move my appointment.”
2. Toolmanage_appointment(action="list", last_name="Whitaker", date_of_birth="1958-03-14", caller_relationship="the patient")
3. Result←Upcoming appointments — APT2001: Wednesday, August 19 at 10:30 AM with Doctor Elena Alvarez
4. NoteThe agent doesn't speak the appointment ID. It's there for the next tool call.
5. Toolfind_open_times(patient_status="established", last_name="Whitaker", date_of_birth="1958-03-14")
6. Toolmanage_appointment(action="reschedule", appointment_id="APT2001", new_slot_id="SLOT-0818-1330", caller_relationship="the patient")
7. Result←Rescheduled to Tuesday, August 18 at 1:30 PM with Doctor Elena Alvarez.
8. Agent“You're moved to Tuesday, August 18 at 1:30 with Doctor Alvarez.”

The caller's relationship to the patient controls what the agent discloses. The agent gives an appointment list only to the patient, a parent, or a guardian. It tells any other adult that the patient must authorize access in writing. Any caller who gives the exact identity can cancel or reschedule for the patient. The agent never speaks an appointment ID or a slot ID out loud, because those IDs exist for the next tool call.

Answering a policy question mid-task

A policy question doesn't interrupt the task in progress. The agent answers from the published guide, then continues where it stopped:

1. Caller“Do you have parking? And can I get an interpreter?”
2. Toolread_practice_information()
3. Result←The complete published guide — hours, location, parking, insurance, refills, results, referrals, privacy, accessibility, after-hours care.
4. Agent“There's free parking behind the building, and interpreter services are free — just tell us the language when you book.”
5. NoteOne guide answers both questions. The agent then returns to the booking in progress.

The guide is a set of Markdown files under src/clinic/practice_info/, one per topic: hours, location, insurance, refills, results, referrals, privacy, accessibility, and after-hours care. One tool takes no arguments, joins the files, and returns the whole guide, and the model picks the relevant part. A topic parameter forces the agent to classify the question first, and a question such as "parking plus interpreters" fits more than one topic. To change practice policy, edit a Markdown file. The prompt stays the same.

Routing a message

Refills, results, billing, referrals, nurse callbacks, and records requests all become one chart message, routed to the correct team. The agent tells the caller what actually happened. A message is a request, not an approval.

1. Caller“I need a refill on my lisinopril.”
2. Tooltake_message(kind="prescription_refill", last_name="Whitaker", date_of_birth="1958-03-14", summary="Lisinopril refill requested")
3. Result←Routed the prescription refill request. The front desk cannot approve or send a refill. The nurse will review it, and routine requests are processed within two business days.
4. Agent“I've sent that to the nurse. Routine refills are processed within two business days.”
5. Caller“Can you make it happen today?”
6. Tooltake_message(kind="prescription_refill", last_name="Whitaker", date_of_birth="1958-03-14", summary="Send it immediately")
7. Result←That prescription refill request was already routed. No second message was sent. The front desk cannot approve or send a refill.
8. Agent“It's already with the nurse — the front desk can't approve refills, and routine ones take up to two business days.”

Messages are durable. If the caller asks for faster action, a second call to the tool returns "That prescription refill request was already routed. No second message was sent," together with the same policy text. The agent then explains the timeline instead of queueing a duplicate. Callback times come from the tool, never from the model.

Insurance uses the same pattern. update_insurance replaces the carrier, member ID, and group number on the chart with the details from the caller's current card. The agent then confirms the update in one sentence and ends the turn.

Collecting pre-visit intake

Pre-visit intake is the only job that continues across several turns, and the prompt drives it instead of a state machine. The agent identifies the patient, then asks one question per turn: the reason for the visit and its duration, medications and supplements, allergies and reactions, ongoing conditions, and the preferred pharmacy. It calls the tool once, with the caller's own answers:

1. Agent“What's bringing you in, and how long has it been going on?”
2. Caller“My knee's been swollen about two weeks.”
3. Agent“Any medications or supplements you're taking?”
4. Caller“Lisinopril and vitamin D.”
5. NoteAllergies, conditions, and pharmacy follow the same pattern: one question per turn, nothing batched.
6. Toolrecord_previsit_intake(last_name="Whitaker", date_of_birth="1958-03-14", chief_complaint="swollen knee", symptom_duration="two weeks", medications=["lisinopril", "vitamin D"], allergies=["penicillin - hives"], conditions=["high blood pressure"], pharmacy="Bridge Street Pharmacy")
7. Result←Saved the pre-visit answers.
8. Agent“That's everything saved for your visit.”

An empty list means that the caller said "none." The agent must not invent an empty list or a "not applicable" answer to finish early. It records what it hears, and doesn't normalize, interpret, or diagnose it.

Escalating an emergency

Emergency handling comes before the rest of the workflow. The agent works from the meaning of what the caller says, not from a keyword list. It records the escalation before it asks for identity or does any front-desk work:

1. Caller“I've got heavy pressure in my chest going down my arm.”
2. NoteNo identity, no scheduling. The escalation happens first.
3. Toolrecord_emergency_escalation(reported_symptoms="heavy chest pressure spreading down the left arm")
4. Result←Emergency escalation recorded. Give the appropriate direction now.
5. Agent“Please hang up and call 911 now, or have someone take you to the nearest emergency department. Don't drive yourself and don't wait for a callback.”

For thoughts of self-harm or harm to others, the agent directs the caller to call or text 988 now. If anyone is in immediate danger, it directs them to 911 or the nearest emergency department. For unclear emotional language, such as "I'm not coping," the agent asks one direct question instead of making an inference.

After an escalation, the ordinary call is over. The agent does no booking, no message, and no intake, even if the caller makes the symptoms sound less serious. Common complaints such as a cough, a sore throat, an earache, a rash, a sore knee, or a headache aren't emergencies without a red flag.

Where the behavior lives

Each layer makes one kind of decision. This split keeps the workflows out of one long prompt:

Running the session on xAI models

src/agent.py composes the session. One AgentSession(...) call holds every model and audio setting:

@server.rtc_session(
    agent_name="xai-patient-intake",
    on_session_end=on_session_end,
)
async def patient_intake(ctx: JobContext) -> None:
    ctx.log_context_fields = {"room": ctx.room.name}
    clinic = open_clinic(datetime.now())
    session = AgentSession[Visit](
        userdata=Visit(clinic=clinic),
        stt=inference.STT(model="xai/stt-1", language="en"),
        llm=inference.LLM(
            model="xai/grok-4.3",
            extra_kwargs={
                # A front desk answers from the chart in front of it; no thinking budget needed.
                "reasoning_effort": "none",
                "temperature": 0.3,
                "max_completion_tokens": 600,
                "parallel_tool_calls": False,
            },
        ),
        tts=inference.TTS(model="xai/tts-1", voice="carina"),
        expressive=ExpressiveOptions(tts_instructions_append=prompt("expressive")),
        vad=inference.VAD(),
        max_tool_steps=5,
        # Dynamic endpointing gives hesitant, incomplete speech time to continue.
        turn_handling=TurnHandlingOptions(
            turn_detection=inference.TurnDetector(),
            endpointing={"mode": "dynamic", "min_delay": 1.2, "max_delay": 4.0},
        ),
    )
    await session.start(
        agent=PatientIntakeAgent(clinic=clinic),
        room=ctx.room,
        room_options=room_io.RoomOptions(
            audio_input=room_io.AudioInputOptions(
                noise_cancellation=ai_coustics.audio_enhancement(
                    model=ai_coustics.EnhancerModel.QUAIL_VF_S
                ),
            ),
        ),
    )
    await ctx.connect()

Here is why each setting has the value it does:

• xai/grok-4.3 with temperature=0.3 and reasoning_effort: "none". The tools carry the structure, so the model only has to follow instructions and call the tools accurately. A low temperature stops a receptionist from inventing clinical facts, and turning reasoning off keeps the first word fast.
• parallel_tool_calls: False and max_tool_steps=5. Front-desk work is sequential: identify, then search, then book. Parallel calls let a booking race a lookup.
• max_completion_tokens: 600. This caps a long reply. A phone caller doesn't want a long answer.
• inference.TurnDetector() with dynamic endpointing. A caller who recites a date of birth or a member ID pauses in the middle. A min_delay of 1.2 seconds gives that speech time to continue, and dynamic mode only adapts upward from that floor, so max_delay is paid only by a caller who is still talking.
• ExpressiveOptions with tts_instructions_append. Delivery guidance shapes how a line sounds. It doesn't affect what the agent decides.
• Noise cancellation through the ai_coustics plugin. This applies to the caller's audio before turn detection runs.
• AgentSession[Visit] with userdata. The per-call practice is typed session state instead of a module global.

When the call ends, on_session_end calls ctx.make_session_report(), which logs the models the call used and their usage.

Run it

Install the dependencies, then start the agent. Console mode runs the agent in your terminal, and dev mode registers the worker with LiveKit Cloud:

export UV_PROJECT_ENVIRONMENT=src/.venv
uv sync
lk agent console src/agent.py
lk agent dev src/agent.py

Test the workflows

The tools hold the behavior, so you can test most of each workflow without a model in the loop. Construct the agent with greet=False, call its tools in the order a call would, and assert what changed in practice state:

async def test_new_patient_booking_registers_and_books_in_one_tool_call() -> None:
    records = clinic()
    agent = PatientIntakeAgent(clinic=records, greet=False)
    slot = records.open_slots()[0]

    result = await agent.book_appointment(
        patient_status="new",
        last_name="Doe",
        date_of_birth="1989-08-01",
        slot_id=slot.id,
        visit_type="sick_visit",
        reason="foot pain",
        first_name="John",
    )

    patient = records.find_patient("Doe", datetime(1989, 8, 1).date())
    assert patient.registered_on_this_call
    assert records.scheduled_for(patient.chart_id)[0].slot == slot
    assert "Doctor Elena Alvarez" in result
    assert "Tuesday, August 18 at 1:30 PM" in result

Pin the clock with NOW = datetime(2026, 8, 17, 9, 0) so the day names and times in the assertions stay stable. The suite covers each workflow this way. For example, a child search redirects to pediatrics, an exact-time search names the unavailable time, and a duplicate refill refuses to send twice. Two more tests assert the structure: the exact set of eight tools, and the absence of the workflow primitives this agent doesn't use.

To test conversation-level behavior, such as whether the agent escalates a described emergency before it asks for a date of birth, use the testing and evaluation framework.

Extend it

Use this agent as a starting point for your own front-desk workflows:

• Swap the fake practice for a real one. src/clinic/ is framework-free and has no LiveKit imports. Replace it with your electronic health record (EHR) or scheduling client, and the workflows stay the same.
• Take the call over the phone. Attach a SIP trunk and dispatch this worker for inbound calls.
• Escalate to a human. Add WarmTransferTask so a caller who needs a nurse reaches one with context.
• Collect consent first. If you record calls, put a consent step at the front of the conversation. See Collect recording consent with tasks.
• Compare Grok variants. Point inference.LLM at the reasoning variant and rerun your evaluations. The xAI LLM page lists what's available.