What I don't let the model decide

I built a booking assistant you talk to instead of filling in a form. The model chooses which tool to call. It does not choose what to send it — every argument that reaches my calendar is rebuilt in Python first, and that split turned out to be most of what I learned building the thing.

Three services and a conversation#

Three processes, started in three terminals. An email server on localhost:8090 that sends and verifies one-time codes through Azure Communication Services. A calendar server on localhost:8080 that finds free slots and writes events through the Google Calendar API. And an agent — an Azure OpenAI chat completion with both servers’ tools attached — that runs the conversation and decides what to call.

Both servers are small FastAPI apps exposing the same two endpoints: GET /tools to describe what they can do, and POST /call to do it. That is the whole contract between the agent and the things it operates.

There is no web UI, and for a prototype that is a feature rather than a shortcut. The printed tool calls above are how I found almost everything worth knowing about how the model behaves, because the arguments it proposes are right there next to what actually got sent.

The model picks the tool#

The agent does not have a hardcoded list of tools. At startup it fetches /tools from both servers and converts each entry into a function schema, adding a namespace so two servers can never collide:

def list_tools(base, ns):
    """Fetch /tools and build a usable schema for OpenAI tools."""
    r = requests.get(f"{base}/tools", timeout=10)
    tools = r.json().get("tools", [])
    for t in tools:
        name = t["name"]
        if not name.startswith(ns + "."):
            name = f"{ns}.{name}"
        ...

CAL_TOOLS   = list_tools(CAL_BASE,   "calendar")
EMAIL_TOOLS = list_tools(EMAIL_BASE, "email")
ALL_TOOLS   = CAL_TOOLS + EMAIL_TOOLS

So the model is offered calendar.find_free_slot and email.send_email_otp, and a new tool on either server appears in the next run with no agent change at all. The namespace is also how the agent routes: anything starting email. goes to port 8090, everything else to 8080.

The namespace is a client-side fiction, though, and the dispatcher has to undo it before the call goes out — the calendar server was written without prefixes and expects its bare tool names:

base = EMAIL_BASE if name.startswith("email.") else CAL_BASE
payload = {
    "name": name if name.startswith("email.") else name.split(".", 1)[1],
    "arguments": args or {}
}

It is a small asymmetry and I would rather it were not there. It is also the honest shape of two services written a fortnight apart, and it cost one line to absorb in the client rather than a breaking change on a server.

Python picks the arguments#

This is the decision the whole design rests on. When the model asks for a calendar tool, the agent keeps the tool it chose and throws away the arguments it proposed:

desired_dt = parse_when(last_intent, TZINFO)

if not desired_dt:
    desired_dt = (datetime.now(TZINFO) + timedelta(days=1)).replace(hour=10, minute=0)

window_start_iso, window_end_iso = make_window_for(desired_dt, duration)

args = {
    "duration_minutes": duration,
    "window_start_iso": window_start_iso,
    "window_end_iso": window_end_iso,
    "pad_minutes": int(args.get("pad_minutes") or 0),
}

Note the last line. args is not edited — it is rebuilt, and the only thing salvaged from what the model sent is a padding integer. Everything with a timezone in it is computed from the conversation by parse_when, which is a regex and some arithmetic, and by make_window_for, which is four lines of timedelta.

The reasoning is narrow and I think it generalises. A model is genuinely good at which of these tools does this person want — that is a judgement about language, made from context, and there is no closed-form way to compute it. It is much less obviously the right thing to be assembling a timezone-aware ISO-8601 timestamp for tomorrow in Europe/Brussels, because that is not a judgement at all. It is a calculation, and calculations have a correct answer that does not depend on how the request was phrased.

The same instinct shows up again in the booking call, where normalize_booking_args guarantees a summary, a duration and a timezone-aware start and end before an event can be created — filling defaults for anything absent rather than passing through whatever arrived.

Verification is not a prompt rule#

The system prompt is direct about the order of operations:

"• Before any calendar.* booking, you MUST verify the user's email via OTP.\n"
"• Only ask for the OTP code AFTER you've sent one.\n"

That is a request. Capitals do not make it a control, and a prompt cannot be the only thing standing between a stranger and my calendar. So the same rules exist a second time, in Python, where they are checked against session state before any call is dispatched:

if name == "email.verify_email_otp" and SESSION["state"] != "otp_sent":
    return "I need to send you a verification code first. What is your email address?"

if name.startswith("calendar.") and not SESSION["verified"]:
    return "Please verify your email first. What is your email address?"

Both guards return before call_tool is reached, so a model that decides to skip verification does not produce a failed booking — it produces a question about an email address. The user experience of the guard firing is indistinguishable from the assistant being helpful, which is the nicest property it has.

Messages that can’t be wrong#

When a free slot comes back, the agent does not ask the model to tell you about it. It builds the sentence itself and returns immediately, without a second completion:

if slot.get("start") and slot.get("end"):
    SESSION["proposed_start_iso"] = slot["start"]
    SESSION["proposed_end_iso"]   = slot["end"]
    confirm_text = (f"I found a free slot from "
                    f"{slot['start']} to {slot['end']}. "
                    f"Do you want me to book it?")
    return confirm_text

The result is a chat message with raw ISO timestamps in it, which is not pretty. It is also an f-string over the tool’s actual response, so it cannot offer you a slot the calendar did not return. The same two values are cached and then popped when you say yes, so the event that gets created is the exact slot you agreed to rather than a re-derivation of it.

I have come to like the ugliness as a signal. In this codebase, prose that reads a little mechanical is prose that was computed, and anywhere the phrasing is warm and natural a model wrote it. That is a surprisingly useful thing to be able to see at a glance while a system is still being built.

Where I’d take it next#

• Enforce verification at the resource, not just in the client. The agent already sends an X-Verified header on calendar calls; the calendar server should require it. Right now the guard lives in my process, and a guard in the caller protects the caller.
• Give the session somewhere to live. SESSION is a module-level dict, so it is one conversation per process and a restart forgets you. A real deployment needs it keyed per user and stored outside the process.
• A tool for a specific hour. There is find_free_slot, which searches a window, and nothing that answers is 15:00 free? directly. Those are different questions and they deserve different tools.
• Containers, and more than one calendar. Both servers are small enough to ship as images, and the interesting version of this handles a team rather than me.

None of that changes the shape. The part I would keep in anything I build after this is the boundary: the model is excellent at reading a sentence and choosing what should happen next, and it is the wrong tool for producing values that have a correct answer. The interesting design decisions in an agent turn out not to be in the prompt at all. They are in the list of things you refuse to let it decide.