Live context notes

A live context note tells the agent something it can't hear on the call: a payment that went through, an order that shipped, the page the caller is looking at, a supervisor's steer. The agent takes it into account from its next reply. A note never counts as something the caller said. It isn't added to the transcript, it doesn't move a flow, and it doesn't make the agent answer it.

A note can also ask the agent to tell the caller (respond: "when_idle"). The agent then speaks up as soon as the line is free, in its own words and in the caller's language.

Notes work on every live call: phone calls on every carrier, SIP and web calls. A note can come from:

How the agent sees notes

Before each reply, the agent's prompt gets one block after the conversation. The block lists the call's notes oldest first, each with who sent it and how old it is. Notes from the caller's app are marked unverified: the agent is told they never prove a payment, an identity or eligibility, and to describe them as what the caller's app shows.

• Keys replace. A note with a key replaces the earlier note with the same key, so state that changes (a payment status, a cart, the current page) is always one current note. Without a key, notes add up.
• Keys are per sender. A note from the caller's app never replaces a note from your systems, and yours never replaces one of the app's. Your backend, your dashboard users and the apps you installed share one set of keys.
• Your notes come first. When the prompt can't fit every note, it takes your systems' notes first (keyed ones before un-keyed, newest first), then the caller's app's. A busy or tampered page can never crowd your notes out.
• Lines the agent says on its own — a silence check-in, a "take your time" nudge, a goodbye, a welcome back after a hold, a flow step's opener — also have the notes in view, as background.

Telling the caller: respond: "when_idle"

Send respond: "when_idle" when the caller should hear the news:

curl -X POST https://api.telenow.ai/api/sessions/{id}/context \
  -H "X-API-Key: vai_live_…" \
  -H "Content-Type: application/json" \
  -d '{ "text": "Payment of ₹4,500 received. UPI ref 41234.", "key": "payment", "respond": "when_idle" }'

• As soon as the line is free. The agent speaks up once nobody else has the floor: it isn't talking, the caller isn't, and the caller doesn't owe it an answer to a question it just asked. It never talks over the caller.
• If the caller speaks first, the agent's reply tells them instead.
• In a wait the caller asked for ("one moment, I'm paying"), the news ends the wait: the agent tells them and the call carries on.
• Told once. The note counts as told once a reply has carried it, or once the agent's line about it has actually been spoken. A line written but never played tells nobody, and neither does a check-in or a nudge that happened to have the note in view.
• A retry is not told twice. If you send the same note again (same key, text and respond) after the agent already had it in view, it isn't announced again.
• One at a time. A newer when_idle note takes over from an older one that hasn't been spoken yet. The older note is still told in the next reply.
• Your agent's own words. The line is written by the agent's own model, in the caller's language. There is no fixed text.
• Not in a text chat. In a text chat (a web call in chat mode) the agent does not write to the caller about a note on its own: when_idle there works like none. The note answers next_turn, and the agent's next reply tells the caller.

What delivery means

Every accepted note answers with when the agent sees it:

A call where a person has the call for good — a manual (softphone) call, a takeover by your team, or a transfer that went through — has no agent to tell. Notes there are refused with 409 no_agent.

Size: how much fits

A note has to fit in the room the agent's prompt leaves beside the conversation, and beside the notes the call already holds. Your notes count first, then the caller's app's. A note that doesn't fit is refused with 413 too_large, and the answer's maxChars says how many of its characters would fit right now.

• maxChars counts Unicode code points, not bytes and not JavaScript string units. In JavaScript, cut with Array.from(text).slice(0, maxChars).join(''). A plain text.slice(0, n) can cut an emoji in half. The SDKs repair half an emoji before sending, but the cut changes the text.
• The room shrinks as the call grows. "Fits right now" is the honest answer. Keep notes short and current: one keyed note per piece of state.
• When the call's notes outgrow what the model could ever hold, the oldest are dropped to make way: the caller's app's notes first, then your un-keyed notes. Your keyed notes are never dropped. The call page marks what was dropped.
• On a speech-to-speech agent, the room is the realtime model's own.

"Still here, just busy": activity

When the caller goes quiet because they are busy — paying, reading a message, typing in a form — tell the agent, so it doesn't ask "are you still there?":

curl -X POST https://api.telenow.ai/api/sessions/{id}/activity -H "X-API-Key: vai_live_…"

{ "success": true, "data": { "nextCheckinInMs": 20000 } }

• It restarts the agent's silence check-in. In a wait the caller asked for, it restarts that wait's own clock instead: its nudges and its close count from now, and the wait goes on. It never makes the agent speak.
• nextCheckinInMs is how long until the agent would speak up unprompted: its check-in or, with check-ins off, its silence hang-up. Ping again before it runs out while the caller is still busy.
• null means nothing is armed right now. Examples: a hold, a wait the agent is still acknowledging, check-ins and the silence hang-up both off, or a speech-to-speech agent. Try again once the conversation moves on.
• In the browser, set autoActivity: true in the web SDK and the SDK does this for you while the user types or clicks. A user busy anywhere in the check-in window gets one ping, timed to land before the window runs out. A user who has stopped gets none.

Speech-to-speech agents

On OpenAI Realtime, Gemini 3.8 Live and Gemini 3.1 Flash Live, a note goes silently into the model's own conversation. A later note with the same key replaces it there too. when_idle works the same way: the model speaks up once the line is free. It never interrupts the caller, never while it is still answering them, and never while a tool call is running.

• A note sent before the realtime model has connected waits, and is delivered as soon as it connects.
• Other Gemini Live models refuse notes with 409 engine_unsupported. So does a Custom API agent.

Answers and errors

The REST endpoints answer every state the same way:

On the WebSocket, every contextual_update is answered by context_ack or context_rejected, and every user_activity by activity_ack or activity_rejected, in the order you sent them. Frames sent before start are answered not_live. The caller's app can also be refused disabled (the agent doesn't accept its notes) or rate_limited (over its allowance, below). See the WebSocket reference.

Notes from the caller's app

Notes from the caller's own app are off by default. Turn them on per agent:

• In the builder: Call handling → "Notes from the caller's app".
• In the API: sessionConfig.liveContext.acceptClientNotes: true (see the agent field reference).

The caller controls their own app, so these notes are always marked unverified to the agent, never replace your notes, and get only the room your notes leave. Each call's app shares the browser keypad's allowance: a burst of 16, then 5 a second, and at most 300 per call. Anything more is refused with rate_limited.

A reseller's client can turn the setting on or off for their own agents when their provider grants it. When a client creates an agent, they can leave it as their provider's starter set it.

On a fleet of servers

A call lives on one server. A note or ping that reaches another server is forwarded, once, to the one holding the call, with your own credentials and the address your request came from. If that server hasn't registered the call yet (its first seconds), or doesn't answer in time, you get 503 with Retry-After. A single-server deployment never sees these.

After the call

• The call page shows a "Notes during the call" card. It lists every note with its sender, its key, and what became of it: replaced by a later note, dropped to make room, spoken up about by the agent, or held while the agent was away. While the call is still on, the card refreshes by itself. The debug timeline has a row for each note (context.note), for each when_idle outcome (context.react), and for each activity ping that restarted something (activity), with who sent it.
• Reseller clients see their own calls' notes in the client portal, without who sent them.
• Post-call analysis is grounded on the notes your systems sent: a statement the agent made after a note that supports it is not a hallucination. The judge only sees notes up to the last transcript line it reads. Notes from the caller's app are never grounds.
• Privacy. A note is stored with the call's PII policy applied to its text and key, exactly like a transcript turn. It is deleted with the transcript when content retention runs. The agent itself received the note as you sent it.

Examples

// Node — your backend (@telenow/server)
await tn.calls.sendContext(sessionId, { text: 'Order A123 shipped; tracking AB123', key: 'order', respond: 'when_idle' });
const { nextCheckinInMs } = await tn.calls.sendActivity(sessionId);

# Python — your backend
tn.send_context(session_id, "Order A123 shipped; tracking AB123", key="order", respond="when_idle")
tn.send_activity(session_id)

// Browser — the caller's page (@telenow/client), agent setting on
const call = new TelenowCall({ session, autoActivity: true });
await call.start();
await call.sendContext('Viewing: blue kurta, ₹1,299, sizes M and L', { key: 'page' });

// iOS — the caller's app (TelenowSDK), agent setting on
let delivery = try await call.sendContext("Basket: 2 items, ₹1,240", key: "basket")

When not to use a note

A note reaches the agent's next reply. The agent starts answering as soon as the caller stops talking, so a note sent in reaction to what the caller just said arrives too late for that answer. To answer the caller's current question with your data, give the agent a tool: it calls your API and waits for the result.