Connecting an assistant

Let a language model build and control windows: give it the four tools the package ships, validate what it sends, and run the calls with useUiWindows. The model never writes code, only JSON that the UiRenderer turns into library components.

The flow

1. The user asks: "Show me the latest video of X and a volume control."
2. Your backend sends the request to the model together with the tools from jarvis-react-ui/ui-tools.json.
3. The model answers with tool calls: open_window with a spec, later update_window ("set the volume to 20"), control_player ("skip to 1:30") or close_window.
4. Your UI runs each call with useUiWindows and returns the result, including validation errors, so the model can correct itself.

Try a spec

Edit the JSON on the left: the window on the right follows, and invalid specs show the errors a model would get back.

import { useMemo, useState } from 'react';
import { Callout, Grid, Panel, Textarea } from 'jarvis-react-ui';
import { UiRenderer, validateUiSpec } from 'jarvis-react-ui/generative';
import type { UiWindowSpec } from 'jarvis-react-ui/generative';

const START = JSON.stringify(
    {
        id: 'scan',
        title: 'Scan',
        state: { radius: 200, live: true },
        root: {
            type: 'Stack',
            children: [
                {
                    type: 'Slider',
                    props: {
                        label: 'Radius',
                        min: 50,
                        max: 500,
                        step: 50,
                        value: { $bind: 'radius' },
                        showValue: true,
                        formatValue: '{value} m',
                    },
                },
                { type: 'Switch', props: { label: 'Live', checked: { $bind: 'live' } } },
                {
                    type: 'Pill',
                    props: { variant: 'ok' },
                    children: 'scanning {{radius}} m',
                    visibleIf: { state: 'live' },
                },
            ],
        },
    },
    null,
    2,
);

export default function Playground() {
    const [text, setText] = useState(START);

    const result = useMemo(() => {
        try {
            const spec: unknown = JSON.parse(text);
            const validation = validateUiSpec(spec);
            return validation.ok
                ? { spec: spec as UiWindowSpec, errors: [] }
                : { spec: null, errors: validation.errors.map((e) => `${e.path}: ${e.message}`) };
        } catch (error) {
            return { spec: null, errors: [`Invalid JSON: ${String(error)}`] };
        }
    }, [text]);

    return (
        <Grid columns={2} gap="md" align="start" className="w-full">
            <Textarea
                aria-label="Spec JSON"
                value={text}
                onChange={(e) => setText(e.target.value)}
                rows={18}
                fullWidth
                size="sm"
                spellCheck={false}
            />
            {result.spec !== null ? (
                <Panel title={result.spec.title} className="w-full">
                    <UiRenderer key={text} spec={result.spec} />
                </Panel>
            ) : (
                <Callout variant="error" title="Spec rejected">
                    {result.errors.join('\n')}
                </Callout>
            )}
        </Grid>
    );
}

The tools

The package ships two files, generated from the component types on every build:

Pass ui-tools.json as tools unchanged. The schema is large (about 60 kB), so cache it with prompt caching; it only changes when you update the package.

Running the tool calls in the UI

The tool names map one to one onto useUiWindows:

import type { UiJson, UiMethod, UiState, UseUiWindowsResult } from 'jarvis-react-ui/generative';

/** Runs one tool call from the model; `content` goes back as the tool result. */
export function runUiTool(
    ui: UseUiWindowsResult,
    name: string,
    input: Record<string, UiJson>,
): { content: string; isError: boolean } {
    switch (name) {
        case 'open_window': {
            const result = ui.open(input);
            const errors = result.errors.map((e) => `${e.path}: ${e.message}`).join('\n');
            return { content: result.ok ? 'opened' : errors, isError: !result.ok };
        }
        case 'update_window': {
            const ok = ui.patchState(String(input.id), input.state as UiState);
            return { content: ok ? 'updated' : 'no such window', isError: !ok };
        }
        case 'control_player': {
            const args = input.value === undefined ? [] : [input.value];
            const method = input.method as UiMethod;
            const ok = ui.call(String(input.window_id), String(input.node_id), method, ...args);
            return { content: ok ? 'done' : 'no such player or method', isError: !ok };
        }
        case 'close_window':
            ui.close(String(input.id));
            return { content: 'closed', isError: false };
        default:
            return { content: `unknown tool ${name}`, isError: true };
    }
}

Calling the model

Call the model where your API key lives: a backend, or the main process of an Electron or Tauri app. Never ship the key to the browser. A minimal loop with the Anthropic TypeScript SDK, where sendToUi forwards a tool call to the UI (for example over a WebSocket) and resolves with the result of runUiTool:

import Anthropic from '@anthropic-ai/sdk';
import uiTools from 'jarvis-react-ui/ui-tools.json';

const client = new Anthropic();
const tools = uiTools as Anthropic.Beta.BetaTool[];

export async function ask(
    userText: string,
    sendToUi: (name: string, input: unknown) => Promise<{ content: string; isError: boolean }>,
): Promise<string> {
    const messages: Anthropic.Beta.BetaMessageParam[] = [{ role: 'user', content: userText }];

    while (true) {
        const response = await client.beta.messages.create({
            model: 'claude-opus-5-5',
            max_tokens: 16000,
            betas: ['server-side-fallback-2026-07-01'],
            fallbacks: 'default',
            system: 'You control the windows of a HUD interface. Build them only with the tools.',
            tools,
            messages,
        });
        if (response.stop_reason === 'refusal') return 'The request was declined.';

        messages.push({ role: 'assistant', content: response.content });
        const calls = response.content.filter(
            (b): b is Anthropic.Beta.BetaToolUseBlock => b.type === 'tool_use',
        );
        if (calls.length === 0) {
            return response.content.map((b) => (b.type === 'text' ? b.text : '')).join('');
        }

        // Run all calls, then return all results in one user message.
        const results = await Promise.all(calls.map((c) => sendToUi(c.name, c.input)));
        messages.push({
            role: 'user',
            content: calls.map((c, i) => ({
                type: 'tool_result' as const,
                tool_use_id: c.id,
                content: results[i]?.content ?? '',
                is_error: results[i]?.isError,
            })),
        });
    }
}

The fallbacks option lets the API retry on another model if a request is declined; drop it if you don't want that.

Safety

The model can only use what the allowlist (UI_REGISTRY) permits. Specs cannot set classes, styles, raw HTML or arbitrary event handlers; URLs must be http(s) or relative paths; WebFrame keeps its sandbox. open validates every spec before rendering, and UiRenderer drops anything that slips through. Treat model output like any other untrusted input: ask for confirmation before acting on emit events that change data outside the UI.