Note: This video and podcast was generated using AI, adapting the original content and technical insights created by the author of the blog post.

We are in the middle of a fundamental shift in how users interact with software. For decades, the paradigm has been static: designers and engineers agree on a panel layout, engineers bind panel controls to a view model, and that view model talks to the document model. This works well when your feature set and workflows are mostly known in advance. But AI assistants break this model. Users express intent in natural language, and the system figures out what controls to present.

This creates an architectural challenge that existing frontend patterns don’t address: How do you build a UI layer that can dynamically compose whatever controls an LLM decides are relevant without hand‑coding every possible combination?

This article dives deep into an architecture for solving this problem at scale. We will explore:

• Why AI assistant UIs are fundamentally different from static panels
• A dual-purpose action definition pattern that serves both LLM and UI needs
• A type-safe declarative schema for dynamic forms
• The runtime mechanics of lazy loading, replay, and multi‑action composition
• Patterns you can apply to your own AI‑powered applications

In other words, when every possible interface can’t be pre-built, you build the system that builds interfaces.

Problem: AI assistants break static UI assumptions

For something like an Adobe Express‑style assistant, the system can propose hundreds of actions: “increase opacity,” “swap background,” “add a drop shadow,” “animate this group,” or “apply a cinematic intro preset.” The important detail is not just the number of actions, but that the combination and ordering of actions is not known at design time. The assistant might decide to show the following:

• A single slider for opacity.
• A compound form with a color picker, a border thickness slider, and an “Apply to all pages” toggle.
• A preset editor surfaced from a remote service that returns a JSON schema describing the controls.

Consider what happens when an LLM responds to “give this image a red border and make it slightly transparent”:

{
  "actions": [
    { "action": "setBorderStyle", "parameters": { "color": "#ff0000", "size": 30 } },
    { "action": "setOpacity", "parameters": { "opacity": 0.8 } }
  ]
}

The assistant needs to display:

• A color picker for the border color
• A slider for the border thickness
• A slider for the opacity

These controls must appear together, in a dynamically composed UI, derived entirely from the LLM’s response. And there might be 100+ different actions, each with different UI requirements. If every new action requires a hand‑crafted panel, bespoke bindings, and one‑off wiring to the document model, your assistant platform effectively does not scale. The result is a brittle UI layer that constrains what your AI can safely suggest.

This is exactly the gap a generative UI layer has to close: the assistant can freely generate or retrieve action specifications, and the client can deterministically turn these specs into safe, consistent, accessible UI.

What “generative UI” really means

Generative UI isn’t about chatbots streaming text. It’s about interfaces that materialize at runtime based on AI‑interpreted user intent.

The core insight: every action the AI can suggest should declare what UI it needs. Not “here’s a panel with a slider,” but “I need a slider with min 0, max 100, current value 50.”

This inverts the traditional relationship. Instead of UI that knows about actions, you have actions that describe their UI requirements.

const SetOpacityAction = {
  // For the LLM
  actionId: "setOpacity",
  description: "Changes transparency of selected elements",
  parameters: { type: "number", min: 0, max: 1 },

  // For the UI
  uiSettings: {
    componentType: "Slider",
    generatePayload: (inputs) => ({
      min: 0,
      max: 100,
      value: inputs.opacity * 100,
      label: "Opacity",
    }),
    parsePayload: (payload) => ({
      opacity: payload.value / 100,
    }),
  },
};

The same definition feeds both sides:

• The LLM understands what the action does and how to call it.
• The UI knows exactly what control to render and how to map between UI state and action parameters.

Most AI integrations treat these as separate concerns: they have one schema for the LLM and separate UI code for each feature. This works at a small scale but becomes a maintenance nightmare as you add actions.

A more scalable pattern is a single AssistantAction definition that serves both “masters”:

interface AssistantAction {
  // === LLM-facing properties ===

  // The ID the LLM uses to invoke this action
  actionId: string;

  // Natural language description for the LLM (max 256 chars)
  // Try to complete: "This action..."
  description: string;

  // Categories for organizational context
  categories: string[];

  // Parameter schema the LLM uses to construct invocations
  parameters: ParameterSchema;

  // Few-shot examples for the LLM
  examples?: string[];

  // === UI-facing properties ===

  // What types of elements this applies to
  applicableTo?: string[];

  // Whether this action can be undone
  isUndoable: boolean;

  // The declarative UI specification
  uiSettings?: {
    componentType: ComponentType;
    generatePayload: (inputs: ActionInputs) => ComponentPayload;
    parsePayload: (payload: ComponentPayload) => ActionInputs;
  };
}

The LLM sees actionId, description, categories, parameters, and examples. It uses these to understand what the action does and how to invoke it with the right parameters.

The UI layer sees uiSettings, which specifies what controls to render and how to transform between action inputs and UI state.

Both perspectives are defined in a single file for a single action. When you add a new action, you define both its LLM interface and its UI in one place.

Approach: declarative schema and bidirectional transforms

The core idea is to move from “UI as code” to “UI as data.” We define a declarative forms framework where each control is a parameterized type described by a payload and callbacks. At runtime, the assistant (or another service) chooses which controls to show by sending a description.

The framework is built on a simple idea: bidirectional transforms between action state and UI state.

Every action with UI declares:

1. componentType: what kind of control (Slider, ColorPicker, Picker, Flex)
2. generatePayload: convert action inputs → component props
3. parsePayload: convert component state → action inputs

Here’s a border action with multiple controls:

const SetBorderAction = {
  actionId: "setBorderStyle",
  uiSettings: {
    componentType: "Flex",
    generatePayload: (inputs) => ({
      direction: "column",
      children: [
        inputs.size !== undefined && {
          key: "size",
          componentType: "Slider",
          label: "Thickness",
          min: 0,
          max: 150,
          value: inputs.size,
        },
        inputs.color !== undefined && {
          key: "color",
          componentType: "ColorPicker",
          label: "Color",
          value: inputs.color,
        },
      ].filter(Boolean),
    }),
    parsePayload: (payload) =>
      payload.children.reduce((acc, child) => {
        acc[child.key] = child.value;
        return acc;
      }, {} as Record<string, unknown>),
  },
};

Now the action can express a composed UI: a flex column containing a slider and a color picker, generated automatically from its own inputs.

Type-safe component mapping

With 100+ actions, you want the compiler catching mistakes, not your users.

You can model component payloads as a map:

interface ComponentPayloadMap {
  Slider: { value: number; min: number; max: number; label: string };
  ColorPicker: { value: string; label?: string };
  Picker: {
    value: string;
    label: string;
    options: { value: string; label: string }[];
  };
  Flex: {
    direction: "row" | "column";
    children: ChildPayload[];
  };
}

Then define a helper type that picks the right payload type based on componentType:

type UISettings<TInputs> = {
  [K in keyof ComponentPayloadMap]: {
    componentType: K;
    generatePayload: (inputs: TInputs) => ComponentPayloadMap[K];
    parsePayload: (payload: ComponentPayloadMap[K]) => Partial<TInputs>;
  };
}[keyof ComponentPayloadMap];

When you write componentType: “Slider”, TypeScript now knows that:

• generatePayload must return { value: number; min: number; max: number; label: string }.
• parsePayload will receive that same shape.

Wrong payload shape? Compile error.

Teams can extend the map for custom components via declaration merging:

declare module "./types" {
  interface ComponentPayloadMap {
    ShadowControl: {
      offsetX: number;
      offsetY: number;
      blur: number;
      color: string;
    };
  }
}

Your generative UI framework stays open for extension while preserving type safety for every component.

Lazy loading: only load what you need

You don’t want to import 100 components upfront just in case the AI might use them. A registry pattern defers loading until the AI actually suggests an action that needs a given component:

class ComponentRegistry {
  private cache = new Map<string, any>();
  private loaders = new Map<string, () => Promise<any>>();

  register(type: string, loader: () => Promise<any>) {
    this.loaders.set(type, loader);
  }

  async get(type: string) {
    if (!this.cache.has(type)) {
      const loader = this.loaders.get(type);
      if (!loader) {
        throw new Error(`No loader registered for component type "${type}"`);
      }
      const component = await loader();
      this.cache.set(type, component);
    }
    return this.cache.get(type);
  }
}

// Registration
registry.register("Slider", () => import("./Slider"));
registry.register("ColorPicker", () => import("./ColorPicker"));

The first time the AI suggests an opacity change, the Slider component loads. Subsequent uses hit the cache. Components that the AI never suggests are never loaded.

Rendering the dynamic UI

Putting it together, here’s how AI output becomes interactive controls:

async function renderActionUI(invocation: ActionInvocation, actionDef: AssistantAction) {
  const { componentType, generatePayload, parsePayload } = actionDef.uiSettings!;

  // Get component (lazy loads if needed)
  const component = await registry.get(componentType);

  // Generate initial payload from action inputs
  const payload = generatePayload(invocation.inputs);

  // Render with callbacks
  return component.render(payload, {
    onInput: (newPayload: ComponentPayload) => {
      // Real-time feedback while dragging
      replayAction(invocation, parsePayload(newPayload), { preview: true });
    },
    onChange: (newPayload: ComponentPayload) => {
      // Commit when released
      replayAction(invocation, parsePayload(newPayload), { commit: true });
    },
  });
}

The replay pattern is key: UI changes don’t directly mutate document state. They trigger the action again with new inputs. The action remains the single source of truth for what happens to the document and how it is recorded in history.

Handling multiple actions

When the AI suggests several actions at once, UI components should appear together and not as disjoint pieces.

async function renderAllActionUIs(invocations: ActionInvocation[]) {
  // Deduplicate: same action on same element = one UI
  const deduped = deduplicateByActionAndTarget(invocations);

  // Render each
  const elements = await Promise.all(
    deduped
      .filter((inv) => inv.actionDef.uiSettings)
      .map((inv) => renderActionUI(inv, inv.actionDef)),
  );

  // Compose into container
  return elements;
}

The deduplication step prevents redundant UI. If the AI suggests two opacity changes on the same element, users see one slider, not two.

Real examples

Slider: opacity control

uiSettings: {
  componentType: "Slider",
  generatePayload: (inputs) => ({
    min: 0,
    max: 100,
    step: 1,
    value: (inputs.opacity ?? 1) * 100,
    label: "Opacity",
  }),
  parsePayload: (p) => ({ opacity: p.value / 100 }),
}

Color picker: border color

uiSettings: {
  componentType: "ColorPicker",
  generatePayload: (inputs) => ({
    value: inputs.color ?? "#000000",
    label: "Border color",
  }),
  parsePayload: (p) => ({ color: p.value }),
}

Picker: border style dropdown

uiSettings: {
  componentType: "Picker",
  generatePayload: (inputs) => ({
    value: inputs.style ?? "solid",
    label: "Style",
    options: [
      { value: "solid", label: "Solid" },
      { value: "dashed", label: "Dashed" },
      { value: "dotted", label: "Dotted" },
    ],
  }),
  parsePayload: (p) => ({ style: p.value }),
}

Composed: multiple controls in a flex container

uiSettings: {
  componentType: "Flex",
  generatePayload: (inputs) => ({
    direction: "column",
    children: [
      {
        key: "size",
        componentType: "Slider",
        label: "Size",
        min: 0,
        max: 150,
        value: inputs.size ?? 50,
      },
      {
        key: "color",
        componentType: "ColorPicker",
        label: "Color",
        value: inputs.color ?? "#ff0000",
      },
      {
        key: "style",
        componentType: "Picker",
        label: "Style",
        value: inputs.style ?? "solid",
        options: [
          { value: "solid", label: "Solid" },
          { value: "dashed", label: "Dashed" },
          { value: "dotted", label: "Dotted" },
        ],
      },
    ],
  }),
  parsePayload: (p) =>
    Object.fromEntries(p.children.map((c: any) => [c.key, c.value])),
}

The complete flow

1. User: “Add a thick red border.”
2. LLM returns: { action: “setBorderStyle”, params: { color: “#ff0000”, size: 50 } }.
3. The action executes, and the border appears on the canvas.
4. The generative UI layer renders: slider + color picker materialize using uiSettings.
5. User drags the slider → parsePayload → replay action with updated inputs → border updates live.
6. User releases → commit to undo stack.

The UI didn’t exist until the AI decided the user needed it. It materialized from a lightweight specification, bound itself to reactive state, and disappeared when no longer relevant.

Conclusion

The shift from static panels to generative UI is not just a technical refactor. It is a rethinking of who owns the interface. In a traditional application, designers and engineers negotiate a fixed contract. In an AI-powered application, the assistant becomes a runtime author, and the UI layer has to be ready for whatever it proposes.

The architecture described here meets that challenge by inverting a longstanding assumption. Instead of UI that knows about actions, you have actions that know about their UI. Every action ships with a specification: what control to render, how to initialize it from action inputs, and how to map user interactions back to action parameters. The LLM and the interface layer each get exactly what they need, derived from the same source of truth.

This is the right mental model for AI-native UI at scale. The future is not less UI. It is UI that assembles itself in response to AI‑interpreted user intent. When every possible interface can’t be pre‑built, you build the system that builds interfaces. That’s declarative forms for AI assistants.