Menu
Language
Theme
useAudio, useImages, useVideo, and useFiles assemble multimodal blocks from the protocol stream. Each hook returns a reactive { current } handle whose value is an array of media handles (one per message containing a matching block in the target namespace).
Runnable example: The
multimodalstorybook in the streaming cookbook streams story text, generated images, audio narration, and video from scoped graph nodes.
Every media handle exposes the same core fields, plus media-specific extras:
| Field | Type | Description |
|---|---|---|
id |
string |
Stable id for the media block. |
messageId |
string |
Owning message id. |
partialBytes |
Uint8Array \| undefined |
Live-growing byte buffer during streaming. |
blob |
Blob \| undefined |
Settled Blob — populated on message-finish. |
objectURL |
string \| undefined |
Object URL for the settled Blob (auto-revoked). |
error |
unknown |
Fail-loud error surface. |
mimeType |
string \| undefined |
Declared MIME type. |
Media-specific extras:
| Hook | Extras |
|---|---|
useAudio |
transcript?: string, duration?: number, sampleRate?: number |
useImages |
width?: number, height?: number |
useVideo |
duration?: number, width?: number, height?: number |
useFiles |
filename?: string, size?: number |
<script lang="ts">
import { useAudio, useImages, useVideo, useFiles } from "@langchain/svelte";
const audios = useAudio(stream);
const images = useImages(stream);
const videos = useVideo(stream);
const files = useFiles(stream);
</script>
{#each images.current as image (image.id)}
<img src={image.objectURL} alt="" />
{/each}Each accepts an optional target argument (subagent / subgraph / { namespace }) to scope the subscription — see Selectors.
Pass a getter (() => target) when the target comes from $state and should re-bind reactively.
useMediaURLuseMediaURL turns a media handle into a stable blob URL you can pass to <audio/img/video src>. It also returns a { current } handle; pass a getter so the URL tracks the selected media handle reactively:
<script lang="ts">
import { useAudio, useMediaURL } from "@langchain/svelte";
const audios = useAudio(stream);
const latest = $derived(audios.current.at(-1));
const url = useMediaURL(() => latest);
</script>
{#if url.current}
<audio controls src={url.current}></audio>
{/if}useMediaURL(() => undefined) returns { current: undefined } — safe to call unconditionally.
useAudioPlayer / useVideoPlayeruseAudioPlayer and useVideoPlayer provide opinionated player handles with play/pause/seek state, built on top of the media hooks:
<script lang="ts">
import { useAudio, useAudioPlayer } from "@langchain/svelte";
const audio = useAudio(stream);
const latest = $derived(audio.current.at(-1));
const player = useAudioPlayer(() => latest, { autoPlay: true });
</script>
<button onclick={() => player.toggle()}>
{player.isPlaying ? "Pause" : "Play"}
</button>
<progress value={player.currentTime} max={player.duration}></progress>useVideoPlayer mirrors the same shape for video handles.
Subscribe to a scoped audio-media stream. Each handle is yielded
on its first matching content-block-start, exposes
.partialBytes for live access, settles .blob / .objectURL /
.transcript on
Subscribe to a scoped image-media stream. Pair with useMediaURL
for <img src>.
Subscribe to a scoped video-media stream. Pair with useMediaURL
for <video src>.
Subscribe to a scoped file-media stream. Pair with useMediaURL
for an <a download href> target.
Resolve the lazy MediaBase.objectURL promise into a
reactive string suitable for <audio src>, <img src>,
<video src>, or <a download href>. Reads .current for the
latest URL, or undefined
Progressive audio playback for AudioMedia handles with a
uniform surface across PCM (streamed) and container
(HTMLAudioElement) strategies.
The Svelte binding mirrors the React / Vue equivalents: r
Bind a VideoMedia handle to a caller-owned <video>
element.
Svelte idioms:
videoRef accepts a raw HTMLVideoElement, a $state binding,
or a getter. Use Svelte 5's bind:this to assign aPlayer controls + live state returned by useAudioPlayer.
Live state is exposed via getters on a stable object so templates
can read player.status / player.currentTime without a .value
hop. Impe
Controls + live state returned by useVideoPlayer. Mirrors
AudioPlayerHandle on the shared subset so callers only ever
learn one shape.
Shared surface across every media handle returned by MediaAssembler.
The handle is live while its parent message is active:
partialBytes is a snapshot of all bytes received so far.stream iShared surface across every media handle returned by MediaAssembler.
The handle is live while its parent message is active:
partialBytes is a snapshot of all bytes received so far.stream iShared surface across every media handle returned by MediaAssembler.
The handle is live while its parent message is active:
partialBytes is a snapshot of all bytes received so far.stream iShared surface across every media handle returned by MediaAssembler.
The handle is live while its parent message is active:
partialBytes is a snapshot of all bytes received so far.stream i