Menu
Language
Theme
useAudio, useImages, useVideo, and useFiles assemble multimodal blocks from the protocol stream. Each hook returns an array of media handles (one per message containing a matching block in the target namespace).
Runnable example: The
multimodalstorybook in the streaming cookbook streams story text, generated images, audio narration, and video from scoped graph nodes.
Every media handle exposes the same core fields, plus media-specific extras:
| Field | Type | Description |
|---|---|---|
id |
string |
Stable id for the media block. |
messageId |
string |
Owning message id. |
partialBytes |
Uint8Array \| undefined |
Live-growing byte buffer during streaming. |
blob |
Blob \| undefined |
Settled Blob — populated on message-finish. |
objectURL |
string \| undefined |
Object URL for the settled Blob (auto-revoked). |
error |
unknown |
Fail-loud error surface. |
mimeType |
string \| undefined |
Declared MIME type. |
Media-specific extras:
| Hook | Extras |
|---|---|
useAudio |
transcript?: string, duration?: number, sampleRate?: number |
useImages |
width?: number, height?: number |
useVideo |
duration?: number, width?: number, height?: number |
useFiles |
filename?: string, size?: number |
import { useAudio, useImages, useVideo, useFiles } from "@langchain/react";
const audios = useAudio(stream);
const images = useImages(stream);
const videos = useVideo(stream);
const files = useFiles(stream);Each accepts an optional target argument (subagent / subgraph / { namespace }) to scope the subscription — see Selectors.
useMediaURLuseMediaURL turns a media handle into a stable blob URL you can pass to <audio/img/video src>:
import { useAudio, useMediaURL } from "@langchain/react";
function AudioReply({ stream }: { stream: AnyStream }) {
const audios = useAudio(stream);
const latest = audios.at(-1);
const url = useMediaURL(latest);
return url ? <audio controls src={url} /> : null;
}useMediaURL(undefined) returns undefined — safe to call unconditionally.
useAudioPlayer / useVideoPlayeruseAudioPlayer and useVideoPlayer provide opinionated player handles with play/pause/seek state, built on top of the media hooks:
import { useAudio, useAudioPlayer } from "@langchain/react";
function AudioPlayer({ stream }: { stream: AnyStream }) {
const audio = useAudio(stream).at(-1);
const player = useAudioPlayer(audio, { autoPlay: true });
return (
<div>
<button onClick={player.toggle}>{player.isPlaying ? "Pause" : "Play"}</button>
<progress value={player.currentTime} max={player.duration} />
</div>
);
}useVideoPlayer mirrors the same shape for video handles.
Subscribe to a scoped audio-media stream. Returns an array of
AudioMedia handles, one per message containing at least one
AudioBlock in the target namespace.
Each handle is yielded on its first mat
Subscribe to a scoped image-media stream. See useAudio for
shared semantics; pair with useMediaURL for <img src>.
Subscribe to a scoped video-media stream. See useAudio for
shared semantics; pair with useMediaURL for <video src>.
Subscribe to a scoped file-media stream. See useAudio for
shared semantics; pair with useMediaURL for an
<a download href> target.
Resolve the lazy MediaBase.objectURL promise into a string
suitable for direct use in <audio src> / <img src> /
<video src> / <a href download>. Returns undefined until the
URL is available.
Progressive audio playback for AudioMedia handles with a
uniform surface across PCM (streamed) and container (HTMLAudioElement)
strategies.
Behaviour
Bind a VideoMedia handle to a caller-owned <video> element.
Contract
<video ref={videoRef}> and styles it however
they like; the hook never injects DOM nor overrides lPlayer controls + live state returned by useAudioPlayer.
Shape is shared with useVideoPlayer where possible — learn one surface, use it for both.
Controls + live state returned by useVideoPlayer. Mirrors AudioPlayerHandle on the shared subset so callers only ever learn one shape.
Shared surface across every media handle returned by MediaAssembler.
The handle is live while its parent message is active:
partialBytes is a snapshot of all bytes received so far.stream iShared surface across every media handle returned by MediaAssembler.
The handle is live while its parent message is active:
partialBytes is a snapshot of all bytes received so far.stream iShared surface across every media handle returned by MediaAssembler.
The handle is live while its parent message is active:
partialBytes is a snapshot of all bytes received so far.stream iShared surface across every media handle returned by MediaAssembler.
The handle is live while its parent message is active:
partialBytes is a snapshot of all bytes received so far.stream i