Skip to content

Schemas

langsmith.schemas

Schemas for the LangSmith API.

Attachments module-attribute 

Attachments = dict[str, Union[tuple[str, bytes], Attachment, tuple[str, Path]]]

Attachments associated with the run. Each entry is a tuple of (mime_type, bytes), or (mime_type, file_path)

Attachment

Bases: NamedTuple

Annotated type that will be stored as an attachment if used.

Examples:

.. code-block:: python

    from langsmith import traceable
    from langsmith.schemas import Attachment


    @traceable
    def my_function(bar: int, my_val: Attachment):
        # my_val will be stored as an attachment
        # bar will be stored as inputs
        return bar

BinaryIOLike

Bases: Protocol

Protocol for binary IO-like objects.

METHOD DESCRIPTION
read

Read function.
seek

Seek function.
getvalue

Get value function.

read 

read(size: int = -1) -> bytes

Read function.

seek 

seek(offset: int, whence: int = 0) -> int

Seek function.

getvalue 

getvalue() -> bytes

Get value function.

ExampleBase

Bases: BaseModel

Example base model.

Config

Configuration class for the schema.

ExampleCreate

Bases: BaseModel

Example upload with attachments.

METHOD DESCRIPTION
__init__

Initialize from dict.

__init__ 

__init__(**data)

Initialize from dict.

ExampleUpsertWithAttachments

Bases: ExampleCreate

Example create with attachments.

METHOD DESCRIPTION
__init__

Initialize from dict.

__init__ 

__init__(**data)

Initialize from dict.

AttachmentInfo

Bases: TypedDict

Info for an attachment.

Example

Bases: ExampleBase

Example model.

METHOD DESCRIPTION
__init__

Initialize a Dataset object.
__repr__

Return a string representation of the RunBase object.

attachments class-attribute instance-attribute 

attachments: Optional[dict[str, AttachmentInfo]] = Field(default=None)

Dictionary with attachment names as keys and a tuple of the S3 url and a reader of the data for the file.

url property 

url: Optional[str]

URL of this run within the app.

Config

Configuration class for the schema.

__init__ 

__init__(
    _host_url: Optional[str] = None, _tenant_id: Optional[UUID] = None, **kwargs: Any
) -> None

Initialize a Dataset object.

__repr__ 

__repr__()

Return a string representation of the RunBase object.

ExampleSearch

Bases: ExampleBase

Example returned via search.

Config

Configuration class for the schema.

AttachmentsOperations

Bases: BaseModel

Operations to perform on attachments.

ExampleUpdate

Bases: BaseModel

Example update with attachments.

METHOD DESCRIPTION
__init__

Initialize from dict.

Config

Configuration class for the schema.

__init__ 

__init__(**data)

Initialize from dict.

DataType

Bases: str, Enum

Enum for dataset data types.

DatasetBase

Bases: BaseModel

Dataset base model.

Config

Configuration class for the schema.

DatasetTransformation

Bases: TypedDict

Schema for dataset transformations.

Dataset

Bases: DatasetBase

Dataset ORM model.

METHOD DESCRIPTION
__init__

Initialize a Dataset object.

url property 

url: Optional[str]

URL of this run within the app.

Config

Configuration class for the schema.

__init__ 

__init__(
    _host_url: Optional[str] = None,
    _tenant_id: Optional[UUID] = None,
    _public_path: Optional[str] = None,
    **kwargs: Any,
) -> None

Initialize a Dataset object.

DatasetVersion

Bases: BaseModel

Class representing a dataset version.

RunBase

Bases: BaseModel

Base Run schema.

A Run is a span representing a single unit of work or operation within your LLM app. This could be a single call to an LLM or chain, to a prompt formatting call, to a runnable lambda invocation. If you are familiar with OpenTelemetry, you can think of a run as a span.

METHOD DESCRIPTION
__repr__

Return a string representation of the RunBase object.

id instance-attribute 

id: UUID

Unique identifier for the run.

name instance-attribute 

name: str

Human-readable name for the run.

start_time instance-attribute 

start_time: datetime

Start time of the run.

run_type instance-attribute 

run_type: str

The type of run, such as tool, chain, llm, retriever, embedding, prompt, parser.

end_time class-attribute instance-attribute 

end_time: Optional[datetime] = None

End time of the run, if applicable.

extra class-attribute instance-attribute 

extra: Optional[dict] = Field(default_factory=_default_extra)

Additional metadata or settings related to the run.

error class-attribute instance-attribute 

error: Optional[str] = None

Error message, if the run encountered any issues.

serialized class-attribute instance-attribute 

serialized: Optional[dict] = None

Serialized object that executed the run for potential reuse.

events class-attribute instance-attribute 

events: Optional[list[dict]] = None

List of events associated with the run, like start and end events.

inputs class-attribute instance-attribute 

inputs: dict = Field(default_factory=dict)

Inputs used for the run.

outputs class-attribute instance-attribute 

outputs: Optional[dict] = None

Outputs generated by the run, if any.

reference_example_id class-attribute instance-attribute 

reference_example_id: Optional[UUID] = None

Reference to an example that this run may be based on.

parent_run_id class-attribute instance-attribute 

parent_run_id: Optional[UUID] = None

Identifier for a parent run, if this run is a sub-run.

tags class-attribute instance-attribute 

tags: Optional[list[str]] = None

Tags for categorizing or annotating the run.

attachments class-attribute instance-attribute 

attachments: Union[Attachments, dict[str, AttachmentInfo]] = Field(default_factory=dict)

Attachments associated with the run. Each entry is a tuple of (mime_type, bytes).

metadata property 

metadata: dict[str, Any]

Retrieve the metadata (if any).

revision_id property 

revision_id: Optional[UUID]

Retrieve the revision ID (if any).

latency property 

latency: Optional[float]

Latency in seconds.

Config

Configuration class for the schema.

__repr__ 

__repr__()

Return a string representation of the RunBase object.

Run

Bases: RunBase

Run schema when loading from the DB.

METHOD DESCRIPTION
__init__

Initialize a Run object.
__repr__

Return a string representation of the RunBase object.

session_id class-attribute instance-attribute 

session_id: Optional[UUID] = None

The project ID this run belongs to.

child_run_ids class-attribute instance-attribute 

child_run_ids: Optional[list[UUID]] = None

Deprecated: The child run IDs of this run.

child_runs class-attribute instance-attribute 

child_runs: Optional[list[Run]] = None

The child runs of this run, if instructed to load using the client These are not populated by default, as it is a heavier query to make.

feedback_stats class-attribute instance-attribute 

feedback_stats: Optional[dict[str, Any]] = None

Feedback stats for this run.

app_path class-attribute instance-attribute 

app_path: Optional[str] = None

Relative URL path of this run within the app.

manifest_id class-attribute instance-attribute 

manifest_id: Optional[UUID] = None

Unique ID of the serialized object for this run.

status class-attribute instance-attribute 

status: Optional[str] = None

Status of the run (e.g., 'success').

prompt_tokens class-attribute instance-attribute 

prompt_tokens: Optional[int] = None

Number of tokens used for the prompt.

completion_tokens class-attribute instance-attribute 

completion_tokens: Optional[int] = None

Number of tokens generated as output.

total_tokens class-attribute instance-attribute 

total_tokens: Optional[int] = None

Total tokens for prompt and completion.

prompt_token_details class-attribute instance-attribute 

prompt_token_details: Optional[dict[str, int]] = None

Breakdown of prompt (input) token counts.

Does not need to sum to full prompt token count.

completion_token_details class-attribute instance-attribute 

completion_token_details: Optional[dict[str, int]] = None

Breakdown of completion (output) token counts.

Does not need to sum to full completion token count.

first_token_time class-attribute instance-attribute 

first_token_time: Optional[datetime] = None

Time the first token was processed.

total_cost class-attribute instance-attribute 

total_cost: Optional[Decimal] = None

The total estimated LLM cost associated with the completion tokens.

prompt_cost class-attribute instance-attribute 

prompt_cost: Optional[Decimal] = None

The estimated cost associated with the prompt (input) tokens.

completion_cost class-attribute instance-attribute 

completion_cost: Optional[Decimal] = None

The estimated cost associated with the completion tokens.

prompt_cost_details class-attribute instance-attribute 

prompt_cost_details: Optional[dict[str, Decimal]] = None

Breakdown of prompt (input) token costs.

Does not need to sum to full prompt token cost.

completion_cost_details class-attribute instance-attribute 

completion_cost_details: Optional[dict[str, Decimal]] = None

Breakdown of completion (output) token costs.

Does not need to sum to full completion token cost.

parent_run_ids class-attribute instance-attribute 

parent_run_ids: Optional[list[UUID]] = None

List of parent run IDs.

trace_id instance-attribute 

trace_id: UUID

Unique ID assigned to every run within this nested trace.

dotted_order class-attribute instance-attribute 

dotted_order: str = Field(default='')

Dotted order for the run.

This is a string composed of {time}{run-uuid}.* so that a trace can be sorted in the order it was executed.

Example
  • Parent: 20230914T223155647Z1b64098b-4ab7-43f6-afee-992304f198d8
  • Children:
  • 20230914T223155647Z1b64098b-4ab7-43f6-afee-992304f198d8.20230914T223155649Z809ed3a2-0172-4f4d-8a02-a64e9b7a0f8a
  • 20230915T223155647Z1b64098b-4ab7-43f6-afee-992304f198d8.20230914T223155650Zc8d9f4c5-6c5a-4b2d-9b1c-3d9d7a7c5c7c

in_dataset class-attribute instance-attribute 

in_dataset: Optional[bool] = None

Whether this run is in a dataset.

url property 

url: Optional[str]

URL of this run within the app.

input_tokens property 

input_tokens: int | None

Alias for prompt_tokens.

output_tokens property 

output_tokens: int | None

Alias for completion_tokens.

input_cost property 

input_cost: Decimal | None

Alias for prompt_cost.

output_cost property 

output_cost: Decimal | None

Alias for completion_cost.

input_token_details property 

input_token_details: dict[str, int] | None

Alias for prompt_token_details.

output_token_details property 

output_token_details: dict[str, int] | None

Alias for output_token_details.

input_cost_details property 

input_cost_details: dict[str, Decimal] | None

Alias for prompt_cost_details.

output_cost_details property 

output_cost_details: dict[str, Decimal] | None

Alias for completion_cost_details.

id instance-attribute 

id: UUID

Unique identifier for the run.

name instance-attribute 

name: str

Human-readable name for the run.

start_time instance-attribute 

start_time: datetime

Start time of the run.

run_type instance-attribute 

run_type: str

The type of run, such as tool, chain, llm, retriever, embedding, prompt, parser.

end_time class-attribute instance-attribute 

end_time: Optional[datetime] = None

End time of the run, if applicable.

extra class-attribute instance-attribute 

extra: Optional[dict] = Field(default_factory=_default_extra)

Additional metadata or settings related to the run.

error class-attribute instance-attribute 

error: Optional[str] = None

Error message, if the run encountered any issues.

serialized class-attribute instance-attribute 

serialized: Optional[dict] = None

Serialized object that executed the run for potential reuse.

events class-attribute instance-attribute 

events: Optional[list[dict]] = None

List of events associated with the run, like start and end events.

inputs class-attribute instance-attribute 

inputs: dict = Field(default_factory=dict)

Inputs used for the run.

outputs class-attribute instance-attribute 

outputs: Optional[dict] = None

Outputs generated by the run, if any.

reference_example_id class-attribute instance-attribute 

reference_example_id: Optional[UUID] = None

Reference to an example that this run may be based on.

parent_run_id class-attribute instance-attribute 

parent_run_id: Optional[UUID] = None

Identifier for a parent run, if this run is a sub-run.

tags class-attribute instance-attribute 

tags: Optional[list[str]] = None

Tags for categorizing or annotating the run.

attachments class-attribute instance-attribute 

attachments: Union[Attachments, dict[str, AttachmentInfo]] = Field(default_factory=dict)

Attachments associated with the run. Each entry is a tuple of (mime_type, bytes).

metadata property 

metadata: dict[str, Any]

Retrieve the metadata (if any).

revision_id property 

revision_id: Optional[UUID]

Retrieve the revision ID (if any).

latency property 

latency: Optional[float]

Latency in seconds.

Config

Configuration class for the schema.

__init__ 

__init__(_host_url: Optional[str] = None, **kwargs: Any) -> None

Initialize a Run object.

__repr__ 

__repr__()

Return a string representation of the RunBase object.

RunTypeEnum

Bases: str, Enum

(Deprecated) Enum for run types. Use string directly.

RunLikeDict

Bases: TypedDict

Run-like dictionary, for type-hinting.

RunWithAnnotationQueueInfo

Bases: RunBase

Run schema with annotation queue info.

METHOD DESCRIPTION
__repr__

Return a string representation of the RunBase object.

last_reviewed_time class-attribute instance-attribute 

last_reviewed_time: Optional[datetime] = None

The last time this run was reviewed.

added_at class-attribute instance-attribute 

added_at: Optional[datetime] = None

The time this run was added to the queue.

id instance-attribute 

id: UUID

Unique identifier for the run.

name instance-attribute 

name: str

Human-readable name for the run.

start_time instance-attribute 

start_time: datetime

Start time of the run.

run_type instance-attribute 

run_type: str

The type of run, such as tool, chain, llm, retriever, embedding, prompt, parser.

end_time class-attribute instance-attribute 

end_time: Optional[datetime] = None

End time of the run, if applicable.

extra class-attribute instance-attribute 

extra: Optional[dict] = Field(default_factory=_default_extra)

Additional metadata or settings related to the run.

error class-attribute instance-attribute 

error: Optional[str] = None

Error message, if the run encountered any issues.

serialized class-attribute instance-attribute 

serialized: Optional[dict] = None

Serialized object that executed the run for potential reuse.

events class-attribute instance-attribute 

events: Optional[list[dict]] = None

List of events associated with the run, like start and end events.

inputs class-attribute instance-attribute 

inputs: dict = Field(default_factory=dict)

Inputs used for the run.

outputs class-attribute instance-attribute 

outputs: Optional[dict] = None

Outputs generated by the run, if any.

reference_example_id class-attribute instance-attribute 

reference_example_id: Optional[UUID] = None

Reference to an example that this run may be based on.

parent_run_id class-attribute instance-attribute 

parent_run_id: Optional[UUID] = None

Identifier for a parent run, if this run is a sub-run.

tags class-attribute instance-attribute 

tags: Optional[list[str]] = None

Tags for categorizing or annotating the run.

attachments class-attribute instance-attribute 

attachments: Union[Attachments, dict[str, AttachmentInfo]] = Field(default_factory=dict)

Attachments associated with the run. Each entry is a tuple of (mime_type, bytes).

metadata property 

metadata: dict[str, Any]

Retrieve the metadata (if any).

revision_id property 

revision_id: Optional[UUID]

Retrieve the revision ID (if any).

latency property 

latency: Optional[float]

Latency in seconds.

Config

Configuration class for the schema.

__repr__ 

__repr__()

Return a string representation of the RunBase object.

FeedbackSourceBase

Bases: BaseModel

Base class for feedback sources.

This represents whether feedback is submitted from the API, model, human labeler, etc.

type instance-attribute 

type: str

The type of the feedback source.

metadata class-attribute instance-attribute 

metadata: Optional[dict[str, Any]] = Field(default_factory=dict)

Additional metadata for the feedback source.

user_id class-attribute instance-attribute 

user_id: Optional[Union[UUID, str]] = None

The user ID associated with the feedback source.

user_name class-attribute instance-attribute 

user_name: Optional[str] = None

The user name associated with the feedback source.

APIFeedbackSource

Bases: FeedbackSourceBase

API feedback source.

type class-attribute instance-attribute 

type: Literal['api'] = 'api'

The type of the feedback source.

metadata class-attribute instance-attribute 

metadata: Optional[dict[str, Any]] = Field(default_factory=dict)

Additional metadata for the feedback source.

user_id class-attribute instance-attribute 

user_id: Optional[Union[UUID, str]] = None

The user ID associated with the feedback source.

user_name class-attribute instance-attribute 

user_name: Optional[str] = None

The user name associated with the feedback source.

ModelFeedbackSource

Bases: FeedbackSourceBase

Model feedback source.

type class-attribute instance-attribute 

type: Literal['model'] = 'model'

The type of the feedback source.

metadata class-attribute instance-attribute 

metadata: Optional[dict[str, Any]] = Field(default_factory=dict)

Additional metadata for the feedback source.

user_id class-attribute instance-attribute 

user_id: Optional[Union[UUID, str]] = None

The user ID associated with the feedback source.

user_name class-attribute instance-attribute 

user_name: Optional[str] = None

The user name associated with the feedback source.

FeedbackSourceType

Bases: Enum

Feedback source type.

API class-attribute instance-attribute 

API = 'api'

General feedback submitted from the API.

MODEL class-attribute instance-attribute 

MODEL = 'model'

Model-assisted feedback.

FeedbackBase

Bases: BaseModel

Feedback schema.

id instance-attribute 

id: UUID

The unique ID of the feedback.

created_at class-attribute instance-attribute 

created_at: Optional[datetime] = None

The time the feedback was created.

modified_at class-attribute instance-attribute 

modified_at: Optional[datetime] = None

The time the feedback was last modified.

run_id instance-attribute 

run_id: Optional[UUID]

The associated run ID this feedback is logged for.

trace_id instance-attribute 

trace_id: Optional[UUID]

The associated trace ID this feedback is logged for.

key instance-attribute 

key: str

The metric name, tag, or aspect to provide feedback on.

score class-attribute instance-attribute 

score: SCORE_TYPE = None

Value or score to assign the run.

value class-attribute instance-attribute 

value: VALUE_TYPE = None

The display value, tag or other value for the feedback if not a metric.

comment class-attribute instance-attribute 

comment: Optional[str] = None

Comment or explanation for the feedback.

correction class-attribute instance-attribute 

correction: Union[str, dict, None] = None

Correction for the run.

feedback_source class-attribute instance-attribute 

feedback_source: Optional[FeedbackSourceBase] = None

The source of the feedback.

session_id class-attribute instance-attribute 

session_id: Optional[UUID] = None

The associated project ID (Session = Project) this feedback is logged for.

comparative_experiment_id class-attribute instance-attribute 

comparative_experiment_id: Optional[UUID] = None

If logged within a 'comparative experiment', this is the ID of the experiment.

feedback_group_id class-attribute instance-attribute 

feedback_group_id: Optional[UUID] = None

For preference scoring, this group ID is shared across feedbacks for each

run in the group that was being compared.

extra class-attribute instance-attribute 

extra: Optional[dict] = None

The metadata of the feedback.

Config

Configuration class for the schema.

FeedbackCategory

Bases: TypedDict

Specific value and label pair for feedback.

value instance-attribute 

value: float

The numeric value associated with this feedback category.

label instance-attribute 

label: Optional[str]

An optional label to interpret the value for this feedback category.

FeedbackConfig

Bases: TypedDict

Represents how a feedback value ought to be interpreted.

type instance-attribute 

type: Literal['continuous', 'categorical', 'freeform']

The type of feedback.

min instance-attribute 

min: Optional[float]

The minimum value for continuous feedback.

max instance-attribute 

max: Optional[float]

The maximum value for continuous feedback.

categories instance-attribute 

categories: Optional[list[FeedbackCategory]]

If feedback is categorical, this defines the valid categories the server will accept. Not applicable to continuous or freeform feedback types.

FeedbackCreate

Bases: FeedbackBase

Schema used for creating feedback.

feedback_source instance-attribute 

feedback_source: FeedbackSourceBase

The source of the feedback.

feedback_config class-attribute instance-attribute 

feedback_config: Optional[FeedbackConfig] = None

The config for the feedback

id instance-attribute 

id: UUID

The unique ID of the feedback.

created_at class-attribute instance-attribute 

created_at: Optional[datetime] = None

The time the feedback was created.

modified_at class-attribute instance-attribute 

modified_at: Optional[datetime] = None

The time the feedback was last modified.

run_id instance-attribute 

run_id: Optional[UUID]

The associated run ID this feedback is logged for.

trace_id instance-attribute 

trace_id: Optional[UUID]

The associated trace ID this feedback is logged for.

key instance-attribute 

key: str

The metric name, tag, or aspect to provide feedback on.

score class-attribute instance-attribute 

score: SCORE_TYPE = None

Value or score to assign the run.

value class-attribute instance-attribute 

value: VALUE_TYPE = None

The display value, tag or other value for the feedback if not a metric.

comment class-attribute instance-attribute 

comment: Optional[str] = None

Comment or explanation for the feedback.

correction class-attribute instance-attribute 

correction: Union[str, dict, None] = None

Correction for the run.

session_id class-attribute instance-attribute 

session_id: Optional[UUID] = None

The associated project ID (Session = Project) this feedback is logged for.

comparative_experiment_id class-attribute instance-attribute 

comparative_experiment_id: Optional[UUID] = None

If logged within a 'comparative experiment', this is the ID of the experiment.

feedback_group_id class-attribute instance-attribute 

feedback_group_id: Optional[UUID] = None

For preference scoring, this group ID is shared across feedbacks for each

run in the group that was being compared.

extra class-attribute instance-attribute 

extra: Optional[dict] = None

The metadata of the feedback.

Config

Configuration class for the schema.

Feedback

Bases: FeedbackBase

Schema for getting feedback.

id instance-attribute 

id: UUID

The unique ID of the feedback.

created_at instance-attribute 

created_at: datetime

The time the feedback was created.

modified_at instance-attribute 

modified_at: datetime

The time the feedback was last modified.

feedback_source class-attribute instance-attribute 

feedback_source: Optional[FeedbackSourceBase] = None

The source of the feedback. In this case

run_id instance-attribute 

run_id: Optional[UUID]

The associated run ID this feedback is logged for.

trace_id instance-attribute 

trace_id: Optional[UUID]

The associated trace ID this feedback is logged for.

key instance-attribute 

key: str

The metric name, tag, or aspect to provide feedback on.

score class-attribute instance-attribute 

score: SCORE_TYPE = None

Value or score to assign the run.

value class-attribute instance-attribute 

value: VALUE_TYPE = None

The display value, tag or other value for the feedback if not a metric.

comment class-attribute instance-attribute 

comment: Optional[str] = None

Comment or explanation for the feedback.

correction class-attribute instance-attribute 

correction: Union[str, dict, None] = None

Correction for the run.

session_id class-attribute instance-attribute 

session_id: Optional[UUID] = None

The associated project ID (Session = Project) this feedback is logged for.

comparative_experiment_id class-attribute instance-attribute 

comparative_experiment_id: Optional[UUID] = None

If logged within a 'comparative experiment', this is the ID of the experiment.

feedback_group_id class-attribute instance-attribute 

feedback_group_id: Optional[UUID] = None

For preference scoring, this group ID is shared across feedbacks for each

run in the group that was being compared.

extra class-attribute instance-attribute 

extra: Optional[dict] = None

The metadata of the feedback.

Config

Configuration class for the schema.

TracerSession

Bases: BaseModel

TracerSession schema for the API.

Sessions are also referred to as "Projects" in the UI.

METHOD DESCRIPTION
__init__

Initialize a Run object.

id instance-attribute 

id: UUID

The ID of the project.

start_time class-attribute instance-attribute 

start_time: datetime = Field(default_factory=lambda: now(utc))

The time the project was created.

end_time class-attribute instance-attribute 

end_time: Optional[datetime] = None

The time the project was ended.

description class-attribute instance-attribute 

description: Optional[str] = None

The description of the project.

name class-attribute instance-attribute 

name: Optional[str] = None

The name of the session.

extra class-attribute instance-attribute 

extra: Optional[dict[str, Any]] = None

Extra metadata for the project.

tenant_id instance-attribute 

tenant_id: UUID

The tenant ID this project belongs to.

reference_dataset_id instance-attribute 

reference_dataset_id: Optional[UUID]

The reference dataset IDs this project's runs were generated on.

url property 

url: Optional[str]

URL of this run within the app.

metadata property 

metadata: dict[str, Any]

Retrieve the metadata (if any).

tags property 

tags: list[str]

Retrieve the tags (if any).

__init__ 

__init__(_host_url: Optional[str] = None, **kwargs: Any) -> None

Initialize a Run object.

TracerSessionResult

Bases: TracerSession

A project, hydrated with additional information.

Sessions are also referred to as "Projects" in the UI.

METHOD DESCRIPTION
__init__

Initialize a Run object.

run_count instance-attribute 

run_count: Optional[int]

The number of runs in the project.

latency_p50 instance-attribute 

latency_p50: Optional[timedelta]

The median (50th percentile) latency for the project.

latency_p99 instance-attribute 

latency_p99: Optional[timedelta]

The 99th percentile latency for the project.

total_tokens instance-attribute 

total_tokens: Optional[int]

The total number of tokens consumed in the project.

prompt_tokens instance-attribute 

prompt_tokens: Optional[int]

The total number of prompt tokens consumed in the project.

completion_tokens instance-attribute 

completion_tokens: Optional[int]

The total number of completion tokens consumed in the project.

last_run_start_time instance-attribute 

last_run_start_time: Optional[datetime]

The start time of the last run in the project.

feedback_stats instance-attribute 

feedback_stats: Optional[dict[str, Any]]

Feedback stats for the project.

session_feedback_stats instance-attribute 

session_feedback_stats: Optional[dict[str, Any]]

Summary feedback stats for the project.

run_facets instance-attribute 

run_facets: Optional[list[dict[str, Any]]]

Facets for the runs in the project.

total_cost instance-attribute 

total_cost: Optional[Decimal]

The total estimated LLM cost associated with the completion tokens.

prompt_cost instance-attribute 

prompt_cost: Optional[Decimal]

The estimated cost associated with the prompt (input) tokens.

completion_cost instance-attribute 

completion_cost: Optional[Decimal]

The estimated cost associated with the completion tokens.

first_token_p50 instance-attribute 

first_token_p50: Optional[timedelta]

The median (50th percentile) time to process the first token.

first_token_p99 instance-attribute 

first_token_p99: Optional[timedelta]

The 99th percentile time to process the first token.

error_rate instance-attribute 

error_rate: Optional[float]

The error rate for the project.

id instance-attribute 

id: UUID

The ID of the project.

start_time class-attribute instance-attribute 

start_time: datetime = Field(default_factory=lambda: now(utc))

The time the project was created.

end_time class-attribute instance-attribute 

end_time: Optional[datetime] = None

The time the project was ended.

description class-attribute instance-attribute 

description: Optional[str] = None

The description of the project.

name class-attribute instance-attribute 

name: Optional[str] = None

The name of the session.

extra class-attribute instance-attribute 

extra: Optional[dict[str, Any]] = None

Extra metadata for the project.

tenant_id instance-attribute 

tenant_id: UUID

The tenant ID this project belongs to.

reference_dataset_id instance-attribute 

reference_dataset_id: Optional[UUID]

The reference dataset IDs this project's runs were generated on.

url property 

url: Optional[str]

URL of this run within the app.

metadata property 

metadata: dict[str, Any]

Retrieve the metadata (if any).

tags property 

tags: list[str]

Retrieve the tags (if any).

__init__ 

__init__(_host_url: Optional[str] = None, **kwargs: Any) -> None

Initialize a Run object.

BaseMessageLike

Bases: Protocol

A protocol representing objects similar to BaseMessage.

content instance-attribute 

content: str

The content of the message.

additional_kwargs instance-attribute 

additional_kwargs: dict[Any, Any]

Additional keyword arguments associated with the message.

type property 

type: str

Type of the Message, used for serialization.

DatasetShareSchema

Bases: TypedDict

Represents the schema for a dataset share.

dataset_id instance-attribute 

dataset_id: UUID

The ID of the dataset.

share_token instance-attribute 

share_token: UUID

The token for sharing the dataset.

url instance-attribute 

url: str

The URL of the shared dataset.

AnnotationQueue

Bases: BaseModel

Represents an annotation queue.

id instance-attribute 

id: UUID

The unique identifier of the annotation queue.

name instance-attribute 

name: str

The name of the annotation queue.

description class-attribute instance-attribute 

description: Optional[str] = None

An optional description of the annotation queue.

created_at class-attribute instance-attribute 

created_at: datetime = Field(default_factory=lambda: now(utc))

The timestamp when the annotation queue was created.

updated_at class-attribute instance-attribute 

updated_at: datetime = Field(default_factory=lambda: now(utc))

The timestamp when the annotation queue was last updated.

tenant_id instance-attribute 

tenant_id: UUID

The ID of the tenant associated with the annotation queue.

AnnotationQueueWithDetails

Bases: AnnotationQueue

Represents an annotation queue with details.

rubric_instructions class-attribute instance-attribute 

rubric_instructions: Optional[str] = None

The rubric instructions for the annotation queue.

id instance-attribute 

id: UUID

The unique identifier of the annotation queue.

name instance-attribute 

name: str

The name of the annotation queue.

description class-attribute instance-attribute 

description: Optional[str] = None

An optional description of the annotation queue.

created_at class-attribute instance-attribute 

created_at: datetime = Field(default_factory=lambda: now(utc))

The timestamp when the annotation queue was created.

updated_at class-attribute instance-attribute 

updated_at: datetime = Field(default_factory=lambda: now(utc))

The timestamp when the annotation queue was last updated.

tenant_id instance-attribute 

tenant_id: UUID

The ID of the tenant associated with the annotation queue.

BatchIngestConfig

Bases: TypedDict

Configuration for batch ingestion.

use_multipart_endpoint instance-attribute 

use_multipart_endpoint: bool

Whether to use the multipart endpoint for batch ingestion.

scale_up_qsize_trigger instance-attribute 

scale_up_qsize_trigger: int

The queue size threshold that triggers scaling up.

scale_up_nthreads_limit instance-attribute 

scale_up_nthreads_limit: int

The maximum number of threads to scale up to.

scale_down_nempty_trigger instance-attribute 

scale_down_nempty_trigger: int

The number of empty threads that triggers scaling down.

size_limit instance-attribute 

size_limit: int

The maximum size limit for the batch.

size_limit_bytes instance-attribute 

size_limit_bytes: Optional[int]

The maximum size limit in bytes for the batch.

LangSmithInfo

Bases: BaseModel

Information about the LangSmith server.

version class-attribute instance-attribute 

version: str = ''

The version of the LangSmith server.

license_expiration_time class-attribute instance-attribute 

license_expiration_time: Optional[datetime] = None

The time the license will expire.

batch_ingest_config class-attribute instance-attribute 

batch_ingest_config: Optional[BatchIngestConfig] = None

The instance flags.

LangSmithSettings

Bases: BaseModel

Settings for the LangSmith tenant.

id instance-attribute 

id: str

The ID of the tenant.

display_name instance-attribute 

display_name: str

The display name of the tenant.

created_at instance-attribute 

created_at: datetime

The creation time of the tenant.

FeedbackIngestToken

Bases: BaseModel

Represents the schema for a feedback ingest token.

id instance-attribute 

id: UUID

The ID of the feedback ingest token.

url instance-attribute 

url: str

The URL to GET when logging the feedback.

expires_at instance-attribute 

expires_at: datetime

The expiration time of the token.

RunEvent

Bases: TypedDict

Run event schema.

name instance-attribute 

name: str

Type of event.

time instance-attribute 

time: Union[datetime, str]

Time of the event.

kwargs instance-attribute 

kwargs: Optional[dict[str, Any]]

Additional metadata for the event.

TimeDeltaInput

Bases: TypedDict

Timedelta input schema.

days instance-attribute 

days: int

Number of days.

hours instance-attribute 

hours: int

Number of hours.

minutes instance-attribute 

minutes: int

Number of minutes.

DatasetDiffInfo

Bases: BaseModel

Represents the difference information between two datasets.

examples_modified instance-attribute 

examples_modified: list[UUID]

A list of UUIDs representing the modified examples.

examples_added instance-attribute 

examples_added: list[UUID]

A list of UUIDs representing the added examples.

examples_removed instance-attribute 

examples_removed: list[UUID]

A list of UUIDs representing the removed examples.

ComparativeExperiment

Bases: BaseModel

Represents a comparative experiment.

This information summarizes evaluation results comparing two or more models on a given dataset.

id instance-attribute 

id: UUID

The unique identifier for the comparative experiment.

name class-attribute instance-attribute 

name: Optional[str] = None

The optional name of the comparative experiment.

description class-attribute instance-attribute 

description: Optional[str] = None

An optional description of the comparative experiment.

tenant_id instance-attribute 

tenant_id: UUID

The identifier of the tenant associated with this experiment.

created_at instance-attribute 

created_at: datetime

The timestamp when the comparative experiment was created.

modified_at instance-attribute 

modified_at: datetime

The timestamp when the comparative experiment was last modified.

reference_dataset_id instance-attribute 

reference_dataset_id: UUID

The identifier of the reference dataset used in this experiment.

extra class-attribute instance-attribute 

extra: Optional[dict[str, Any]] = None

Optional additional information about the experiment.

experiments_info class-attribute instance-attribute 

experiments_info: Optional[list[dict]] = None

Optional list of dictionaries containing information about individual experiments.

feedback_stats class-attribute instance-attribute 

feedback_stats: Optional[dict[str, Any]] = None

Optional dictionary containing feedback statistics for the experiment.

metadata property 

metadata: dict[str, Any]

Retrieve the metadata (if any).

PromptCommit

Bases: BaseModel

Represents a Prompt with a manifest.

owner instance-attribute 

owner: str

The handle of the owner of the prompt.

repo instance-attribute 

repo: str

The name of the prompt.

commit_hash instance-attribute 

commit_hash: str

The commit hash of the prompt.

manifest instance-attribute 

manifest: dict[str, Any]

The manifest of the prompt.

examples instance-attribute 

examples: list[dict]

The list of examples.

ListedPromptCommit

Bases: BaseModel

Represents a listed prompt commit with associated metadata.

id instance-attribute 

id: UUID

The unique identifier for the prompt commit.

owner instance-attribute 

owner: str

The owner of the prompt commit.

repo instance-attribute 

repo: str

The repository name of the prompt commit.

manifest_id class-attribute instance-attribute 

manifest_id: Optional[UUID] = None

The optional identifier for the manifest associated with this commit.

repo_id class-attribute instance-attribute 

repo_id: Optional[UUID] = None

The optional identifier for the repository.

parent_id class-attribute instance-attribute 

parent_id: Optional[UUID] = None

The optional identifier for the parent commit.

commit_hash class-attribute instance-attribute 

commit_hash: Optional[str] = None

The optional hash of the commit.

created_at class-attribute instance-attribute 

created_at: Optional[datetime] = None

The optional timestamp when the commit was created.

updated_at class-attribute instance-attribute 

updated_at: Optional[datetime] = None

The optional timestamp when the commit was last updated.

example_run_ids class-attribute instance-attribute 

example_run_ids: Optional[list[UUID]] = Field(default_factory=list)

A list of example run identifiers associated with this commit.

num_downloads class-attribute instance-attribute 

num_downloads: Optional[int] = 0

The number of times this commit has been downloaded.

num_views class-attribute instance-attribute 

num_views: Optional[int] = 0

The number of times this commit has been viewed.

parent_commit_hash class-attribute instance-attribute 

parent_commit_hash: Optional[str] = None

The optional hash of the parent commit.

Prompt

Bases: BaseModel

Represents a Prompt with metadata.

repo_handle instance-attribute 

repo_handle: str

The name of the prompt.

description class-attribute instance-attribute 

description: Optional[str] = None

The description of the prompt.

readme class-attribute instance-attribute 

readme: Optional[str] = None

The README of the prompt.

id instance-attribute 

id: str

The ID of the prompt.

tenant_id instance-attribute 

tenant_id: str

The tenant ID of the prompt owner.

created_at instance-attribute 

created_at: datetime

The creation time of the prompt.

updated_at instance-attribute 

updated_at: datetime

The last update time of the prompt.

is_public instance-attribute 

is_public: bool

Whether the prompt is public.

is_archived instance-attribute 

is_archived: bool

Whether the prompt is archived.

tags instance-attribute 

tags: list[str]

The tags associated with the prompt.

original_repo_id class-attribute instance-attribute 

original_repo_id: Optional[str] = None

The ID of the original prompt, if forked.

upstream_repo_id class-attribute instance-attribute 

upstream_repo_id: Optional[str] = None

The ID of the upstream prompt, if forked.

owner instance-attribute 

owner: Optional[str]

The handle of the owner of the prompt.

full_name instance-attribute 

full_name: str

The full name of the prompt. (owner + repo_handle)

num_likes instance-attribute 

num_likes: int

The number of likes.

num_downloads instance-attribute 

num_downloads: int

The number of downloads.

num_views instance-attribute 

num_views: int

The number of views.

liked_by_auth_user class-attribute instance-attribute 

liked_by_auth_user: Optional[bool] = None

Whether the prompt is liked by the authenticated user.

last_commit_hash class-attribute instance-attribute 

last_commit_hash: Optional[str] = None

The hash of the last commit.

num_commits instance-attribute 

num_commits: int

The number of commits.

original_repo_full_name class-attribute instance-attribute 

original_repo_full_name: Optional[str] = None

The full name of the original prompt, if forked.

upstream_repo_full_name class-attribute instance-attribute 

upstream_repo_full_name: Optional[str] = None

The full name of the upstream prompt, if forked.

ListPromptsResponse

Bases: BaseModel

A list of prompts with metadata.

repos instance-attribute 

repos: list[Prompt]

The list of prompts.

total instance-attribute 

total: int

The total number of prompts.

PromptSortField

Bases: str, Enum

Enum for sorting fields for prompts.

num_downloads class-attribute instance-attribute 

num_downloads = 'num_downloads'

Number of downloads.

num_views class-attribute instance-attribute 

num_views = 'num_views'

Number of views.

updated_at class-attribute instance-attribute 

updated_at = 'updated_at'

Last updated time.

num_likes class-attribute instance-attribute 

num_likes = 'num_likes'

Number of likes.

InputTokenDetails

Bases: TypedDict

Breakdown of input token counts.

Does not need to sum to full input token count. Does not need to have all keys.

audio instance-attribute 

audio: int

Audio input tokens.

cache_creation instance-attribute 

cache_creation: int

Input tokens that were cached and there was a cache miss.

Since there was a cache miss, the cache was created from these tokens.

cache_read instance-attribute 

cache_read: int

Input tokens that were cached and there was a cache hit.

Since there was a cache hit, the tokens were read from the cache. More precisely, the model state given these tokens was read from the cache.

OutputTokenDetails

Bases: TypedDict

Breakdown of output token counts.

Does not need to sum to full output token count. Does not need to have all keys.

audio instance-attribute 

audio: int

Audio output tokens.

reasoning instance-attribute 

reasoning: int

Reasoning output tokens.

Tokens generated by the model in a chain of thought process (i.e. by OpenAI's o1 models) that are not returned as part of model output.

InputCostDetails

Bases: TypedDict

Breakdown of input token costs.

Does not need to sum to full input cost. Does not need to have all keys.

audio instance-attribute 

audio: float

Cost of audio input tokens.

cache_creation instance-attribute 

cache_creation: float

Cost of input tokens that were cached and there was a cache miss.

Since there was a cache miss, the cache was created from these tokens.

cache_read instance-attribute 

cache_read: float

Cost of input tokens that were cached and there was a cache hit.

Since there was a cache hit, the tokens were read from the cache. More precisely, the model state given these tokens was read from the cache.

OutputCostDetails

Bases: TypedDict

Breakdown of output token costs.

Does not need to sum to full output cost. Does not need to have all keys.

audio instance-attribute 

audio: float

Cost of audio output tokens.

reasoning instance-attribute 

reasoning: float

Cost of reasoning output tokens.

Tokens generated by the model in a chain of thought process (i.e. by OpenAI's o1 models) that are not returned as part of model output.

UsageMetadata

Bases: TypedDict

Usage metadata for a message, such as token counts.

This is a standard representation of token usage that is consistent across models.

input_tokens instance-attribute 

input_tokens: int

Count of input (or prompt) tokens. Sum of all input token types.

output_tokens instance-attribute 

output_tokens: int

Count of output (or completion) tokens. Sum of all output token types.

total_tokens instance-attribute 

total_tokens: int

Total token count. Sum of input_tokens + output_tokens.

input_token_details instance-attribute 

input_token_details: NotRequired[InputTokenDetails]

Breakdown of input token counts.

Does not need to sum to full input token count. Does not need to have all keys.

output_token_details instance-attribute 

output_token_details: NotRequired[OutputTokenDetails]

Breakdown of output token counts.

Does not need to sum to full output token count. Does not need to have all keys.

input_cost instance-attribute 

input_cost: NotRequired[float]

The cost of the input tokens.

output_cost instance-attribute 

output_cost: NotRequired[float]

The cost of the output tokens.

total_cost instance-attribute 

total_cost: NotRequired[float]

The total cost of the tokens.

input_cost_details instance-attribute 

input_cost_details: NotRequired[InputCostDetails]

The cost details of the input tokens.

output_cost_details instance-attribute 

output_cost_details: NotRequired[OutputCostDetails]

The cost details of the output tokens.

ExtractedUsageMetadata

Bases: TypedDict

Usage metadata dictionary extracted from a run.

Should be the same as UsageMetadata, but does not require all keys to be present.

input_tokens instance-attribute 

input_tokens: int

The number of tokens used for the prompt.

output_tokens instance-attribute 

output_tokens: int

The number of tokens generated as output.

total_tokens instance-attribute 

total_tokens: int

The total number of tokens used.

input_token_details instance-attribute 

input_token_details: InputTokenDetails

The details of the input tokens.

output_token_details instance-attribute 

output_token_details: OutputTokenDetails

The details of the output tokens.

input_cost instance-attribute 

input_cost: float

The cost of the input tokens.

output_cost instance-attribute 

output_cost: float

The cost of the output tokens.

total_cost instance-attribute 

total_cost: float

The total cost of the tokens.

input_cost_details instance-attribute 

input_cost_details: InputCostDetails

The cost details of the input tokens.

output_cost_details instance-attribute 

output_cost_details: OutputCostDetails

The cost details of the output tokens.

UpsertExamplesResponse

Bases: TypedDict

Response object returned from the upsert_examples_multipart method.

count instance-attribute 

count: int

The number of examples that were upserted.

example_ids instance-attribute 

example_ids: list[str]

The ids of the examples that were upserted.

ExampleWithRuns

Bases: Example

Example with runs.

METHOD DESCRIPTION
__init__

Initialize a Dataset object.
__repr__

Return a string representation of the RunBase object.

runs class-attribute instance-attribute 

runs: list[Run] = Field(default_factory=list)

The runs of the example.

attachments class-attribute instance-attribute 

attachments: Optional[dict[str, AttachmentInfo]] = Field(default=None)

Dictionary with attachment names as keys and a tuple of the S3 url and a reader of the data for the file.

url property 

url: Optional[str]

URL of this run within the app.

Config

Configuration class for the schema.

__init__ 

__init__(
    _host_url: Optional[str] = None, _tenant_id: Optional[UUID] = None, **kwargs: Any
) -> None

Initialize a Dataset object.

__repr__ 

__repr__()

Return a string representation of the RunBase object.

ExperimentRunStats

Bases: TypedDict

Run statistics for an experiment.

run_count instance-attribute 

run_count: Optional[int]

The number of runs in the project.

latency_p50 instance-attribute 

latency_p50: Optional[timedelta]

The median (50th percentile) latency for the project.

latency_p99 instance-attribute 

latency_p99: Optional[timedelta]

The 99th percentile latency for the project.

total_tokens instance-attribute 

total_tokens: Optional[int]

The total number of tokens consumed in the project.

prompt_tokens instance-attribute 

prompt_tokens: Optional[int]

The total number of prompt tokens consumed in the project.

completion_tokens instance-attribute 

completion_tokens: Optional[int]

The total number of completion tokens consumed in the project.

last_run_start_time instance-attribute 

last_run_start_time: Optional[datetime]

The start time of the last run in the project.

run_facets instance-attribute 

run_facets: Optional[list[dict[str, Any]]]

Facets for the runs in the project.

total_cost instance-attribute 

total_cost: Optional[Decimal]

The total estimated LLM cost associated with the completion tokens.

prompt_cost instance-attribute 

prompt_cost: Optional[Decimal]

The estimated cost associated with the prompt (input) tokens.

completion_cost instance-attribute 

completion_cost: Optional[Decimal]

The estimated cost associated with the completion tokens.

first_token_p50 instance-attribute 

first_token_p50: Optional[timedelta]

The median (50th percentile) time to process the first token.

first_token_p99 instance-attribute 

first_token_p99: Optional[timedelta]

The 99th percentile time to process the first token.

error_rate instance-attribute 

error_rate: Optional[float]

The error rate for the project.

ExperimentResults

Bases: TypedDict

Results container for experiment data with stats and examples.

Breaking change in v0.4.32: The 'stats' field has been split into 'feedback_stats' and 'run_stats'.

feedback_stats instance-attribute 

feedback_stats: dict

Feedback statistics for the experiment.

run_stats instance-attribute 

run_stats: ExperimentRunStats

Run statistics (latency, token count, etc.).