Pebblo's safe dataloader is a wrapper for document loaders
Generate batches of documents based on page_content size. Args: docs: List of documents to be batched. max_batch_size: Maximum size of each batch in bytes. Defaults to 100*1024(100KB) Returns: List[List[Document]]: List of batches of documents
Return an absolute local path for a local file/directory, for a network related path, return as is.
Return an absolute source path of source of loader based on the keys present in Document.
Return loader type among, file, dir or in-memory.
Fetch the current Framework and Runtime details.
Fetch size of source path. Source can be a directory or a file.
Pebblo AI application.
Pebblo Framework instance.
Pebblo Indexed Document.
Wrapper for Pebblo Loader API.
Pebblo Safe Loader class is a wrapper around document loaders enabling the data to be scrutinized.
Loader for text data.
Since PebbloSafeLoader is a wrapper around document loaders, this loader is used to load text data directly into Documents.
