Create a concurrent generic document loader using a filesystem blob loader.
from_filesystem(
cls,
path: _PathLike,
*,
glob: str = '**/[!.]*',
exclude: Sequence[str] = (),
suffixes: Optional[Sequence[str]] = None,
show_progress: bool = False,
parser: Union[DEFAULT, BaseBlobParser] = 'default',
num_workers: int = 4,
parser_kwargs: Optional[dict] = None
) -> ConcurrentLoader| Name | Type | Description |
|---|---|---|
path* | _PathLike | The path to the directory to load documents from. |
glob | str | Default: '**/[!.]*'The glob pattern to use to find documents. |
suffixes | Optional[Sequence[str]] | Default: NoneThe suffixes to use to filter documents. If None, all files matching the glob will be loaded. |
exclude | Sequence[str] | Default: ()A list of patterns to exclude from the loader. |
show_progress | bool | Default: FalseWhether to show a progress bar or not (requires tqdm). Proxies to the file system loader. |
parser | Union[DEFAULT, BaseBlobParser] | Default: 'default'A blob parser which knows how to parse blobs into documents |
num_workers | int | Default: 4Max number of concurrent workers to use. |
parser_kwargs | Optional[dict] | Default: NoneKeyword arguments to pass to the parser. |