| Name | Type | Description |
|---|---|---|
url* | Union[str, AnyPath] | Cloud URL to load from. Supports s3://, az://, gs://, file:// schemes. If no scheme is provided, it is assumed to be a local file. If a path to a file is provided, glob/exclude/suffixes are ignored. |
glob | str | Default: '**/[!.]*'Glob pattern relative to the specified path by default set to pick up all non-hidden files |
exclude | Sequence[str] | Default: () |
suffixes | Optional[Sequence[str]] | Default: None |
show_progress | bool | Default: False |
Load blobs from cloud URL or file:.
Example:
.. code-block:: python
loader = CloudBlobLoader("s3://mybucket/id")
for blob in loader.yield_blobs():
print(blob)
patterns to exclude from results, use glob syntax
Provide to keep only files with these suffixes Useful when wanting to keep files with different suffixes Suffixes must include the dot, e.g. ".txt"
If true, will show a progress bar as the files are loaded. This forces an iteration through all matching files to count them prior to loading them.