Fetch all urls concurrently with rate limiting.
Fetch all urls, then return soups for all results.
Async fetch all urls, then return soups for all results.
Scrape data from webpage and return it in BeautifulSoup format.
Asynchronously loads data into Document objects.
| Name | Type | Description |
|---|---|---|
web_path* | str | url of the sitemap. can also be a local path |
filter_urls | Optional[List[str]] | Default: Nonea list of regexes. If specified, only
URLS that match one of the filter URLs will be loaded.
WARNING The filter URLs are interpreted as regular expressions.
Remember to escape special characters if you do not want them to be
interpreted as regular expression syntax. For example, |
parsing_function | Optional[Callable] | Default: None |
blocksize | Optional[int] | Default: None |
blocknum | int | Default: 0 |
meta_function | Optional[Callable] | Default: None |
is_local | bool | Default: False |
continue_on_failure | bool | Default: False |
restrict_to_same_domain | bool | Default: True |
max_depth | int | Default: 10 |
Load a sitemap and its URLs.
Security Note: This loader can be used to load all URLs specified in a sitemap. If a malicious actor gets access to the sitemap, they could force the server to load URLs from other domains by modifying the sitemap. This could lead to server-side request forgery (SSRF) attacks; e.g., with the attacker forcing the server to load URLs from internal service endpoints that are not publicly accessible. While the attacker may not immediately gain access to this data, this data could leak into downstream systems (e.g., data loader is used to load data for indexing).
This loader is a crawler and web crawlers should generally NOT be deployed
with network access to any internal servers.
Control access to who can submit crawling requests and what network access
the crawler has.
By default, the loader will only load URLs from the same domain as the sitemap
if the site map is not a local file. This can be disabled by setting
restrict_to_same_domain to False (not recommended).
If the site map is a local file, no such risk mitigation is applied by default.
Use the filter URLs argument to limit which URLs can be loaded.
See https://python.langchain.com/docs/security
Function to parse bs4.Soup output
number of sitemap locations per block
the number of the block that should be loaded - zero indexed. Default: 0
Function to parse bs4.Soup output for metadata remember when setting this method to also copy metadata["loc"] to metadata["source"] if you are using this field
whether the sitemap is a local file. Default: False
whether to continue loading the sitemap if an error occurs loading a url, emitting a warning instead of raising an exception. Setting this to True makes the loader more robust, but also may result in missing data. Default: False
whether to restrict loading to URLs to the same domain as the sitemap. Attention: This is only applied if the sitemap is not a local file!
maximum depth to follow sitemap links. Default: 10