LangChain Reference home pageLangChain ReferenceLangChain Reference
  • GitHub
  • Main Docs
Deep Agents
LangChain
LangGraph
Integrations
LangSmith
  • Overview
  • MCP Adapters
    • Overview
    • Agents
    • Callbacks
    • Chains
    • Chat models
    • Embeddings
    • Evaluation
    • Globals
    • Hub
    • Memory
    • Output parsers
    • Retrievers
    • Runnables
    • LangSmith
    • Storage
    Standard Tests
    Text Splitters
    ⌘I

    LangChain Assistant

    Ask a question to get started

    Enter to send•Shift+Enter new line

    Menu

    MCP Adapters
    OverviewAgentsCallbacksChainsChat modelsEmbeddingsEvaluationGlobalsHubMemoryOutput parsersRetrieversRunnablesLangSmithStorage
    Standard Tests
    Text Splitters
    Language
    Theme
    Pythonlangchain-classicchainsnatbotcrawler
    Module●Since v1.0

    crawler

    Attributes

    attribute
    logger
    attribute
    black_listed_elements: set[str]

    Classes

    class
    ElementInViewPort

    A typed dictionary containing information about elements in the viewport.

    class
    Crawler

    A crawler for web pages.

    Security Note: This is an implementation of a crawler that uses a browser via Playwright.

    This crawler can be used to load arbitrary webpages INCLUDING content
    from the local file system.
    
    Control access to who can submit crawling requests and what network access
    the crawler has.
    
    Make sure to scope permissions to the minimal permissions necessary for
    the application.
    
    See https://docs.langchain.com/oss/python/security-policy for more information.
    
    View source on GitHub