LangChain Reference home pageLangChain ReferenceLangChain Reference
  • GitHub
  • Main Docs
Deep Agents
LangChain
LangGraph
Integrations
LangSmith
  • Overview
  • MCP Adapters
    Standard Tests
    Text Splitters
    • Overview
    • Agents
    • Callbacks
    • Chains
    • Chat models
    • Embeddings
    • Evaluation
    • Globals
    • Hub
    • Memory
    • Output parsers
    • Retrievers
    • Runnables
    • LangSmith
    • Storage
    ⌘I

    LangChain Assistant

    Ask a question to get started

    Enter to send•Shift+Enter new line

    Menu

    MCP Adapters
    Standard Tests
    Text Splitters
    OverviewAgentsCallbacksChainsChat modelsEmbeddingsEvaluationGlobalsHubMemoryOutput parsersRetrieversRunnablesLangSmithStorage
    Language
    Theme
    Pythonlangchain-classicchainsnatbotcrawler
    Moduleā—Since v1.0

    crawler

    Attributes

    Classes

    View source on GitHub
    attribute
    logger
    attribute
    black_listed_elements: set[str]
    class
    ElementInViewPort
    class
    Crawler

    A typed dictionary containing information about elements in the viewport.

    A crawler for web pages.

    Security Note: This is an implementation of a crawler that uses a browser via Playwright.

    This crawler can be used to load arbitrary webpages INCLUDING content
    from the local file system.
    
    Control access to who can submit crawling requests and what network access
    the crawler has.
    
    Make sure to scope permissions to the minimal permissions necessary for
    the application.
    
    See https://docs.langchain.com/oss/python/security-policy for more information.