[Feature Request]: Only extract visible content

[SimonMayerhofer]

What needs to be done?

I'd like to have an option that the final markdown only has text which is visible on the page. This means any content which is in an element where any parent element has e.g. display: none; or visibility: hidden; set is not extracted.

What problem does this solve?

We contact companies and reference content from their website. Hidden content might be outdated, so they sometimes wonder where we found that info.

Target users/beneficiaries

Marketing/Sales teams

Current alternatives/workarounds

None that I'm aware of. Maybe remove elements with JS after page load which are not visible.

Proposed approach

No response

[ntohidi]

@SimonMayerhofer
Hi. Crawl4AI does not currently have a built-in option to strip visually hidden elements from the final markdown. There's no exclude_hidden_elements flag or similar.
The closest workaround today is injecting JavaScript via CrawlerRunConfig.js_code to remove hidden elements before extraction:

  from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

  remove_hidden_js = """
  (() => {
      const allElements = document.querySelectorAll('body *');
      for (const el of allElements) {
          const style = window.getComputedStyle(el);
          if (style.display === 'none' || style.visibility === 'hidden') {
              el.remove();
          }
      }
  })();
  """

  config = CrawlerRunConfig(
      js_code=remove_hidden_js,
      wait_for="js:() => true",  # ensure JS runs first
  )

  async with AsyncWebCrawler() as crawler:
      result = await crawler.arun(url="https://example.com", config=config)
      print(result.markdown)  # only visible content

This runs in the browser (Playwright) so getComputedStyle resolves inherited display:none from parent elements — exactly what you need.