Skip to Content
PureHTMLdocs

Recipes & Real-World Patterns

Production-grade scraping patterns for modern websites, hydration state, pagination, and resilient selectors.

Real-world web scraping frequently presents challenging DOM patterns: Single Page App (SPA) hydration scripts, paginated listing pages with relative URLs, variant DOM structures caused by A/B testing, and adjacent sibling table layouts. PureHTML handles these scenarios cleanly and declaratively.


1. Scraping Next.js / Nuxt / JSON-LD Data

Modern web applications built with Next.js, Nuxt, Remix, and SEO-oriented sites embed pre-rendered structured state directly within <script> tags (such as script#__NEXT_DATA__, script#__NUXT_DATA__, or script[type="application/ld+json"]).

Extracting from these hydration script tags directly is vastly faster and less brittle than targeting presentation classes in the rendered DOM. You can chain json and jsonpath to query the state directly:

properties:
id:
  selector: script#__NEXT_DATA__
  transform:
    - json
    - 'jsonpath("$.props.pageProps.product.id")'
title:
  selector: script#__NEXT_DATA__
  transform:
    - json
    - 'jsonpath("$.props.pageProps.product.title")'
price:
  selector: script#__NEXT_DATA__
  transform:
    - json
    - 'jsonpath("$.props.pageProps.product.price")'
inStock:
  selector: script#__NEXT_DATA__
  transform:
    - json
    - 'jsonpath("$.props.pageProps.product.inStock")'
jsonLdPrice:
  selector: 'script[type="application/ld+json"]'
  transform:
    - json
    - 'jsonpath("$.offers.price")'
    - number

Web crawlers and list-detail scrapers frequently extract pagination links ("Next Page", "Previous Page") and item links. In most HTML documents, these links are stored as relative URLs (/catalog/page-3 or details/401).

Combine attr(href) with resolve to automatically resolve relative links against the base URL passed to extract():

properties:
currentPage:
  selector: .pagination .current
  transform:
    - 'replace("Page ", "")'
    - trim
    - number
nextPageUrl:
  selector: .pagination a.next
  transform:
    - attr(href)
    - resolve
items:
  selector: ul.items li a
  type: array
  items:
    properties:
      name:
        transform: trim
      url:
        transform:
          - attr(href)
          - resolve

3. Resilient Data Extraction: Combining Union and Default

Real-world web pages often exhibit variant markup due to A/B testing, conditional UI, or optional data:

  • A product price might be in .sale-price, .price, or marked with a .free-text badge.
  • Optional badges or metadata may be missing altogether on some items.

PureHTML's union evaluates candidates in order, and adding a constant entry at the end guarantees a safe fallback value:

selector: .products .product
type: array
items:
properties:
  name:
    selector: .name
    transform: trim
  price:
    union:
      - selector: .sale-price
        transform:
          - 'replace("\\$", "")'
          - trim
          - number
      - selector: .price
        transform:
          - 'replace("\\$", "")'
          - trim
          - number
      - selector: .free-text
        transform:
          - 'replace("FREE", "0")'
          - number
      - constant: 0
  status:
    union:
      - selector: .badge
        transform: trim
      - constant: "Standard"

4. Real-World Case Study: Hacker News Scraper

This production-ready scraper extracts submissions from Hacker News (news.ycombinator.com). It demonstrates multiple advanced patterns working together:

  • Sibling Row Navigation: Hacker News structures posts across adjacent <tr> rows. PureHTML uses CSS adjacent sibling combinators (+tr span.age, +tr span.score) to query metadata from sibling rows within each item.
  • Attribute Extraction: Extracts item ID with attr(id) on the row, and exact ISO timestamp with attr(title).
  • Full Link Resolution: Uses [attr(href), resolve] to ensure outbound article links and relative HN comment links (item?id=...) resolve to canonical absolute URLs.
selector: tr.athing
type: array
items:
properties:
  id:
    transform:
      - attr(id)
      - number
  title:
    selector: span.titleline > a
    transform: trim
  url:
    selector: span.titleline > a
    transform:
      - attr(href)
      - resolve
  points:
    selector: '+tr span.score'
    transform:
      - 'replace(" points", "")'
      - number
  author:
    selector: '+tr a.hnuser'
    transform: trim
  commentsCount:
    selector: '+tr a[href^="item?id="]:last-of-type'
    transform:
      - 'replace(" comments", "")'
      - number

On this page