Chapter 94: Headless Browsers, Crawlers & Scrapers

Automated Screenshots & PDF Generation

High-fidelity document generation, full-page vs. element captures, `@media print` CSS rules, page headers/footers, and dynamic OpenGraph image generation.

LEARNING OBJECTIVES ⌵

Programmatically capture viewport, full-page, and element-level screenshots using Playwright and Puppeteer.
Generate print-ready, multi-page PDF documents from HTML with custom margins, pagination headers, and footers.
Master CSS print styling specifications (@media print, @page, break-inside: avoid, and print-color-adjust: exact).
Implement an automated OpenGraph (OG) social card image generation pipeline.

🎬 INTERACTIVE VISUAL PIPELINE Core Architecture Simulation

🌐

1. Input

Directives & Tags

⚙️

2. Parse

Tokenizer & AST

🌳

3. Layout

Box Model & Flow

🎨

4. Render

GPU Paint & Composite

PHASE 1: INPUT & DIRECTIVES

Browser receives declarative markup stream, parsing tag tokens and initializing component state.

📖 The Mental Model & Story (Intuitive Foundation)

For decades, generating formatted documents like invoices, quarterly reports, and certificates required clunky server-side libraries (like ReportLab or low-level binary PDF writers) where developers manually calculated millimeter coordinates in code. If you wanted to change a font size or add a two-column grid, you had to rewrite mathematical coordinate offsets.

Modern browser automation turns the browser itself into a high-precision digital printing press. You design your document using the full power of modern declarative HTML5 and CSS (Flexbox, Grid, custom web fonts, SVG graphics). Then, headless Chromium rasterizes the layout engine directly to a vector PDF or a high-DPI raster image.

+-----------------------------------------------------------------------------------+
|                        HTML-TO-PRINT RASTERIZATION PIPELINE                       |
+-----------------------------------------------------------------------------------+
|  1. Declarative Markup & Styles (HTML5 + CSS Grid + Flexbox + Web Fonts)          |
|                                        |                                          |
|                                        v                                          |
|  2. Headless Chromium Engine (Blink Layout + Skia 2D Graphics Library)            |
|                                        |                                          |
|            +---------------------------+---------------------------+              |
|            |                                                       |              |
|            v                                                       v              |
|  Raster Image Stream (.png / .jpeg)               Vector Document (.pdf)          |
|  - `page.screenshot({ fullPage: true })`          - `page.pdf({ format: 'A4' })`  |
|  - Device scale factor (1x, 2x Retina, 3x)        - Multi-page pagination         |
|  - Element clips & component masks                - Running headers & footers     |
+-----------------------------------------------------------------------------------+

Technical Deep Dive & Specifications

Screenshot Mechanics: Viewport vs. Full-Page vs. Element

When capturing screenshots in headless environments, you have three primary capture targets:

+-----------------------------------------------------------------------------------+
| Viewport Screenshot: Captures only what is inside the current window frame.        |
| Full-Page Screenshot: Measures scrollHeight and resizes virtual canvas.           |
| Element Screenshot: Computes element.getBoundingClientRect() and crops canvas.    |
+-----------------------------------------------------------------------------------+

Screenshot Method	Configuration Options	Key Behavior
Viewport	`page.screenshot({ path: 'view.png' })`	Captures only the current visible dimensions (e.g. 1280x720).
Full Page	`page.screenshot({ fullPage: true })`	Expands the virtual viewport to capture the entire scrollable DOM height.
Element Level	`locator.screenshot({ path: 'card.png' })`	Renders only the target node bounding box with transparent/background padding.
Clip Box	`page.screenshot({ clip: { x, y, width, height } })`	Captures a precise pixel-coordinate rectangle.

PDF Generation Specifications (`page.pdf()`)

[!NOTE] PDF generation via page.pdf() relies on the Chromium Skia PDF backend and is supported in Chromium-based browsers (Chrome, Edge). Firefox and WebKit do not support the CDP Page.printToPDF protocol.

Key configuration options for page.pdf():

await page.pdf({
  path: 'invoice.pdf',
  format: 'A4',                // 'Letter', 'Legal', 'A3', 'A4', etc.
  printBackground: true,       // CRITICAL: Renders CSS background colors and images
  margin: {
    top: '20mm',
    right: '15mm',
    bottom: '20mm',
    left: '15mm'
  },
  displayHeaderFooter: true,
  headerTemplate: '<div style="font-size: 9px; width: 100%; text-align: right; padding-right: 15mm;">Confidential Document</div>',
  footerTemplate: `
    <div style="font-size: 9px; width: 100%; text-align: center;">
      Page <span class="pageNumber"></span> of <span class="totalPages"></span>
    </div>
  `
});

Special Chromium Header/Footer CSS Classes

When displayHeaderFooter: true is enabled, Chromium injects dynamic metadata using these reserved class names:

: Formatted print date.
: Document <title>.
: Document URL.
: Current page number (1-indexed).
: Total page count.

Essential CSS Rules for Print & Document Export

To ensure PDF documents break cleanly across page boundaries without splitting lines of text or table rows, apply specialized CSS print media queries:

@media print {
  /* 1. Ensure backgrounds and borders print accurately */
  * {
    -webkit-print-color-adjust: exact !important;
    print-color-adjust: exact !important;
  }

  /* 2. Define page dimensions and zero browser margins */
  @page {
    size: A4 portrait;
    margin: 0; /* Let page.pdf margin handle spacing */
  }

  /* 3. Prevent page breaks inside cards, tables, and invoice items */
  .card, tr, .avoid-break {
    break-inside: avoid;
    page-break-inside: avoid;
  }

  /* 4. Force a clean page break before specific sections */
  .page-break {
    break-before: page;
    page-break-before: always;
  }

  /* 5. Hide web-only elements (buttons, navigation bars) */
  .no-print {
    display: none !important;
  }
}

💻 Interactive Code Playground

Here is a complete Node.js script using Playwright that builds a styled invoice in HTML, renders a high-DPI OpenGraph social card screenshot, and compiles a multi-page PDF with pagination.

Starter Code: `screenshot-pdf-generator.mjs`

Line-by-Line Code Breakdown

Line 11 (deviceScaleFactor: 2): Configures Chromium to render at 200% resolution (2 CSS pixels per physical device pixel), producing crisp text and sharp edges in generated PNG screenshots.
Lines 39–43 (@media print): Enforces print-color-adjust: exact so the browser preserves colored backgrounds (e.g. the green .badge and table headers) when printing.
Line 104 (await page.evaluate(() => document.fonts.ready)): Halts execution until all dynamic web fonts are completely downloaded and applied to the layout tree, avoiding Flash of Unstyled Text (FOUT) artifacts.
Line 107 (page.screenshot({ fullPage: false })): Captures the visual frame at the fixed 1200x630 viewport.
Lines 114–133 (page.pdf(...)): Invokes Chromium's vector PDF rasterizer with precise A4 dimensions, automated margins, and running page number footer templates.

Expected Terminal Output

import { chromium } from 'playwright';
import fs from 'node:fs';
import path from 'node:path';

async function generateMediaArtifacts() {
  console.log('[Engine] Launching Chromium instance...');
  const browser = await chromium.launch({ headless: true });

  try {
    const context = await browser.newContext({
      viewport: { width: 1200, height: 630 }, // Exact standard OpenGraph dimensions
      deviceScaleFactor: 2                     // 2x Retina pixel density for crisp text
    });
    const page = await context.newPage();

    // 1. Define rich HTML invoice with print stylesheets
    const invoiceHtml = `
      <!DOCTYPE html>
      <html lang="en">
      <head>
        <meta charset="UTF-8">
        <title>Invoice #INV-2026-089</title>
        <style>
          * { box-sizing: border-box; margin: 0; padding: 0; }
          body {
            font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
            color: #1e293b;
            background: #ffffff;
            padding: 40px;
          }
          .header {
            display: flex;
            justify-content: space-between;
            align-items: center;
            border-bottom: 2px solid #e2e8f0;
            padding-bottom: 20px;
            margin-bottom: 30px;
          }
          .brand { font-size: 24px; font-weight: 800; color: #0284c7; }
          .invoice-meta { text-align: right; }
          .invoice-meta h1 { font-size: 20px; color: #0f172a; }
          
          table { width: 100%; border-collapse: collapse; margin-top: 20px; }
          th { background: #f8fafc; text-align: left; padding: 12px; font-size: 13px; color: #64748b; border-bottom: 1px solid #cbd5e1; }
          td { padding: 12px; border-bottom: 1px solid #f1f5f9; font-size: 14px; }
          .total-row td { font-weight: 700; font-size: 16px; border-top: 2px solid #0f172a; }
          .badge { background: #dcfce7; color: #15803d; padding: 4px 8px; border-radius: 4px; font-size: 12px; font-weight: bold; }

          /* Print-specific rules */
          @media print {
            body { padding: 0; }
            * { -webkit-print-color-adjust: exact !important; print-color-adjust: exact !important; }
            tr { break-inside: avoid; page-break-inside: avoid; }
          }
        </style>
      </head>
      <body>
        <div class="header">
          <div class="brand">⚡ CloudScale Technologies</div>
          <div class="invoice-meta">
            <h1>INVOICE</h1>
            <p><strong>Invoice #:</strong> INV-2026-089</p>
            <p><strong>Status:</strong> <span class="badge">PAID</span></p>
          </div>
        </div>

        <main>
          <table>
            <thead>
              <tr>
                <th>Service Description</th>
                <th style="text-align: center;">Qty</th>
                <th style="text-align: right;">Unit Price</th>
                <th style="text-align: right;">Amount</th>
              </tr>
            </thead>
            <tbody>
              <tr>
                <td>Enterprise Kubernetes Cluster (us-east-1)</td>
                <td style="text-align: center;">1</td>
                <td style="text-align: right;">$1,200.00</td>
                <td style="text-align: right;">$1,200.00</td>
              </tr>
              <tr>
                <td>Edge CDN Data Transfer (10 TB)</td>
                <td style="text-align: center;">10</td>
                <td style="text-align: right;">$80.00</td>
                <td style="text-align: right;">$800.00</td>
              </tr>
              <tr>
                <td>Automated Headless QA Pipeline Runner</td>
                <td style="text-align: center;">1</td>
                <td style="text-align: right;">$350.00</td>
                <td style="text-align: right;">$350.00</td>
              </tr>
              <tr class="total-row">
                <td colspan="3" style="text-align: right;">Total Due:</td>
                <td style="text-align: right;">$2,350.00</td>
              </tr>
            </tbody>
          </table>
        </main>
      </body>
      </html>
    `;

    // 2. Load content into page
    await page.setContent(invoiceHtml, { waitUntil: 'networkidle' });

    // 3. Ensure all web fonts and assets are rendered before capturing
    await page.evaluate(() => document.fonts.ready);

    // 4. Capture a 1200x630 OpenGraph / Social Share Card Screenshot
    const screenshotBuffer = await page.screenshot({
      type: 'png',
      fullPage: false
    });
    console.log(`[Screenshot] Successfully generated PNG image (${screenshotBuffer.length} bytes).`);

    // 5. Generate a high-fidelity A4 Vector PDF
    const pdfBuffer = await page.pdf({
      format: 'A4',
      printBackground: true,
      margin: {
        top: '20mm',
        right: '15mm',
        bottom: '20mm',
        left: '15mm'
      },
      displayHeaderFooter: true,
      headerTemplate: `
        <div style="font-size: 8px; font-family: sans-serif; width: 100%; text-align: right; padding-right: 15mm; color: #94a3b8;">
          CloudScale Systems • Official Receipt
        </div>
      `,
      footerTemplate: `
        <div style="font-size: 8px; font-family: sans-serif; width: 100%; text-align: center; color: #94a3b8;">
          Page <span class="pageNumber"></span> of <span class="totalPages"></span>
        </div>
      `
    });

    console.log(`[PDF] Successfully generated Vector PDF document (${pdfBuffer.length} bytes).`);

  } finally {
    await browser.close();
    console.log('[Engine] Chromium session closed.');
  }
}

generateMediaArtifacts();

[Engine] Launching Chromium instance...
[Screenshot] Successfully generated PNG image (142850 bytes).
[PDF] Successfully generated Vector PDF document (38910 bytes).
[Engine] Chromium session closed.

🏋️ Hands-On Exercise

🎯 The Challenge: Component-Specific Social Share Image Generator

Scenario: You are building an automated OpenGraph (OG) image generation microservice for a blog platform. Given an article title, author name, and read time, your script must render an isolated HTML card element and capture only that element's bounding box as a 2x PNG image.

Instructions:

Create a page with a <div id="og-card"> styled with a dark gradient background, custom typography, and badge icons.
Ensure the card has dimensions width: 1200px; height: 630px;.
Use Playwright's page.locator('#og-card').screenshot() to capture the element.
Verify that the screenshot captures only the card element.

🏁 Starter Code Sandbox

import { chromium } from 'playwright';

async function generateSocialCard(title, author, readTime) {
  const browser = await chromium.launch({ headless: true });
  
  try {
    const page = await browser.newPage({
      viewport: { width: 1400, height: 900 },
      deviceScaleFactor: 2
    });

    const cardHtml = `
      <!DOCTYPE html>
      <html>
      <head>
        <style>
          body { margin: 0; background: #000; display: flex; align-items: center; justify-content: center; min-height: 100vh; }
          #og-card {
            width: 1200px;
            height: 630px;
            background: linear-gradient(135deg, #1e1b4b 0%, #0f172a 100%);
            color: #ffffff;
            padding: 60px;
            display: flex;
            flex-direction: column;
            justify-content: space-between;
            font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
            box-sizing: border-box;
            border: 1px solid #334155;
            border-radius: 16px;
          }
          .tag {
            display: inline-block;
            background: #0284c7;
            color: #ffffff;
            padding: 6px 14px;
            border-radius: 999px;
            font-size: 16px;
            font-weight: 700;
            letter-spacing: 0.05em;
            text-transform: uppercase;
          }
          .title {
            font-size: 52px;
            font-weight: 800;
            line-height: 1.15;
            margin-top: 24px;
            color: #f8fafc;
          }
          .meta {
            display: flex;
            justify-content: space-between;
            align-items: center;
            font-size: 24px;
            color: #94a3b8;
            border-top: 1px solid #334155;
            padding-top: 24px;
          }
        </style>
      </head>
      <body>
        <div id="og-card">
          <div>
            <span class="tag">Engineering Deep Dive</span>
            <h1 class="title">${title}</h1>
          </div>
          <div class="meta">
            <span>Author: <strong style="color: #f1f5f9;">${author}</strong></span>
            <span>⏱️ ${readTime} min read</span>
          </div>
        </div>
      </body>
      </html>
    `;

    await page.setContent(cardHtml, { waitUntil: 'networkidle' });
    await page.evaluate(() => document.fonts.ready);

    // Target specifically the #og-card locator
    const card = page.locator('#og-card');
    const imageBuffer = await card.screenshot({ type: 'png' });

    console.log(`[Success] Captured OG Card element screenshot (${imageBuffer.length} bytes, 2400x1260 physical pixels at 2x DPR).`);
    return imageBuffer;

  } finally {
    await browser.close();
  }
}

generateSocialCard('Mastering Headless Chrome & Playwright', 'Sarah Chen', 8);

Why this works:

locator.screenshot() queries the element's exact bounding box via getBoundingClientRect() and clips the offscreen raster buffer to those precise coordinates, ignoring any extraneous body or margin pixels.

⚠️ Common Pitfalls

Omitting printBackground: true: By default, Chromium follows W3C print specifications and ignores CSS background-color, background-image, and box shadows to save printer ink. For digital PDFs, you must always set printBackground: true and include print-color-adjust: exact in CSS.
Capturing Screenshots Before Web Fonts Load: If you trigger page.screenshot() immediately after page.goto(), external web fonts (Google Fonts, Typekit) may still be transferring over the network. This causes the screenshot to render with fallback system fonts. Always call await page.evaluate(() => document.fonts.ready) before capturing.
Attempting page.pdf() in Firefox or WebKit: page.pdf() is exclusive to Chromium engines. Calling it in Firefox or WebKit throws a runtime error (page.pdf: Printing to PDF is only supported in Chromium).

💡 Pro Tips

Auto-Scroll to Trigger IntersectionObservers / Lazy Images: When taking full-page screenshots of long pages, images below the initial fold are often lazy-loaded via IntersectionObserver. Execute a quick scroll loop before taking the screenshot:

await page.evaluate(async () => {
  await new Promise((resolve) => {
    let totalHeight = 0;
    const distance = 400;
    const timer = setInterval(() => {
      window.scrollBy(0, distance);
      totalHeight += distance;
      if (totalHeight >= document.body.scrollHeight) {
        clearInterval(timer);
        window.scrollTo(0, 0);
        resolve();
      }
    }, 100);
  });
});

Mask Dynamic Elements in Visual Snapshots: Prevent screenshot flakiness caused by changing timestamps, live stock tickers, or avatar images using Playwright's mask option:
```
await page.screenshot({
  mask: [page.locator('.timestamp'), page.locator('.live-ticker')]
});
```

📌 Key Takeaways

Headless Chromium uses its Blink layout engine and Skia 2D graphics library to render pixel-perfect screenshots and vector PDFs.
Use deviceScaleFactor: 2 (or higher) to generate ultra-sharp Retina resolution images for social sharing and OpenGraph cards.
page.pdf() requires Chromium and depends on printBackground: true and CSS print-color-adjust: exact to preserve styling.
Chromium PDF headers and footers support dynamic metadata injection via  and .
Prevent page breaks in table rows and cards using the CSS standard break-inside: avoid.
--

⭐ LEARN: HTML 🌟 ⚔️ QUIZ BATTLE ARENA // ACTIVE

? Question 1 / 3

Why do background colors and gradients appear blank/white when generating a PDF via `page.pdf()` unless explicitly configured?

Topic: HTML Fundamentals

? Question 2 / 3

Which CSS property should be applied to invoice line-item `<tr>` rows to prevent them from being cut in half across a multi-page PDF boundary?

Topic: HTML Fundamentals

? Question 3 / 3

How can you ensure that custom web fonts (e.g. `@font-face` or Google Fonts) are completely loaded before capturing a screenshot?

Topic: HTML Fundamentals

👨‍🚀 CADET FLIGHT DOSSIER

CADET EXPLORER

📖 The Mental Model & Story (Intuitive Foundation)

Technical Deep Dive & Specifications

Screenshot Mechanics: Viewport vs. Full-Page vs. Element

PDF Generation Specifications (page.pdf())

Special Chromium Header/Footer CSS Classes

Essential CSS Rules for Print & Document Export

💻 Interactive Code Playground

Starter Code: screenshot-pdf-generator.mjs

Line-by-Line Code Breakdown

Expected Terminal Output

🏋️ Hands-On Exercise

🎯 The Challenge: Component-Specific Social Share Image Generator

🏁 Starter Code Sandbox

⚠️ Common Pitfalls

💡 Pro Tips

📌 Key Takeaways

Why do background colors and gradients appear blank/white when generating a PDF via page.pdf() unless explicitly configured?

Which CSS property should be applied to invoice line-item <tr> rows to prevent them from being cut in half across a multi-page PDF boundary?

How can you ensure that custom web fonts (e.g. @font-face or Google Fonts) are completely loaded before capturing a screenshot?

PDF Generation Specifications (`page.pdf()`)

Starter Code: `screenshot-pdf-generator.mjs`

Why do background colors and gradients appear blank/white when generating a PDF via `page.pdf()` unless explicitly configured?

Which CSS property should be applied to invoice line-item `<tr>` rows to prevent them from being cut in half across a multi-page PDF boundary?

How can you ensure that custom web fonts (e.g. `@font-face` or Google Fonts) are completely loaded before capturing a screenshot?