Step-by-Step Guide: How to Get Page Source in Puppeteer Effectively

Web scraping is an indispensable technique for data extraction, enabling analysts and developers to capture the full page source for various purposes, from market research to competitive analysis. Utilizing the Web Scraping API, a tool designed to streamline and enhance the efficiency of data retrieval processes can significantly augment the capabilities of web scraping frameworks. One such framework, Puppeteer, is particularly adept at navigating and extracting content from web pages. By employing Puppeteer’s page.content() method, users can effortlessly obtain the complete HTML of a web page, paving the way for in-depth data parsing with utilities like Cheerio. This article provides a comprehensive walkthrough on leveraging Puppeteer in conjunction with a robust web scraping API to achieve efficient and effective page source retrieval.

const puppeteer = require('puppeteer');

async function run() {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.goto("https://httpbin.dev/html");

    let source = await page.content();
    // OR the faster method that doesn't wait for images to load:
    let source = await page.content({"waitUntil": "domcontentloaded"});

    console.log(source);
    browser.close();
}

run();

⚠ Be aware that this command might retrieve the page source before the page fully loads if it’s a dynamic JavaScript page. For more information, see how to wait for a page to load in Puppeteer on the Scrape Network.