Mastering Puppeteer: How to Click on Modal Alerts like Cookie Pop Up

Using Puppeteer for web scraping often involves navigating modal popups, such as Javascript alerts that conceal content and display messages upon page load. For developers and businesses looking to streamline their data acquisition processes, leveraging a web scraping API becomes an indispensable tool. Such APIs simplify the complexities associated with web scraping, providing an efficient, robust solution for handling modal popups and beyond, thus enhancing the overall efficiency of your web scraping endeavors. A prevalent instance is the cookie consent popup, manageable through various Puppeteer techniques.

1. Interacting directly by clicking on provided options like “OK”, “Yes”, or “I Agree”.
2. Removing the modal element entirely from the DOM.

An illustrative example can be seen on the login page of web-scraping.dev/login, where a cookie consent popup appears as the page loads. The strategies for handling such popups in Puppeteer include:

const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch({ headless: false });
    const page = await browser.newPage();
    
    await page.goto('https://web-scraping.dev/login');
    
    // Option #1 - Interact by clicking on the button
    try {
        await page.waitForSelector('#cookie-ok', { timeout: 2000 });
        await page.click('#cookie-ok');
    } catch (error) {
        console.log('No cookie popup detected.');
    }
    
    // Option #2 - Remove the popup and any backdrop from the HTML
    const cookieModal = await page.$('#cookieModal');
    if (cookieModal) {
        await page.evaluate((el) => el.remove(), cookieModal);
    }

    const modalBackdrop = await page.$('.modal-backdrop');
    if (modalBackdrop) {
        await page.evaluate((el) => el.remove(), modalBackdrop);
    }
    
    await browser.close();
})();

This example demonstrates two primary methods for managing modal popups: initiating a click to dismiss the popup, a reliable approach as it might trigger associated functionality such as setting a cookie to prevent future popups; and directly removing popup elements from the DOM, suitable for bypassing login requirements or advertisements.