Mastering Selenium: How to Click on Modal Alerts Like Cookie Pop Up – A Comprehensive Guide

Modal pop-ups, such as cookie consent notifications or login requests, are common challenges when scraping websites with Selenium. These pop-ups typically utilize custom JavaScript to obscure content upon page loading, displaying a message to the user. Efficiently handling these modal alerts can significantly streamline your web scraping projects. Employing a web scraping API provides a robust solution for such scenarios, enhancing your ability to interact with and navigate through these pop-ups. By integrating these APIs into your scraping toolkit, you can bypass modal alerts more effectively, allowing for smoother data extraction processes and optimizing your overall scraping strategy.

To effectively manage modal pop-ups, two prevalent strategies are utilized:

1. Clicking on an affirmative option like “OK” or “Yes” to dismiss the popup.
2. Removing the modal element directly from the Document Object Model (DOM).

An illustrative scenario involves handling a cookie consent popup on the login page of web-scraping.dev/login, demonstrated as follows:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

driver = webdriver.Chrome()

driver.get("https://web-scraping.dev/login")

# Option #1 - Dismissing the popup by clicking the consent button
try:
    cookie_ok_button = WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.ID, "cookie-ok")))
    cookie_ok_button.click()
except TimeoutException:
    print("No cookie popup present.")


# Option #2 - Removing the modal and its backdrop from the HTML
try:
    cookie_modal = WebDriverWait(driver, 2).until(EC.presence_of_element_located((By.ID, "cookieModal")))
    driver.execute_script("arguments[0].remove();", cookie_modal)
except TimeoutException:
    pass

try:
    modal_backdrop = WebDriverWait(driver, 2).until(EC.presence_of_element_located((By.CLASS_NAME, "modal-backdrop")))
    driver.execute_script("arguments[0].remove();", modal_backdrop)
except TimeoutException:
    pass

driver.quit()

This guide explores two approaches to navigate modal pop-ups: directly engaging with the popup through a click, which is typically more reliable as it can trigger associated functionality such as cookie consent; and programmatically removing the popup and any obfuscating overlays from the DOM, a method suited for bypassing obtrusive login or advertisement barriers.