Logo New Black

Comprehensive Guide: How to Capture XHR Requests Playwright with Ease

When utilizing Playwright and Python for web scraping to capture background requests and responses, the integration of a powerful web scraping API can significantly streamline the process. In this context, the page.on() method plays a crucial role, allowing developers to add middleware callbacks for handling request and response events efficiently. This capability is essential for extracting valuable data from dynamic web pages where content loading depends on various asynchronous HTTP requests, including XHR (XMLHttpRequest) and Fetch API calls. By leveraging such an API in conjunction with Playwright’s features, developers can enhance their web scraping solutions, making them more robust and adaptable to complex web environments.

from playwright.sync_api import sync_playwright

def intercept_request(request):
    # we can update requests with custom headers
    if "secret" in request.url :
        request.headers['x-secret-token'] = "123"
        print("patched headers of a secret request")
    # or adjust sent data
    if request.method == "POST":
        request.post_data = "patched"
        print("patched POST request")
    return request

def intercept_response(response):
    # we can extract details from background requests
    if response.request.resource_type == "xhr":
    return response

with sync_playwright() as pw:
    browser = pw.chromium.launch(headless=False)
    context = browser.new_context(viewport={"width": 1920, "height": 1080})
    page = context.new_page()
    # enable intercepting for this page
    page.on("request", intercept_request)
    page.on("response", intercept_response)


These background requests often contain crucial dynamic data. Blocking certain requests can also decrease the bandwidth consumed by the scraper. For more information on this, refer to the guide on blocking resources in Playwright and Python.