ScrapeNetwork

Mastering XPath Selectors in NodeJS: Comprehensive Guide on How to Use Them

Table of Contents

Table of Contents

CSS selectors are predominantly used in the NodeJS and Javascript ecosystems. However, for web scraping, the more robust features of XPath selectors may be required.
Several options are available for XPath selectors. The most popular one in web scraping is the osmosis library:

const osmosis = require("osmosis");

const html = `
<a href="http://scrapenetwork.com/">link 1</a>
<a href="http://scrapenetwork.com/">link 2</a>
`
osmosis
    .parse(html)
    .find('//a/@href')
    .log(console.log);

Another viable option is the xmldom library:

import xpath from 'xpath';
import { DOMParser } from '@xmldom/xmldom'

const tree = new DOMParser().parseFromString(`

    <h1>Page title</h1>
<p>some paragraph</p>
<a href="http://scrapenetwork.com/">some link</a>

`);

console.log({
    // we can extract text of the node, which returns `Text` object:
    title: xpath.select('//h1/text()', tree)[0].data,
    // or a specific attribute value, which return `Attr` object:
    url: xpath.select('//a/@href', tree)[0].value,
});

Related Questions

Related Blogs

Css Selectors
XPath and CSS selectors are vital tools for parsing HTML in web scraping, serving similar purposes with distinct features. While CSS selectors are lauded for...
Data Parsing
Dynamic class names on websites pose a significant challenge for web scraping efforts, reflecting the complexity and ever-evolving nature of the modern web. These classes,...
Data Parsing
Python, in conjunction with BeautifulSoup4 and xlsxwriter, plus an HTTP client-like requests, can be employed to convert an HTML table into an Excel spreadsheet. This...