Cheerio can parse HTMLs while Puppeteer lets you run browser automation. They work pretty well together.
Let’s look at an example. Say I want to parse some data from stake.rocketpool.net, simply doing an http request and parse the result with Cheerio won’t work because it needs to execute some code in a browser to fetch the data.
By examining how the page loads, I noticed that class="loaded" will be added to the <body> tag when it finishes loading.
So you can write the code as:
To select a certain element in the HTML, you can test it out in chrome first:
Then you can write the code using the Cheerio APIs:
Puppeteer also allows you perform complex tasks. Here is an example where it inputs some text in the page and lets you parse the updated HTML:
Sometimes, it’s not easy to figure out the right selector for waitForSelector. A general solution (i.e. it won’t always work) is:
await page.goto(url, { waitUntil: 'networkidle0' });
Web scrapping isn’t the antidote for all situations. It’s slower, more complicated and less reliable (e.g. sometimes you will get the “Navigation timeout” error) than making http requests directly. Thus, in some cases, you might want to consider making http requests directly (using libraries such as axios) if it’s easy to identify the corresponding network request by analyzing the requests using the browser’s developer tools.
Some sites might also give you the HTML containing the data you want from making plain http requests. So you can use Cheerio directly to parse it. Here is an example:
Puppeteer also does not work well with docker. You can check out the guide if you want to dive into the rabbit hole.