# Web Scraping with Puppeteer and Cheerio By [bap2pecs](https://paragraph.com/@0xb2p) · 2022-10-04 --- `Cheerio` can parse HTMLs while `Puppeteer` lets you run browser automation. They work pretty well together. Let’s look at an example. Say I want to parse some data from [stake.rocketpool.net](http://stake.rocketpool.net), simply doing an http request and parse the result with `Cheerio` won’t work because it needs to execute some code in a browser to fetch the data. By examining how the page loads, I noticed that `class="loaded"` will be added to the `` tag when it finishes loading. ![](https://storage.googleapis.com/papyrus_images/c024e0af4a2e8d296aef8a2739dca26f8c1a97b30853661afcc1c98152b9819f.gif) So you can write the code as: ![](https://storage.googleapis.com/papyrus_images/f3ce39266771e01a670fefc84d085a1d0adb37f186a165e47e3116a439775509.png) To select a certain element in the HTML, you can test it out in chrome first: ![](https://storage.googleapis.com/papyrus_images/550764e5a4317c912ed600529a9b84e040fa47f00c544473a62a3149c09702a5.png) Then you can write the code using the `Cheerio` APIs: ![](https://storage.googleapis.com/papyrus_images/f3ed0714a998b2c8acb112f8254a0a9ce26356923f473a6c529d7464fe81adde.png) `Puppeteer` also allows you perform complex tasks. Here is an example where it inputs some text in the page and lets you parse the updated HTML: ![](https://storage.googleapis.com/papyrus_images/3262fe1bf40c718468a7127db3f4ce14b7d3b8cf440b64095c12b28ac80bd3d4.png) Sometimes, it’s not easy to figure out the right selector for `waitForSelector`. A general solution (i.e. it won’t always work) is: await page.goto(url, { waitUntil: 'networkidle0' }); Limitations ----------- Web scrapping isn’t the antidote for all situations. It’s slower, more complicated and less reliable (e.g. sometimes you will get the “Navigation timeout” error) than making http requests directly. Thus, in some cases, you might want to consider making http requests directly (using libraries such as `axios`) if it’s easy to identify the corresponding network request by analyzing the requests using the browser’s developer tools. Some sites might also give you the HTML containing the data you want from making plain http requests. So you can use `Cheerio` directly to parse it. Here is an example: ![](https://storage.googleapis.com/papyrus_images/1820db505031605530d725985d66b7d2acb6ddd5742ed9c87345ae45173f503c.png) `Puppeteer` also does not work well with docker. You can check out the [guide](https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md#running-puppeteer-in-docker) if you want to dive into the rabbit hole. --- *Originally published on [bap2pecs](https://paragraph.com/@0xb2p/web-scraping-with-puppeteer-and-cheerio)*