# $5000 Per Hour Web Scraping Legal Site - Python in Plain English

By [howit](https://paragraph.com/@howit) · 2023-06-10

---

  
  

I answered a an advertisement on Craigslist to build a web scraper for a law firm. I actually built the web scraper in about 30 minutes. The request was to scrape definite data from a legal site and to place the information into an Excel spreadsheet.

Photo by [Tingey Injury Law Firm](https://unsplash.com/@tingeyinjurylawfirm?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText) on [Unsplash](https://unsplash.com/s/photos/legal?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText)

This is how I arrived the price. The customer told me that he does the task daily, it takes about 45 minutes and would pay some $20 per hour to perform it on a regular basis. To this I replied, there are about 200 working days in a year which amount to $4000. I told him that I would build the scraper for a little over 60% of that or $2500 which is equivalent to $5000 per hour based on my development time. To this he happily agreed. I promised him that I would send him a sample later tonight but I wrapped up during my lunch at work and sent it to him that afternoon. Had to wait a little did not want him to know that it was little to no effort on my part. But as I explained to him, he is paying for my expertise not my time. Lawyers are not the only ones who can get great pay for their services lol.

I build the scraper during lunch. The only challenge with the scraper is the output of the columns change and we don’t know what the correct order will be. To handle this nuance, I used a dictionary.

First, I loaded my normal imports. I used an existing scraper to speed up the process.

    import pandas as pd
    import os, re, requests, urllib
    from selenium import webdriver
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.chrome.service import Service
    from selenium.webdriver.common.by import By
    from datetime import datetime, date, timedelta
    from selenium.webdriver.support.select import Select
    from selenium.common.exceptions import NoSuchElementException
    import datetime
    from dateparser.search import search_dates 
    from bs4 import BeautifulSoup
    import glob as glob
    from time import sleep
    

This is the core part of the scraper — really easy to build:

    records = []
    for count in range(1, 5):
        data = {}
        for i in range(1, 10):
            label = driver.find_element(By.XPATH, f'/html/body/div/table[4]/thead/tr/th[{str(i)}]/a').text
            value = driver.find_element(By.XPATH, f'/html/body/div/table[4]/tbody/tr[{str(count)}]/td[{str(i)}]').text
            data.update({label:value})
        records.append(data)
    

And finally, here is a sample out of the columns — can’t show actual data contains PII and I don’t want to personally responsible.

    ['Case Number', 'Name', 'Date of Birth', 'Party Type', 'Court',       'Case Type', 'Case Status', 'Filing Date', 'Case Caption']
    

Yes it was really that easy. I imagine a lot of other law firms have similar need to I will use them as a target market going forward.

_More content at_ [**_PlainEnglish.io_**](https://plainenglish.io/)_. Sign up for our_ [**_free weekly newsletter_**](http://newsletter.plainenglish.io/)_. Follow us on_ [**_Twitter_**](https://twitter.com/inPlainEngHQ)_,_ [**_LinkedIn_**](https://www.linkedin.com/company/inplainenglish/)_,_ [**_YouTube_**](https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw)_, and_ [**_Discord_**](https://discord.gg/GtDtUAvyhW)_._\*\*

**_Interested in scaling your software startup_**_? Check out_ [**_Circuit_**](https://circuit.ooo/?utm=publication-post-cta)_._

---

*Originally published on [howit](https://paragraph.com/@howit/5000-per-hour-web-scraping-legal-site-python-in-plain-english)*