I answered a an advertisement on Craigslist to build a web scraper for a law firm. I actually built the web scraper in about 30 minutes. The request was to scrape definite data from a legal site and to place the information into an Excel spreadsheet.

Photo by Tingey Injury Law Firm on Unsplash

This is how I arrived the price. The customer told me that he does the task daily, it takes about 45 minutes and would pay some $20 per hour to perform it on a regular basis. To this I replied, there are about 200 working days in a year which amount to $4000. I told him that I would build the scraper for a little over 60% of that or $2500 which is equivalent to $5000 per hour based on my development time. To this he happily agreed. I promised him that I would send him a sample later tonight but I wrapped up during my lunch at work and sent it to him that afternoon. Had to wait a little did not want him to know that it was little to no effort on my part. But as I explained to him, he is paying for my expertise not my time. Lawyers are not the only ones who can get great pay for their services lol.

I build the scraper during lunch. The only challenge with the scraper is the output of the columns change and we don’t know what the correct order will be. To handle this nuance, I used a dictionary.

First, I loaded my normal imports. I used an existing scraper to speed up the process.

import pandas as pd
import os, re, requests, urllib
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from datetime import datetime, date, timedelta
from selenium.webdriver.support.select import Select
from selenium.common.exceptions import NoSuchElementException
import datetime
from dateparser.search import search_dates 
from bs4 import BeautifulSoup
import glob as glob
from time import sleep

This is the core part of the scraper — really easy to build:

records = []
for count in range(1, 5):
    data = {}
    for i in range(1, 10):
        label = driver.find_element(By.XPATH, f'/html/body/div/table[4]/thead/tr/th[{str(i)}]/a').text
        value = driver.find_element(By.XPATH, f'/html/body/div/table[4]/tbody/tr[{str(count)}]/td[{str(i)}]').text
        data.update({label:value})
    records.append(data)

And finally, here is a sample out of the columns — can’t show actual data contains PII and I don’t want to personally responsible.

['Case Number', 'Name', 'Date of Birth', 'Party Type', 'Court',       'Case Type', 'Case Status', 'Filing Date', 'Case Caption']

Yes it was really that easy. I imagine a lot of other law firms have similar need to I will use them as a target market going forward.

More content at PlainEnglish.io. Sign up for our free weekly newsletter. Follow us on Twitter, LinkedIn, YouTube, and Discord.**

Interested in scaling your software startup? Check out Circuit.