Look, I’m not here to pick sides. Donald Trump is merely my scape-goat for my using of the term “fake news”, but whether or not you’re a Republican or Democrat, social-media junkie or forested hermit, you have to admit that our rapidly digitalizing, polarized world is running into a certain problem: fake news. Whether it’s Facebook and the whole spectrum of monolithic technology companies feeding us only information that we *want *to hear to increase advertising revenue, we are losing touch on diversified information, and as consumers, are facing the likelihood that whatever we are fed may not even be accurate. This is significant — given the average adult, according to eMarketer, spends 3 hours and 54 minutes per day on our phones. That’s a lot of time to be fed information we don’t even know is accurate. As a data scientist, my goal is to harness the good of the information age to catalyze change in society, and hopefully progress within this field and likeminded researchers over time will reduce the negatives.

This project, my self-dubbed “fake news classifier”, will aim to use natural language processing to classify whether a news is fake or not fake. First, let’s define fake news:

As we see in the table above, fake news is prevalent over a wide array of mediums. For sake of use, we’ll be focusing on classifying satirical content (given that I, embarrassingly, have more than once, fallen to believing a The Onion news article).

We’ll be pulling from two Reddit subs, r/TheOnion and r/NotTheOnion to build our data. The Onion satisfies our requirement of satirical news, and Not The Onion is a subreddit which provides a diverse range of insane, but real news. Even Reddit frequenters often get these two mixed up.

The Onion even has to issue a warning!

We can see from above that r/TheOnion has about 100,000 readers, while r/NotTheOnion has over 15 million (that’s roughly 150x more readers or a LOT more, for you non-math junkies out there). Given this fact, we can expect a ton more posts from Not The Onion, and likely a lot more diversity in terms of news sources — given that The Onion is literally its own publication. This may skew a little bit of the details, as we’ll see later on, as The Onion posts are likely written in a certain homogeneous way, and may affect our analysis, but for now we’ll forge on!

For our data collection, we used the Pushshift API (pushshift.io) to collect about 15,500 posts (the limit it hit) from r/TheOnion and 17,500 posts from r/NotTheOnion.

You can see how I used the PSAW wrapper (based off the Pushshift API) to extract the posts below:

You could definitely set time.sleep to be less than 1 second. I was playing it way too safe.

Time spans for both Subreddits are as listed below:

1. **The Onion **subreddit pulled posts all the way back from October 3, 2013 to July 5, 2019.

2. **Not The Onion **subreddit (17,500) posts pulled posts back only from March 29, 2019 to July 6, 2019.

Objectively, the time span disparity between both Subreddits is not only due to the fact that Not The Onion has 150x more readers than The Onion, but also that The Onion is its own independent news site — while Not The Onion references a diverse range of “credible news sources”.

As we see above, The Onion mostly links The Onion related sites, while Not The Onion links relatively balanced news sources (surprisingly The Guardian is linked on both Subreddits). Looking at diversity of authors, we find something similar.

The author ‘dwaxe’ posts over 4,000 times — a pretty ridiculous number from the outset. Looking at the contrast between frequency of posts of The Onion and Not The Onion, the top poster in Not The Onion only has 350 posts. Considering the much higher number of readers, this may seem surprising, but makes sense when you consider the fact that the authors of The Onion subreddit are probably editors/posters from the publication itself.

All of this is to say: since the time disparity is large (a lot happens in a 6 year gap), and the domain linked of The Onion posts are mostly homogeneous, our model may not take into account an accurate representation of the current day. This is something we’ll definitely think about in future buildups of this project.

In addition, since these posts are all links to their own independent news articles, no post has a body — and thus what we will be focusing the entirety of our natural language analysis on are the titles of the articles. For the rest of the project, we’ll be imputing 1 to refer to The Onion and 0 to refer to Not The Onion.

To add onto our list of features, I used the packages VADER and TextBlob (yes, yes, I know…) to conduct sentiment analysis on the titles. I’ll show you how I implemented these two packages — starting with VADER:

from nltk.sentiment.vader import SentimentIntensityAnalyzersia = SentimentIntensityAnalyzer()corpus = list(df['title'])
sentimentscores = []for i in corpus:
    score = sia.polarity_scores(i)
    score['title'] = i
    sentimentscores.append(score)

    sentimentdf = pd.DataFrame(sentimentscores)
sentimentdf.drop(columns=['title'], inplace = True)

VADER gives you four scores, as explained by its documentation:

• The Positive, Negative and Neutral scores represent the proportion of text that falls in these categories. These scores add up to 1.

• The Compound score is a metric that calculates the sum of all the ratings which have been normalized between -1(most extreme negative) and +1 (most extreme positive).

We see our VADER scores below:

VADER Sentiment Scores

As we see, none of the scores are significantly different. But we can see that the compound score for Not The Onion posts are roughly 0.1 units less than that of the Onion — suggesting that the titles are a tad bit more negative.

Our implementation of TextBlob sentiment analysis is about the same, so I’ll skip that (see https://textblob.readthedocs.io/en/dev/ if you want to see more documentation), and go straight to business! TextBlob gives you two scores that range from 0 to 1:

1. Polarity describes positive and negative emotions in the text, where 0 is most negative and 1 is most positive.

2. Subjective sentences generally refer to personal opinion, emotion or judgment whereas objective refers to factual information. This range from 0 where it is most objective to 1 where it is most subjective.

We see our TextBlob sentiment scores below:

TextBlob Sentiment scores

We see some confirmations from the VADER analysis, with Not The Onion posts being slightly more negative, but also get some new information: subjectivity in The Onion posts is noticeably higher. We’ll see how sentiment analysis factors into our model later on.

Well. None of the scores are *that *correlated with each other — which makes sense considering this is a classification problem. But it’s helpful to know that sentiment analysis (especially that of Textblob) has some effect. Moving forward, we’ll use TextBlob’s sentiment analysis as our additional features.

Before vectorizing, we use WordNetLemmatizer and RegexpTokenizer to process the titles. We can see this below:

for i in range(len(listoftitles)):
    tokenized = tokenizer.tokenize(listoftitles[i].lower())
    lemmatizelist = []
    for word in tokenizedi:
        lemmatizedword = lemmatizer.lemmatize(word)
        lemmatizelist.append(lemmatizedword)
    listoftitles[i] = " ".join(lemmatizelist)soupedtitles = [BeautifulSoup(title).get_text() for title in listoftitles]cleanedsoup = [re.sub("[^a-zA-Z]", " ", i) for i in soupedtitles]arraysoup = np.ravel(cleanedsoup)df['title'] = arraysoup

The Tokenizer will separate our individual titles into a list of words, and the Lemmatizer will stem the word to its “root”. We then use regular expressions to remove any of the non-alphabetical characters, and replace the title array to our now processed title array!

Using CountVectorizer, we analyze the frequencies of words in both The Onion and Not The Onion subreddits:

Unigrams

The top 5 unigrams that appear in The Onion are “life”, “new”, man”, “ha”, and “trump”.

The top 5 unigrams that appear in Not The Onion are “man”, “say”, “woman”, “trump”, and “police”.

Bigrams

The top 5 bigrams that appear in The Onion are “year old”, “white house”, “donald trump”, “news source”, and “onion america”.

The top 5 bigrams that appear in Not The Onion are “year old”, “florida man”, “white house”, “molecule freedom”, and “police say”.

Stop Words

I take the intersection of the top unigrams and top bigrams that occur in both Subreddits, the word ‘onion’, and create a custom stop words list including the original English stop words list from Scikit-Learn. This will be useful not only in our modeling process but also in our coefficient analysis post-modeling.

I fit 7 models to the training set, with testing accuracy of the models being:

1. Logistic Regression with Count Vectorizer (86.9%)

2. Logistic Regression with Count Vectorizer + Sentiment Analysis (80.6%)

3. Logistic Regression with TfidfVectorizer (88.6%)

4. SVC Model with TfidfVectorizer (88.4%)

5. Multinomial Naive Bayes with TfidfVectorizer (87.8%)

Through trial and test, I found that the best models involve TfidfVectorizer without sentiment analysis. The original models were quite overfit (with >98% accuracy on the training set). For my final 2 models, I wanted to remove the features that were zeroed-out by the Logistic Regression with the L1 penalty and fit them into more models to potentially fix overfitting.

6. Gradient Boost Classifier (76%)

7. Logistic Regression with TfidfVectorizer (85.8% on test set, and 91.9% on the training set — which was the best model and one I chose, taking into account the bias-variance trade-off, as well as it being the model with the best interpretability)

We chose our best model, taking into account the bias-variance tradeoff, to be our Logistic Regression with the TfidfVectorizer without sentiment analysis. With a train-test-split with a 10% test size, this achieved a 91.9% accuracy on the training set with a 85.8% accuracy on the testing data. Obviously this wasn’t the best testing accuracy, but it definitely reduced variance — which is also an aspect of good modeling.

I plotted a Confusion Matrix of our model to our test scores:

Confusion Matrix of Model

With the following metrics:

Sensitivity: 84.45%
Specificity: 87.03%
Precision: 85.16%
Accuracy: 85.82%
Misclassification Rate: 14.18%

It’s clear that there are a fair amount of false negatives in our test set prediction (which in this case, are posts that are predicted to be citing real news, but are actually fake news). This is definitely an issue we have to address going forward — even if it may sacrifice accuracy.

Our most impactful features

Plotting the top 10 features that affected our model in either direction, we see that:

The words ‘life’, ‘said’, ‘quiz’, ‘nation’, ‘news’, ‘blog’ and ‘incredible’ increased the odds of the post being in The Onion subreddit the most.

The words ‘arrested’, ‘police’, ‘florida’, ‘mueller report’, ‘cocaine’, and ‘jail’ increased the odds of the post being in Not The Onion subreddit the most.

Features listed above affected the model’s classification ability a substantial amount. For instance, if the word ‘life’ was in the post’s title, relative to other words, the post had a 1531737x odds to be in The Onion. As we mentioned in our earlier discussion about the dataset, given that The Onion is drawing from homogeneous posters and domains (only The Onion news articles), it makes sense that certain words that are used often in either subreddit greatly affect the model’s classification capabilities.

This model was not perfect. There was still a fair bit of variance in the best model, with a high-enough bias, given the circumstances and the implications of not being able to detect fake news effectively. However, it was definitely better than the baseline, and considering that the model could detect Not The Onion posts with an accuracy of up to 88.5% accuracy makes it OK for my first voyage into Natural Language Processing and API web scraping.

If I were to do the project all over again, I would definitely gather more satirical news from a wider variety of sources than The Onion. Since the only text we could really analyze was the posts’ title (which is sadly fair, given that most of us in “The Internet Age” really only scan a news article’s title and make judgments from that), I would want to be able to create a database of news stories that extracted the actual news’ contents and made analyses based on that.

All things considered though, I definitely learned a ton through this project and can’t wait to continue building up my skills! For now, I’ll leave you all with these: