import matplotlib
if not hasattr(matplotlib.RcParams, "_get"):
    matplotlib.RcParams._get = dict.get

Cleaning Tweets and Social Media Data in Python#

This notebook provides a step-by-step guide to cleaning tweets and other social media data using Python. The script is based on a tutorial by Catris Code and adapted by Monika Barget (November 2022).

Note: This version runs entirely in your browser via Live Code (Pyodide). There is no shared server or fixed data folder, so you upload your own file below and download your results at the end — nothing is saved automatically.

Step 1: Import Required Libraries#

We begin by importing the libraries needed for data processing, text cleaning, and the upload/download widgets.

Note: we no longer import or use nltk. Live Code runs Python inside your browser (via Pyodide), which cannot make the kind of network request nltk.download() needs, so this cell would otherwise fail with a urlopen error unknown url type: https message every time. We use a plain offline stopword list further down instead.

import numpy as np
import re
import io
import base64
import ipywidgets as widgets
from IPython.display import display, HTML, clear_output

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[1], line 5
      1 import numpy as np
      2 import re
      3 import io
      4 import base64
----> 5 import ipywidgets as widgets
      6 from IPython.display import display, HTML, clear_output

ModuleNotFoundError: No module named 'ipywidgets'

Step 2: Upload Your Tweets File#

Click Upload below and choose a .txt file from your own computer (one tweet per line). There is no fixed input folder any more — each person uploads their own file, and it only exists in your browser tab for this session.

tweets = []
uploaded_filename = None

upload_widget = widgets.FileUpload(accept='.txt', multiple=False, description='Upload tweets')
upload_output = widgets.Output()

def _handle_upload(change):
    global tweets, uploaded_filename
    with upload_output:
        clear_output()
        if not upload_widget.value:
            print("No file selected yet.")
            return
        # ipywidgets >=8 gives a tuple of dicts; ipywidgets 7 gives a dict keyed by filename
        value = upload_widget.value
        if isinstance(value, dict):
            item = next(iter(value.values()))
            uploaded_filename = next(iter(value.keys()))
        else:
            item = value[0]
            uploaded_filename = item['name']
        raw_bytes = bytes(item['content'])
        text = raw_bytes.decode('utf-8', errors='replace')
        tweets = text.splitlines()
        print(f"Loaded '{uploaded_filename}' with {len(tweets)} lines.")
        print("First 5 lines:")
        for t in tweets[:5]:
            print(" -", t)

upload_widget.observe(_handle_upload, names='value')
display(upload_widget, upload_output)

Step 3: Define Custom Stopwords#

Stopwords are common words (e.g., ‘the’, ‘and’) that can be filtered out. We use a simple offline list here — no internet download needed, so this works the same for everyone running Live Code.

my_stopwords = ["http", "https", "#", "@"]
print("Custom stopwords:", my_stopwords)

Step 4: Define the Cleaning Function#

This function performs the following operations:

Convert text to lowercase
Remove special characters, URLs, and punctuation
Filter out stopwords

def clean_tweet(tweet):
    if isinstance(tweet, float):  # covers NaN-like float values; np.float was removed from modern NumPy
        return ""
    temp = tweet.lower()
    temp = re.sub("'", "", temp)  # Remove apostrophes to preserve contractions
    temp = re.sub(r'/[A-Za-z0-9_]+', '', temp)  # Remove mentions (e.g., @user)
    temp = re.sub(r'http\S+', '', temp)  # Remove URLs
    temp = re.sub('[()!?]', ' ', temp)  # Remove specific punctuation
    temp = re.sub('\\[.*?\\]', ' ', temp)  # Remove square brackets and content
    temp = re.sub("[^a-z0-9]", " ", temp)  # Keep only alphanumeric
    temp = temp.split()
    temp = [w for w in temp if w not in my_stopwords]  # Filter stopwords
    temp = " ".join(word for word in temp)
    return temp

Step 5: Apply Cleaning Function to Tweets#

Process all uploaded tweets using the clean_tweet function. Run Step 2 first and make sure a file is uploaded, or this cell will simply report that there’s nothing to clean yet.

if not tweets:
    print("No tweets loaded yet — go back to Step 2 and upload a file first.")
    results = []
else:
    results = [clean_tweet(tw) for tw in tweets]
    print("First 5 cleaned tweets:")
    for r in results[:5]:
        print(" -", r)

Step 6: Download Cleaned Tweets#

Because this notebook runs inside your browser, there’s no local disk to save to directly. Running the cell below builds a download link — click it to save the cleaned file to your own machine, wherever you’d like.

if not results:
    print("Nothing to download yet — complete Steps 2 and 5 first.")
else:
    output_text = "\n".join(results)
    output_bytes = output_text.encode('utf-8')
    b64 = base64.b64encode(output_bytes).decode('utf-8')

    if uploaded_filename:
        base_name = uploaded_filename.rsplit('.', 1)[0]
        download_name = f"{base_name}_cleaned_with_Python.txt"
    else:
        download_name = "cleaned_tweets.txt"

    href = f'<a download="{download_name}" href="data:text/plain;base64,{b64}" target="_blank">\u2b07\ufe0f Download {download_name}</a>'
    display(HTML(href))
    print("Cleaning complete. Click the link above to save your results.")

Cleaning Tweets and Social Media Data in Python

Contents