Tools

DHM280 · Project 2

Three kinds of tools keep this archive running: one to collect the tweets, one to make the giant files usable, and outside tools that turn IDs back into tweets.

Screenshot of the Tweets Sampling Toolkit repository on GitHub
The Tweets Sampling Toolkit on GitHub. Screenshot, Social Media Lab

The tool chain

  • Getting the data: Twitter’s COVID-19 Streaming Endpoint, read by a custom script the lab wrote.
  • Processing the data: the lab’s own Tweets Sampling Toolkit.
  • Putting it out: monthly download files and a public dashboard.
  • Using it afterwards: rehydration tools like Hydrator and Twarc, which rebuild full tweets from IDs.

Up close: the Tweets Sampling Toolkit

The Tweets Sampling Toolkit is a free Python 3 program the lab posted on GitHub. It exists because of a very practical problem. One month of this archive can be over 100 million tweet IDs, a file several gigabytes in size, and rehydrating all of it would take a very long time. The toolkit reads these files one line at a time, so it can handle them without needing a powerful computer.

It does a few simple but useful things:

  • Pulls a random sample, either a set number of IDs or a percentage, with no duplicates.
  • Splits one huge file into smaller ones.
  • Compares datasets using union, difference, and intersection. For example, a researcher can check which tweets in the archive are missing from their own collection and fetch only those.
  • Counts the IDs in a file.

Uses outside this project: nothing about it is specific to COVID. It works on any list of tweet IDs, so someone studying an election or a protest movement could use it just as easily.

Link to DH: Hydrator

A DH tool, and a broken one

To turn IDs back into tweets, the archive points users to Hydrator, a free app from Documenting the Now, a DH project focused on archiving social media ethically. It is a nice example of DH projects building on each other’s tools.

The problem is that Hydrator’s own GitHub page now says Twitter’s API changes make it “no longer a useful application.” The main route the project recommends for getting tweets back has stopped working.

How good is the documentation?

The toolkit itself is well documented. The GitHub page explains how to install it, what each function does, and what file formats it expects, and it even suggests the archive as test data. It is released under an open BSD-3-Clause license, so anyone can reuse or adapt it.

The weak spot is the first link in the chain. I couldn’t find the lab’s collection script published anywhere alongside the archive, so the step where tweets were actually pulled from Twitter can’t be checked or reused.

← DataMethods →