Data

DHM280 · Project 2

The data is public tweets about COVID-19, shared not as the tweets themselves but as tweet IDs: long numbers that each point to one tweet.

Screenshot of the archive Downloads page listing monthly tweet ID datasets
The Downloads page: 4.17 billion tweet IDs in monthly datasets. Screenshot, Social Media Lab

What the data is

Every item in the archive is a public tweet about COVID-19. That ties straight back to the first goal: giving researchers the raw conversation so they can study how information, and misinformation, moved through it.

Why tweet IDs instead of tweets?

Twitter’s rules didn’t allow researchers to share full tweets, but they did allow IDs. There is an ethical upside too. If someone deletes a tweet, its ID no longer leads anywhere, so people keep some control over what they posted.

What goes in, what comes out

Input

A live stream of every public tweet Twitter itself flagged as COVID-related, pulled from Twitter’s COVID-19 Streaming Endpoint by a custom script the lab wrote. Twitter said the stream had no volume limits.

Output: files

Two files a month: every COVID-19 tweet ID, plus a vaccine-only subset (tweets containing words starting with vaccin*, vacin*, or vax*).

Output: dashboard

Monthly totals; the split between original tweets, retweets, and replies; location and language breakdowns; the most followed and most active accounts; and the most shared links.

Is it shared?

Yes, freely, for non-commercial research. Each monthly file has a permanent DOI link through figshare, so citations to the data won’t break. To get the full tweets back, users “rehydrate” the IDs with tools like Hydrator or the Python library Twarc, which is where things get complicated (see Tools).

Documentation and openness

On openness, the project does very well: free data, a permanent home for it, and a tool to help people use it. Documentation is mostly solid too, apart from the collection gap noted on the Goals page. The catch is that the openness only goes so far. The IDs are open, but rebuilding the tweets depends on access that Twitter controls.

← GoalsTools →