{"id":36,"date":"2026-11-02T18:14:03","date_gmt":"2026-11-02T18:14:03","guid":{"rendered":"https:\/\/classprojects.digitalwcu.org\/DESocial\/tools\/"},"modified":"2026-11-02T19:11:30","modified_gmt":"2026-11-02T19:11:30","slug":"tools","status":"publish","type":"page","link":"https:\/\/classprojects.digitalwcu.org\/DESocial\/tools\/","title":{"rendered":"Tools"},"content":{"rendered":"<p class=\"dh-eyebrow wp-block-paragraph\">DHM280 \u00b7 Project 2<\/p><p class=\"dh-lead wp-block-paragraph\">Three kinds of tools keep this archive running: one to collect the tweets, one to make the giant files usable, and outside tools that turn IDs back into tweets.<\/p><figure class=\"wp-block-image size-full dh-hero dh-shot\"><img loading=\"lazy\" decoding=\"async\" width=\"1280\" height=\"760\" src=\"\/DESocial\/wp-content\/uploads\/2026\/11\/shot-tools.jpg\" alt=\"Screenshot of the Tweets Sampling Toolkit repository on GitHub\" class=\"wp-image-47\" srcset=\"https:\/\/classprojects.digitalwcu.org\/DESocial\/wp-content\/uploads\/2026\/11\/shot-tools.jpg 1280w, https:\/\/classprojects.digitalwcu.org\/DESocial\/wp-content\/uploads\/2026\/11\/shot-tools-300x178.jpg 300w, https:\/\/classprojects.digitalwcu.org\/DESocial\/wp-content\/uploads\/2026\/11\/shot-tools-1024x608.jpg 1024w, https:\/\/classprojects.digitalwcu.org\/DESocial\/wp-content\/uploads\/2026\/11\/shot-tools-768x456.jpg 768w\" sizes=\"auto, (max-width: 1280px) 100vw, 1280px\" \/><figcaption class=\"wp-element-caption\">The Tweets Sampling Toolkit on GitHub. Screenshot, Social Media Lab<\/figcaption><\/figure><div class=\"wp-block-group dh-section is-layout-flow wp-block-group-is-layout-flow\"><h2 class=\"wp-block-heading\">The tool chain<\/h2><ul class=\"wp-block-list\"><li><strong>Getting the data:<\/strong> Twitter&#8217;s COVID-19 Streaming Endpoint, read by a custom script the lab wrote.<\/li><li><strong>Processing the data:<\/strong> the lab&#8217;s own Tweets Sampling Toolkit.<\/li><li><strong>Putting it out:<\/strong> monthly download files and a public dashboard.<\/li><li><strong>Using it afterwards:<\/strong> rehydration tools like Hydrator and Twarc, which rebuild full tweets from IDs.<\/li><\/ul><\/div><div class=\"wp-block-group dh-section is-layout-flow wp-block-group-is-layout-flow\"><h2 class=\"wp-block-heading\">Up close: the Tweets Sampling Toolkit<\/h2><p class=\"wp-block-paragraph\">The <a href=\"https:\/\/github.com\/SocialMediaLab\/Tweets_Sampling_Toolkit\" target=\"_blank\" rel=\"noopener\">Tweets Sampling Toolkit<\/a> is a free Python 3 program the lab posted on GitHub. It exists because of a very practical problem. One month of this archive can be over 100 million tweet IDs, a file several gigabytes in size, and rehydrating all of it would take a very long time. The toolkit reads these files one line at a time, so it can handle them without needing a powerful computer.<\/p><p class=\"wp-block-paragraph\">It does a few simple but useful things:<\/p><ul class=\"wp-block-list\"><li>Pulls a <strong>random sample<\/strong>, either a set number of IDs or a percentage, with no duplicates.<\/li><li><strong>Splits<\/strong> one huge file into smaller ones.<\/li><li><strong>Compares datasets<\/strong> using union, difference, and intersection. For example, a researcher can check which tweets in the archive are missing from their own collection and fetch only those.<\/li><li><strong>Counts<\/strong> the IDs in a file.<\/li><\/ul><p class=\"wp-block-paragraph\"><strong>Uses outside this project:<\/strong> nothing about it is specific to COVID. It works on any list of tweet IDs, so someone studying an election or a protest movement could use it just as easily.<\/p><\/div><div class=\"wp-block-group dh-section is-layout-flow wp-block-group-is-layout-flow\"><h2 class=\"wp-block-heading\">Link to DH: Hydrator<\/h2><div class=\"wp-block-group dh-card dh-callout dh-warn is-layout-flow wp-block-group-is-layout-flow\"><h3 class=\"wp-block-heading\">A DH tool, and a broken one<\/h3><p class=\"wp-block-paragraph\">To turn IDs back into tweets, the archive points users to <a href=\"https:\/\/github.com\/DocNow\/hydrator\" target=\"_blank\" rel=\"noopener\">Hydrator<\/a>, a free app from Documenting the Now, a DH project focused on archiving social media ethically. It is a nice example of DH projects building on each other&#8217;s tools.<\/p><p class=\"wp-block-paragraph\">The problem is that Hydrator&#8217;s own GitHub page now says Twitter&#8217;s API changes make it &#8220;no longer a useful application.&#8221; The main route the project recommends for getting tweets back has stopped working.<\/p><\/div><\/div><div class=\"wp-block-group dh-section is-layout-flow wp-block-group-is-layout-flow\"><h2 class=\"wp-block-heading\">How good is the documentation?<\/h2><p class=\"wp-block-paragraph\">The toolkit itself is well documented. The GitHub page explains how to install it, what each function does, and what file formats it expects, and it even suggests the archive as test data. It is released under an open BSD-3-Clause license, so anyone can reuse or adapt it.<\/p><p class=\"wp-block-paragraph\">The weak spot is the first link in the chain. I couldn&#8217;t find the lab&#8217;s collection script published anywhere alongside the archive, so the step where tweets were actually pulled from Twitter can&#8217;t be checked or reused.<\/p><\/div><p class=\"dh-pager wp-block-paragraph\"><a href=\"\/DESocial\/data\/\">\u2190 Data<\/a><a href=\"\/DESocial\/methods\/\">Methods \u2192<\/a><\/p>","protected":false},"excerpt":{"rendered":"<p>DHM280 \u00b7 Project 2 Three kinds of tools keep this archive running: one to collect the tweets, one to make the giant files usable, and outside tools that turn IDs back into tweets. The tool chain Up close: the Tweets Sampling Toolkit The Tweets Sampling Toolkit is a free Python 3 program the lab posted [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":6,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-36","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/classprojects.digitalwcu.org\/DESocial\/wp-json\/wp\/v2\/pages\/36","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/classprojects.digitalwcu.org\/DESocial\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/classprojects.digitalwcu.org\/DESocial\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/classprojects.digitalwcu.org\/DESocial\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/classprojects.digitalwcu.org\/DESocial\/wp-json\/wp\/v2\/comments?post=36"}],"version-history":[{"count":2,"href":"https:\/\/classprojects.digitalwcu.org\/DESocial\/wp-json\/wp\/v2\/pages\/36\/revisions"}],"predecessor-version":[{"id":85,"href":"https:\/\/classprojects.digitalwcu.org\/DESocial\/wp-json\/wp\/v2\/pages\/36\/revisions\/85"}],"wp:attachment":[{"href":"https:\/\/classprojects.digitalwcu.org\/DESocial\/wp-json\/wp\/v2\/media?parent=36"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}