The scholarship is titled “Looking Forward,” by Ed Summers, published on July 19, 2023. This explains the goal of trying to develop tools and cultivate a community of practice for collecting social media. It states how over 500 citations have been made to twarc as a tool for gathering and hydrating Twitter datasets, also influencing other software products such as the Social Feed Manager. As part of Twitter Research outreach, a number of guidelines have been independently created by Software Carpentry, data scientists, and even Twitter itself. Twitter’s Academic Research Product Track provided enhanced access that allowed users to search the entire archive of tweets that were made public starting in 2006. Although scraping procedures are common and access to corporate social media platform API’s has been approaching its end for a little while now, they are still fragile due to the interface changes, a lack of documentation, and significant issues with repeatability and provenance. Although social media APIs are usually implemented on top of the web, the creation of specific infrastructure has been necessary for key management, authentication, URL routes, and data representations, resulting in fragmented approaches and decreased collaboration across projects. The best way to preserve and study social media platforms is to embrace the use of general-purpose web archiving methods and technologies. The majority of systems still use some type of web interface, so the research community must support organizations that offer services. Further development and socialization are also required for specialized methodological techniques to use WARC data in research and tools for evaluating web archive data. A more diversified community of individuals who can use web and social media archiving tools to generate data collections are needed as well. This will allow the number of collections available for research if we attract a wider range of users, opening up new possibilities in terms of what constitutes important online data and how to acquire it, as well as how to establish a more inclusive historical record. While some aspects of traditional archiving practices are still relevant today, Other aspects offer new directions for working with social media data in the post-open API era, including collaborating with individuals to develop collections, building publicly accessible digital repositories of archival materials, respecting one’s rights to consent to the collection of their information, and working with individuals to donate their collections data.
This project has an open data repository and clear documentation that the science can be used well.
After the acquisition of Twitter by Elon Musk, the changes to the Twitter API have resulted in the complete dismantling of DocNow’s tools. While it still technically functions (if you still have API keys), twarc is basically useless unless you want to pay $100/month to retrieve 10,000 tweets ($0.01 per tweet), or $5000/month for up to 1 million tweets. These prices and the new policies around deletion of Twitter data are designed for commercial use, and to terminate academic research and study of the platform. The DocNow application, which was built around Academic Research Product Track quotas and historical search no longer functions with the new API regime. Twitter has revoked the keys for Hydrator and the instance we have been running at community.docnow.io.
Ed Summers from Looking Forwards Jul 19, 2023
