The Viral Texts project analyzed 2.7 million pages, derived from 500 newspapers from the 19th century. Around 650 articles were reprinted 50 times over. With the right to free reprinting and the similar language by the united states congress, how could we provide a method to locating the viral articles with textual evidence. One of the challenges harped on throughout the project was text ruse and why did different news outlets choose to reuse certain articles. As we know things can go viral very fast in todays day in age, during the 19th century some texts circulated for 50 years.
- Text reuse deduction: Excluding republication made from the same source multiple times. This text reuse will be held over a multiple year time span unlike other projects in the past done on a singular blog. There could be near duplicates, unfortunately due to boilerplate elements documents from the same website.
- N-gram indexing: The first step is to give a particular n-gram to each article. During the first pass the n-grams were set into a fixed number of bins, then during the second pass, singleton n-grams were discarded. Then being filtered by using (n-1) and this method has been deemed efficient in detecting shared n grams.
- Extracting and Ranking candidate pairs: Once the crew receives an inverted index of each n-gram, they start ranking them by different categories containing reprinted texts. “Documents containing the same n-gram are paired. N-grams that generate more then a certain number of pairs are eradicated, this is because these are just filled with similar phrases and not a repeating text.” Repeating n-grams are given a score in which they are given a grade based off the distinctiveness which will be used to balance recall and precision.
- Computing local alignments: The main goal is to increase precision while maintaining a high recall. The method is called Smith-Waterman dynamic programming algorithm, which identifies regions of high overlap. This will align n-grams between documents.
- Intrinsic Evaluation: This evolution takes data from pools derives from text reuse methods to create a pseudo-relevant set of document pairs. They are pulled form previous ranking pairs and are determined by the length of the local alignment for each pair. Skip bigram features help speed the process up while maintaining precision. However it still is not full-proof.
- Network Connections: This step measures the influence of officials and newspaper outlets controlling what articles go reprinted. This method found that geography did play a major role on how frequently a reprint occurred. For example New York prints were the most common to be reprinted, which would make sense since it is so densely populated and at this time was a developing international phenomenon.
The project meets best practices for Digital Humanities because they used clean practices and a public data base while achieving their goal in developing a mathematical equation to hunt down viral texts.