Uncovering the Past: How AI is Rewriting History from Old Newspapers

Author: Denis Avetisyan

Researchers are leveraging advanced artificial intelligence to analyze vast archives of historical newspapers, revealing hidden patterns and offering fresh perspectives on past events.

The distribution of documents across time reveals how the volume of available content ebbs and flows, fundamentally shaping the observed evolution of topics within the analysis-a pattern indicative of how data density itself becomes a critical, often overlooked, factor in understanding temporal trends.

This review demonstrates the superior performance of neural topic modeling, specifically BERTopic, for extracting thematic trends and insights from large-scale newspaper collections.

Extracting meaningful themes from vast historical newspaper collections remains challenging due to evolving language, data noise, and sheer volume. This study, ‘Automating Historical Insight Extraction from Large-Scale Newspaper Archives via Neural Topic Modeling’, addresses these limitations by applying BERTopic, a neural topic modeling technique, to a corpus of articles concerning nuclear power and safety from 1955-2018. Our results demonstrate that BERTopic surpasses traditional methods in capturing dynamic shifts in public discourse and revealing nuanced connections between related themes. How can these advancements in automated topic modeling further illuminate complex historical trends and inform contemporary understandings of public opinion?

The Echo of Absence: Uncovering Latent Themes

Conventional methods of text analysis, such as manual coding or simple keyword searches, frequently encounter limitations when applied to extensive collections of documents. These approaches often fail to capture the subtle, interwoven themes that characterize complex texts, instead relying on pre-defined categories that may not accurately reflect the content’s inherent structure. The sheer volume of data in large corpora can overwhelm analysts, leading to superficial interpretations and a loss of valuable insights. Consequently, nuanced thematic structures – the underlying patterns of meaning – remain hidden, hindering a comprehensive understanding of the information contained within the documents. This inability to effectively process and interpret large-scale textual data underscores the need for more sophisticated analytical techniques capable of uncovering these latent themes.

Topic modeling represents a powerful shift in how researchers and analysts approach large volumes of text. Rather than relying on manual coding or keyword searches, it employs statistical algorithms to identify the underlying thematic structure within a collection of documents. These methods don’t search for pre-defined topics; instead, they discover abstract topics based on patterns of word co-occurrence. The result is a structured overview of content, where each document is represented as a mixture of these learned topics, and each topic is characterized by a distribution of words. This allows for automated content summarization, efficient information retrieval, and a deeper understanding of the prevailing themes within the data, moving beyond simple keyword analysis to reveal nuanced relationships and hidden insights.

Traditional topic modeling techniques, such as Latent Dirichlet Allocation (LDA) and Non-negative Matrix Factorization (NMF), operate under simplifying assumptions that can limit their ability to fully capture the intricacies of textual data. These methods often represent documents as mixtures of topics, and topics as distributions over words, but they struggle with nuanced semantic relationships, polysemy – where a single word has multiple meanings – and the complex co-occurrence of concepts. Specifically, LDA assumes a ‘bag-of-words’ approach, disregarding word order and context, while NMF, though capable of capturing some non-linear relationships, can be sensitive to initialization and may produce less coherent topics when dealing with high-dimensional data. Consequently, these earlier methods may identify broad themes, but often fall short in discerning subtle connections and the hierarchical structure often present within large text corpora, necessitating more advanced approaches for deeper thematic analysis.

Neural Topic Models represent a significant advancement in uncovering thematic structure within large text collections. Unlike traditional methods which often rely on distributional assumptions and limited feature spaces, these models harness the capacity of neural networks to learn complex, non-linear relationships between words and documents. By embedding words into continuous vector spaces, they capture semantic similarities that go beyond simple co-occurrence, allowing for the identification of more nuanced and coherent topics. This approach moves beyond merely identifying frequently occurring terms; instead, it aims to understand the underlying conceptual themes that connect documents, leading to topic representations that are both more interpretable and more reflective of the content’s true meaning. The result is a powerful tool for tasks ranging from document summarization and information retrieval to social media analysis and trend detection, offering insights previously inaccessible through conventional topic modeling techniques.

BERTopic: A System Adapting to the Noise

BERTopic distinguishes itself as a topic modeling technique by integrating transformer-based embeddings with clustering algorithms. Traditional topic models often rely on bag-of-words approaches or latent Dirichlet allocation, which can struggle with semantic nuance. BERTopic leverages Sentence Transformers to generate dense vector representations of documents, capturing contextual relationships between words. These embeddings are then reduced in dimensionality using techniques like UMAP to facilitate efficient clustering. The HDBSCAN algorithm identifies robust clusters within the embedded space, each representing a coherent topic. This combination allows BERTopic to automatically discover and represent topics in a manner that is both computationally efficient and semantically meaningful, offering improvements in topic coherence and interpretability compared to earlier methods.

Sentence Transformers are utilized to generate dense vector representations, also known as embeddings, of input text. Unlike traditional methods like bag-of-words or TF-IDF which focus on word frequency, Sentence Transformers are pre-trained on large datasets to understand semantic relationships between sentences. This allows the model to map semantically similar texts to vectors that are close to each other in vector space, even if they don’t share many of the same words. The resulting embeddings are typically several hundred dimensions, capturing nuanced meaning and contextual information, and serving as the foundation for downstream topic modeling tasks by providing a numerical representation of text content.

UMAP (Uniform Manifold Approximation and Projection) addresses the computational challenges posed by high-dimensional sentence embeddings generated by models like Sentence Transformers. These embeddings, while semantically rich, can be inefficient for clustering algorithms due to the “curse of dimensionality”. UMAP performs non-linear dimensionality reduction, projecting the embeddings into a lower-dimensional space – typically 2 to 5 dimensions – while preserving the topological structure of the original data. This reduced representation allows HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) to efficiently identify dense clusters. HDBSCAN is particularly effective because it doesn’t require a pre-defined number of clusters and is robust to varying densities within the data, effectively grouping similar documents into topics and identifying outliers as noise.

C-TF-IDF (Class-based TF-IDF) is employed to identify keywords that best represent each discovered topic within BERTopic. Unlike traditional TF-IDF, which calculates term frequency across the entire corpus, C-TF-IDF calculates term frequency within each identified topic cluster separately. This ensures that keywords specific to a particular topic are prioritized, even if those terms are less frequent in the overall document collection. The term frequency is then weighted by the inverse document frequency, calculated based on the number of topics, providing a more refined weighting scheme. The resulting keywords are ranked by their C-TF-IDF scores, allowing for the selection of the most representative terms for each topic, thereby improving interpretability and facilitating a clearer understanding of the identified themes.

Historical Echoes: BERTopic and the Impresso Dataset

Historical text mining, when integrated with the BERTopic modeling technique, offers a comprehensive approach to analyzing large collections of historical documents such as the Impresso Dataset. This framework leverages natural language processing to extract meaningful themes and patterns from digitized texts, overcoming limitations inherent in manual analysis. BERTopic utilizes a class-based TF-IDF procedure to create dense clusters of semantically similar words, followed by a technique called UMAP for dimensionality reduction, and finally, HDBSCAN for identifying clusters of topics. The resulting topics are then represented by keywords, allowing for quantifiable analysis of thematic prevalence and evolution within the historical corpus. This methodology provides researchers with a systematic and reproducible method for investigating historical trends and public discourse.

Analysis of the Impresso Dataset using BERTopic enables the identification and tracking of public sentiment changes concerning nuclear technologies over time. The Impresso Dataset, comprising a collection of historical texts related to nuclear issues, is processed by BERTopic to create a dynamic representation of topic prevalence. This allows researchers to quantify shifts in public opinion, pinpoint emerging concerns – such as safety, waste disposal, or proliferation – and trace their evolution across different periods. The method generates topic clusters representing distinct sentiments, which are then analyzed to determine temporal trends and correlations with specific historical events or policy changes, providing a granular view of public perception beyond simple positive or negative classifications.

Application of BERTopic to the Impresso Dataset facilitates the detection of temporal changes in public discourse regarding nuclear technologies. Analysis of document clusters generated over time reveals evolving concerns, such as shifts from initial optimism regarding atomic energy to increased apprehension following events like the Chernobyl disaster and Fukushima Daiichi nuclear accident. This method moves beyond simple keyword frequency analysis, identifying subtle thematic variations within the corpus and allowing researchers to map the trajectory of public sentiment, including the emergence and decline of specific anxieties and expectations related to nuclear power and its societal impact. The granularity of BERTopic enables the differentiation of closely related but distinct public concerns, offering a more precise historical record than traditional methods.

Evaluations conducted using the Impresso Dataset demonstrate that BERTopic consistently surpasses the performance of Latent Dirichlet Allocation (LDA) and Non-negative Matrix Factorization (NMF) in identifying subtle changes in thematic focus within historical texts. Specifically, BERTopic achieves higher topic coherence scores, as measured by UMass and CV metrics, and exhibits demonstrably improved topic relevance, indicated through human evaluation of topically representative documents. These quantitative and qualitative results suggest that BERTopic’s contextualized embedding approach, utilizing transformers, effectively captures the semantic relationships crucial for discerning nuanced shifts in historical narratives, a capability often lacking in the more statistically-driven approaches of LDA and NMF.

Visualizations of topic evolution reveal distinct patterns in public discourse surrounding the Harrisburg, Chernobyl, Fukushima, and Hiroshima disasters.

The Shape of Understanding: Topic Quality and Coverage

Topic coherence serves as a vital benchmark for evaluating the quality of thematic modeling, quantifying the semantic relatedness of terms within a given topic. A high coherence score indicates that the words frequently appearing together within a topic are genuinely associated, fostering interpretability for those examining the results. Essentially, it measures whether a topic represents a conceptually meaningful cluster, rather than a random assortment of words; a topic about “artificial intelligence” should prominently feature terms like ‘machine learning’, ‘neural networks’, and ‘algorithms’, rather than unrelated concepts. Consequently, maximizing topic coherence is paramount in ensuring that identified themes are not only statistically significant but also readily understandable and insightful for researchers and analysts.

While identifying coherent themes within a body of text is essential, a truly comprehensive topic modeling approach also requires assessing topic diversity. This metric quantifies how distinct each identified topic is from the others, preventing the emergence of redundant or heavily overlapping themes. Without considering diversity, a model might identify multiple topics that essentially describe the same underlying concept, offering limited insight into the full range of subjects present in the data. A high degree of topic diversity, therefore, indicates that the model has successfully captured a broader spectrum of ideas, providing a more nuanced and informative representation of the content.

Evaluations reveal that BERTopic consistently generates topics with significantly higher coherence scores when contrasted with traditional methods like Latent Dirichlet Allocation (LDA) and Non-negative Matrix Factorization (NMF). This heightened coherence indicates that the themes identified by BERTopic are more semantically consistent and internally structured, resulting in topics that are demonstrably easier for humans to understand and interpret. The model’s ability to distill meaningful representations from text data allows it to avoid fragmented or ambiguous groupings, offering a clearer and more insightful overview of the underlying content. Consequently, BERTopic provides a robust framework for uncovering the core narratives within a corpus, fostering a more nuanced comprehension of complex information.

Evaluations reveal that BERTopic consistently identifies a more expansive spectrum of themes within a given dataset when contrasted with traditional topic modeling approaches like Latent Dirichlet Allocation (LDA) and Non-negative Matrix Factorization (NMF). This heightened topic diversity isn’t simply an increase in the number of topics, but rather a demonstrable ability to pinpoint more distinct and previously obscured subject areas. The model achieves this by effectively partitioning the semantic space, minimizing overlap between identified topics and ensuring a more comprehensive representation of the underlying data. Consequently, BERTopic provides a richer and more nuanced understanding of complex datasets, offering insights that might be lost when employing classical methods prone to generating redundant or highly similar thematic clusters.

Beyond the Static: Tracking Evolving Themes

Traditional topic modeling techniques, while capable of identifying prevalent themes within a body of text, often present a snapshot in time, failing to account for the dynamic nature of language and thought. These static analyses treat a collection of documents as a singular entity, overlooking how topics emerge, evolve, and potentially fade away over time. Consequently, crucial nuances in evolving narratives – shifts in public concern, changes in perspective, or the introduction of novel concepts – remain hidden. This limitation hinders a comprehensive understanding of complex datasets, particularly those spanning extended periods, as the relationships between ideas and their temporal context are not fully explored. A static view essentially offers a fragmented picture, missing the crucial story of how a conversation unfolded rather than simply what was discussed.

Traditional topic modeling often delivers a snapshot of themes within a text corpus, but fails to capture their fluidity over time. Dynamic topic modeling overcomes this by charting the evolution of topics – how their prevalence and underlying meanings shift across a sequence of documents. This allows researchers to move beyond simply identifying what subjects are discussed, to understanding how those discussions change, revealing emerging concerns, fading interests, and the subtle alterations in perspective that characterize evolving narratives. By tracing these thematic trajectories, dynamic models offer a more nuanced and complete picture of complex phenomena, particularly within historical datasets where long-term trends and the shifting landscape of public discourse are crucial to understanding the past.

The power of dynamic topic modeling truly shines when applied to historical datasets, offering a unique lens through which to examine the evolution of public thought. By tracing the rise and fall of different themes over decades or even centuries, researchers can uncover long-term trends and patterns in societal concerns that would remain hidden with static analyses. This approach moves beyond simply identifying what people discussed in a given period, to reveal how those discussions changed, reflecting shifts in cultural values, political priorities, and technological advancements. For example, analyses of digitized newspapers or archived correspondence can demonstrate the gradual emergence of environmental consciousness, the changing rhetoric surrounding social movements, or the impact of major events on collective anxieties, ultimately providing a richer and more nuanced understanding of the past.

Recent advancements in topic modeling have yielded not only improved analytical accuracy but also significant gains in computational efficiency, as demonstrated by the BERTopic framework. Traditional methods often struggle with large datasets, requiring substantial processing time to identify and track thematic trends; however, BERTopic utilizes a streamlined approach that dramatically reduces these demands. This faster processing allows researchers to analyze extensive corpora – such as decades of news articles or millions of social media posts – in a fraction of the time previously required, enabling more dynamic and responsive investigations into evolving narratives. The increased speed doesn’t come at the cost of precision; BERTopic maintains strong performance while offering a practical advantage for time-sensitive research and real-time monitoring of public discourse.

The pursuit of automated insight from historical archives, as demonstrated by this work with BERTopic, inevitably embraces a degree of inherent instability. The system doesn’t construct understanding; it cultivates it from the chaos of textual data. The algorithm identifies thematic shifts, but those shifts themselves are rarely clean or definitive – a guarantee of perfect categorization is simply a contract with probability. The findings reveal that BERTopic excels at capturing these dynamic changes, yet acknowledges that stability is merely an illusion that caches well. As Tim Bern-Lee observed, “Data is just stuff. It’s the relationships between the data that are meaningful.” This research doesn’t impose meaning; it reveals the pre-existing relationships within the archives, a process of discovery, not construction.

What Lies Beneath?

The demonstrated efficacy of neural topic modeling against brittle, hand-engineered features is predictable. Every precision achieved is merely a temporary reprieve from the inevitable drift of language, the semantic erosion of even the most carefully constructed lexicon. The true challenge isn’t identifying themes within these archives, but modeling the archive itself as a decaying ecosystem – a substrate for future misinterpretations. Expect the next generation of tools to wrestle not with “what happened,” but “how confidently can we claim to know?”

Current approaches treat topic models as stable representations. This is a fiction. Themes aren’t fixed stars; they are currents, eddies, and ultimately, the silt that settles at the bottom of a changing sea. The coming work will necessitate dynamic models-not just tracking topic evolution, but anticipating points of fracture, where the narrative breaks down into unintelligible fragments. These systems will need to quantify their own uncertainty, to acknowledge that every insight is provisional.

The pursuit of ‘historical insight’ implies a destination. A more honest endeavor accepts that the archive isn’t a map to the past, but a mirror reflecting the biases of the present. The next iteration will not be about extracting knowledge, but about meticulously cataloging the methods-and inherent limitations-of its extraction. The true product will be a rigorous accounting of what remains unknowable.

Original article: https://arxiv.org/pdf/2512.11635.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/