Text Mining for Historical Analysis
Course Materials Digital Humanities NLP Government Data

Text Mining for Historical Analysis

This methods book provides a practical introduction to the R programming language for text mining historical records. And more than just a code cookbook, it offers a critical perspective to handling our human history. It is the companion guide to The Dangerous Art of Text Mining by Jo Guldi.

About this Course

Computer-powered methods are changing the way that we access information about society. New methods help us to detect change over time, to identify influential figures, and to name turning points. What happens when we apply these tools to a million congressional debates or tweets?

I designed and collaborated on a series of Jupyter Notebooks for Jo Gulid’s class, “Text Mining as Historical Method,” to explore questions like these. The Notebooks for the class provide an introduction to the cutting-edge methodologies of textual analysis that are transforming the humanities. They are geared towards both computationalists as well as those with a background in the humanities (but not code), and are designed to teach the skills of analyzing texts as data for evidence of change over time.

For the full course materials, including the interactive Jupyter Notebooks used by the class, please see the digital history repository on GitHub.

Becuase this class encourages exploring discourses that have shaped our culture throughout time, students have access to a diverse range of data sets for their own research projects. These data sets include: Reddit posts from the Push Shift API from 2008 to 2015; the 19th-century Hansard Parliamentary debates of Great Britain; the Stanford Congressional Records; the Dallas, Texas and Houston, Texas City Council Minutes; literature from Project Gutenberg; metadata from the NovelTM Datasets for English-Language Fiction, 1700-2009; and corporate reports from EDGAR, the Electronic Data Gathering, Analysis, and Retrieval system.

The skills practiced in this course begin with basic word counts and visualization techniques and extend to the high-level application of machine learning modules to tell digital history, modules such as spaCy, sci-kit learn, and gensim for word embeddings. By the end of the course students are able to perform comparative analyses and observe how langauge chnages over time.

You may also like

See all Course Materials
PANGeA

PANGeA

PANGeA is a system that uses large language models (LLMs) to create narrative content for turn-based RPGs based on game designers' high-level criteria. It introduces a novel validation system for handling free-form text input during development and gameplay, employing "self-reflection" techniques, enabling small/local LLMs to perform comparably to foundational models. It enriches player-NPC interactions by generating personality-biased non-playable characters (NPCs). It improves AI accuracy through crowdsourcing mechanics. PANGeA houses a server with a custom memory system that provides context for LLM generation. The server's REST interface enables integration with any game engine.

GAME-KG

GAME-KG

Knowledge graphs (KGs) can augment large language models (LLMs) while also providing an explainable set of facts that can be inspected by a human. Explainability is valuable for fields that may otherwise avoid LLMs due to hallucinations, such as human trafficking analysis. Creating KGs poses challenges, however. KGs parsed from documents may include explicit connections (those directly stated in a document) but miss implicit connections (those evident to a human, but not directly stated). This research introduces GAME-KG, an approach to modifying explicit and implicit KG connections by crowdsourcing feedback through video games.

Agent-Driven, Game-Based Learning: Personalized CS Education for Diverse Students

Agent-Driven, Game-Based Learning: Personalized CS Education for Diverse Students

AI agents can personalize education by identifying students’ strengths, weaknesses, and personalities to generate content tailored to them. This work presents "personalized education agents" deployed in an educational version of Minecraft. Agents bridge concepts from lessons to "big picture" thinking by creating connections between STEM and interdisciplinary topics, such as the Language Arts. Agents translate student progression and learning outcomes to teachers for their assessment of student progress.