BrockNLP Lab Authors Accepted to NAACL 2025

Congratulations to BrockNLP Researchers Sangmitra Madhusudan (Undergraduate Researcher), Robert Morabito (MSc Student), Skye Reid (Undergraduate Researcher), Nikta Gohari Sadr (MSc Student), and Ali Emami (Director) for the acceptance of their upcoming publication, Fine-Tuned LLMs are “Time Capsules” for Tracking Societal Bias Through Books*, to NAACL 2025!

Their work fine-tunes Large Language Models on BookPAGE, a collection of 593 fictional books across seven decades, to analyze the evolution of societal bias across generations. Their findings reveal troubling sparks in bias with respect to gender and religious biases especially.

Their work has been accepted to the upcoming NAACL 2025 conference to be held in Albuquerque, New Mexico from April 29th to May 4th.

Abstract:

Large Language Models (LLMs) have shown impressive performance on various benchmarks, yet their ability to engage in deliberate reasoning remains questionable. We present NYT-Connections, a collection of 358 simple word classification puzzles derived from the New York Times Connections game. This benchmark is designed to penalize quick, intuitive “System 1” thinking, isolating fundamental reasoning skills. We evaluated six recent LLMs, a simple machine learning heuristic, and humans across three configurations: single-attempt, multiple attempts without hints, and multiple attempts with contextual hints. Our findings reveal a significant performance gap: even top-performing LLMs like GPT-4 fall short of human performance by nearly 30%. Notably, advanced prompting techniques such as Chain-of-Thought and Self-Consistency show diminishing returns as task difficulty increases. NYT-Connections uniquely combines linguistic isolation, resistance to intuitive shortcuts, and regular updates to mitigate data leakage, offering a novel tool for assessing LLM reasoning capabilities.