This is a past event
In everyday life, a majority of the inferences we make are defeasible. That is, they are highly plausible but are based on incomplete information and can therefore be impacted or defeated by new information. Defeasible inferences often arise from generalisations, which are readily capture in language by generics (i.e., generalisations without quantifiers). Generics are notable because they express general rules (e.g., birds can fly) while allowing for exceptions (e.g., emus can’t fly). When inferences are drawn from generics, their exceptions provide a natural source of weakening or defeating evidence; we exploit this to uncover the challenges that generics and defeasible reasoning pose for LLMs. In this talk, we will probe defeasible reasoning about property inheritance based on generics. We will first present a framework for generics that we use to generate a dataset of instantiations and exceptions for generics while maintaining specific semantic relations. Using this data, we probe how LLMs respond to generics and exceptions, in comparison to human studies from psychology. We also construct a dataset of defeasible inheritance reasoning problems and show that a wide range of LLMs struggle at this task, in a variety of ways.
- Speaker
- Emily Allaway
- Venue
- Meston G05 and Microsoft Teams