This is a past event
Abstract:
Retrieval system effectiveness can be measured in two quite different ways: by monitoring the behaviour of users and gathering data about the ease and accuracy with which they accomplish certain specified information-seeking tasks; or by using numeric effectiveness metrics to score system runs in reference to a set of relevance judgements. In the second approach, the effectiveness metric is chosen in the belief that it predicts ease or accuracy. This work explores that link, by analysing the assumptions and implications of a number of effectiveness metrics, and exploring how these relate to observable user behaviours. Data recorded as part of a user study included user self-assessment of search task difficulty; gaze position; and click activity. Our results show that user behaviour is influenced by a blend of many factors, including the extent to which relevant documents are encountered, the stage of the search process, and task difficulty. These insights can be used to guide development of batch effectiveness metrics.
Bio:
PAUL THOMAS is a research scientist with the CSIRO in Canberra, Australia, and an adjunct at the Australian National University Research School of Computer Science. His research interests are around information retrieval -- particularly distributed retrieval and users' behaviour -- and around evaluating search engines and other online information tools.
- Speaker
- Paul Thomas
- Hosted by
- Nava Tintarev
- Venue
- Meston Lecture Theatre 3 (also known as Meston 147, which is on the first floor of the Meston building)