Monday, 17 April 2023 @ 14:00 CEST
On-site:
University of Vienna
Seminarraum 5
Kolingasse 14–16
1090 Vienna
Online:
https://univienna.zoom.us/j/65485292899?pwd=d1l0SWxLV3NPL3hYL0lOZXN6UWwzZz09
Meeting ID: 654 8529 2899
Passcode: DS
Evaluation and learning with structured test sets
Abstract:
Behavioural testing -- verifying system capabilities by validating human-designed input-output pairs -- is an alternative evaluation method of natural language processing systems proposed to address the shortcomings of the standard approach: computing metrics on held-out data. While behavioural tests capture human prior knowledge and insights, there has been little exploration on how to leverage them for model training and development. With this in mind, we explore behaviour-aware learning by examining several fine-tuning schemes using HateCheck, a suite of functional tests for hate speech detection systems. To address potential pitfalls of training on data originally intended for evaluation, we train and evaluate models on different configurations of HateCheck by holding out categories of test cases, which enables us to estimate performance on potentially overlooked system properties. The fine-tuning procedure led to improvements in the classification accuracy of held-out functionalities and identity groups, suggesting that models can potentially generalise to overlooked functionalities. However, performance on held-out functionality classes and i.i.d. hate speech detection data decreased, which indicates that generalisation occurs mostly across functionalities from the same class and that the procedure led to overfitting to the HateCheck data distribution.
Bio:
Benjamin Roth is an accomplished computational linguist who earned a Bachelor of Science degree in Computational Linguistics from Saarland University in 2004, including a study program at the University of Edinburgh in 2006. He obtained a Master of Science degree in Language Science and Technology from Saarland University in 2010 and completed research internships at the Xerox Research Center Europe and Google Research from 2009 to 2011. He was awarded the Google Europe Doctoral Fellowship in Natural Language Processing from 2011 to 2014 and completed his Doctorate in Computer Science (Dr.-Ing.) at Saarland University. He worked as a postdoctoral researcher at the University of Massachusetts, Amherst from 2014 to 2015. In 2016, Roth was appointed as an Adjunct Professor of Computational Linguistics at Ludwig-Maximilians-University Munich, a position he held until 2018. He then served as a Temporary Academic Council member at the same institution from 2018 to 2020. Since September 2020, he has been a Professor of Digital Text Sciences at the Institute for European and Comparative Linguistics and Literary Studies and in the Data Mining Research Group at the University of Vienna.
Benjamin Roth's main research interests are:
- Extraction of knowledge from text with deep learning and statistical methods
- Interdisciplinary applications of natural language processing
- Knowledge-supervised learning and weak supervision
- Learning from combinations of structured and unstructured data (e.g., graphs and text)
- Explainability of neural network-based models for natural language processing
