Beyond Accuracy: Behavioral Testing of NLP models with CheckList
May 8, 2020 · Inspired by principles of behavioral testing in software engineering, we introduce CheckList, a task-agnostic methodology for testing NLP models.
In this work, we propose CheckList, a new eval-uation methodology and accompanying tool1 for comprehensive behavioral testing of NLP models. CheckList guides users in what to test, by provid-ing a list of linguistic capabilities, which are appli-cable to most tasks.
GitHub - marcotcr/checklist: Beyond Accuracy: Behavioral Testing of NLP ...
This repository contains code for testing NLP Models as described in the following paper: Beyond Accuracy: Behavioral Testing of NLP models with CheckList Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, Sameer Singh Association for Computational Linguistics (ACL), 2020
comprehensive behavioral testing of NLP models. CheckListguides users in what to test, by provid-ing a list of linguistic capabilities, which are appli-cable to most tasks. To break down potential ca-pability failures into specific behaviors, CheckList introduces different test types, such as prediction invariance in the presence of certain ...
Beyond Accuracy: Behavioral Testing of NLP Models with Checklist ...
We illustrate the utility of CheckList with tests for three tasks, identifying critical failures in both commercial and state-of-art models. In a user study, a team responsible for a commercial sentiment analysis model found new and actionable bugs in an extensively tested model.
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
Feb 5, 2021 · A CheckList is a new evaluation methodology for comprehensive functional testing of NLP models where we have a matrix of Linguistic Capabilities such as Named Entity Switching, Negations, Robustness to Typos, etc and Test Types like Minimum Functionality Test(MFT), Directional Expectation(DIR) and Invariance Test(INV) on X and Y axis respectively.
Semantically equivalent adversarial rules for debugging nlp models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 856-865).
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
Jan 1, 2020 · Behavioral testing, as proposed by Ribeiro et al. (2020), uses input-output pairs tailored to evaluate a model capability to correctly handle certain language properties. The...
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
5 days ago · We illustrate the utility of CheckList with tests for three tasks, identifying critical failures in both commercial and state-of-art models. In a user study, a team responsible for a commercial sentiment analysis model found new and actionable bugs in …
In this work, we propose CheckList, a new evaluation methodology and accompanying tool for comprehensive be-havioral testing of NLP models. CheckList guides users in what to test, by providing a list of linguistic capabilities, which are applicable to most tasks.
- Some results have been removed