Chinese Grammatical Error Diagnosis

Introduction

Unlike the English learning setting for which many learning technologies have been developed, those to support Asian language learners are relatively rare. In response, the NLPTEA (Natural Language Processing Techniques for Educational Applications) workshops were organized to provide a forum where international participants can share knowledge on the computer-assisted techniques for Asian language learning and teaching. In NLPTEA workshops, a series of shared tasks for Chinese grammatical error diagnosis was hosted (Yu et al., 2014; Lee et al., 2015b; Lee et al., 2016). The goal of these tasks is to develop NLP techniques to automatically diagnose grammatical errors in Chinese sentences written by Chinese as a foreign language learners. The grammatical errors are broadly categorized into 4 error types: word ordering, redundant, missing, and incorrect selection of linguistic components (also called PADS error types, denoting errors of Permutation, Addition, Deletion, and Selection, correspondingly). In the first edition in NLPTEA 2014 workshop, a sentence with/without one of grammatical errors is given, and the developed system should indicate whether it contains a grammatical error and further identify the error type if any error occurs. In the second edition in NLPTEA 2015 workshop, in addition to detecting whether a given sentence contains a grammatical error and identify the error type, the system should also indicate the range of occurring errors. In the third edition in NLPTEA 2016 workshop, the task is basically the same, except that a sentence may contain more than one errors and the HSK learner corpus is also included for the task.


Summary

Shared Task NLPTEA 2014 Task
(Yu et al., 2014)
NLPTEA 2015 Task
(Lee et al., 2015)
NLPTEA 2016 Task
(Lee et al., 2016)
Examples

Example 1:
Input: (sid=C1-1876-2) 對社會國家不同的影響
Output: C1-1876-2, Missing

Example 2:
Input: (sid=A2-0775-2) 我起床很早
Output: A2-0775-2, Disorder

Example 1:
Input: (sid=B2-0080) 他是我的以前的室友
Output: B2-0080, 4, 4, Redundant

Example 2:
Input: (sid=B1-1193) 吳先生是修理腳踏車的拿手
Output: B1-1193, 11, 12, Selection

Example 1:
Input: (sid=A2-0011-1) 我聽到你找到工作。恭喜恭喜!
Output: A2-0011-1, 2, 3, S
A2-0011-1, 9, 9, M

Example 2:
Input: (sid=00038800464) 我真不明白。她们可能是追求一些前代的浪漫。
Output: 00038800464, correct

Data Source TOCFL Learner Corpus TOCFL Learner Corpus TOCFL Learner Corpus
&
HSK Dynamic Composition Corpus
Training Set 1,506 writings
(5,607 errors)
2,205 sentences
(2,205 errors)

TOCFL Track:
10,693 sentences (24,492 errors)

HSK Track:
10,071 sentences (24,797 errors)

Test Set

This set consists of 1,750 testing sentences. Half of these instances contained no grammatical errors. Another half included one grammatical error.

This set consists of 1,000 testing sentences. Half of these sentences contained no grammatical errors, while the other half included one error.

TOCFL Track:
3,528 sentences
(1703 correct & 1,825 errors)

HSK Track:
3,011 sentences
(1,539 correct & 1,472 errors)

Technical Challenges

Error Detection
Error Identification

Error Detection (binary class categorization problem)
Error Identification (multi-class categorization problem)
Error Position (sequence labeling problem)

Evaluation Metrics

False Positive Rate
Detection-/Identification-level Acc./Prec./Recall/F1

False Positive Rate
Detection-level Accuracy/Precision /Recall/ F1
Identification-level Accuracy/Precision /Recall/ F1
Position-level Accuracy/Precision /Recall/ F1

Registered Teams

13 teams (4 from Taiwan, 3 from China, and the remaining 6 from Japan, USA, UK, Germany, New Zealand, and Russia)

13 teams (4 from Taiwan, 3 from China, 2 from USA, 2 from UK, 1 from Japan, and 1 from Poland)

15 teams (8 from China, 4 from Taiwan, 1 from Dublin, 1 from Germany, and 1 private firm)


Download

Click here to download all data sets and evaluation tools.


References


Contact

Lung-Hao Lee

Assistant Professor

Department of Electrical Engineering

National Central University

lhlee@ee.ncu.edu.tw