[CaCL] 1/24: Linzen et al. (2016)

[Marten van Schijndel]

For CaCL on 1/24 I'll be facilitating discussion of the following paper:

Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg

The success of long short-term memory (LSTM) neural networks in language
processing is typically attributed to their ability to capture
long-distance statistical regularities. Linguistic regularities are
often sensitive to syntactic structure; can such dependencies be
captured by LSTMs, which do not have explicit structural
representations? We begin addressing this question using number
agreement in English subject-verb dependencies. We probe the
architecture’s grammatical competence both using training objectives
with an explicit grammatical target (number prediction, grammaticality
judgments) and using language models. In the strongly supervised
settings, the LSTM achieved very high overall accuracy (less than 1%
errors), but errors increased when sequential and structural information
conflicted. The frequency of such errors rose sharply in the
language-modeling setting. We conclude that LSTMs can capture a
non-trivial amount of grammatical structure given targeted supervision,
but stronger architectures may be required to further reduce errors;
furthermore, the language modeling signal is insufficient for capturing
syntax-sensitive dependencies, and should be supplemented with more
direct supervision if such dependencies need to be captured.

http://tallinzen.net/media/papers/linzen_dupoux_goldberg_2016_tacl.pdf

(Fun fact: This is the paper from TaCL, not the one we read last time.
Just by way of explaining my mistake last CaCL.)


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: OpenPGP digital signature
URL: <http://lists.osu.edu/pipermail/cacl/attachments/20170121/9fd84d7c/attachment.sig>