[CaCL] Reading for 9/14

[Clark, Christian]

Hi CaCL members,

Here's the reading for our next meeting, on 9/14.

Contextual Distortion Reveals Constituency: Masked Language Models are Implicit Parsers (Li and Lu 2023)

Paper: https://aclanthology.org/2023.acl-long.285.pdf

Zoom: https://osu.zoom.us/j/93495474520?pwd=NWFRZzF6QnZrQWdheThxbWJyNjJ4dz09<https://urldefense.com/v3/__https://osu.zoom.us/j/93495474520?pwd=NWFRZzF6QnZrQWdheThxbWJyNjJ4dz09__;!!KGKeukY!2TWrTWTrANlP6_byTlK5ZLL4FcVlqFnyd9wv4X2M3zr-UJY-3iNs13eazubwOhgODDjOzblyZP_95P1NaUWjzw$>

Abstract:
Recent advancements in pre-trained language models (PLMs) have demonstrated that these models possess some degree of syntactic awareness. To leverage this knowledge, we propose a novel chart-based method for extracting parse trees from masked language models (LMs) without the need to train separate parsers. Our method computes a score for each span based on the distortion of contextual representations resulting from linguistic perturbations. We design a set of perturbations motivated by the linguistic concept of constituency tests, and use these to score each span by aggregating the distortion scores. To produce a parse tree, we use chart parsing to find the tree with the minimum score. Our method consistently outperforms previous state-of-the-art methods on English with masked LMs, and also demonstrates superior performance in a multilingual setting, outperforming the state-of-the-art in 6 out of 8 languages. Notably, although our method does not involve parameter updates or extensive hyperparameter search, its performance can even surpass some unsupervised parsing methods that require fine-tuning. Our analysis highlights that the distortion of contextual representation resulting from syntactic perturbation can serve as an effective indicator of constituency across languages.

----
Christian Clark
Ph.D. Student
Department of Linguistics
The Ohio State University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/cacl/attachments/20230908/5f1e6754/attachment.html>