<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: black;">
Hi CaCL members,</div>
<div style="text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: black;">
<br>
</div>
<div class="elementToProof" style="text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: black;">
On 4/4 we will discuss "Pushdown Layers: Encoding Recursive Structure in Transformer Language Models " by Murty et al. (2023).</div>
<div style="text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: black;">
<br>
</div>
<div style="text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: black;">
<b>Paper:</b></div>
<div class="elementToProof" style="text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: black;">
<a href="https://aclanthology.org/2023.emnlp-main.195.pdf" id="LPlnk891981" class="OWAAutoLink" data-loopstyle="linkonly">https://aclanthology.org/2023.emnlp-main.195.pdf</a></div>
<div class="elementToProof" style="text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 0px; font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 15px; color: rgb(36, 36, 36);">
<br>
</div>
<div style="text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: black;">
<b>Abstract:</b></div>
<div class="elementToProof" style="text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 0px; font-family: -apple-system, BlinkMacSystemFont, "segoe ui", Roboto, "helvetica neue", Arial, "noto sans", sans-serif, "apple color emoji", "segoe ui emoji", "segoe ui symbol", "noto color emoji"; font-size: 16px; color: rgb(33, 37, 41);">
<span style="background-color: rgb(248, 249, 250);">Recursion is a prominent feature of human language, and fundamentally challenging for self-attention due to the lack of an explicit recursive-state tracking mechanism. Consequently, Transformer language models
poorly capture long-tail recursive structure and exhibit sample-inefficient syntactic generalization. This work introduces Pushdown Layers, a new self-attention layer that models recursive state via a stack tape that tracks estimated depths of every token
in an incremental parse of the observed prefix. Transformer LMs with Pushdown Layers are syntactic language models that autoregressively and synchronously update this stack tape as they predict new tokens, in turn using the stack tape to softly modulate attention
over tokens—for instance, learning to “skip” over closed constituents. When trained on a corpus of strings annotated with silver constituency parses, Transformers equipped with Pushdown Layers achieve dramatically better and 3-5x more sample-efficient syntactic
generalization, while maintaining similar perplexities. Pushdown Layers are a drop-in replacement for standard self-attention. We illustrate this by finetuning GPT2-medium with Pushdown Layers on an automatically parsed WikiText-103, leading to improvements
on several GLUE text classification tasks.</span></div>
<div style="text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: black;">
<br>
</div>
<div style="background-color: rgb(255, 255, 255); margin: 0px;">
<div style="margin: 0px; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: black;">
<span style="background-color: white;">----</span></div>
<div style="background-color: white; margin: 0px;"></div>
<div style="text-align: left; background-color: white; margin: 0px; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: black;">
<b>Christian Clark</b></div>
<div style="background-color: white; margin: 0px;">
<div style="text-align: left; margin: 0px; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: black;">
Ph.D. Student</div>
<div style="text-align: left; margin: 0px; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: black;">
Department of Linguistics</div>
<div style="text-align: left; margin: 0px; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: black;">
The Ohio State University</div>
</div>
</div>
<div class="elementToProof" style="text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="background-color: rgb(255, 255, 255); margin: 0px;"></div>
<div style="background-color: rgb(255, 255, 255); margin: 0px;"></div>
</body>
</html>