<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=big5">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:新細明體;
panose-1:2 2 5 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@新細明體";
panose-1:2 1 6 1 0 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;
mso-ligatures:none;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="ZH-TW" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt">Hi all,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt">This is a friendly reminder that the paper we will discuss this week (3/28) is “Contrastive Decoding: Open-ended Text Generation as Optimization” (Li et al., 2023). Please find the abstract and
the link to the paper in the following.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt">Best,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt">Yi-Chien<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt"><o:p> </o:p></span></p>
<div id="mail-editor-reference-message-container">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;font-family:"新細明體",serif;color:black">寄件者</span></b><b><span lang="EN-US" style="font-size:12.0pt;color:black">:
</span></b><span lang="EN-US" style="font-size:12.0pt;color:black">CaCL <cacl-bounces+lin.4434=osu.edu@lists.osu.edu>
</span><span style="font-size:12.0pt;font-family:"新細明體",serif;color:black">代表</span><span lang="EN-US" style="font-size:12.0pt;color:black"> Lin, Yi Chien via CaCL <cacl@lists.osu.edu><br>
</span><b><span style="font-size:12.0pt;font-family:"新細明體",serif;color:black">日期</span></b><b><span lang="EN-US" style="font-size:12.0pt;color:black">:
</span></b><span style="font-size:12.0pt;font-family:"新細明體",serif;color:black">星期四</span><span lang="EN-US" style="font-size:12.0pt;color:black">, 2024</span><span style="font-size:12.0pt;font-family:"新細明體",serif;color:black">年</span><span lang="EN-US" style="font-size:12.0pt;color:black">3</span><span style="font-size:12.0pt;font-family:"新細明體",serif;color:black">月</span><span lang="EN-US" style="font-size:12.0pt;color:black">7</span><span style="font-size:12.0pt;font-family:"新細明體",serif;color:black">日</span><span style="font-size:12.0pt;color:black">
</span><span style="font-size:12.0pt;font-family:"新細明體",serif;color:black">下午</span><span lang="EN-US" style="font-size:12.0pt;color:black">2:33<br>
</span><b><span style="font-size:12.0pt;font-family:"新細明體",serif;color:black">收件者</span></b><b><span lang="EN-US" style="font-size:12.0pt;color:black">:
</span></b><span lang="EN-US" style="font-size:12.0pt;color:black">cacl@lists.osu.edu <cacl@lists.osu.edu><br>
</span><b><span style="font-size:12.0pt;font-family:"新細明體",serif;color:black">主旨</span></b><b><span lang="EN-US" style="font-size:12.0pt;color:black">:
</span></b><span lang="EN-US" style="font-size:12.0pt;color:black">[CaCL] Reading for 3/28: Contrastive Decoding: Open-ended Text Generation as Optimization<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt">Hi All,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt">CaCL will not meet next week (3/14) and the week after (3/21). Our next meeting will be on 3/28 – we will be discussing “Contrastive Decoding: Open-ended Text Generation as Optimization” (Li et
al., 2023).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:12.0pt">Paper:</span></b><span lang="EN-US" style="font-size:12.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt"><a href="https://urldefense.com/v3/__https:/aclanthology.org/2023.acl-long.687/__;!!KGKeukY!wlPhBq-744it4BSBM1kqsSqUz8AchcHxOLByey3L_tmrI0-AoqwOd4rZ-5mlxmMGcDbWz22zoigqnTkuu_U$">https://aclanthology.org/2023.acl-long.687/</a>
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:12.0pt">Abstract:</span></b><span lang="EN-US" style="font-size:12.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt">Given a language model (LM), maximum probability is a poor decoding objective for open-ended generation, because it produces short and repetitive text. On the other hand, sampling can often produce
incoherent text that drifts from the original topics. We propose contrastive decoding (CD), a reliable decoding approach that optimizes a contrastive objective subject to a plausibility constraint. The contrastive objective returns the difference between the
likelihood under a large LM (called the expert, e.g. OPT-13B) and a small LM (called the amateur, e.g. OPT-125M), and the constraint ensures that the outputs are plausible. CD is inspired by the fact that the failures of larger LMs (e.g., repetition, inco-
herence) are even more prevalent in smaller LMs, and that this difference signals which texts should be preferred. CD requires zero additional training, and produces higher quality text than decoding from the larger LM alone. It also works across model scales
(OPT-13B and GPT2-1.5B) and significantly outperforms four strong decoding algorithms (e.g., nucleus, top-k) in automatic and human evaluations across wikipedia, news and story domains.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt">Best,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt">Yi-Chien<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</body>
</html>