[CaCL] 9/7: Retentive Network: A Successor to Transformer for Large Language Models

[Schuler, William]

Here’s a site that walks through the math:

https://medium.com/ai-fusion-labs/retentive-networks-retnet-explained-the-much-awaited-transformers-killer-is-here-6c17e3e8add8
________________________________
From: CaCL <cacl-bounces+schuler=ling.osu.edu at lists.osu.edu> on behalf of Oh, Byung-Doh via CaCL <cacl at lists.osu.edu>
Sent: Thursday, August 31, 2023 2:32:57 PM
To: cacl at lists.osu.edu <cacl at lists.osu.edu>
Subject: [CaCL] 9/7: Retentive Network: A Successor to Transformer for Large Language Models

Hello everyone,

Next week, we'll discuss the following paper on Retentive Network:

Retentive Network: A Successor to Transformer for Large Language Models
https://arxiv.org/pdf/2307.08621.pdf<https://urldefense.com/v3/__https://arxiv.org/pdf/2307.08621.pdf__;!!KGKeukY!wVnlcLm6OYgby9mskEjEVwuxODKw4np1MdzKBXRGQYdbLsBcN5EvAk8puzKon_4vT7_Czz8cEmdCdsRLzc4$>
In this work, we propose Retentive Network (RETNET) as a foundation architecture for large language models, simultaneously achieving training parallelism, low-cost inference, and good performance. We theoretically derive the connection between recurrence and attention. Then we propose the retention mechanism for sequence modeling, which supports three computation paradigms, i.e., parallel, recurrent, and chunkwise recurrent. Specifically, the parallel representation allows for training parallelism. The recurrent representation enables low-cost O(1) inference, which improves decoding throughput, latency, and GPU memory without sacrificing performance. The chunkwise recurrent representation facilitates efficient long-sequence modeling with linear complexity, where each chunk is encoded parallelly while recurrently summarizing the chunks. Experimental results on language modeling show that RETNET achieves favorable scaling results, parallel training, low-cost deployment, and efficient inference. The intriguing properties make RETNET a strong successor to Transformer for large language models. Code will be available at https://aka.ms/retnet<https://urldefense.com/v3/__https://aka.ms/retnet__;!!KGKeukY!wVnlcLm6OYgby9mskEjEVwuxODKw4np1MdzKBXRGQYdbLsBcN5EvAk8puzKon_4vT7_Czz8cEmdCtufTM5M$>.

Best,
Byung-Doh

=================
Byung-Doh Oh (he/him/his)
Ph.D. Candidate
Department of Linguistics
The Ohio State University

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/cacl/attachments/20230907/110312e6/attachment.html>