Compressing Documents to Augment Language Models More Efficiently

Recent years have witnessed the enormous growth of language models like GPT-3, with hundreds of billions of parameters.

Recent years have witnessed the enormous growth of language models like GPT-3, with hundreds of billions of parameters.

One common technique to enhance their performance is retrieval augmentation – prepending relevant documents to provide more context. However, this adds significant overhead.

Researchers from the University of Washington and Allen Institu…