Improving language models by retrieving from trillions of tokens
Uncategorized
Improving language models by retrieving from trillions of tokens
We explore an alternate path for improving language models: we augment transformers with retrieval over a database of text passages including web pages, books, news and code. We call our method RETRO, for “Retrieval Enhanced TRansfOrmers”.
Uncategorized

Improving language models by retrieving from trillions of tokens

We explore an alternate path for improving language models: we augment transformers with retrieval over a database of text passages including web pages, books, news and code. We call our method RETRO, for “Retrieval Enhanced TRansfOrmers”.