Jay van Zyl @ ecosystem.Ai

Jay van Zyl @ ecosystem.Ai

Perceiver AR: general-purpose, long-context autoregressive generation

We develop Perceiver AR, an autoregressive, modality-agnostic architecture which uses cross-attention to map long-range inputs to a small number of latents while also maintaining end-to-end causal masking. Perceiver AR can directly attend to over a hun…