What is the core idea?

• Inductive biases are baked into models, thus while increasing efficiency, they limit the model’s performance to one particular modality.

  • Enter Perceiver, a model designed to handle arbitrary configurations of different modalities.

• Perceiver is built upon Transformers

• By introducing a small set of latent units, the authors are able to introduce an attention bottleneck through which inputs must pass.

  • Eliminates the quadratic scaling problem of the classical Transformer
  • Decouples network depth from input’s size

• Allows scaling to hundreds of thousands of inputs like ConvNets

• Employs asymmetric attention mechanism to iteratively distill inputs
  • Hence, can scale to handle very large inputs
• Model is competitive or outperforms strong specialised models across various modalities

How is it realized (technically)?

• 

• Architecture
  • Cross attention module - maps a byte array and a latent array to a latent array
    • A byte array can be something like a pixel array etc.. (the input)
      • Generally large size $M$
    • Size of latent array is typically much smaller, and is a hyperparameter - size $N$
    • So, $M << N$.
  • Transformer tower maps a latent array to a latent array
  • In essence, the cross-attention module projects a high-dimensional input byte array to a fixed-dimensional latent bottleneck
    • Then processes it using a deep stack of Transformer-style self-attention blocks in the latent space
  • Cross attention and the Transformer are applied in alternation
    • Allows Perceiver to iteratively attend to input byte array by alternating between cross attention and the latent self-attention blocks.
  • Asymmetric attention
    • Authors use the query-key-value (QKV) attention
      • Problem with QKV in Transformers is quadratic self-attention in input dimensionality
        • In standard QKV attention, $Q \in \mathbb{R}^{M \times D}, K \in \mathbb{R}^{M \times C}, \text{ and } V \in \mathbb{R}^{M \times C}$ where $C$ and $D$ are channel dimensions and $M$ is the input size.
        • So $softmax(QK^T)V$ is $\mathcal{O}(M^2)$
      • Introduce assymmetry
        • $Q$ is a projection of the learned latent array with index dimension $N << M$, where $N$ is a hyperparameter.
        • $K$ and $V$ are projections of the inputer byte array
        • Resulting cross attention operation is $\mathcal{O}(MN)$
  • Cross attention module takes the shape of input $Q$
    • Induces a bottleneck
    • which the authors exploit by building deep Transformers in the latent space - low cost of $\mathcal{O}(N^2)$.
    • Hence latent Transformer has complexiity $\mathcal{O}(LN^2)$, when considered as a function of the number of layers.
      • notice that a standard Transformer would be $\mathcal{O}(LM^2)$
  • Overall architecture - $\mathcal{O}(MN + LN^2)$
    • Input size ($M$) and depth ($L$) are decoupled
      • Allows construction of very large networks on large-scale data
  • Latent transformer uses GPT-2 architecture
    • based on the decoder of the orginal Transformer
  • Iterative cross-attention
    • Severity of the bottleneck may restrict the network’s ability to capture all of the necessary details from the input signal
      • Hedge against this by adding multiple cross-attend layers
        • Allows latent array to iteratively extract information
• Position encodings
  • Permutation invariance and position information
    • Attention is a permutation-invariant operation which is preserved by the Perceiver model.
    • However permutation invariance does not allow exploitation of spatial relationships in the input data.
    • Hence, they also include position encodings
  • Scalable Fourier Features
    • Authors use a parameterization of Fourier features
      • allows direct representation of the position structure of the input data
      • allows control over the number of frequency bands in the position encoding indepdently of the cutoff frequency
      • uniformly sample all frequencies up to a target resolution.
      • Frequency parametrized to take the values $[\sin(f_k\pix_d), \cos(f_k\pix_d)]$,
        • where $f_k$ is the $k^{th}$ band of a bank of frequencies spaced equally between 1 and $\frac{\mu}{2}$. $\frac{\mu}{2}$ can be naturally interpreted as the Nyquist frequency, where $\mu$ is the target sampling rate.
        • $x_d$ is the value of input position along the $d^{th}$ dimension - for images $d = 2$. $x_d \in [-1, 1]$.
      • related to the NeRF position encoding scheme (Mildenhall et al., 2020).
  • Position encodings are generally applicable.
    • Feature based approach allows the network to learn how to use (or ignore) the position structure
    • Position encoding can be easily adapted to a new domain as Fourier features are trivial to adapt as long as the input dimension is relatively small and known
    • Position encodings can be extended to multimodal data - each domain has it’s own position encoding.

How well does the paper perform?

• Pretty competitively with other models
• 
  • Top-1 validation accuracy (in %). Compares baseline from literature, baseline with Fourier Features, Transformer with Fourier Features against Perceiver. Perceiver performs as well as baseline.
• 
  • Top-1 validation accuracy (in %). Shows the Perceiver model’s resiliency to permuted dataset. This is due to global attention. Input RF refers to the Input Receptive Field (RF) in pixels - this is the size of the local neighbourhood used for 2D convolutions
• 
  • Very comparable results. In future work, hope to incorporate the insights that are also used in CNN-14 (Kong et al., 2020).
• 
  • Top-1 validation accuracy (in %) on Model-Net40. PointNet++ uses extra geometric features and other techniques not used in the blue models.

TL;DR

• Uses iterative asymmetric cross-attention to decouple network depth from input size allowing for large networks
• Motivated to be a generalisable architecture - and thus, Is a multimodal architecture
• Performs competitively against specialised models with inductive biases baked in.