Silicon Perception Inc · open source

A recurrent CNN vision-language-action model

No attention, no positional encoding. The model's memory is a stack of 2-D register grids; every clock tick one small convolutional network reads the current token together with the registers and writes the registers back.

The target is a vision-language-action model on a single chip at 100 Hz or more, with no external components: no off-chip memory, every weight stored on the chip itself. A CNN makes that possible because it is parameter-efficient: the same small kernels run at every grid position and every tick, so a few million weights drive a state of hundreds of thousands of values.

Block diagram of one 40 ms clock tick: token, image and audio grids and five registers are concatenated, mixed by a 1x1 convolution, transformed by eight 3x3 convolutions, projected, and written back into the registers as a cascade; three decoders produce text, voice and image tokens.
One 40 ms clock tick of the design (v451): three token streams (text, voice, image), a frozen image encoder at 2 Hz, fixed log-mel hearing at 25 Hz, and the five-register cascade. Select the figure to open it full size.
125,440values of state: 5 registers × 512 channels × 7 × 7
20.1 Mtrained parameters, small enough to keep every weight on-chip
128 s → 1 tickregister time scales, from slow topic to the current word
0 attentionlayers; the gradient stops at the registers (no backpropagation through time)

How a tick works

  1. The current token's embedding is normalized and copied to every position of a 7 × 7 grid.
  2. That grid, the image and sound grids, and the five registers are concatenated and mixed by a 1 × 1 convolution.
  3. Eight 3 × 3 convolutions with BatchNorm and ReLU transform the result, and a 1 × 1 convolution projects it to register width.
  4. The registers are written as a cascade: each slow register moves 1/W of the way toward what the slower ones have not yet captured (W = 3200, 800, 200, 50 ticks); the last holds the remainder, so the registers always sum to the new output.
  5. A small tree of convolutions collapses the grid to one vector, and a frozen output layer gives the next token.

Results

The published model (run v472) on minimal-pair tests, scored the same way for every model: each sentence is read with no context, and the one with the lower total loss counts as the model's choice.

ModelSizeBLiMP (grammar)ComVE (common sense)
pythia-14m14 M69.764.7
pythia-31m31 M71.166.9
babyllama-10m-202410 M words70.063.7
SmolLM2-135M135 M80.979.3
VLA v47220.1 M trained69.865.2

BLiMP: 1,005 pairs spread over its 67 paradigms. ComVE: 999 pairs. Held-out FineWeb: 4.560 nats/token.

Samples

Unprompted, from a cleared state, at the published step. Fluent, not factual:

Before our first election, some may be interested in polling. However, after a few days at the start of the summer period, the team is busy building up an outdoor camp during a coach-planned tour.
The range comes in several sizes to suit your needs. These are the two choices here are plenty of decorations that you want.

Try it

git clone https://github.com/siliconperception/vla && cd vla
pip install -r requirements.txt
python vla.py --pretrained --generate --n 100 --prompt "The little girl"