PART TWO · MAKING THE SENTENCE STAND UP · type a sentence and watch it turn into a tensor

A sentence stands up and becomes a tensor.

Type anything below. It will be split into tokens, the horizontal line of words will rotate into a vertical stack — one row per token — and that stack will march left to right through the model, one column per stage, until only the last row is left.

One word on words: in this view a column is a stage — the words, the ids, the embeddings, then each of the 24 layers. A row is one token. And 896 is how long each row is, not a count of columns.

01

The whole pipeline

— —
The view scrolls itself to follow the action. Drag the scrubber to move by hand, or scroll the strip sideways.
02

Reading the strip

What each column is

  1. Words. Your sentence, cut into tokens. Roughly one per word, but a leading space belongs to the token and long or unusual words get split into pieces.
  2. Ids. Each token is looked up in the tokenizer's list and becomes an integer id. Qwen2.5’s tokenizer has 151,665 of them. This is the last point at which anything is text.
  3. Embeddings. Each id selects one row of a table with 151,936 rows (the tokenizer’s 151,665 plus padding), each row 896 numbers long. It is a copy, not a multiplication. Now the sentence is a grid: one row per token, 896 numbers along each row.
  4. Layers 1 to 24. Each column is one transformer layer. The grid keeps its exact shape all the way through — same number of rows, same 896 per row — but the numbers change at every column.
  5. The last row. Only the bottom row leaves the stack. It is normed once more and becomes h, the 896 numbers that decide the next word.

Three things the strip shows that a diagram usually hides

  1. The turn. A sentence is horizontal to a reader and vertical to a model. That rotation is the whole of "tokenization plus embedding", and it is where text stops and arithmetic starts.
  2. The rows are not lanes. Watch the faint arcs inside a layer: row 3 reads rows 0 to 3. Depth is the sequential axis, not the rows. On this page a column is a stage of the strip; the 896 numbers along a row are never columns, and the rest of the series calls the transformer blocks layers throughout.
  3. Rows 0 to n−2 stop at the last layer. They did their work by being attended to, and their keys and values stay in the KV cache for the next token. Only the last row goes on: its 896 numbers are normed and sent to the LM head.
Where the numbers come from. The token ids are real for common words and illustrative otherwise, since a full 151,936-entry vocabulary is not embedded in this page. The cell values are illustrative throughout. The shapes, the counts and the order are exact.
03

One column at a time

The same run, but only the newest column is on screen. Press step and it advances: words, then ids, then embeddings, then one layer at a time. The column is always the same shape. What changes is what is in it.

step 0 of 28
Words
—
—