Lists That Agree

Scenario

Amber studio keeps glaze recipes as ordered lists of four knobs: kaolin, feldspar, silica, and a colorant. Two recipes sit on the bench this morning.

The house white:

$$a = [3,\ -1,\ 2,\ 0.5]$$

The guest celadon:

$$b = [1,\ 2,\ -1,\ 4]$$

The minus on the house white is not a typo. The second slot is feldspar relative to a baseline: this batch is short one part feldspar. If you drop the minus you are mixing a different glaze, and the rest of the arithmetic will lie to you politely.

The question is not "what is a vector?" The question is: when do these two recipes agree, and how much of the house white is already living along the celadon?

Those two measurements — agreement, and "the piece of one list that lives along another" — are the whole of gradient descent in miniature. A loss will later be a hill over a list of weights. A gradient will be a second list. The inner product of those two lists is how much the hill agrees with a walk you are considering. If you cannot see that agreement in four slots, you will not see it in $10^{11}$.

A few answers look almost right and are wrong:

• $5$ — you dropped the minus on $a$’s feldspar ($3\cdot 1 + 1\cdot 2 + 2\cdot(-1) + 0.5\cdot 4 = 3 + 2 - 2 + 2 = 5$).
• $10.5$ — you added slots inside each recipe first and then combined them ($(3 - 1 + 2 + 0.5) + (1 + 2 - 1 + 4) = 4.5 + 6 = 10.5$). Agreement needs partner slots multiplied.
• $[4,\ 1,\ 1,\ 4.5]$ — you added the recipes. That is a third glaze, not a number that says how they agree.
• $9$ — you dropped every sign ($|3| + |-2| + |-2| + |2| = 9$). Agreement respects directions.

Keep the minus. Multiply partners. Add those products. The agreement is $1$.

The walk is live on the media Worker (176.4 s, cover at 66 s): a narrated plate of the four partner products and their sum. It is served from the SuperLearn media Worker; there is no second copy in the app bundle.

A list is an order, not a pile

The brackets on $a$ are doing work. They say the first slot is kaolin, always. Swap the first two slots and you have $[-1,\ 3,\ 2,\ 0.5]$: a glaze that is short kaolin and heavy feldspar. Same digits, different object. In the rest of this course a weight vector is the same kind of object. The 17th coordinate of a linear layer is not "a number we happen to have." It is that direction in parameter space.

You can add two lists only when every slot has a partner:

$$a + b = [4,\ 1,\ 1,\ 4.5]$$

That third glaze is a mix: three parts house kaolin plus one part guest kaolin, and so on. Scaling is the other primitive: $\frac{1}{2} a$ is the same recipe, halved. Those two acts — add, and scale — are the linear algebra this course needs. We will not rebuild all of linear algebra here; we need just enough to read a gradient.

The length of a list is not the count of its slots. The Euclidean length is how far the arrow sits from the origin:

$$\|a\| = \sqrt{3^2 + (-1)^2 + 2^2 + 0.5^2} = \sqrt{9 + 1 + 4 + 0.25} = \sqrt{14.25}$$

A long list is a strong recipe, or a steep slope, depending on what the list is. Length does not know about agreement: two long lists can point opposite ways.

Agreement is multiply-then-add

The dot product (or inner product) of two lists $a$ and $b$ of equal length is the sum of their coordinate-wise products:

$$a \cdot b = \sum_{i=1}^n a_i b_i$$

For our glaze recipes:

$$a \cdot b = 3\cdot 1 + (-1)\cdot 2 + 2\cdot(-1) + 0.5\cdot 4 = 3 - 2 - 2 + 2 = 1$$

When $a \cdot b > 0$, the recipes lean in the same direction. When $a \cdot b < 0$, they oppose each other. When $a \cdot b = 0$, they are orthogonal: neither recipe has anything to say about the other.

Notice that length squared is self-agreement:

$$\|a\|^2 = a \cdot a = 14.25, \qquad \|b\|^2 = b \cdot b = 1^2 + 2^2 + (-1)^2 + 4^2 = 22$$

The piece of one recipe that lives along another

Suppose you want to express the house white $a$ as a piece that points along the celadon $b$, plus a leftover residual $r$:

$$a = c\,b + r$$

We require that the leftover $r$ shares zero agreement with $b$, meaning $r \cdot b = 0$. Substitute $r = a - c\,b$:

$$(a - c\,b) \cdot b = 0 \;\implies\; a \cdot b - c\,(b \cdot b) = 0 \;\implies\; c = \frac{a \cdot b}{\|b\|^2}$$

For the Amber studio numbers:

$$c = \frac{1}{22}$$

The projection of $a$ onto $b$ is therefore:

$$\operatorname{proj}_b(a) = \frac{1}{22} b = \left[\frac{1}{22},\ \frac{2}{22},\ -\frac{1}{22},\ \frac{4}{22}\right]$$

Now verify the residual. The leftover glaze is:

$$r = a - \frac{1}{22} b = \left[3 - \frac{1}{22},\ -1 - \frac{2}{22},\ 2 + \frac{1}{22},\ 0.5 - \frac{4}{22}\right] = \left[\frac{65}{22},\ -\frac{24}{22},\ \frac{45}{22},\ \frac{7}{22}\right]$$

Take its dot product with $b$:

$$r \cdot b = \frac{65(1) + (-24)(2) + 45(-1) + 7(4)}{22} = \frac{65 - 48 - 45 + 28}{22} = \frac{0}{22} = 0$$

The residual really is orthogonal to $b$. The leftover glaze shares nothing with celadon.

Two lengths are worth keeping, because they are the triangle in the figure below. The piece along $b$ has length

$$\|c\,b\| = c\,\|b\| = \frac{1}{22}\sqrt{22} = \frac{1}{\sqrt{22}} \approx 0.213$$

and the leftover has length $\sqrt{14.25 - 0.0455} \approx 3.769$. Those two are square to each other, so together they rebuild the house white: $0.213^2 + 3.769^2 = 14.25$. A right triangle whose legs are measured, not sketched.

Cosine: agreement without scale

If you double the guest recipe to $2b$, the dot product doubles: $a \cdot (2b) = 2(a \cdot b) = 2$. But the recipes do not point any more or less in the same direction than they did before.

The cosine normalizes by list lengths so size cannot fake alignment:

$$\cos\theta = \frac{a \cdot b}{\|a\|\,\|b\|} = \frac{1}{\sqrt{14.25 \cdot 22}} = \frac{1}{\sqrt{313.5}} \approx 0.0565$$

Because $\cos\theta \approx 0.0565$ is close to $0$, the house white and guest celadon are nearly perpendicular. Put the number back through the inverse cosine and the angle is $\arccos(0.0565) \approx 86.76^\circ$ — six tenths of a degree short of a right angle.

Worked example table

Every intermediate value computed on the home numbers:

Slot	Knob	$a_i$	$b_i$	$a_i b_i$	$a_i^2$	$b_i^2$
1	kaolin	3	1	3	9	1
2	feldspar	−1	2	−2	1	4
3	silica	2	−1	−2	4	1
4	colorant	0.5	4	2	0.25	16
sum				1	14.25	22

Summary: $a \cdot b = 1$, $\|a\|^2 = 14.25$, $\|b\|^2 = 22$, $c = 1/22$, $r \cdot b = 0$, $\cos\theta \approx 0.0565$.

Interactive workbench

Use the lab below to inspect how slot pairings affect agreement. Flip the feldspar sign to see the agreement jump from 1 to 5, shuffle slots to see how order alters the dot product, or scale $b$ to confirm that the projection foot and cosine remain stable.

OPT-P1-AGREE

OPT-P1-ANATOMY

Try it: Transfer example

Work this example by hand before revealing the solution:

Given two lists:

$$p = [1,\ -2,\ 2], \qquad q = [2,\ 1,\ 2]$$

Find $p \cdot q$, the projection scale $c$ of $p$ onto $q$, the residual $r$, and verify $r \cdot q = 0$.

Reveal solution

1. Inner product: $$p \cdot q = 1(2) + (-2)(1) + 2(2) = 2 - 2 + 4 = 4$$
2. Length of $q$: $$\|q\|^2 = 2^2 + 1^2 + 2^2 = 4 + 1 + 4 = 9 \implies \|q\| = 3$$
3. Projection scale: $$c = \frac{p \cdot q}{\|q\|^2} = \frac{4}{9}$$
4. Residual: $$r = p - \frac{4}{9} q = \left[1 - \frac{8}{9},\ -2 - \frac{4}{9},\ 2 - \frac{8}{9}\right] = \left[\frac{1}{9},\ -\frac{22}{9},\ \frac{10}{9}\right]$$
5. Orthogonality check: $$r \cdot q = \frac{1(2) - 22(1) + 10(2)}{9} = \frac{2 - 22 + 20}{9} = 0$$
6. Cosine: $$\|p\| = \sqrt{1^2 + (-2)^2 + 2^2} = 3 \implies \cos\theta = \frac{4}{3 \cdot 3} = \frac{4}{9}$$

Runnable code verification

Run this in Python to check the arithmetic:

import numpy as np

a = np.array([3, -1, 2, 0.5])
b = np.array([1, 2, -1, 4])

print(a @ b, a @ a, b @ b)          # 1.0 14.25 22.0

# Projection scale and residual
c = (a @ b) / (b @ b)
r = a - c * b

print(np.round(r * 22, 6), r @ b)   # [65. -24. 45. 7.] ~0 (2.2e-16)
print(round((a @ b) / np.sqrt((a @ a) * (b @ b)), 4))  # 0.0565

Why does r @ b print 2.2e-16 instead of exactly 0? Standard 64-bit IEEE floating-point arithmetic has finite precision; $2.2 \times 10^{-16}$ is zero to the printer's patience.

Failure modes to watch for

• Adding slots inside each list: Adding kaolin to silica gives $4.5$ and $6$, which has nothing to do with agreement.
• Dropping a minus: If you treat feldspar as $+1$, the dot product becomes $5$. That is a different glaze.
• Mismatched list lengths: You cannot compute the inner product of a 4-slot recipe with a 3-slot recipe. Unmatched slots have no partner.
• Confusing length with agreement: A vector can be huge while sharing zero agreement with another vector.
• Treating the residual as parallel to $b$: By construction, $r$ is strictly orthogonal to $b$. It holds the variation that $b$ cannot explain.

Why this is the language of a loss

A loss $f(w)$ is a hill over a list of parameters $w$. In Chapter P3, we will meet the gradient $\nabla f(w)$, which is a second list of the exact same length. The directional derivative — how fast the loss climbs as you walk in direction $d$ — is nothing other than the inner product $\nabla f(w) \cdot d$.

In deep neural networks, weights and gradients contain billions of numbers. But the geometry is identical: multiplying partner slots and adding them up to determine how much two directions agree.

The desk problem at the end uses different lists. Do not reuse $1$ or $22$ as a quiz answer; work the desk lists by hand.

Next: P2 The Price of a Nudge — what happens when we tweak a single knob by a tiny amount.

What this page claims

Claim	Status
Amber studio glaze recipes, house white and guest celadon	ILLUSTRATION: a teaching scenario, not a measured batch
$a \cdot b = 1$, $\|a\|^2 = 14.25$, $\|b\|^2 = 22$, $c = 1/22$, $r \cdot b = 0$, $\cos\theta \approx 0.0565$	COMPUTED: numpy on the page's own arrays; the same asserts run in tools/opt-p1-narration/manim/p1_lists_that_agree.py and opt-p1-agree.test.ts
Still opt_p1_agree	ILLUSTRATION: a teaching diagram served from the media Worker
Lesson video	shipped: Worker mp4 + cover (about 176.4 s walk, HTTP 200)
Agreement is the directional derivative of a loss	PUBLISHED: standard textbook description, not proved on this page