Original Substack post

You See Neural Nets Wrong

A lot of beginner explanations make neural networks look like mystical webs of circles. That picture is memorable, but it is also the wrong thing to internalize. A neural network is not primarily a diagram. It is a composition of functions, and most of those functions are matrix multiplications followed by simple nonlinearities.

The clean mental model is:

$$z_1=X{W}_1+b_1$$ $$h_1=\sigma (z_1)$$ $$z_2=h_1{W}_2+b_2$$ $$\hat{y}=f(z_2)$$

That is the forward pass. Data comes in as a matrix, weights transform it, biases shift it, nonlinearities bend it, and the next layer repeats the same pattern.

Why the circle diagram misleads

The circle diagram makes you think neuron by neuron. That is useful for the first five minutes, but real models are not implemented neuron by neuron. They are implemented as dense tensor operations.

For one neuron, you can say:

$$z=w_1{x}_1+w_2{x}_2+...+w_n{x}_n+b$$

For a whole layer, this becomes:

$$Z=XW+b$$

That single equation is the layer. The same thing that looked like many little arrows is really a batch matrix multiplication.

What learning means

Learning means changing the weights so the function produces better outputs. The loss measures how wrong the prediction is:

$$L(\hat{y},y)$$

Backpropagation computes how much each parameter contributed to the loss:

$$\frac{\partial L}{\partial W}$$

Then the optimizer nudges weights in the opposite direction:

$$W\leftarrow W-\eta \frac{\partial L}{\partial W}$$

This is not magic. It is bookkeeping through a chain of matrix operations.

The important thing to memorize

Do not memorize neural nets as circles connected by arrows. Memorize them as repeated blocks:

1. multiply by weights
2. add bias
3. apply nonlinearity
4. compute loss
5. propagate gradients backward
6. update weights

That picture scales. MLPs, CNNs, transformers, and diffusion models all become less mysterious when you ask: what tensor shape enters this block, what operation transforms it, and where do gradients flow?

If you can track shapes, operations, and gradients, neural networks stop being diagrams and start being programs.