AI is (was) just functions

2026-08-28

When we think about AI today we immediately think about LLMs, Agents, MoE and Transformers. This is the result of basically the last 10 years of developments, but still many people struggle to understand the heart of this incredible topic that is reshaping the world we live in.

Modern AI systems are HUGE and non-deterministic. Their capabilities are emergent, not strictly programmed, meaning that we can't be sure about the output. It's just structural, unpredictability is an inherent property of their scale and complexity.

It’s like building a super complex car crash simulator. We code the laws of physics in every detail and build the most realistic 3D version of the car and road we can think of. When we start the simulation, we know the car will be hitting the wall, but since we are simulating millions of interacting variables down to the microscopic variations in the steel, we can never predict the exact shape of the twisted metal. We know the car will crumple, and we can test its general safety, but the exact arrangement of the debris will be completely unique every single time we run it.

In fact, if there's something where many people failed, it is predicting future developments, on both reductive and exaggerated extremes, regarding the capabilities of these models.

Still, it's essential for us and the generations to come to understand how these systems work, to get the intuitions that got us here, and to address the criticalities and dangers of these nets.

To get a good insight of what the heck is happening under the hood, it is good to see a neural net as a function.

When we talk about neural nets we're not talking about GPT or Opus, as those are really complex architectures that stem from the concept of a pure neural net.

Let's say we have a simple neural net that takes 2 inputs and tries to predict an output using only 1 neuron.

Our neuron is just a linear function: it takes 2 inputs, multiplies them by its weights, and gives a resulting expression.

𝑓 (𝑥_{1}, 𝑥_{2}) = 𝑥_{1} \cdot 𝑤_{1} + 𝑥_{2} \cdot 𝑤_{2}

In Calculus 2 this is called a multivariable function; it represents a plane in a 3D space.

We have 2 input variables, $𝑥_{1}$ and $𝑥_{2}$ , and the third dimension will represent our output.

When we do a forward pass on the network, we just extract the third value.

Our weights represent the coefficients of the lines that define our plane. Our goal is to adjust those coefficients such that the plane "fits", approximates, our data.

That's what we do during training.

When we train our network, the perspective shifts: the weights become our variables, while the inputs are treated as fixed constants that we want to fit.

We need to evaluate the output of our network, we can simply compare it to our desired output ( $𝑋_{exp}$ ):

loss = 𝑓 (𝑥_{1}, 𝑥_{2}) - 𝑋_{exp}

This function will give us a measure of how well our neural net is approximating our data.

Since our goal is to "fit" our data, we want $𝑓 (𝑥_{1}, 𝑥_{2})$ to approach $𝑋_{exp}$ , which is equivalent to saying we want to find the combination of weights where the function $loss$ approaches 0.

If we plot the loss function in weights coordinates, what we want to do is basically move along the plane until we intersect with the 0 plane.

w₁: 0.50 w₂: 0.50 Loss: -0.75

How do we do find the right combination?

We can compute the partial derivative of the loss with respect to each parameter. This will tell us the direction where are moving if we increase the parameter.

The derivatives with respect to $𝑤_{1}$ and $𝑤_{2}$ in our simple case will represent the steepness of the 2 lines that define the plane that our $loss$ function represents in the 3D space, in $𝑤_{1}$ and $𝑤_{2}$ coordinates. (In our simple case the derivatives happen to be just our inputs...)

Let's assume $\frac{𝜕 loss}{𝜕 𝑤_{1}}$ is positive. It means that, with the same $𝑥_{1}$ and $𝑥_{2}$ input, if we increase $𝑤_{1}$ we will get a bigger loss, we move upward.

To decrease the loss, we must walk down the slope along the $𝑤_{1}$ axis, meaning we decrease $𝑤_{1}$ .

By finding the right combination the situation will look like so

We just fitted our single example.

What if we have more data?

So far we trained of dummy neuron on 1 example of data, what if we want to teach it how to approximate more inputs?

Let's train our network on 1 batch of 2 examples, 2 inputs each (say $𝑥_{1}, 𝑥_{2}, 𝑦_{1}, 𝑦_{2}$ ).

We will have 2 outputs:

𝑓 (𝑥_{1}, 𝑥_{2}) = 𝑥_{1} \cdot 𝑤_{1} + 𝑥_{2} \cdot 𝑤_{2}

𝑓 (𝑦_{1}, 𝑦_{2}) = 𝑦_{1} \cdot 𝑤_{1} + 𝑦_{2} \cdot 𝑤_{2}

We define the loss to be the mean error:

loss = \frac{(𝑓 (𝑥_{1}, 𝑥_{2}) - 𝑋_{exp}) + (𝑓 (𝑦_{1}, 𝑦_{2}) - 𝑌_{exp})}{2}

We can see here that if the 2 errors have different signs we will have $loss = 0$ , while we actually are not fitting our data. That's the reason why we normally get rid of signs by using absolute values or squares. The normal preferred way is squaring because it has some useful properties like penalizing more big errors and less smaller ones, it's differentiable at 0...

loss = \frac{{(𝑓 (𝑥_{1}, 𝑥_{2}) - 𝑋_{exp})}^{2} + {(𝑓 (𝑦_{1}, 𝑦_{2}) - 𝑌_{exp})}^{2}}{2}

The function now will look like a cup in space.

The idea is the same, move along the cup to reach its minimum.

w₁: 0.50 w₂: 0.50 Loss (MSE): 0.81

If we expand the loss function we notice that now $𝑤_{1}$ and $𝑤_{2}$ will have their derivative with respect to 2 inputs each.

loss = \frac{{((𝑥_{1} \cdot 𝑤_{1} + 𝑥_{2} \cdot 𝑤_{2}) - 𝑋_{exp})}^{2} + {((𝑦_{1} \cdot 𝑤_{1} + 𝑦_{2} \cdot 𝑤_{2}) - 𝑌_{exp})}^{2}}{2}

When we compute the partial derivatives of the loss with respect to $𝑤_{1}$ and $𝑤_{2}$ we know from the sum rule of partial derivatives that the partial derivative of a sum of two functions is equal to the sum of their individual partial derivatives.

So we just sum the partial derivatives of the single error expressions with respect to $𝑤_{1}$ and $𝑤_{2}$ :

\frac{𝜕 loss}{𝜕 𝑤_{1}} = \frac{𝜕}{𝜕 𝑤_{1}} (\frac{{(𝑓 (𝑥_{1}, 𝑥_{2}) - 𝑋_{exp})}^{2}}{2}) + \frac{𝜕}{𝜕 𝑤_{1}} (\frac{{(𝑓 (𝑦_{1}, 𝑦_{2}) - 𝑌_{exp})}^{2}}{2})

The vector of all the partial derivatives is called gradient:

\nabla loss = (\begin{matrix} \frac{𝜕 loss}{𝜕 𝑤_{1}} \\ \frac{𝜕 loss}{𝜕 𝑤_{2}} \end{matrix})

Our goal is to make all the partial derivatives approach zero, meaning our parameters cannot be "optimized" more.

To compute partial derivatives, in programming, we normally would apply the chain rule to break up our expression in smaller ones that have simple derivation like:

\frac{𝜕 {(𝑓 (𝑥_{1}, 𝑥_{2}) - 𝑋_{exp})}^{2}}{𝜕 𝑤_{1}} = \frac{𝜕 𝑋^{2}}{𝜕 𝑤_{1}} = \frac{𝜕 𝑋^{2}}{𝜕 𝑋} \cdot \frac{𝜕 𝑋}{𝜕 𝑤_{1}}

As before, we compute the gradient for each of our parameters and adjust them in order to fit our training data.

If the derivative is positive it means if we increase our parameter the loss will go up, hence we want to decrease it
If the derivative is negative it means if we increase our parameter the loss will go down, hence we want to increase it

So each time we just go in the opposite direction of the sign of the derivative. We usually step by a fixed quantity proportionally to the derivative (called Learning Rate, $LR$ ):

𝑤_{1^{'}} = 𝑤_{1} - LR \cdot \frac{𝜕 loss}{𝜕 𝑤_{1}}

𝑤_{2^{'}} = 𝑤_{2} - LR \cdot \frac{𝜕 loss}{𝜕 𝑤_{2}}

We will end up like so:

This iterative process, called gradient descent, is the algorithm at the hearth of machine learning. Of course our introduction is really dumb, for istance it would be better to add a "bias" to our neurons so that we can "adjust" our plane in another dimension in space, or add non-linearity to it more sparse data since right now we are fitting data with a flat plane...

But the core intuition still is: - Define what our network should learn to predict - Define a policy to determine the output of our network, a loss function - Run gradient descent and "fit" the data