1.3 Computation & Mathematical Systems

When engineers first learn linear algebra, we are taught to think in terms of concepts: matrices, vectors, transformations, subspaces, eigenvalues. It feels elegant and orderly—a clean language for describing relationships between quantities.

But when that same mathematics enters a real computer, the environment changes. What was once a purely logical system now becomes a computational system, living inside hardware with limits, quirks, and trade-offs. In this new environment, algorithms behave differently—not because the math changed, but because the world around the math changed.

Mathematics Describes; Computation Executes

A mathematical system defines how numbers should behave. A computational system defines how numbers can behave.

In mathematics, a vector is an abstract object. In computation, it is a block of memory with alignment constraints, cache behavior, and fixed precision.

In mathematics, a matrix multiplication is a concept. In computation, it is a sequence of operations that must navigate:

• memory bandwidth
• floating-point precision
• CPU/GPU vector units
• cache hierarchy
• parallelism and synchronization

The mathematics tells us what the answer should be. The computational system determines whether we can get the answer at all.

The Two Invisible Enemies: Stability and Conditioning

When engineers say “the algorithm is unstable,” or “the problem is ill-conditioned,” they are describing failures that come from the interaction between mathematics and the constraints of computation.

These two concepts govern almost every numerical method:

• Conditioning — how sensitive the problem is to small changes in input
• Stability — how sensitive the algorithm is to small errors during computation

Mathematics may offer a beautiful solution to a problem, but if the problem is ill-conditioned, even perfect computation would amplify tiny input errors into meaningless output.

Likewise, an algorithm may be mathematically correct, but if it is unstable, small floating-point errors introduced along the way will grow uncontrollably until the final result becomes unreliable.

Together, these two forces determine whether your model behaves sensibly, or whether it spirals into nonsense.

Computation Has Rules of Its Own

Computational systems have behaviors that do not appear in math textbooks:

• Catastrophic cancellation — subtracting nearly equal numbers destroys precision
• Non-associativity — (a + b) + c ≠ a + (b + c)
• Precision loss through accumulation
• Overflow and underflow
• Memory access patterns affecting performance

Understanding these rules is not optional. They determine whether an algorithm behaves as intended.

Why Expertise in Mathematical Systems Matters for AI

Modern AI systems—LLMs, vision models, optimization pipelines, simulation engines—are not just mathematical constructs. They are computational systems built on mathematical foundations.

Every training step, every gradient update, every embedding computation, every optimization step depends on delicate interactions between:

• floating-point precision
• matrix factorizations
• iterative solvers
• GPU kernel execution
• numerical stability of operations

This is why two models with identical architectures can behave differently on different hardware. It’s why training can “explode” mid-epoch. It’s why inference can become inconsistent across platforms.

It is not just math. It is math under the constraints of computation.

Bridging the Two Worlds

A numerical engineer is someone who can see both layers of reality:

• the mathematical system (the ideal world)
• the computational system (the physical world)

When you understand both, things that once seemed mysterious—NaNs during training, unpredictable model outputs, unstable optimization, divergence in simulations—begin to make sense. Solutions that once seemed impossible become practical.

And perhaps most importantly: you gain the ability to predict failure before it happens.

Now that we’ve seen how mathematical systems behave differently once they enter the world of computation, it’s time to examine where these issues surface in the real world. The next section walks through concrete examples of numerical failures—some dramatic, some subtle—that demonstrate why numerical linear algebra is not optional knowledge for modern engineers.

Let’s continue to 1.4 A Brief Tour of Real-World Failures.