3.2 Measuring Errors

3.2 Measuring Errors

In numerical computation, errors are not bugs. They are not signs of failure. They are not even unusual. They are the normal state of the universe. Every computation—no matter how trivial it looks on paper—accumulates imperfections as it moves through floating-point arithmetic.

But here is the uncomfortable truth: Most engineers are not taught how to measure these imperfections correctly.

We measure the wrong thing. We compare the wrong values. We assume the wrong baseline. And as a result, we misunderstand the behavior of our own algorithms.

To build systems that work, we need to understand what “error” really means. More importantly, we need to understand which kinds of errors matter.

Absolute Error: The Most Misleading Truth

The simplest measure of error is the absolute error:

absolute error = |computed − true|

It’s intuitive, easy to explain, and almost always insufficient.

Imagine you compute a value that should be 1,000,000 and instead get 1,000,001. That’s an absolute error of 1.

Now imagine a value that should be 0.000001, but you compute 0.000002. Absolute error is still 1e−6.

Mathematically the errors look identical. But practically, the first is irrelevant—and the second is catastrophic.

Absolute error answers the question:

“How far off are we in raw units?”

But most of the time, this is not the question we actually need to ask.

Relative Error: How Wrong Are We Compared to the Truth?

Relative error reframes the question:

relative error = |computed − true| / |true|

This measure tells you how large the error is compared to the value itself.

Using the earlier example:

• value should be 1,000,000 → relative error = 1e−6
• value should be 0.000001 → relative error = 1

One result is essentially perfect. The other is entirely wrong.

Relative error exposes sensitivity. It reveals whether the computation deserves our trust. But it has its limits too—particularly when the “true” value is very close to zero.

And in numerical computation, “very close to zero” happens constantly. In fact, entire classes of algorithms (like orthogonalization methods) rely on repeatedly subtracting nearly identical values, where relative error becomes undefined or meaningless.

Backward Error: The Most Honest Kind of Wrong

In practical systems, we often cannot measure the true value directly. Sometimes the exact answer is unknown. Sometimes it is impossible to compute exactly. Sometimes it doesn’t even exist as a single number—only as the solution to an equation.

This is where backward error enters.

Backward error reframes the narrative:

“Instead of asking how wrong the answer is, ask how much the original problem must have been perturbed for our answer to be exact.”

This feels philosophical, but it is profoundly practical. A result with a large forward error can still be considered excellent if the backward error is small.

Example: solving Ax = b.

• Forward error asks: How far is our computed x̂ from the true x?
• Backward error asks: For which A + δA and b + δb is x̂ the exact solution?

Backward error tells us how trustworthy the computation is within the context of the original data. It is the closest we can get to understanding “How wrong is the world I’m computing inside?” rather than “How wrong is the number I printed?”

Forward vs Backward Error: A Tale of Two Perspectives

Forward error answers:

“Did we compute the right number?”

Backward error answers:

“Are we solving a slightly different, but legitimate, problem?”

These two perspectives rarely align. A solution can have:

• small backward error and large forward error, or
• large backward error and small forward error.

Numerical analysts consider backward error the more fundamental measure because it reflects the true nature of floating-point computation: computers solve nearby problems exactly, not exact problems approximately.

Propagation of Error: The Invisible Tide

Errors do not sit still. They move. They grow. They shrink. They interact. Sometimes they cancel each other out. Sometimes they amplify each other dramatically.

Understanding error propagation is crucial for:

• deep learning (gradients can explode or vanish),
• optimization algorithms (conditioning affects convergence),
• simulation (tiny perturbations lead to divergent trajectories),
• matrix factoring (different algorithms propagate different errors).

An algorithm that is stable under one norm may be unstable under another. An operation that is safe in isolation may be dangerous when chained repeatedly. Error propagation is not about individual operations—it is about the overall architecture of computation.

Why Measuring Error Correctly Matters

Misunderstanding error measurement leads to:

• trusting results that are unreliable,
• rejecting results that are perfectly valid,
• choosing inefficient solvers out of fear,
• debugging symptoms instead of causes,
• misdiagnosing instability as “bad data” or “bad model.”

When engineers learn to measure error correctly, something shifts: they stop fighting their tools and start understanding them.

Error measurement transforms from a chore into an instrument—one that guides design, predicts behavior, and prevents catastrophic failures before they happen.

Transition: From Error to Sensitivity

Now that we understand how to measure errors, a deeper question appears:

“How sensitive is the underlying problem to those errors?”

Two algorithms can produce identical numerical errors—but one collapses under tiny perturbations while the other remains stable. Two matrices can produce identical solutions—but one amplifies noise dramatically, while the other absorbs it calmly.

To understand why, we must explore the idea of conditioning—the sensitivity of a problem itself—as well as the stability of the algorithm chosen to solve it.

Next: 3.3 Conditioning of Problems vs Stability of Algorithms