8.2 Rayleigh Quotient

8.2 Rayleigh Quotient

If the power method teaches us how eigenvectors emerge through repeated multiplication, the Rayleigh quotient teaches us how eigenvalues reveal themselves through the geometry of a vector. It is, in many ways, the moment where numerical linear algebra becomes almost poetic: a single expression captures the “best possible” eigenvalue estimate associated with a given direction in space.

The formula is simple:

R(x) = (xᵀ A x) / (xᵀ x)

Yet those few symbols encode a tremendous amount of meaning. The Rayleigh quotient tells you:

• how much the matrix A stretches the vector x, on average, and
• in what direction that stretching aligns.

When x happens to be an eigenvector, the Rayleigh quotient collapses to the exact eigenvalue. But even when x is only close to an eigenvector, the quotient becomes an extraordinarily precise estimator. This makes it the backbone of many modern eigenvalue algorithms.

The geometric intuition

To understand why the Rayleigh quotient works, picture a matrix as a transformation of space — rotating, stretching, compressing. Every vector that passes through this transformation changes in both direction and magnitude.

The Rayleigh quotient measures the ratio of how much A stretches x in the direction of x itself. In other words:

“How aligned is A x with x, and by how much?”

If x is an eigenvector, A x points exactly in that same direction, scaled by λ. So the quotient simply returns λ.

If x is not an eigenvector, the quotient returns something in between — a weighted combination of the true eigenvalues, biased toward the directions where x has more “influence.”

The Rayleigh quotient as an energy

In physics and optimization, the Rayleigh quotient appears as a kind of “energy” associated with a state vector:

• In structural engineering, it measures vibrational frequencies.
• In quantum mechanics, it corresponds to expected energy levels.
• In statistics, it forms the basis of PCA directions.

Across all of these fields, the same principle applies:

The Rayleigh quotient identifies the extremal eigenvalues.

Extrema and eigenvalues

One of the most beautiful properties of the Rayleigh quotient is:

The Rayleigh quotient reaches its maximum at the eigenvector associated with the largest eigenvalue, and its minimum at the eigenvector associated with the smallest eigenvalue.

This gives us a powerful variational interpretation:

• You can find eigenvalues by optimizing the Rayleigh quotient.
• You can find eigenvectors by looking at where the quotient reaches extrema.

This variational viewpoint is not just theoretical — it is the foundation for methods like the power method, inverse iteration, and the Lanczos algorithm.

Rayleigh quotient in power iteration

When the power method produces a vector xₖ that has almost converged to the dominant eigenvector, the Rayleigh quotient:

R(xₖ)

provides a remarkably accurate estimate of the corresponding eigenvalue — much more accurate than simply computing ratios of components.

This is why modern implementations of the power method almost always compute:

λₖ = (xₖᵀ A xₖ) / (xₖᵀ xₖ)

at each iteration.

Rayleigh quotient in inverse iteration

When combined with inverse iteration (especially shifted inverse iteration), the Rayleigh quotient becomes a tool for accelerating convergence.

Given an approximate eigenvector x:

• Compute R(x) = λ̂, an estimated eigenvalue.
• Use λ̂ as a shift for the inverse iteration.
• Repeat.

This feedback loop dramatically improves convergence and forms the basis of the Rayleigh Quotient Iteration method, which converges cubicly for symmetric matrices — one of the fastest rates achievable by any practical eigenvalue algorithm.

Why the Rayleigh quotient matters

The Rayleigh quotient is important not because it is simple, but because it is unreasonably effective. It shows up everywhere:

• PCA: directions maximizing variance
• Spectral clustering: Fiedler vectors from graph Laplacians
• Mechanical engineering: vibration modes
• Quantum mechanics: ground-state energies
• Optimization: quadratic forms

Whenever a system has a natural “dominant mode,” the Rayleigh quotient is the tool that reveals it.

The bridge to modern eigenvalue algorithms

So far, the journey through eigenvalues has been intuitive:

• Power method — discover dominant behavior.
• Inverse iteration — discover specific behavior.
• Rayleigh quotient — measure behavior precisely.

But what if we want:

• all eigenvalues?
• or all eigenvectors?
• or a way to reduce any matrix to a form where those computations are fast and stable?

This is where the next major character in our story enters — an algorithm so fundamental that nearly every linear algebra library depends on it:

The QR algorithm.

It is the general-purpose engine behind dense eigenvalue solvers, and its core idea is a breathtaking combination of orthogonal transformations, matrix factorization, and elegant iteration.

Let’s step into the world of the QR algorithm and see how it systematically extracts eigenvalues.