Step 02 · Densities & Matrices
The Anatomy of a Covariance Matrix
A covariance matrix is a small square of numbers that remembers, for every pair of quantities, how they wander together. Reading its anatomy is the key to every multivariate statement that follows.
Building the square
List the quantities as rows and as columns. In the cell where a quantity meets itself, place its variance. In the cell where one quantity meets another, place their covariance, a measure of how they move in step. The result is symmetric, because the covariance of the first with the second equals the covariance of the second with the first. The diagonal therefore carries the spread and the off-diagonal carries the association.
What the off-diagonal says
A positive covariance says that when the first quantity runs above its centre, the second tends to run above its centre too. A negative covariance says the opposite. A covariance near zero says the two wander without a settled relationship, although it does not, by itself, prove independence; two quantities can be plainly related and still show zero covariance if the relationship is curved and symmetric. Correlation rescales covariance to the interval between minus one and one, which makes comparisons across pairs of different sizes fair.
Positive definite, in plain terms
A genuine covariance matrix must be positive definite. In plain terms this says that every direction you might look along carries a positive spread. Equivalently, no quantity formed as a weighted sum of the listed variables can have zero variance unless every weight is zero. When a matrix fails this test, the numbers it contains cannot be read as a real arrangement of covariances, and any density built from it would collapse.
The link to the density
Once a centre and a positive definite covariance matrix are fixed, the multivariate normal density is determined. Its exponent is a squared distance, but a distance measured with the matrix as its ruler, so that directions of large spread are treated as short journeys. This is why the covariance matrix is so often called the shape of the density.
Reading a matrix at a glance
Three habits make a matrix quick to read. First, check the diagonal: the variances should be positive and roughly the size you expect. Second, check the symmetry: each off-diagonal cell must equal its mirror. Third, check the off-diagonal pattern: a block of large positive numbers suggests a group of quantities that move together. Practise those three glances and the matrix stops being a wall of symbols and becomes a small portrait of the data.
Two cautionary examples
Consider two quantities that both vary wildly but never in step. Their covariance sits near zero, and a reader might mistake this for independence. It is not. Independence is a stronger statement about how every value of one quantity pairs with every value of the other, and a near-zero covariance can hide a curved relation that is symmetric about the centre. The matrix reports association only in its straight-line form.
Consider next three quantities in which the third is nearly a sum of the first two. The matrix then contains a near-dependence among its rows, and its smallest direction of spread approaches zero. A density built on such a matrix becomes exceedingly tall and narrow, and small changes in the entries move it a great deal. Noticing this near-dependence before fitting anything is one of the quiet skills the matrix rewards.