Chapter 20: The Kalman Filter

Estimating Unobserved Components (Potential GDP, NAIRU)

“The Kalman filter is the Swiss Army knife of state estimation: it handles missing data, real-time updating, and likelihood evaluation — all in a unified recursive algorithm.”

Cross-reference: Principles Ch. 6 (potential GDP and the output gap as unobserved); Appendix B (DSGE estimation via MLE with Kalman filter) [P:Ch.6, P:AppB]

20.1 The State-Space Form¶

Many macroeconomic concepts are not directly observed: potential output, the natural rate of unemployment (NAIRU), the natural rate of interest, and long-run inflation expectations are all inherently unobservable quantities inferred from observed data. The state-space model provides the unified statistical framework for this inference; the Kalman filter is the optimal algorithm for doing so recursively in real time.

Definition 20.1 (Linear Gaussian State-Space Model). The state-space model consists of two equations:

where:

• $\mathbf{y}_t \in \mathbb{R}^p$ is the observation vector (observed variables: GDP, unemployment, interest rates).

• $\boldsymbol\alpha_t \in \mathbb{R}^m$ is the state vector (unobserved variables: potential output, NAIRU, natural rate).

• $H \in \mathbb{R}^{p\times m}$ is the observation matrix (links states to observables).

• $F \in \mathbb{R}^{m\times m}$ is the transition matrix (law of motion of the state).

• $R \in \mathbb{R}^{p\times p}$ is the measurement noise covariance.

• $Q \in \mathbb{R}^{m\times m}$ is the state noise covariance (process noise).

• $\mathbf{d}$ and $\mathbf{c}$ are constant terms (intercepts).

Assumption: $\boldsymbol\varepsilon_t$ and $\boldsymbol\eta_t$ are mutually uncorrelated at all leads and lags; the initial state $\boldsymbol\alpha_1 \sim \mathcal{N}(\mathbf{a}_1, P_1)$.

The state-space form is extremely general: every ARMA model, every DSGE model (after log-linearization), and every unobserved components (UC) model can be written in this form. Chapter 28 will show that the log-linearized DSGE model reduces to exactly this state-space form, with the Kalman filter providing the likelihood.

20.2 The Kalman Filter Algorithm¶

The Kalman filter solves the linear filtering problem: given observations $\mathbf{y}_1, \ldots, \mathbf{y}_t$, what is the best (minimum mean squared error) estimate of the unobserved state $\boldsymbol\alpha_t$?

Definition 20.2 (Filtered Estimate). The filtered mean $\mathbf{a}_{t|t} \equiv \mathbb{E}[\boldsymbol\alpha_t | \mathbf{y}_1, \ldots, \mathbf{y}_t]$ and filtered covariance $P_{t|t} \equiv \text{Var}[\boldsymbol\alpha_t | \mathbf{y}_1, \ldots, \mathbf{y}_t]$ are the posterior mean and variance of the state conditional on all information up to time $t$.

The predicted mean $\mathbf{a}_{t|t-1} \equiv \mathbb{E}[\boldsymbol\alpha_t|\mathbf{y}_1,\ldots,\mathbf{y}_{t-1}]$ and predicted covariance $P_{t|t-1} \equiv \text{Var}[\boldsymbol\alpha_t|\mathbf{y}_1,\ldots,\mathbf{y}_{t-1}]$ are the prior to the $t$-th observation.

Algorithm 20.1 (Kalman Filter).

Initialization: $\mathbf{a}_{1|0} = \mathbf{a}_1$, $P_{1|0} = P_1$.

For $t = 1, 2, \ldots, T$:

Prediction step (propagate the prior through the transition equation):

Prediction error and variance:

Kalman gain:

Update step (incorporate the new observation):

In APL, the prediction and update steps are single matrix expressions:

The full Kalman filter iterates these two steps over $T$ observations, storing the filtered estimates $\{\mathbf{a}_{t|t}, P_{t|t}\}$ and the innovations $\{v_t, F_t\}$.

20.3 Derivation of the Kalman Gain as MMSE Estimator¶

Theorem 20.1 (Optimality of the Kalman Gain). The Kalman gain $K_t = P_{t|t-1}H'F_t^{-1}$ minimizes the mean squared error of the updated state estimate among all linear functions of $(\mathbf{a}_{t|t-1}, \mathbf{y}_t)$.

Proof. The general linear update: $\hat{\boldsymbol\alpha}_{t|t} = \mathbf{a}_{t|t-1} + G\mathbf{v}_t$ for some $m\times p$ matrix $G$. The update error:

Since $\mathbf{v}_t = H(\boldsymbol\alpha_t - \mathbf{a}_{t|t-1}) + \boldsymbol\varepsilon_t$:

The MSE matrix:

Taking the derivative with respect to $G$ and setting to zero:

Solving: $P_{t|t-1}H' = G(HP_{t|t-1}H' + R) = GF_t$, so $G^* = P_{t|t-1}H'F_t^{-1} = K_t$. $\square$

The Joseph form for numerical stability: The update equation $P_{t|t} = (I-K_tH)P_{t|t-1}$ can become non-positive-definite due to numerical errors. The numerically stable Joseph form is:

This is algebraically equivalent but guaranteed to remain symmetric positive semi-definite.

20.4 The Kalman Smoother¶

The Kalman filter produces filtered estimates $\mathbf{a}_{t|t}$ — based on information up to time $t$. The Kalman smoother produces smoothed estimates $\mathbf{a}_{t|T}$ — based on all $T$ observations. Smoothed estimates are generally more accurate and are required for computing the log-likelihood efficiently.

Algorithm 20.2 (Rauch–Tung–Striebel Smoother).

Run the Kalman filter forward to obtain $\{\mathbf{a}_{t|t}, P_{t|t}\}$ for $t = 1, \ldots, T$.

Then run backward for $t = T-1, T-2, \ldots, 1$:

with terminal condition $\mathbf{a}_{T|T}$ and $P_{T|T}$ from the filter.

Interpretation: The smoother revises the filtered estimate at each date using information from all subsequent observations. For the output gap, the smoother tells us what we would have estimated for 2006Q2 if we had known the entire 2007–2012 sequence of GDP data — a retrospective revision.

20.5 Maximum Likelihood Estimation via the Kalman Filter¶

The Kalman filter generates the prediction error decomposition of the log-likelihood — the key formula connecting the filter to parameter estimation.

Theorem 20.2 (Prediction Error Decomposition of Log-Likelihood). Under the Gaussian assumption, the log-likelihood of the state-space model parameters $\boldsymbol\theta = (F, H, Q, R, \mathbf{a}_1, P_1)$ given observations $\{\mathbf{y}_t\}_{t=1}^T$ is:

where $\mathbf{v}_t = \mathbf{y}_t - H\mathbf{a}_{t|t-1} - \mathbf{d}$ and $\mathbf{F}_t = HP_{t|t-1}H' + R$ are the prediction errors and their covariances generated by the Kalman filter.

Proof sketch. By the chain rule of conditional probability:

Each conditional $p(\mathbf{y}_t | \cdot)$ is Gaussian with mean $H\mathbf{a}_{t|t-1} + \mathbf{d}$ (the predicted observation) and variance $\mathbf{F}_t$. Taking logs gives the stated formula. $\square$

Practical estimation: MLE maximizes $\mathcal{L}(\boldsymbol\theta)$ over model parameters $\boldsymbol\theta$. The gradient $\partial\mathcal{L}/\partial\boldsymbol\theta$ can be computed analytically (or by automatic differentiation) and used with a quasi-Newton optimizer (BFGS, Chapter 24). The Kalman filter evaluates $\mathcal{L}$ at each candidate $\boldsymbol\theta$ in $O(Tm^3)$ time — fast enough for models with state dimension $m \lesssim 50$.

20.6 Macroeconomic Applications¶

20.6.1 Estimating the Output Gap¶

The simplest unobserved components (UC) model for the output gap:

State vector: $\boldsymbol\alpha_t = (\tau_t, g_t, c_t, c_{t-1})'$. Observation: $y_t = \tau_t + c_t$.

The four unknowns are the signal-to-noise ratios $\{q_\tau = \sigma_\tau^2/\sigma_c^2, q_g = \sigma_g^2/\sigma_c^2\}$ and AR coefficients $(\phi_1, \phi_2)$. MLE via the Kalman filter estimates these from GDP data.

20.6.2 The Laubach–Williams Natural Rate Model¶

Laubach and Williams (2003) estimate the natural rate of interest $r^n_t$ as an unobserved state in a system that links $r^n_t$ to the output gap and inflation:

The system is cast in state-space form with unobserved $r^n_t$ and estimated by MLE. The LW estimates show $r^n_t$ declining from approximately 3.5% in the early 1980s to near 0% by 2015 — consistent with the secular stagnation narrative of Principles Ch. 39 [P:Ch.39.4].

20.7 Worked Example: Output Gap via UC Model¶

Python¶

Julia¶

using LinearAlgebra, Optim

function kalman_filter(y, F, H, Q, R, a0, P0)
    T = length(y); m = length(a0); p = size(H, 1)
    a = copy(a0); P = copy(P0); log_lik = 0.0
    a_store = zeros(T, m)
    for t in 1:T
        a_pred = F * a; P_pred = F * P * F' + Q
        v = [y[t]] - H * a_pred
        Fv = H * P_pred * H' + R
        log_lik -= 0.5*(log(det(Fv)) + dot(v, Fv\v) + p*log(2π))
        K = P_pred * H' * inv(Fv)
        a = a_pred + K * v; P = (I(m) - K*H) * P_pred
        a_store[t,:] = a
    end
    log_lik, a_store
end

println("Kalman filter implemented. Use Optim.jl to maximize log-likelihood over parameters.")

R¶

# R — Kalman filter via KFAS
library(KFAS)

# Simple local level model (trend extraction)
T <- 200; set.seed(42)
y <- cumsum(rnorm(T, 0.005, 0.002)) + rnorm(T, 0, 0.01)  # trend + noise

model <- SSModel(y ~ SSMtrend(2, Q=list(matrix(NA),matrix(NA))),
                 H=matrix(NA))
fit <- fitSSM(model, inits=c(log(0.001), log(0.001), log(0.01)))
out <- KFS(fit$model)

plot(out$alphahat[,1], type='l', col='red', main='Filtered trend (KFAS)')
lines(y, col='blue', alpha=0.5)
legend('topleft', c('Observed','Filtered trend'), col=c('blue','red'), lty=1)

20.8 Programming Exercises¶

Exercise 20.1 (APL — Full Kalman Filter Loop)¶

Implement the complete Kalman filter in APL as a dfn that takes (y F H Q R a0 P0) and returns the filtered states and log-likelihood. Use {predict_step ⍵} and {update_step ⍵} as helper dfns called within the main loop. The log-likelihood accumulation should use ln_det F_t + v'F^{-1}v at each step. Verify on simulated data from a known state-space model.

Exercise 20.2 (Python — Laubach–Williams Replication)¶

Implement a simplified Laubach–Williams (2003) model:

Cast in state-space form with state $(\hat x_t, \pi_t, r^n_t, g_t)'$ and observations $(x_t, \pi_t)'$. Estimate parameters by MLE via the Kalman filter. Plot the estimated natural rate path over 1960–2019 and compare to the LW published estimates.

Exercise 20.3 (Julia — Kalman Smoother)¶

Extend the Kalman filter implementation to include the RTS smoother (backward pass). Compare: (a) filtered output gap $c_{t|t}$ (using only information up to $t$); (b) smoothed output gap $c_{t|T}$ (using all information). Show that the smoothed estimates have lower uncertainty (smaller $P_{t|T}$ diagonal elements) than the filtered estimates.

Exercise 20.4 — DSGE Likelihood ($\star$)¶

The log-linearized NK model from Chapter 18 can be written in state-space form with state $\boldsymbol\alpha_t = (\hat x_t, \hat\pi_t, r^n_t)'$, observation $\mathbf{y}_t = (\hat x_t + e_t^x, \hat\pi_t + e_t^\pi)'$ (output gap and inflation observed with measurement error). (a) Write the $F$, $H$, $Q$, $R$ matrices explicitly in terms of the NK model parameters. (b) Simulate 100 periods of data from the model with $\phi_\pi = 1.5$. (c) Use the Kalman filter to compute $\mathcal{L}(\phi_\pi)$ for $\phi_\pi \in [0.5, 3.0]$ and plot the likelihood surface. Verify the MLE is near the true value.

20.9 Chapter Summary¶

Key results:

• The state-space model has a measurement equation $\mathbf{y}_t = H\boldsymbol\alpha_t + \boldsymbol\varepsilon_t$ and transition equation $\boldsymbol\alpha_{t+1} = F\boldsymbol\alpha_t + \boldsymbol\eta_t$; virtually all dynamic models reduce to this form.

• The Kalman filter is a two-step recursion: predict step ($\mathbf{a}_{t|t-1} = F\mathbf{a}_{t-1|t-1}$, $P_{t|t-1} = FPF'+Q$) and update step ($\mathbf{a}_{t|t} = \mathbf{a}_{t|t-1}+K_t\mathbf{v}_t$, $P_{t|t} = (I-K_tH)P_{t|t-1}$).

• The Kalman gain $K_t = P_{t|t-1}H'F_t^{-1}$ is the MMSE linear estimator (proved as Theorem 20.1).

• The RTS smoother revises filtered estimates backward in time, producing more accurate estimates $\mathbf{a}_{t|T}$ using all $T$ observations.

• The prediction error log-likelihood $\mathcal{L} = -\frac{1}{2}\sum_t[\ln|F_t| + v_t'F_t^{-1}v_t]$ enables MLE estimation of state-space model parameters via the Kalman filter.

• In APL: P_pred ← (F+.×P+.×⍉F)+Q and K ← P+.×(⍉H)+.×⌹Fv — the prediction and gain steps as single expressions; the full filter as {update predict ⍵}⍣T⊢init_state.

Next: Chapter 21 — GMM and Maximum Likelihood