The Nowcasting Problem

State Space Models & the Kalman Filter — Session 1

Tyler Sotomayor · Columbia University

What did the economy do this morning?

  • GDP for quarter Q: published ~4 weeks after Q ends, then revised for years
  • Meanwhile, every single day:
    • weekly unemployment claims
    • daily asset prices, card spending, search behavior
    • hourly electricity load
    • continuous news text
  • Nowcasting = estimating the present from that flood

\hat{y}_{Q\mid t} = \mathbb{E}\left[\, y_Q \mid \Omega_t \,\right]

Three things make it hard

  1. Mixed frequency — quarterly target, monthly/weekly/daily indicators
  2. The ragged edge — asynchronous releases, different publication lags: the newest data is always incomplete
  3. Real-time discipline — \Omega_t is the data as it looked then, not as revised today

Rule for this course: a backtest on revised data is inadmissible evidence. We test against archived vintages (ALFRED).

One framework eats all three

State space form (Giannone–Reichlin–Small 2008):

\begin{aligned} \alpha_{t+1} &= T\,\alpha_t + R\,\eta_t, & \eta_t &\sim \mathcal{N}(0, Q)\\ y_t &= Z\,\alpha_t + \varepsilon_t, & \varepsilon_t &\sim \mathcal{N}(0, H) \end{aligned}

  • Latent state \alpha_t = “the economy right now”
  • Each indicator = a noisy, frequency-appropriate measurement of it
  • AR, VAR, factor models, trend/cycle, mixed frequency: all special cases of (T,R,Q,Z,H)

The Kalman filter — predict

Beliefs: \alpha_t \mid Y_s \sim \mathcal{N}(a_{t\mid s},\, P_{t\mid s}) — two moments are sufficient.

Prediction step — push beliefs through the dynamics:

a_{t+1\mid t} = T\, a_{t\mid t}, \qquad P_{t+1\mid t} = T\, P_{t\mid t}\, T' + R Q R'

Uncertainty grows while we wait for data.

The Kalman filter — update

Data arrives; compute the innovation (the surprise):

v_{t+1} = y_{t+1} - Z\, a_{t+1\mid t}, \qquad F_{t+1} = Z P_{t+1\mid t} Z' + H

Revise beliefs in proportion — the Kalman gain:

a_{t+1\mid t+1} = a_{t+1\mid t} + K_{t+1} v_{t+1}, \qquad K_{t+1} = P_{t+1\mid t} Z' F_{t+1}^{-1}

K = how much attention this release deserves — derived, not assumed.

Why nowcasters love this machine

  • Missing data costs nothing — no observation? skip the update, let uncertainty fan out. That is the ragged edge, solved.
  • News decomposition — updates are linear in surprises, so nowcast revisions attribute exactly to each release (“claims moved it -0.2pp”)
  • Likelihood for free — the same recursion delivers ML estimation

The local level model

\alpha_{t+1} = \alpha_t + \eta_t, \qquad y_t = \alpha_t + \varepsilon_t

  • Filter becomes an adaptive exponential smoother
  • One number rules it: signal-to-noise q = \sigma^2_\eta / \sigma^2_\varepsilon
  • Steady-state gain: \;\bar{K} = \dfrac{\bar P}{\bar P + \sigma^2_\varepsilon}, \;\dfrac{\bar{P}}{\sigma_\varepsilon^2} = \dfrac{q + \sqrt{q^2+4q}}{2}
  • High q → chase the data · Low q → trust the model

Now you drive

Open the playground:

tylersotomayor.com → courses → Kalman filter playground

Move \sigma_\eta and \sigma_\varepsilon. Watch the gain converge to \bar K. Switch on the data outage and watch the ragged-edge logic happen.

Next session

  • Many indicators, one latent factor: the dynamic factor model
  • Replicating the Weekly Economic Index (Lewis, Mertens, Stock & Trivedi 2022)
  • Same Kalman engine — only (T,R,Q,Z,H) change
  • Before class: read Lewis et al. (2022); do Practicum 0