The Nowcasting Problem

State Space Models & the Kalman Filter — Session 1

Tyler Sotomayor · Columbia University

What did the economy do this morning?

  • GDP for quarter Q: published ~4 weeks after Q ends, then revised for years
  • Meanwhile, every single day:
    • weekly unemployment claims
    • daily asset prices, card spending, search behavior
    • hourly electricity load
    • continuous news text
  • Nowcasting = estimating the present from that flood

\hat{y}_{Q\mid t} = \mathbb{E}\left[\, y_Q \mid \Omega_t \,\right]

Forecast, nowcast, or backcast?

The name depends on where the target’s reference period sits relative to the information cutoff.

Target period at cutoff r Name Question
Future Forecast What will happen?
Current or recently ended Nowcast What is happening now?
Earlier but incomplete Backcast What was happening then?

The model can be identical in all three cases. What changes is the target date relative to the information available at vintage r.

Every release lives on two clocks

Clock What it records
Reference period When the economic activity occurred
Release date When a value for that period became public
  • A vintage is the dataset available at one chosen cutoff date.
  • July payroll employment may be released in August and revised in September.
  • It always measures July—but the value visible to the forecaster changes.

Real-time rule: place a value in the model by reference period; admit it to the information set only on its release date.

Three things make it hard

  1. Mixed frequency — quarterly target, monthly/weekly/daily indicators
  2. The ragged edge — asynchronous releases, different publication lags: the newest data is always incomplete
  3. Real-time discipline — \Omega_t is the data as it looked then, not as revised today

Rule for this course: a backtest on revised data is inadmissible evidence. We test against archived vintages (ALFRED).

One framework eats all three

State space form (Giannone–Reichlin–Small 2008):

\begin{aligned} \alpha_{t+1} &= T\,\alpha_t + R\,\eta_t, & \eta_t &\sim \mathcal{N}(0, Q)\\ y_t &= Z\,\alpha_t + \varepsilon_t, & \varepsilon_t &\sim \mathcal{N}(0, H) \end{aligned}

  • Latent state \alpha_t = “the economy right now”
  • Each indicator = a noisy, frequency-appropriate measurement of it
  • AR, VAR, factor models, trend/cycle, mixed frequency: all special cases of (T,R,Q,Z,H)

The Kalman filter — predict

Beliefs: \alpha_t \mid Y_s \sim \mathcal{N}(a_{t\mid s},\, P_{t\mid s}) — two moments are sufficient.

Prediction step — push beliefs through the dynamics:

a_{t+1\mid t} = T\, a_{t\mid t}, \qquad P_{t+1\mid t} = T\, P_{t\mid t}\, T' + R Q R'

Uncertainty grows while we wait for data.

The Kalman filter — update

Data arrives; compute the innovation (the surprise):

v_{t+1} = y_{t+1} - Z\, a_{t+1\mid t}, \qquad F_{t+1} = Z P_{t+1\mid t} Z' + H

Revise beliefs in proportion — the Kalman gain:

a_{t+1\mid t+1} = a_{t+1\mid t} + K_{t+1} v_{t+1}, \qquad K_{t+1} = P_{t+1\mid t} Z' F_{t+1}^{-1}

K = how much attention this release deserves — derived, not assumed.

Why nowcasters love this machine

  • Missing observations are handled directly — no release? skip that update and let uncertainty fan out. The release calendar determines which rows are available at each vintage.
  • News decomposition — updates are linear in surprises, so nowcast revisions attribute exactly to each release (“claims moved it -0.2pp”)
  • Likelihood for free — the same recursion delivers ML estimation

The local level model

\alpha_{t+1} = \alpha_t + \eta_t, \qquad y_t = \alpha_t + \varepsilon_t

  • Filter becomes an adaptive exponential smoother
  • One number rules it: signal-to-noise q = \sigma^2_\eta / \sigma^2_\varepsilon
  • Steady-state gain: \;\bar{K} = \dfrac{\bar P}{\bar P + \sigma^2_\varepsilon}, \;\dfrac{\bar{P}}{\sigma_\varepsilon^2} = \dfrac{q + \sqrt{q^2+4q}}{2}
  • High q → chase the data · Low q → trust the model

Work Through the Session 1 Playground

Open the Session 1 playground, then use the five labs in order:

  1. calculate one release update;
  2. filter through missing observations;
  3. build a real-time data vintage;
  4. aggregate monthly growth into quarterly GDP; and
  5. decompose a nowcast revision into release news.

Exercises, hints, and worked solutions

  1. Start with the Session 1 exercises.
  2. Open Hint only when you need a direction.
  3. Compare your work with the Detailed solution, which explains every step and the intuition behind it.

The website keeps both panels collapsed. The solutions companion also provides a printable PDF with all 10 hints and solutions expanded.

Next session

  • Many indicators, one latent factor: the dynamic factor model
  • Replicating the Weekly Economic Index (Lewis, Mertens, Stock & Trivedi 2022)
  • Same Kalman engine — only (T,R,Q,Z,H) change
  • Before class: read Lewis et al. (2022); do Practicum 0