The Kalman Filter is a beautiful algorithm, probably among the top math algorithms of the 20th century. It gives a very fast solution to a specific optimization problem (optimal estimation of hidden/indirectly observed states in a linear dynamic system). But for practitioners, the computations and derivations are far less relevant than recognising the circumstances where state space / Kalman model are relevant/appropriate, and how to set up such a model (defining the transition and observation matrices of the model).
In this sense, the Kalman Filter in time-dynamic prediction/forecasting problems is like gradient descent in ML prediction problems, or the solution to the Normal equations in linear regression or FFT for discrete Fourier transforms. The algorithms are mathematical pieces of art, but the useful practical skills are how to set up your problems so that they're amendable to the solutions.
In case of Kalman / state space models, the main things that are estimated are positions of things that move in time, but not (directly) observed. These "things" can be concrete (drones, rockets in space, robots on Mars) or abstract (an idealized compuer cursor/pointer, the water level in a river, inflation or unemployment in macro-economics). When concrete "things" are estimated by traditional engineers and the dynamics are governed by physics, the tool of choice is usually (employer-paid) Matlab. For fuzzier economics and forecasting problems, nowadays there's a very good state space / Kalman Filter implementation in Python statsmodels by Chad Fulton[1], which closely follows the academic works from Harvey, Durbin and Koopman. The challenge in these problems is usually setting up a plausible underlying dynamics model, more so than the estimation/prediction part. Just like in linear regression, finding appropriate features and functional forms are the main practical challenge, not calculating the parameters and actual predictions.
In this sense, the Kalman Filter in time-dynamic prediction/forecasting problems is like gradient descent in ML prediction problems, or the solution to the Normal equations in linear regression or FFT for discrete Fourier transforms. The algorithms are mathematical pieces of art, but the useful practical skills are how to set up your problems so that they're amendable to the solutions.
In case of Kalman / state space models, the main things that are estimated are positions of things that move in time, but not (directly) observed. These "things" can be concrete (drones, rockets in space, robots on Mars) or abstract (an idealized compuer cursor/pointer, the water level in a river, inflation or unemployment in macro-economics). When concrete "things" are estimated by traditional engineers and the dynamics are governed by physics, the tool of choice is usually (employer-paid) Matlab. For fuzzier economics and forecasting problems, nowadays there's a very good state space / Kalman Filter implementation in Python statsmodels by Chad Fulton[1], which closely follows the academic works from Harvey, Durbin and Koopman. The challenge in these problems is usually setting up a plausible underlying dynamics model, more so than the estimation/prediction part. Just like in linear regression, finding appropriate features and functional forms are the main practical challenge, not calculating the parameters and actual predictions.
[1] http://www.chadfulton.com/fulton_statsmodels_2017/sections/1...