They literally "do context", as they do in-context learning. Also, factuality can effectively be solved by chain-of-thought prompting[0] and the use of external tools[1], also see toolformerzero[2] for a simple example.
The "predicting words" part of the language models is a building block, but the actual applications being built around the LLMs are using that predictive ability in very dynamic ways along with external memory, feedback and sophisticated control methods to attenuate and channel their behavior.
