I think Sam is referring to transition from "Deep" to "Long" learning [1]. What new emergent properties, if any, will 1 billion tokens unlock?

[1] https://hazyresearch.stanford.edu/blog/2023-03-27-long-learn...