Decades ago, prior to the existence of personal computers, when a "computer" was a glassed-in room staffed by lab-coat-wearing technicians (picture John Von Neumann standing next to the first stored-program computer: https://www.theguardian.com/technology/2012/feb/26/first-com...), someone reduced an entire printed book (or more than one) to a word-token decision tree, at great cost and effort, just to see what would happen.
I can't find the original paper, but with an appropriate amount of pseudorandomness to avoid dead ends, this primitive algorithm would generate the occasional sentence that almost made sense and that bore little resemblance to the original data.
Because of the state of computer technology it was a massive effort and a source of general astonishment. I suspect we're now recreating that minimal environment, this time with better ways to curate the data for small size and maximum drama.
Let's remember that a modern GPT isn't far removed from that scheme -- not really.
Decades ago, primarily for show, computer technicians wore lab coats, in a glassed-in facility at the University of Colorado. To gain access to the computer one would punch a set of 80-column cards and pass them through a little window into the hands of a person who apparently lived in a separate, elevated, dimension. Debugging an otherwise trivial program was often a multi-week nightmare.
It was my first exposure to the world of computing. Ten years later, hand calculators appeared and the ridiculousness of the entire show was revealed for all to see.
I can't find the original paper, but with an appropriate amount of pseudorandomness to avoid dead ends, this primitive algorithm would generate the occasional sentence that almost made sense and that bore little resemblance to the original data.
Because of the state of computer technology it was a massive effort and a source of general astonishment. I suspect we're now recreating that minimal environment, this time with better ways to curate the data for small size and maximum drama.
Let's remember that a modern GPT isn't far removed from that scheme -- not really.