What you described is basically an input/output pair. The input is the sentence so far, and the output is the next token. You build your dataset by splitting the raw text corpus into sentences, paragraphs or documents, and for each of these chunks generate input/target pairs by taking the sentence up to the Nth token as input and that token as output. You do this for each token in your corpus' chunks.

For further reference you can lookup "next-token prediction objective".