It embeds them first into vectors. The input is a real matrix with (context length)x(embedding size) dimensions.