Dockerfiles are just a few lines of shell script and a single shell command away from a fully reproducible image that can be run anywhere. They are also reusable across projects. This is a point where you can easily carry over your productivity if you are constantly starting new projects. Your latest project will be much further ahead on day 1 than your first project. Compare that to manually typing in commands on a server to set your production environment up. You'll forget things (massive drop in productivity) or have to write them down (massive drop in productivity). You'll redo the same stuff over and over (massive drop in productivity). For what gain? Just to stay purist?

You are taking the advice way too far. The advice is do things that increase your productivity and many things that increase productivity are also easy to setup once you know them. I'm not saying that you should learn containers for the sake of your startup. I am saying that tools you already know well often require very little effort to setup and pay dividends immediately.