Setting up a CI server is not a user story. It doesn't deliver any value to the customer on its own, and thus is not really something that should be in the customer backlog. It should be rolled into the first story done on the project, as it's a part of setting up the development environment. Similarly, you probably didn't have a story for creating the git repository, nor one for installing your text editor.
You work with your customer to decide what end-user bugfixes and features to prioritize, but it's your job to make technical decisions. That's why they hired you. Don't push those decisions back onto them.
The problem isn't so much of setting up a CI server as much as the fact that the task requires some time and resources from potentially other people. It might need approval for a simple VM with some disks, for example. But more importantly, use of such a server oftentimes means you need some time from other engineers to setup CI tasks including QA, security, etc. Anything that requires team consensus potentially requires meetings and some formalization.
But really, management that doesn't keep up conceptually with the business trends of software engineering management are low performers in the same way as engineers that refuse to learn how to improve their code even if it doesn't directly impact their immediate codebase (functional programming patterns as an embedded software engineer comes to mind).
You work with your customer to decide what end-user bugfixes and features to prioritize, but it's your job to make technical decisions. That's why they hired you. Don't push those decisions back onto them.