The specific worry about Net Neutrality was that ISPs would use their monopoly power to censor specific sources and/or self-preference their own businesses. It's something that should have been expanded to large online platforms rather than being disposed of entirely.
As you said, it was a worry, but ending Net Neutrality about enforcing government censorship was never even an argument being made at all by any sides of the issue.
Justice Brett Kavanaugh[0] argued that Net Neutrality specifically violated the 1st Amendment because Comcast should decide what speech they do and don't retransmit.
