The performance impact depends substantially on whether you've configured it to fsync the binlog on every group commit.
Also, it's important to consider that replication and backups serve different purposes. Backups alone are insufficient for high availability, change data capture, point-in-time recovery / undoing a bad change, etc.
Turning it off cause you're not using it seems reasonable, but I'm surprised it has a big effect on performance. Sequential appends to a file are pretty easy as long as you're not doing so many writes per second that there's contention on the write.
I've used bin log for almost decades and never experienced a big performance impact. This even holds for write heavy MySQL instances in ancient times where servers had spinning disks.
Thinking back about it, I think the biggest issue was the size, not performance. For a write-intensive app, the bin long quickly got to tens of GBs and filled the entire disk, which was a problem when running the app on smaller VPSs.
You can tune binlog_expire_logs_seconds to control how long old binlog files stay around. The default is 2592000 seconds (30 days) which is often too long.