This is 100% correct. As a hiring manager, the attitude of candidates was probably equally important as their skills when it comes to the decision of whether to hire. The baseline of is this someone I would enjoy working with every day? Is this someone my team would enjoy working with every day? If it's not a solid yes I'm going to pass on you.
This must be some kind of cultural difference in the purpose of jobs. Where I come from, the purpose of work is get things done, put food on the table, and hopefully not make the world a worse place in the process. Enjoying the company of everyone you work with is not a significant consideration.
You might be missing out on a lot of great people because of your high first impression bar. And I don’t mean good workers but bad social skills, but people who are actually really chill to work with even if they don’t show it so quickly. This is why most culture fit filters really don’t work well, people have a wrong or too narrow impression of who they would enjoy working with everyday.