For me, it wasn't the tones or sounds that made Mandarin tricky - once you get used to letting your mouth/tongue make shapes/be places you've trained it not to be the vowle/consonants aren't that bad, and fi you've ever played an instrument/sung, tones are manageable. It was the stupid pictographs. You're literally just memorizing shapes that didn't correspond to anything was the absolute worst. It got better once you learned to recognize the basic forms that were getting composed into parts of other characters, but that really discouraged me/was the hardest part. I basically gave up on written Chinese and focused on pinyin, because that made sense to my head.
Source: took two years of Mandarin and lived in Taiwan for a few months
David Moser wrote an essay telling people to focus more on Pinyin and not stress about characters until much later. I totally agree with him. Maybe one day the CCP will force everyone to Pinyin and I can finally read without having to look up half a sentence.
Moving everything to pinyin probably would introduce a different set of difficulties for a foreigner: while a native Chinese speaker knows the spoken language and context well enough to disambiguate homophones, it's unlikely a Chinese learner would have the same ability.
Source: took two years of Mandarin and lived in Taiwan for a few months