Difficult to speak (either Mandarin or Cantonese) for most Westerners, too, since we're not used to tonal languages.

But the writing system is the killer. It allows for tremendous nuance of expression, but the absurd complexity isn't worth it, imo. Japanese kanji (the more-complex characters imported from China) is similarly frustrating, but at least there are fewer characters: c. 3800, and most Japanese adults know "only" about a thousand. The kana (hiragana for native Japanese words, katakana for "import words") is easier to deal with, there being a little over in each. Kana are more of an actual (phonetic) alphabet, rather than being essentially pictographs.