Re: Been listening to Milton Babbitt for the first time in ages,
I think Trung is partly right by saying that the ear naturally hears stuff that doesn't obey any of the physics-based rules of harmony as jarring. One could counter that there are regional forms of music that don't fit comfortably into Western harmonic language, but as far as I can tell most other forms of music have at least some other organizing principle, whether it be a scale, a set of rhythmic ideas, a certain set of timbres. I think the problem with serialism is that the organizing principle is purely academic, and not even smartly academic. It's trying to find something "pure" out of the equal-tempered scale without recognizing that that scale is a functional compromise to begin with.
The other reason the ears of the general listener haven't adjusted to serialism is that the general listener has never bothered to listen to it, and even the subset that has bothered has mostly not done so for very long.
I've also never heard a note of Babbitt and probably heard under half an hour of serialism in my life, so am not really qualified to answer this question.