You do have a point about them being more pop-oriented than most hip hop groups but I don't think that's really an issue for me, since I like a lot of poppy hip hop, The Beastie Boys being an obvious example (though they're basically in their own category).
Yeah I should probably listen to "Speakerboxxx" at some point. "Love Below" has its moments but tons of filler and generally can barely be classified as hip hop. I mean, he pulls it off pretty damn well but I still don't really want a new version of Prince when I listen to Outkast (especially since I'm not really a Prince fan either).
The main reason I prefer the debut is because it sounds the closest to the type of hip hop I listen to, which makes sense- it's more modestly produced and they were still developing their style. Plus it has that nice mid-90's vibe.
"I'm sure I'm in the minority for preferring their later albums."
Yes, this is where you differ from the typical Outkast fan, who tends to rank "Aquemini" or "ATLiens" above everything else. And I think my opinion that the first 3 are better than what followed is a pretty common one as well.