Re: I was hoping there'd be something I really liked on this list
The head in Japed is a statue head, hence the Simpson's reference, so it shouldn't be leaking stuff. I think those are just leaves of a plant he's running through. As far as "really really early stuff," that one is his third published novel, and it's pretty good but not essential.
He wrote classic short fiction before Solar Lottery. SL was 1955, his first few stories are from 1952, and Second Variety came out in 1953. That's his most anthologized story.