I think the two characters you cite as the weakest were probably meant to become more interesting later - Shepherd Book with his unclear past (implied military background) and Inara with her romantic tension with Mal. I like Inara, in part because the actor is very good in the role. By the same token, I'd cite the doctor as the weakest character. (Incidentally, Wikipedia informs me that Zac Efron's first onscreen performance was as the doctor in a flashback.)
I think the common denominator between Whedon's shows, at their best, is their ability to marry humorous light tone with serious emotional stakes (without the combination feeling forced) and to gradually tease out the unique implications of their premises (the latter of which is where Firefly falls short, since the "space Western" idea is not all that interesting).
You think Caleb on BtVS was a better villain than the Mayor? I'd put him near the bottom of the list - he's very underwritten.