Well yeah, except for the whole race angle
which isn't exactly a minor part of the film. But Peele is apparently an admitted horror geek so there's a lot of visible influences at play, particularly the various versions of the age-old bodysnatchers premise. I'm completely with you though- a horror film doesn't need to have extreme violence. Or *any* violence, for that matter.