“Daddy Diller” was a more-or-less human man.
Daddy Diller, later described by his daughter the witch Phyllis Diller as “a real swinger”, was a monstrous hunchbacked bellringer. At some point, he married a woman, sister to Madam Mim and Medusa, who later became an undead mummy (and an elegant, if hideous, one at that). Diller had, by all appearances, a brother in the form of the wolfman Harry, Phyllis's Uncle. Unlike his undead wife, it would seem that Diller had died by 1952, as Phyllis spoke of him in the past tense when an image of him appeared on her crystal-ball TV.
Behind the scenes
He is, of course, patterned after popular film depictions of Victor Hugo's Quasimodo, long before Disney debuted their own, actual version of the character; the Notre-Dame-style Hunchback was, for a time, part of the roster of classic monsters alongside the Wolfman and the Vampire, though he was later downplayed.