I think the way his dad psychologically treated him was much worse, because it ruined his whole self-esteem (even if he pretends to have the biggest ego on earth). He spends his whole life to prove him wrong. We don't know what else he did to him. And yes, I am talking about sexual abuse. Not that I would assume that FOX would dare to touch that subject (they are conservative cowards in any way).
Maybe now that his dad died he has the liberty to live for his own goals. If he still knows what they are.