True. Casual way of agreeing. Grammatically the phrase is a double-negative so ‘not’ and ‘untrue’ cancel out to become=true. The double negative modifies what otherwise might sound like a severe comment. (Compare to If it quacks like a duck or Manner of speaking.)

“To say my boss is self-centered is not altogether untrue.”