Actually there are several exceptions. If the baby's life was in
danger (even moderately so), it may not be performed until the baby is
healthy enough. If the baby was born by Caesarean, it may not be
performed on Shabbat or holiday. If the baby was born at twilight
(between sunset and darkness), when a halakhic doubt exists as to whether
it was born on the "day before" or the "day after", it may not be
performed on a Shabbat or holiday. Barring these, yes it is performed on
Yom Kippur itself (except there is no celebratory meal, of course).
Mike Turniansky