No, you can live off of it- you just need to get involved with a lot of projects all at the same time. Voice acting takes, what- half an hour? Longer if they want you to re-do some parts (which they usually don't, because they're on expensive recording studio time?)
So you pick up four voice acting gigs (if you're lucky) and then you do whatever odds and ends you can at those places, until you become irreplaceable.
Like Mark Hamil. Seriously, he was the Joker for Batman the Animated series, and most likely did fifty or sixty different voices for other characters (as filling).
-kv