From what I have heard/read, it was Hugo except maybe in some of the stunts - in other words, anytime V walked or gestured or moved or fenced or the like. It was originally supposed to be James Purefoy, but he left early in the filming and Hugo was called in at the last minute to take over (sort of like Viggo in LOTR, I think).