Whether you're casting an original character, a complex character with apparent contradictions in his nature—the rogue with a soft, gooey center, for instance—or an archetype like Gandalf of the Ring Trilogy and Dumbledore of the Harry Potter series, you need to know what a voice actor needs to be able to do in order to bring that character to life. You need to know what aspects of the voice are negotiable and which aren't.
A character's age or accent, for example, are generally not negotiable unless you are willing to change the very heart of your production. If you need a French accent, but find a voice actor who does British accents, but who is a genius at pulling off other difficult aspects of the character, it is perfectly all right to change the character to fit the actor—if you've found a gem of an actor. You have to make the call in the end, but you have to at least know what you want in the beginning, even if you find you are unable to get it.
You may even need special skills from a voice actor, like being able to do a child's voice. Chances are, you are not going to be able to find a child who is a voice actor, after all. But there are things you can do to create the illusion. Bart Simpson, for instance, is voiced by Nancy Cartwright.
All you really need when casting is a basic idea of the character's voice: gender, age, accent or lack thereof, and things like “smooth,” “rough,” “low,” “high,” “squeaky,” “sultry”--that sort of thing. And you need to know what kind of person he is, whether he's confident, sneaky, whatever. Then go shopping and see what happens.