Tark is right. You have two weapon sets so why not use both. You basically can fit it to your play style.
I am dual wielding wants as one helps out with some aoe, so for large groups I can spam my wands and ember lighting (also for barrels). But when these one big guy I sometimes find it faster to beat him with a staff.
Thats just how I am doing it. I am not 100% familiar with how each stat contributes to magic and defense, so I might revise my stragety once i have more familiarity in that area.