Not sure if you're still around to get the response. But just in case. Most of the skills for the summoner work well with the limitations of the controller vs mouse, the one exception being the teleport.
This may be surprising, but your most important summon will not be from any of your own skills. The B&B of a summoner is skeletons. Although not permanent, they are fast, aggressive, do awesome damage (much more than imps), and you can have many of them out at a time. For each skeleton spell you have, you can keep well over a dozen skeletons out at a time (assuming you cast every 3 seconds). Don't put a single point into imps (or do, but then respec out). I recommend having 2 different skeleton spells at a time. This means you can cast skeleton every 1.5 seconds and easily maintain an army of 30 skeletons (usually you don't need that many). I found many of the other summons poor for various reasons: zombies are very slow, imps die very easily and need corpses meaning that when you need them the most (i.e. when its hard to kill enemies) they aren't available, you only get one golem, the skeleton archers are hard to place and have low damage/health. However the archers at least offer ranged damage so I usually keep one of those spells too. For the same reason, I like the beam golem. The regular golem is basically like a couple of permanent skeletons for a 10 point investment, not worth it at all. That said, the build goes like this:
Required: these will make your skeletons extremely damaging, these all must be maxed
10 points - pet mastery
10 points - charm mastery
10 points - burning bind
Ember Shield, 1 point - scales with defense/magic stat somewhat, although it doesn't absorb a ton or magical, it's a good "early warning" system for taking damage. Don't underestimate that when you have 50 things on the screen at once.
Terror, 1-4 points - knockback and fear effect. Your summons don't hold aggro extremely well, these means often monsters will charge right past them into you. This is the best spell to keep enemies off you. It's cheap enough to spam. Doesn't work on everything, but most things. Keep in mind late game most melee creatures will keep you in just a few hits.
Teleport, 1 point on PC, I don't know how well it works with controller. Self explanatory. Surprisingly cheap.
Spell Mastery, 1 point: gives +2 mana per second. Usually one point here is enough. The idea is to skeleton and burning bind spam almost nonstop.
Beam Golem, 10 points: good damage, and ranged which you don't have much of
Ember Sentry, 1 point: detection system, minor damage.
2 highest level skeleton spells (especially blood skeleton once you get it).
Web - fantastic spell, helps tremendously with crowd control/survival.
skeleton archer spell (or something else).
Mostly defense, + whatever you need to equip items.
Note that I didn't talk about your damage, via wand or via spells. That's because it will pale compared to your summons. Right now I'm in my early 20's, moving towards this build. My skeletons when burning bound already hit for close to 1K each, around once every other second. So with 20 skeletons that would be 10 K DPS. It's just not worth the points or stats to try to raise your own damage.
For gear, just maximize your survivability try to find gear with bonuses to pets.