This is actually a tough question! I say, spend as much time as you can, because there is so much to see in this culturally, socially, and historically rich environment. Summer is the best time to come!
If you are flying in from a short distance, you should give yourself at least four full days, and make sure the trip includes the weekend!
If you are flying in from far away, definitely take a couple of weeks and add St. Petersburg to your list. It is a four hour train ride north.