I think the toughest part of sweeping has got to be the continuity portion of it.
Let's say you're doing a sweep, any number of strings for that matter, you start from the 1st high E string, go back down, and once you're about to hit the lowest string of the arpeggio or scale, you have to decide whether it's gonna be a down stroke or upstroke.
This for me is probably the hardest bit to master.
I feel sweeping is most based on the right hand and since I'm lefthanded player playing righthanded it's kinda hard for me to get. Legato has come more naturally to me but the whole up and down motion has been something I've shyed away from somewhat.
Keep at it and try to master the movement of the right hand. After that it's all just coordination between the two hands.
Try the sweeps in the solo of Shy Boy, that one is a real challenging one, probably the most commonly used Vai sweep, 4 string sweep, same shape just played at 3 different positions. You can find it at approximately 2.00 onwards.
Keep at it dude you'll be shredding in no time