The springs need to be symmetrical, with a normal set of strings the pulling force from the strings will also be even and balanced:
From D'addario website:
- string-note-inches/mm-tension
PL013 E 0.013/0.33 12.43 kgf
PL017 B 0.017/0.43 11.93 kgf
XSG026 G 0.026/0.66 14.69 kgf
XSG036 D 0.036/0.91 15.46 kgf
XSG046 A 0.046/1.17 13.83 kgf
XSG056 E 0.056/1.42 11.66 kgf
the last column denotes the string force (tension) as you can see it is pretty balanced, a lot of guitarist make the faulty assumption that the thicker strings will have more tension, but since their pitch is so much lower this is not the case. In fact here the middle strings have the hardest pull.
For reference, a regular set of 10s:
string-note-inches/mm-tension
- PL010 E 0.010/0.25 7.35 kgf
PL013 B 0.013/0.33 6.98 kgf
PL017 G 0.017/0.43 7.53 kgf
XSG026 D 0.026/0.66 8.25 kgf
XSG036 A 0.036/0.91 8.71 kgf
XSG046 E 0.046/1.17 7.76 kgf
As you can see, again the High E is 7.35 the Low E 7.76, not a big difference at all.
Adding up all the tensions will give you a total of 47kgf for the 10s and 80kgf for the 13s.
Since 3 springs are ok for 10s (47/3=~16kgf) you will need five (80=5*16kgf) springs to proberly balance things.
Alternatively (and since you are detuning to D) you can use four, and screw the spring retainer in deeper. (the difference will probably be no more than 2 to 3mm)
You can easily try this, put the springs on in a straight, symmetrical, 2+2 pattern, if necessary screw the spring retainer in (evenly! a quarter twist on the bottom one, a quarter twist on the top, etc. until the high E is to tune again) If your guitar is tuned correctly now and you do this right you won't even have to retune.
So why is this causing tuning instability? The forces on the post are unbalanced, but more importantly, with one spring angled the trem will be pulled slightly sideways, causing the rounded knife edge to ride up the post.