Ok.. I'm not the most experienced guy on floating trems, only been using them about a year. I have found a method that is simple and works great for me. I did something similar to what you did initially... trying to tune like a solid bridge guitar and the tremolo ended up doing a nose dive!
First - You need to get the tremolo level, if you don't know what it looks like when level... tell us what kind you have or hit google for a picture. You will need to know "level" for good stability. Then put your fine tuners in the middle of their position.
Second - Using small blocks of wood... or a screwdriver handle... or anything really (a pkg of wood shims is cheap and works great), you need to remove the cover on the back of the guitar and expose the tremolo springs and "block" (where they attach). Using blocks of wood (or whatever) in front and back of the tremolo block, set the tremolo so it is level on the front of the guitar. The tremolo block will usually be pretty close to a right angle to the strings.
Third - Now proceed to tune the guitar as you normally would - to E or Eb or whatever. With the tremolo "blocked" it will stay level. (stretch your strings good or it will go out of tune easily until they settle in)
4th - remove the blocks that are keeping the tremolo level... it will probably go out of tune - sharp or flat. Now just adjust the spring "claw", usually 2 screws. If the strings went sharp, unscrew them little by little (if you turn 1 screw a quarter turn, then do the same to the other), until you are back in tune. If they went flat, then tighten the screws until back in tune.
This will get you just about perfect... then lock your nut... and use the fine tuners to tweak it until perfect! I use the same process if I am going to change tunings... like from E to Eb or drop D.
Check tremolo level after the process is done and make sure it is still set up right.
More info here...
http://www.jemsite.com/tech-setup-a-questions