first off, good choice on the gear, the 2496 is a great card and the direct box is a great lil unit.
What do you have your buffers set at in the Control panel in midiman's mixer. Lock your rate and prefereably play at 128 or 256 for your buffers. Make sure you have the proper driver chosen in Cubase under the VST tabs (midiman asio driver)
Right now my midiman is very stable at 5ms under high loads (p4 2.

and the card always impresses me with it's great fidelity.
You may have to be more speicific with what options you have tweaked or not. Can you also list your other hardware and OS?
Try what I said and if not post again in this thread and Ill try help as much as I can.