Keep the RG770DX, you won't find another as they're quite rare! and its much better than Jem505!!!
In what way is it much better than a 505? Are you confusing the Japanese Prestige Jem505 and the older Korean/Indonesian Jem555?
Regardless, they are different guitars. I would personally prefer the 505 hands down due to the 5 piece Prestige Jem carve neck. They are both Japanese made guitars, both basswood, so I suppose you can consider that a wash.
The thing is that the 505 will be less likely (in my opinion only) to go up in value as it was a 'cheaper' Jem, although it is a maple boarded Japanese Jem so it was the first since the DYs were discontinued. The 770 will remain a more upscale RG, with the bound neck and shark fin inlays and such, so that should be something you take into consideration. For example, the RG might be worth 650-700 now, and the 505 more like 900, but the RG may be worth over a grand in 5 years the way they keep going up and there's no way to tell with the Jem.
Play the Jem and if you have to have it go for it. If you play it and they are both pretty much the same in your hand then I would keep the RG. In a perfect world I would save up for the Jem and let that RG grow in value.
Just so you are aware, I have a 505 and I like it A LOT so I may be biased, but I have owned plenty of RG 7xx guitars over the years as well.