You have to wrench like a gorilla to damge a truss rod.
If only the open E is buzzing and it doesn't buzz when fretted at the first fret, check and make sure your nut is high enough on the wound side. If it's not too low I suggest you loosen the truss rod to give the neck a little more relief. The upper registers are a little trickier, especially if you say the action isn't that low. You may have a high fret in the higher frets, or maybe your action is too low (if you're trying to set it to the specs of Joes guitars)