Lowering your action will never get rid of buzz. It'll only increase it.
If you're only having problems in one part of the neck, check your action and the bow of the neck. Here's how to check it:
Hold down your E and A strings (i'm assuming you mean the BOTTOM E and A, according to pitch, not vertical orientation) at the first fret. Now, with the pinky of your right hand, Press down on the 22nd (or 24th) fret. Tap around the 12th fret or so with your index finger or thumb and make sure you have enough clearance all around.
If you don't have a couple of millimetres of clearance, your neck is probably too straight or has a slight back bow. For clean action, your neck should have a very slight forward bow, which you can adjust with the
truss rod. See the tech section here for detailed setup instructions.
