You should also use a digital caliper and a referece surface to make sure you are on center.
First remove the slide, it make this a lot easier. If you have a precision vise put the slide in with the sight up and the top of the sight parallel and flush to the top of the vise jaws. Using the inside measuring blades of the caliper check to see if the distace from the sides of the slide to the sides of the sight are the same both sides.
To adjust simply open the vise and raise the slide up enough to get the drift into place. An Aluminum drift will not dent the sight, but brass might. Remember to move the front sight in the opposite direction you want to move your hit point. You are hitting left, so move your sight to the right...
But, I would leave the front on the centerline of the slide and move the rear.