You will only see it when you have words in your picture, or the statue of liberty for example (she would raise the wrong arm). I assume every panorama will be wrong on the sphere.

If you use U Scale -1 on the sphere the light does not match anymore. So you have to set U Scale to -1 on the IBL too.

As far as I know it does not matter where the IBL light is placed (maybe it does matter if it is above or under the ground, but did not test that).
But it does matter where the specular only light is placed and maybe a sun light for shadows.
A specular only light is correct, because a IBL does not produce specular highlights. Render without one and everthing looks somehow flat.

I don't have a "real" HDRI and therefore I'm not able to tell you how that works or if you need specular only light there.