Was discussing this with a friend recently. He points out that photons aren't super-intelligent. If a blue photon hits your eye (aka from the sky), it also hits your skin. That's, in a nutshell, the principle of the HDRI render; what is in the surrounding image is also the available light on the object you are rendering!