Look here, I make for you a quick diagram regarding the grey diffuse and how it response to the light and HDR IBL... usual there is not specular effect in HDR IBL ( fake GI ) only of there is a strong light.. the specular effect you see sometimes are just reflection on the surface that reflect lighter objects, the true specular show finally when the sun light light up the surface, this is different in the studio or using a photo flash.. every object in nature reflect light, the space is dark b/c there are not much objects close together to create this effect not even the atmosphere that scattering the light changing it to light yellow thanks to the blue photons .... but the most truth about that is that the real world have not colors at all, everything is in grey scale ( black& white) even the wonderful red dress is not red at all, not even in your poser scene, but thanks to our brains that transform the grey scale we can see the surfaces in colors... with other words the black and white photography is how the world would looks for real if our brains did not transfer it to colors.. as by the dogs that see the world as it is for real...

_________________________________________________________

"Surrender to what it is - Let go of what was - Have faith in what will be "