Yes and no. If you are working in a true radiosity environment then unseen objects can have a big influence. Otherwise not very much except in reflections and shadows as noted.

Pass no temptation lightly by, for one never knows when it may pass again!