Well guess I better read the manual again. If I turn of f caustics and just use the map...it doesn't apply caustics and it actually seems to be taking longer than the original render. So caustics must be seperate and at the moment not compatible with IM. Anybody else using Irradiance Maps?

No need to think outside the box....
    Just make it invisible.