Irradiance maps only contain Global Illumination information.  You're right, caustics is not part of GI. There's no reason not to use caustics with an irradiance map, but I'd probably render my irradiance map without caustics first and save it. let us know how it turns out. The manual says to even try rendering your map without textures first. Yeah, the glittering refections are from caustics.