By modeling room I assume you mean the vertex modeler room.  I have never found it necessary to set the modeling level in there.  I usually just subdivide once and that yields good results.  Could you send up a screen capture showing your exact render settings?  A few things to try.

Anti-Aliasing can be set to best with no problem, I always use .5 for object and shadow accuracy.  If I am using indirect lighting, I usually set the intensity to 80% or so.  This for some reason seems to make softer shadows and a bit less intense and more realistic lighting.  You can usually get away with setting Lighting quality to good, and accuracy to 4 pixels.  If you are not using refractions or light through transparency make sure to turn them off.  I usually use the improved edges and light through transparency functions under indirect lighting as well, but it is a resource hog.

Oh yeh, set the tile size to 32 or even 16 if you are having memory issues.  Sometimes this helps with renders.

If you are getting the error while still in the moddeling room, well...  I have had that problem a few times.  It seems to be rather obscure as to what causes it.

I hope this helps.