Hi,
The error is related to memory allocation, which occurred while compiling the geometry in your scene( or while loading geometry/scene objects by the V-ray for rendering, in it's in-built data structure). To me it seems like a bottle-neck in between the amount of Memory available for V-Ray and objects in the scene.
Here is what I can suggest apart from the tips given above:
**Try to save the file once , and make following changes inside System tab of Vray:
Max tree depth-Decrease this from it's default value, this will make the render slower but Vray might be able to proceed with the rendering.
Min leaf size:- Leave this to default i.e. 0
Face/level coef:-Increase this value may be up to 1
Default geometry:-Set this to Auto.
**YOu can try using V-ray proxies it will make the scene lighter and thus will cause Vray to proceed further and assign resources to other vital parts of rendering.
Keep us posted about the progress!
Cheers,
Shashi Kant
Blogs:
http://learningfluids.blogspot.in/
http://evocatematte.blogspot.in/