Resuming a render that has crashed or aborted is an advanced feature which requires a little extra care for how rendered images are written to disk. This is also known as checkpointing, or append mode. This feature only applies to renders writing to the filesystem, which means it will only work for disk/batch renders and not preview or live renders.
When append mode is active, Arnold will skip the rendering of any tiles in the image file that are complete, instead of starting over. It will only skip finished tiles; partial tiles will need to be re-rendered.
In order to resume a render, it must be initially set up properly so that Arnold can pick up where it left off in the even a render is aborted for some reason in the middle.
- Each output channel must use either the EXR or TIFF driver (
- Each driver must be set to tiled mode
- Each driver must have append enabled
- Each render output must specify a temporary render file location that is outside of Katana's temp directory, so that the file doesn't get deleted from the file system when the render stops
We will examine these in detail below.
Output Channel Setup
For each AOV you intend to include in your render, you will need an output channel set up via the
ArnoldOutputChannelDefine node. In that node, among other things you can adjust the driver used to write the image. For renders to resume they must be written using a tiling image format, requiring the use of the EXR or TIFF driver. The
tiled parameter must be enabled for resuming to make any difference (as scanline renders wait to write the image until entire scanlines are done, and often this means the image won't be written at all until it is nearly finished anyway). And finally
append must be enabled or else Arnold will always restart the render regardless of what tiles were already in the rendered image on disk.
If the image was empty (no tiles) or the image was not on disk at all, Arnold will just start from the beginning of the render. Turning off append mode makes sense if you intend to overwrite a previous render that changed – you don't want to keep any tiles already written to disk. Append mode generally makes sense when a render crashed or was aborted by the user or farm management software for some reason and you want to avoid losing as much render progress as possible.
Render Output Setup
Each render output (which has one output channel it pulls from) will need an explicit
RenderOutputDefine node. In it, you will need to select the channel defined in the corresponding
ArnoldOutputChannelDefine, and set the tempRenderLocation parameter to a location that will survive the render aborting, crashing, or Katana itself exiting. When that temp location is left empty, Katana auto-generates a location within its temp session directory (for example, located in /tmp/katana_tmpdir_6142), but when the render exits in any way it will delete the temporary render file. With that file gone, Arnold cannot resume the render and has to start over. Setting an explicit location for the render will allow the file to stick around so that the next render attempt Arnold can pick up where it left off.