Custom procedurals are compiled shader programs (.dll, .so, .dylib), that allow generating Arnold nodes on the fly.
You can find an example of how to write a simple Arnold procedural here.
The best way to support custom procedurals in MtoA is to write a MtoA extension. When no extension exists, it's necessary to inform MtoA that a new type of Maya node can be created for this procedural.
First, the procedural dll must be placed in a folder recognized by Arnold by setting the variable ARNOLD_PLUGIN_PATH.
In order to differentiate such procedurals with the usual procedurals that already have a Maya counterpart, it is necessary to set a specific metadata "maya.procedural" in this Arnold node.
This metadata can either be set:
In a separate ascii metadata .mtd file, placed in the same folder as the procedural library. It needs to have the same name of the procedural dll, but with a .mtd extension. For example, a library myProc.dll can have metadata set in myProc.mtd. Below is an example of how this should be set:
[node myProc] maya.name STRING "myProc" maya.procedural BOOL true
Here the "maya.name" is optional, it can be set to choose the name of the Maya node type.
With this setup, MtoA will recognize that this procedural type is a custom procedural that can be created in Maya. It will then appear in the menu "Arnold->Custom Shapes".
This will create a new node in Maya, containing all the parameters defined in the procedural, as well as the common procedural parameters.
When this node is rendered/exported, all of the procedural attributes will be automatically translated to Arnold.