Alembic workflow for Deadline over a VPN
This blog will outline my process to get a shaded alembic file rendered on a render farm through a VPN. It will be a no-brainer after reading this blog, but a 5 day RnD process and understanding the back-end of Maya, Arnold and Deadline was needed to get an efficient result.
Please note all images are work in progress and do not represent final
Intro
I’m currently in the middle of directing an animated short (very short 30 second) film as part of my final year of animation at university. Due to the global pandemic, and the inability to travel to campus, the film has been made 100% remotely.
Our project is entirely 3D with a traditional referencing pipeline, using Maya and Arnold. We had decided in pre-production to use the university’s 40 slave render farm. I have previously used the render farm for personal projects. This would usually involve:
Transferring my Maya files to an external SSD
Physically driving to campus
Transfer the Maya files off the drive and onto the render farm’s network
Repath the files to the ones on the render farm network
Set off the render and pick it up the next day
COVID-19 and Melbourne’s stage 4 lock downs have made it pretty difficult to head to university. However, I’ve been lucky enough to be able to VPN into the university’s network to use the render farm for our film.
I was super excited seeing Deadline on my home machine, but there was always going to be new challenges. Hence, the start of a 5 day RnD and head-banging session.
Clarisse Render + Nuke Bash comp
Clarisse and Its Powerful Scatter Tools
Our film has an army of alien soldiers lined up in formation in a large hall . Yes, my tutors were also concerned about the idea of what would probably be a crowd simulation. And ‘no’ we have not been taught or had even worked on any crowd simulations previously. It shouldn’t be that hard right…
I had recently been getting my hands dirty with Isotropix Clarisse. Clarisse is a very user-friendly layout, look-dev and lighting tool with some very powerful scatter options, and has probably been one of the smoothest introductions to a software package I’ve encountered. The student license is only $90 AUD a year (gotta love a student discount). After 2 months of playing around in the software and building countless mountains with trees on top, I decided to use it for my film.
Our animators animated 3 different chanting movements on our soldier rig (for a particular shot). Each animation then had a 1-2 plus/minus frame offset to create a sense of difference. So we had a total of 9 different animations. I exported each animation as alembic (.abc) and referenced it into Clarisse. I then used a simple point array and scattered the animation. The scale, rotation and position were slightly varied to make each soldier seem unique.
I originally wanted to match the lighting in Clarisse to the Maya scene I prepared earlier. I would render the soldiers in Clarisse and the other elements in Arnold then comp both passes in Nuke. However, there were over 70 unique lights in the Maya scene, and it would be a nightmare to reproduce in Clarisse. In addition, I would have to render the Clarisse shots on my machine, as the render farm does not have Cnodes set up.
Clarisse with the scattered soldiers
Back to Maya
I decided to export the Clarisse scatter as alembic and import it into a Maya scene. I then imported the shaders from the soldier models and added it to each object. There were 80 soldiers and I filtered the outliner to select each part and assign the correct shader. This was a bit of a grind. I kept the soldiers in their own scene and referenced that scene into the “master scene.” This workflow worked out nicely when all the media was on my local drive.
I repathed the Maya files to the render farm network, and copied the alembic file across too. This was quite slow as I only had a read/write speed of 1mb/s. Little to my surprise, problem number two arrived. The soldiers were not rendering at all.
First farm render - missing soldiers
Some Results
After some Googling, I found out that the alembic needed to be imported from the render farm network and not my local drive. This is due to Maya not being able to repath an alembic after it has been imported. My slow read/write speeds made importing from the render farm network incredibly slow and navigating Maya was basically impossible (despite my mediocre NBN plan).
More Googling, I found other students had a similar problem rendering alembic on a university render farm. Their alembic files disappear on render too but they found a solution by baking the animation first.
I gave it a go and my Maya scene increased by a few hundred MBs. I chucked it back onto the render farm. This time it seemed to only render objects with no deformation. I decided to render it for the laughs.
The bake worked but not completely.
I went into the bake options and decided to bake the deformation (“control points” checked) of the body. I was lucky enough to have 64gb ram as Maya was chewing up around 40gb to bake all the animation over 86 frames. This made the Maya file a massive 6gb. Referencing this into the master scene took over 30 minutes to open. And then transferring it to the render farm network was even worse. Despite the slow speeds, the render farm managed to render the scene. Though it was incredibly slow with very low AA samples and low global illumination samples. (aa2/d1/s1/t0/sss1/vol0). According to the logs, it took Deadline 30 minutes to load in the 6gb reference file into the master scene. I decided to crank up the sampling in a text editor but Deadline just couldn’t handle the data and failed it.
Opening the ASCII file in Sublime to make quick edits
The Efficient Result
I originally imported my alembic from Clarisse into Maya but that is where I went wrong.
I decided to reference in the alembic instead. I imported my shaders the same way but applied it to the reference. I uploaded my entire project window, including the alembic, and repathed the reference to the one on the render farm network. This took a bit of time to load but not as long as a 6gb Maya file. And it worked. The file rendered and looks exactly the same as a render from my local drive/computer.
I did find that you cannot reference in the alembic into a Maya scene and then reference that Maya scene into another as you will get some strange results. See below:
referenced alembic file into a Maya scene (soldiers only) that is referenced into another Maya scene (master scene)
So I referenced the alembic straight into the master scene, added my shaders and reapthed to the render farm network.
Conclusion
There is still relatively long load times and slow read/write speeds but referencing the alembic straight into the master scene was the most efficient of all the outcomes. This way, I can work locally first, then when it is time to render, repath my files to the ones on the render farm network.
An even faster result would be to remote into a machine on campus and repath textures in Maya via remote desktop. This will ensure I get gigabit speeds when reading and writing to the render farm network.
Enjoy the current raw render at (5/3/3/0/3/0)