Setting up virtual clusters for Compressor to maximize your rendering power w/ video


I'm just going to lay it out there. Apple's Qmaster, a part of Final Cut Studio, can be a cruel mistress. It's a very powerful feature that promises to make video processing enormously faster. Mostly though, it's quirky and temperamental. But we shall persevere. Apple's support forums another outlets have many tips, and many a trails of tears. This is what I've managed to get going on my setup, but I'm always looking for tips/tricks because every minute battling Compressor is a minute lost.

Essentially, Apple's Qmaster in conjunction with Compressor allows users to utilize multiple computers or multiple processors/cores to greatly speed up processing. Compressor preps and submits the job, and Qmaster runs it. In this article, I'll cover how I set up Qmaster on a single Mac with multiple processor cores. I found this to be the most stable and reliable system, and generally the most overall productive for typical video stations. If your job is cranking out tons of compressed video, or you just bored and have multiple computers not doing anything, a multiple computer cluster may be advantageous, but generally I find I spend more time troubleshooting than I save rendering. Still, however, I'll cover this adventure in a later installment.

Mac OS X shares its processing power among other apps, and keeps horse power in reserve for whatever you may want to throw at it. If you have a multi-core Mac, specifically the big octo 8 core system, you'll have much to gain since you'll likely be using 50% or less of your available CPU power on each core. Making a QuickCluster, you can treat each core as an individual computer, and let Qmaster feed all they can eat.

The steps are to first, run Apple Qmaster in the System Preferences. From there, you'll make your QuickCluster. A QuickCluster is Apple's way of making a simple cluster without going through a whole lot of fuss.

Next, have it Share and Managed for Compressor. If you have Shake, you can use that, otherwise leave those unchecked.

Then, in Options for selected Service, you want to add the number of instances to use. I use all 8 cores on my system. If you have needs for higher processing tasks, you may want to reserve one or two cores to keep the computer usable. I max it out and have no problem doing web and email while it chugs along and it doesn't seem to choke the OS. I'd say go full out and see how it works for you. Plus it's fun to see all those cores maxed out. YMMV, so pull out the stop watch and see what works for you.

Next, name your cluster what you would like, and select to include unmanaged services.

Then start sharing.

I don't believe exporting to the cluster from the timeline is supported. It never worked for me anyway, so a key step is to export the movie to QuickTime. No need to make it self-contained. This export will add a few minutes to the process, but the cluster will more than make up for it on larger projects. One point on that note, if I'm working on something small, I skip the cluster and just do a standard render on the computer. Last steps are to drag it into Compressor, add your compression preset, then Submit. When you submit, you'll see your created cluster in the Cluster drop down.

From there, open up your Monitor and Activity Viewer and watch it go. It's also helpful to have some type of CPU monitor that show individual cores, plus have the Qmaster in the menu bar under the Advanced tab. This way you can keep an eye on things for when it just simple doesn't want to work.

So, now what can go wrong? Well, all sorts of things. The biggest thing I notice is that Qmaster just dies. You'll submit something to the cluster and it will only run on one core. Or, the cluster won't even appear in Compressor. First thing I try is to go into Qmaster and stop sharing, and then hold down the option key, which should change the start button to the Reset. If that doesn't work, I try a cool little app call Compressor Repair. This will manually restart all the associated processes. If that doesn't work, I reboot. If that still doesn't work, I start to get that feeling. I'll usually fuss around with it, but this usually mean it's time to remove and reinstall. If you're lucky, you can just reinstall Final Cut Studio, or maybe less with Pacifier. A few times, however, I had to do a full out fresh install and reset up the workstation. That is a real pain.

Since we're talking speed, you absolutely need enough available memory. If you're system is paging, you're not helping yourself. Either shut everything until you have enough memory to run the jobs, or add more memory. The jobs are handed by the system, so actually you don't need any apps open. Just keep an eye on those memory page outs. You can monitor this in the Activity Viewer. Hard drives don't seem to matter here. Simple SATA drives seem to have no problems reading/writing faster than the processors can compress. Video cards are not at all important. Network will come into play when doing a multi Mac cluster. Largely rendering is pure brute force.

So, good luck. If it a shot, share any tips, and let us know how it goes.

- Mike Flaminio

HD Version of this video is available.


Setting up a QuickCluster for Apple Compressor from Mike Flaminio on Vimeo.