For this test I'm going to be using a few set ups to get some test results. First I'll be using both the Atomos Ninja-2 and the SmallHD DP4-EVF with my Sony SLT-a99 on my newly rebuilt shoulder rig in some real world testing. I'll also be using it to record from my Canon XA-10 and from a GoPro Hero3 Black. In all cases I'll be recording in both devices at the same time for some A-B side by side samples.
The XA-10 is a camera I keep expecting to grow out of but always find a use for. Last year it was Canon's smallest, least expensive pro camcorder (now replaced with the XA-20). It has all the features standard to a pro video camera which aren't present on DSLR style cameras making it a perfect camera to set and forget. I honestly wasn't sure what to expect from pairing the Ninja with this camera but it's on the compatible list of cameras so it needs to be tried. The XA-10 is the most storage friendly camera I have as it has 64GB of internal memory and two SD slots. With the ability to span recordings from one slot to the other, the capacity for long recordings is unmatched...especially with lower quality compression. As a favour I filmed a local seniors association christmas theatre show using the XA10. It was the perfect opportunity to see what the Ninja could do.
The first thing I noticed was the weight. Clamped to the handle of a fluid head combined with a lightweight camera like the XA-10, the fluid head had to be locked often to keep it from falling over. It took some repositioning before I found a neutral balance. Compared to the small, light XA-10, the Ninja is a boat anchor.
The other thing I noticed was that the screen was far brighter than the built in screen on the camera...not brightness brighter but exposure brighter. Images which were properly exposed (or appeared to be) on the camera screen looked blown and washed out on the Ninja. This was both annoying and made it difficult to use the Ninja-2 without the built in monitor.
I tested zebras and peaking on the Ninja and found that with both turned on, the image becomes so cluttered and noisy combined with the above exposure issues that it's nearly impossible to see what's going on in the scene. This is especially true with scenes with small, fine detail...like the wide shot of a choir.
That said, the ease of use was phenominal. The XA-10 touch screen interface is bulky and cumbersome by comparison. As a recorder, it's simply amazing.
The audio meters on the Ninja-2 are also much more advanced and finely detailed than those built into the XA10...I liked this a lot. Switching meters off on the camera de-clutters the screen some what...a good thing if the screen is already small and cluttered.
The filming took place in a community centre gymnasium with some harshly coloured LED stage lights and little else other than the fluorescents in the ceiling...a good test in other words as I wanted deep shadows...typically where AVCHD falls flat.
****As a quick aside...I'm having issues with the USB dock for reading the hard drives. As of this writing, I'm using the hard drive in a different enclosure (modified to fit the Atomos drive casings) while I go through the trouble ticket process with Atomos...which hasn't impressed me so far****
The results of this test were a bit shocking. There's almost no difference between the footage recorded at 24p 24Mbps AVCHD and the ProResHQ recorded by the Ninja! The video looks identical and only freeze frames blown up to 400% or more show the slight differences. I didn't even bother posting a comparison video...it's just not worth it. Below are TIFF rips from the footage actual size and blown up so you can see for yourself.
This is the original frame...
And here are the blow ups of the XA-10 and Ninja-2 side by side...the only place you can see a difference...
If you look closely enough, there is a tiny bit more detail in the Ninja-2 though in many of the shots that just resulted in more gain noise than actual detail. The AVCHD compression does a good job of cleaning up the noise to reduce storage requirements.
I can only conclude that in this case the AVCHD codec is optimized for the small sensor in the XA-10 and that other than for workflow benefits, there isn't much reason to use the Ninja with this camera. In my case that's a relief. I can continue to use the XA-10 as I always have without worry that I could be getting more out of it.
My test with the a99 as much more controlled. In this case I set up a simple scene in my living room and recorded using several combinations of camera and Ninja settings to see what would result. At higher ISO ambient conditions, the results were similar to that with the XA-10. I think there's just too much noise when the ISO or gain is high for the Ninja to really stretch it's legs. When I set up another scene in front of some big windows and got the ISO down to 400, then I started seeing some interesting differences.
At 24p, as with the XA-10, the difference can barely be noticed. In the first shot of my dog Ellie sleeping, and at an ISO of 3200, I can't tell any difference between the internal 24Mbps AVCHD codec and ProResHQ. In the second shot of a couple of my cameras, there are slight differences in sharpness and detail but you have to blow up the image several times it's normal size to even begin to notice. This scene was shot at ISO 400. Both were shot with a Nikkor 180mm ED Ai-S at f/2.8
This second example is just of the cameras but at various camera and Ninja settings. I was surprised to see that there were strange interlacing problems with shots taken at 30p (60i according to the camera). These issues reduced detail in the Minolta's faux leather grip and in the lettering on the front of the lens significantly. There was also a moire introduced on the ridges of the focus ring. This was recorded by the Ninja-2 when recording at 60i but not at 30p with a 2:2 pulldown. With the pulldown the images look as sharp as they do in 24p. In this example at least the Ninja-2, set up correctly, showed a great improvement of image quality. I likely didn't notice this before because I almost never shoot 30p. The majority of my footage is 24p, with 60p shot for overcranking.
On that note, it appears to be possible to record 60p in the camera and 30p in the Ninja-2 at the same time without issue. The only problem being selecting an appropriate shutter speed.
If you work with ProRes a lot, there are definite benefits to using the Ninja. With enough footage recorded, transcoding to ProRes HQ alone can take at least several hours even with a fast computer. If you're like me and do everything on a MacBook Pro, that pretty much stakes claim to your computer resources and sets your project behind from the start.
Speaking of MacBook Pro computers, the drive dock which comes with the Ninja-2 has dual USB ports. One for data and a supplimental one for extra power. The Macbook pro only has one USB on each side making plugging both in without a hub impossible.
So where does that leave me after this completely unscientific test? Well I have to say that it's an impressive piece of technology but doesn't quite live up to all the hype. As a monitor, I wasn't impressed. It does an OK job but compared to my SmallHD DP4, it really fails. The Newer models of Atomos recorders now come with the ability to adjust screen settings using a monitor calibration unit but this option isn't available on the Ninja.
As a recorder, I'm very happy with the controls and the ease of use. I like editing in ProRes so this can save me a lot of time. Even more if I edit directly off of the hard drives that the Ninja-2 records to. I have to admit that I was expecting amazing things from the ability to record straight to a very high bitrate ProRes file but the results didn't knock my socks off. I'm not sure if that's because the latest AVCHD codecs are that good or because the HDMI output is still only 8 bit 4:2:2 compared to the 4:2:0 captured in camera. I'm sure in time I'll find scenes where the Ninja puts that to good use but in the examples I've tested with, they're nearly identical. It may come into play when images require a lot of colour grading and post processing.
In the writing of this article I found a good one about recorders at B&H Photo which explains a lot of these funny terms and technologies. If you're curious about reading a thing or two about it, check it out here.
Because of what the unit can do, and despite what it doesn't to well enough for me, it's still been a good purchase and I'll be using it on a regular basis with my video work. Because of my disappointment with the monitor quality, I might have been better off with the Blackmagic Hyperdeck Shuttle but I'm not a fan of designs with built in batteries which have to be recharged...seems like a recipe for a bad day in the field. The bottom line is that this is a piece of hardware which compliments the rest of the system and doesn't have to take the place of the internal codec.