MiniDV and its 25 Mbps variations (DVCAM, DVCPro) are notoriously poor performers when it comes to bluescreen and greenscreen compositing (i.e. chroma key.) The difficulties stem from the NTSC 4:1:1 digital sampling format — for every four luminance samples, there is only one sample for each of the two colour difference channels. This can be seen in the figures below.
DV 4:1:1 Sampling 
|
 |
Y (luminance) one sample per pixel |
| U (colour difference B-Y) one sample per four pixels | ||
| V (colour difference R-Y) one sample per four pixels |
Note the grayscale images on the right which depict the three channels. The luminance channel (Y) at top is a sharp representation of the final image. The two colour difference channels (U and V) are of much lower resolution, made up of 4-pixel-wide horizontal blocks.
Generally speaking, 4:1:1 sampling is fine for ordinary images. The human eye perceives detail mainly through differences in brightness, so the lack of resolution in the colour channels has little adverse effect.
But in the case of bluescreen/greenscreen compositing, differences in colour are fundamental to the matting process. A clear distinction between the background colour and foreground subject is required to generate a clean matte edge.
Most compositing software, even high-end packages such as Ultimatte and Primatte, stumble over the blockiness of 4:1:1 sampling. DV composites typically exhibit an objectionable edge around the matted foreground element. (See figures below.)
 Typical DV Composite |
 "Blocky" Matte Edge |
Dealing with this has become an on-going challenge for many people. Solutions have ranged from applying a matte choker to hide the rough edges (at the expense of fine edge detail) to simply giving up and using more expensive (4:2:2 or BetaSP) equipment.
The best research I've seen on the topic is presented by Philip Williams at www.philipwilliams.com/greenscreen.aspx. In his tests, he discovered that re-digitizing the DV footage through an analogue connection resulted in a cleaner composite. (Though, slightly at the expense of overall image quality.)
The analogue signal, being a wave, effectively acted like an anti-aliasing filter, smoothing out the blockiness of the colour difference channels. Even though no colour resolution was added, it appeared to be enough to help Boris FX generate a cleaner key. This led me to conduct some tests which led to the solution described below.
Ultimately, the 4-pixel-wide blocks in the colour difference channels (U and V) need to be blurred. Good compositing software – those that support transparency and soft edges – will tolerate the low resolution colour channels, just not the hard-edge, blocky digitization of 4:1:1 sampling. The results of my test on a still image are shown below, using Premiere, Photoshop and the Primatte Keyer from Photron USA, Inc (Sold through Red Giant Software.)
DV Artifacts Smoothed 
|
 |
Y (intact) |
| U after four-pixel horizontal blur | ||
| V after four-pixel horizontal blur |
 Cleaner Composite |
 No Blockiness |
Admittedly, this technique is a bit labour-intensive, but it was my only option for conducting the test at the time. However, it does demonstrate the power of using these two programs in tandem when no other solution is handy.
The natural choice would be to use a plug-in to process the video clips. So, I hunted for a package that could handle the task...
Synthetic Aperture makes a plug-in package called Video Finesse. One of the filters, SA Gaussian Blur, allows you to apply a blur (horizontal, vertical or both) to just the chrominance channel. A horizontal-only blur with a radius of 2 to 4 would likely be appropriate. (Naturally, this filter would be applied before any compositing steps.)
I don't have a licensed copy of this package to conduct a test but will post results if the situation changes.
As many people have pointed out, the easiest solution is simply to use a better video format that doesn't exhibit these problems. But in reality, most people using MiniDV are doing so because they can't afford anything else. Generally they have more time than money, and are willing to spend the extra effort to overcome technical (and financial) hurdles. This solution has been presented for those resourceful individuals.
If you give it a try, please let me know of your results. I'd be interested to hear of any successes or problems you encounter. I can also post (or link to) any information you wish to share. Email me