Motion Picture Inpainting on Aged Films - Motion Estimation and Defect Block Detection

Definition: Motion picture inpainting refers to the detection of damages in video frames and repairing them.

Video inpainting, as we have discussed in the article entitled “Digital Inpainting,” has a different challenge issue compared to image inpainting. Most image inpainting methods require the user to select a target area. This requirement is not possible on video due to the large number of video frames. Detection of damages in video frames is a must. Damages in video frames include spikes (usually in a bright intensity) and long vertical lines (usually in bright or dark intensity). The former is due to dust and dirt, which occurs randomly and mostly appears in one frame. The later is accidentally produced in film development and usually occurs in the same position for an unknown duration (from several frames to several minutes). Detections of the two types of damages are different problems. Both problems need to look at the damage from temporal and spatial domains. A video inpainting project at Tamkang University uses the following strategy for video inpainting on spikes, as illustrated below:

Motion Estimation and Defect Block Detection

Subdivide each frame in the defect video into a number of 2 by 2 blocks
Compare each block in frame F i with a block in the same position in frame F k , where i-1 = k
= i+3
If the SAD (Sum of Absolute Differences) is large, the block could be a noise
Compare a noise block to its surrounding, merge adjacent blocks with a small difference
Defect Block Inpainting
If there exists good candidate source blocks in frame F i-1 or frame F i+1
Use the candidate source block to inpaint the damaged block in frame F i
Otherwise
Use a multi-resolution image inpainting technique on frame F i

The results of color and B&W video inpainting are shown in Figure 1 and Figure 2, respectively.