Other Free Encyclopedias » Online Encyclopedia » Encyclopedia - Featured Articles » Contributed Topics from A-E

Digital Camera Image Processing - CFA Basics, Image Processing for Color Estimation, Camera Image Compression, Video-Demosaicking, CFA Image Indexing

sensor imaging using spectral

Rastislav Lukac and Konstantinos N. Plataniotis
University of Toronto, Toronto, Canada

Definition: Digital imaging devices, such as digital camera, contain built-in image processing systems for applications such as computer vision, and multimedia and surveillance.

Digital imaging solutions are becoming increasingly important due to the development and proliferation of imaging-enabled consumer electronic devices, such as digital cameras, mobile phones, and personal digital assistants. Because its performance, flexibility, and reasonable expenses digital imaging devices are used extensively in applications ranging from computer vision, multimedia, sensor networks, surveillance, automotive apparatus, to astronomy.

Information about the visual scene is acquired by the camera by first focusing and transmitting the light through the optical system, and then sampling the visual information using an image sensor and an analog-to-digital (A/D) converter. Typically, zoom and focus motors control the focal position of the lens. Optical aliasing filter and an infrared blocking filter to eliminate infrared light are also included in apparatus Interacting with the sensor, there are other control mechanisms which are used to determine the exposure based on the scene content. Depending on the measured energy in the sensor, the exposure control system changes the aperture size and the shutter speed, by interacting with the gain controller to capture the sensor values using a charge coupled device (CCD) or complementary metal oxide semiconductor (CMOS) sensor. Based on the number of sensors used in camera hardware, conventional digital cameras can be differentiated as three-sensor and single-sensor devices.

The design of a three-sensor device (Figure 1) follows the trichromatic theory of color vision. Since an arbitrary color is matched by superimposing appropriate amounts of three-primary colors, and the sensor in the pipeline considered here has a monochromatic nature, professional digital cameras acquire color information using three sensors with Red®, Green (G) and Blue (B) color filters having different spectral transmittances.

The sensor is usually the most expensive component of the digital camera. To reduce cost and complexity, digital camera manufacturers often use a single CCD or CMOS sensor covered by a color filter array (CFA). The acquired image is a gray-scale image and thus, digital image processing solutions should be used (Figure 2) to generate a camera output comparable to the one obtained using a three-sensor device (Figure 1).

After an A/D conversion, the acquired image data undergo various preprocessing operations, such as linearization, dark current compensation, flare compensation and white balance. The order of the operations differs from manufacturer to manufacturer. The objective of preprocessing is to remove noise and artifacts, eliminate defective pixels, and produce an accurate representation of the captured scene. After the sensor image data are preprocessed, the image processing is used to perform estimation/interpolation operations on the sensor values in order to reconstruct full color representation of an image and/or modify its spatial resolution. Note that the order, complexity and actual form of image processing operations depend on the form of the CFA employed in an imaging pipeline.

CFA Basics

In the single-sensor imaging pipeline depicted in Figure 2, each pixel of the raw, CFA sensor image has its own spectrally selective filter (Figure 3). The specific arrangements of color filters in the CFA vary between the camera manufacturers which commonly use RGB CFAs. Alternative solutions include arrays constructed using Cyan-Magenta-Yellow (CMY) complementary colors, color systems with mixed primary/complementary colors, and arrays with more than three colors with spectrally shifted color.

Among these, the Bayer pattern (Figure 4) is commonly used due to simplicity of the subsequent processing steps. This pattern contains twice as many G components compared to R or B components reflecting the fact that the spectral response of Green filters is close to the luminance response of human visual system.

Image Processing for Color Estimation

After the CFA image is captured, numerous image processing operations can be implemented in the camera pipeline (Figure 5). The most critical step in a single-sensor imaging pipeline is a process called demosaicking or CFA interpolation. Demosaicking is used to estimate the two missing components in each spatial location of a CFA sensor image and convert a gray-scale, mosaic-like, CFA image to a full-color image. It performs spectral interpolation operations. Since demosaicking solutions usually introduce various visual impairments, such as blurred edges and color shifts, the quality of demosaicked camera images should be enhanced using demosaicked image postprocessing. By operating on a demosaicked, full-color, image input the postprocessor performs full color image enhancement. Apart from demosaicking and demosaicked image postprocessing, zooming operations are probably the most commonly performed processing operations in digital cameras. Digital image zooming can be performed either on the CFA domain or the demosaicked full-color domain. By increasing the spatial resolution of the captured image, image zooming techniques perform spatial interpolation operations. Finally, it should be noted that while demosaicking constitutes a mandatory step, both CFA and color image zooming as well as demosaicked image postprocessing can be viewed as the optional steps.

From the above listing is evident that the demosaicking, postprocessing/enhancement and zooming steps are fundamentally different, although they employ similar, if not identical, image processing concepts. It should be also mentioned that the research in the area of demosaicking has recently culminated due to the commercial proliferation of digital still cameras, which are the most popular acquisition devices in use today. However, both demosaicked image postprocessing and CFA image zooming constitute a novel application of great importance to both the end-users and the camera manufacturers. The similarity of the camera image processing steps listed in Figure 5, along with the limited resources in single-sensor imaging devices suggests that the target is to unify these processing steps in order to provide an integrated, cost-effective, imaging solution to the end user.

The visual impairments often observed in processed camera images can be attributed not only to procedural limitations but to the spatial and spectral constraints imposed during processing . Spatial constraints relate to the size of the supporting area and the form of the shape-mask used in camera image processing as well as the ability of the solution to follow the structural content of the captured image. Spectral constraints relate to the utilization of the essential spectral characteristics of the captured image during the estimation process. As it is shown in Figure 6, to eliminate both the spatial and spectral constraints, a camera image processing solution should employ both an edge-sensing mechanism and a spectral model, respectively. Such a solution can produce an image which faithfully represents the structural and spectral characteristics of the input scene.

The required performance is obtained using the so-called unbiased, data-adaptive spectral estimator, which can be defined as follows.

where w ( i,j ) is an edge-sensing weight, and f (·) is a function implementing the spectral model’s concept. Such an estimator operates over the color components available in both the estimation location ( p,q ) and the neighboring locations ( i,j ) ? with ? denoting the area of support. It also generalizes numerous processing solutions, which may be constructed by changing the form of the spectral model, as well as the way the edge-sensing weights are calculated. The choice of these two construction elements essentially determines the characteristics and the performance of the single-sensor imaging solution.

Depending on the application requirements and implementation constraints, an imaging solution can be implemented either in a digital camera or in a companion personal computer (PC). The architecture shown in Figure 7 is suitable for a computationally efficient method useful for practical, cost-effective camera solutions operating in the real-time constraints. Potentially demanding solutions may be included in the architecture shown in Figure 8. Unlike the conventional solution, using the PC-based pipeline the end-user can control the processing operation, change the parameter setting of the processing solution, and re-run the procedure if the output image does not satisfy the quality requirements.

Camera Image Compression

The outputted camera images (Figures 7 and 8) are usually stored either in a CFA format or as full-color images. Most professional digital cameras follow the so-called tagged image file format for electronic photography (TIFF-EP) for image storage. In this format, the raw CFA image is stored along with additional information, such as the details about camera setting, spectral sensitivities, and illuminant used. Consumer digital cameras store the full-color image in a compressed format using the Joint Photographic Experts Group (JPEG) standard. However, exchangeable image file (EXIF) format has been popularized due to its easy implementation and possibilities to store additional (metadata) information about the camera and the environment.

Current research in camera image compression indicates that the use of JPEG compression on the CFA image data may lead to new and promising imaging solutions which can be of great interest in wireless imaging-enabled devices since it allows for the transmission of significantly less information compared to the full-color image compression solutions. Further improvements may be expected using JPEG2000 which is a new standard for digital still image compression.


Recent advances in hardware and digital image/signal processing have allowed to record motion video or image sequences using digital still image or digital video single sensor cameras. Such a visual input can be viewed as a three-dimensional image signal or a time sequence of two-dimensional still images, and it usually exhibits significant correlation in both the spatial and temporal sense. Single-sensor camera image processing solutions devised for still images can also be used in single-sensor video cameras, however by omitting the essential temporal characteristics, conventional camera image processing methods, viewed as spatial solution which process separately the individual frames, produce an output video sequence with motion artifacts. Therefore, a well designed video processing technique should follow the representation of the signal, and utilize all the available information during operation. Such a spatiotemporal solution (Figure 9) should use the spatial, temporal, spectral and structural characteristics of the captured CFA video to produce a full-color, demosaicked, image sequence at the output.

CFA Image Indexing

CFA image indexing (Figure 10) constitutes one of the most novel developments in imaging enabled consumer electronics. A single-sensor captured image is connected to digital databases using embedded metadata. Depending on the application, the metadata can vary in the type and amount of the information to be processed. For example, images captured by common digital cameras can be automatically indexed using the camera’s identification number, ownership information and a time stamp. In imaging enabled phones, the advanced, satellite tracking based solutions can be used to provide location stamps. Furthermore, metadata can be completed by adding semantic content through the mobile phone’s or pocket device’s keyboard. To unify the approach for imaging solutions shown in Figures 7 and 8, the acquired images should be indexed directly in the capturing device by embedding metadata information in the CFA domain using the available data hiding solution. It should be mentioned that the CFA indexing approach can also be used to embed the metadata information which is achieved in EXIF format. Since such information is currently written in the header of the stored file only, it can easily lost by changing the file format or modifying the captured image using any conventional graphic software.

The metadata information should be extracted from either the CFA images or the demosaicked images in personal image databases using PC software commonly available by camera manufacturers (Figure 10). Alternatively, conventional public image database tools such as the well-known World Wide Media eXchange (WWMX) database can be used instead. Thus, the CFA image indexing approach helps to authenticate, organize and retrieve images in digital databases.

Digital Camera Raw [next] [back] Digital Biometrics

User Comments

Your email address will be altered so spam harvesting bots can't read it easily.
Hide my email completely instead?

Cancel or