Most image fusion methods merge two or more images based on pixel information. Usually the extracted pixel information from all images should correspond to the same scene location. If the camera would not have moved during the capture of multiple images or the scene had not changed, then a pixel at location (u,v) in one image does correspond to a pixel at location (u,v) in another one. In practical applications however the two latter constraints do not always hold true. Mostly the camera is moving (even when using a tripod the camera often moves a bit due to e.g. camera chasis vibrations) or some parts of the scene are moving (see figure 1 for an example). In order to establish the correspondences between pixels in one image with the ones from another image a process called Image Registration is needed. Aside the geometric misalignment (e.g. image shifted to the lower right corner in figure 1), images may also exhibit a photometric misalignment which are typically caused by different camera parameters (e.g. auto exposure control of the camera). For an example the left image in figure 1 was captured with a much longer exposure time than the right image, still both images show the same scene.

Fig. 1: Due to a camera movement all pixels in the right image are somehow shifted towards the lower right corner. Changes within the scene like the two walking persons (green rectangle) cause some local pixel changes.

The problem of image registration is very old and many methods are available [1,2,3]. However because of the large amounts of different approaches very often it is not quite clear which method should be used. To shed a bit of light into the large field of image registration, this section categorises the most popular methods and highlights pros and cons of each category. In the following figure 2 a possible categorisation is depicted.

Fig. 2: Overview of different registration methods.

A first distinction between different registration algorithms can be made based on how pixel changes are introduced in two different images. In many cases the movement of the camera (e.g. physically moving it or by changing zoom) causes pixels to become disaligned between two images. This movement however can be described in mathematical terms and is applied to all pixels at the same time. Hence any registration algorithm that builds on the assumption that a camera movement globally effected all pixels is called a global registration method. Parts of between two captured images may also change if the camera was not moved at all. This might be due to dynamic scene motion like a walking person. Such a complex movement can not be described mathematically described for all pixels in one closed form. Hence each pixel has an own motion from one image to another. Method that try to estimate the motion of each pixel individually are called local registration methods.

Global registration methods are much easier to compute. In most applications the pixel motion can be at least approximated by a global transform. Local registration methods are much more general, but very expensive to compute. Global registration methods can be applied on small tiles of an image to each some level of local registration. However because global registration methods ususally are designed to use a global, large set of pixels, applying these to very small tiles degrades their performance.

In this section we will focus on global registration methods. These can be further subdivided based on how information about the camera motion is extracted from the images. Either all pixels are directly used or only a sparse subset of pixels are used. The first type of methods are called dense registration methods. These either work on intensity values (Intensity-based Registration Methods) or they transform the images into the frequency domain (Frequency-based Registration Methods). The latter type of methods are called Feature-based Registration Methods. Features are the few sparsely selected pixel locations that are used for registration. The way how these few "distinctive pixel locations" (features) are selected is crucial for the performance.

References

[1] L.G. Brown. A survey of image registration techniques. ACM Computing Surveys, 1992.

[2] B. Zitova and J. Flusser. Image registration methods: a survey. Image and Vision Computing, 2003.

[3] R. Szeliski. Computer Vision: Algorithms and Applications. Springer, 2010.