 Open Access
 Total Downloads : 8
 Authors : M. Sangeetha, Dr. A. Senthilrajan
 Paper ID : IJERTCONV4IS21008
 Volume & Issue : ICRET – 2016 (Volume 4 – Issue 21)
 Published (First Online): 24042018
 ISSN (Online) : 22780181
 Publisher Name : IJERT
 License: This work is licensed under a Creative Commons Attribution 4.0 International License
Super Resolution – A Review
M. Sangeetha
Alagappa University
Dr. A. Senthilrajan Director,Computer Centre Alagappa University,Karaikudi Tamil Nadu,India
Abstract— Super resolution reconstruction is the technique of increasing the resolution of the image. The set of low resolution images (LR) are combined to form one or more high resolution image (HR).The super resolution reconstruction is possible only when the LR images acquired need to have different information of the same scene. In this paper existing super resolution reconstruction techniques of spatial and frequency domain methods are analyzed. Recent contributions were also studied.
Keywords: Super Resolution; Nonuniform interpolation; SR reconstruction.
1 .INTRODUCTION
Super resolution (SR) is a method to create high resolution image (HR) from one or more low resolution images (LR).In digital image processing the extraction of high frequency components of an image is of great importance. An application of SR includes satellite imaging where multiple images are taken from the same location and SR could be used to get more information for a certain object. In surveillance, region of interest is desired. Further, it is used in medical imaging such as in magnetic resonance imaging (MRI) and computer tomography (CT) with the possibility of more images with limited resolution. Another useful application would be conversion of SDTV to HDTV signal to provide the demand for HD quality.SR algorithms can be classified as i) Interpolation based ii) Example based iii) Reconstruction based methods. Interpolation based SR algorithms are fast but the results may lack some of the fine details. In examplebased SR algorithms detailed textures are elucidated by searching through a training set of LR/HR images. They need a careful selection of the training images, otherwise erroneous details may be found. Alternatively, reconstructionbased SR algorithms apply various smoothness priors and impose the constraint that when properly down sampled, the HR image should reproduce the original LR image[1].
The effort to attain the very high resolution coincides with technical limitations. Charged coupled device (CCD) or complementary metaloxidesemiconductor (CMOS) sensors are widely used to capture image signals. Spatial
resolution of the image is determined mainly by the number of sensor elements per unit area. The straight solution to increase spatial resolution is to increase the density of the pixel per unit area by reducing the size of the pixel. However, as the pixel size decreases, the amount of light impact on each sensor element also decreases and more shot noise is generated [2]. Another way to enhance the spatial resolution is to increase the chip size. However, the increase in the capacitance decreases charge transfer rate. Due to the cost of high precision optics and image sensors limits scientific applications. Superresolution is the term generally applied to the problem of transcending the limitations of optical imaging systems through the use of image processing algorithms, which presumably are relatively inexpensive to implement [7].
The major advantage of the signal processing approach is that it may cost less and the existing LR imaging systems can still be used. The SR technique is possible only with the availability of multiple LR images of the same scene. If the LR images are shifted by integer units, then each image contains the same information, and thus there is no new information that can be used to reconstruct an HR image [3].Each LR image should provide new information to acquire a HR image. The subpixel shift between the images is obtained by capturing the same scene from multiple cameras positioned at different angles or by multiple captures by using single camera. The subpixel accuracy is estimated and pixels are aligned in the HR grid using non uniform interpolation as presented in the figure.

OBSERVATION MODEL
When capturing a digital image, the original image of the scene cannot be obtained. Generally the captured image has several kinds of degradations due to the hardware limitation of the optical system. The acquired image suffers from various defects such as optical blur caused by the size of the lens, minimal sensor size leads to sensor blur, motion blur due to limited aperture time. The inadequate sensor density results in aliasing effects
kth warped image
kth warped image
Desired HR image X
Continuous Scene
Sampling without Aliasing
Sampling without Aliasing
Sampling
Translation Rotation, etc.
Translation Rotation, etc.
Motion
Blur Down Sampling
LR
Optical Blur Motion Blur Sensor Blur, etc
Optical Blur Motion Blur Sensor Blur, etc
Down sampling with aliasing
Down sampling with aliasing
Observations
Sensor noise
Sensor noise
Fig 1. The Observation model relating HR images to LR images.
thus create a reduced spatial resolution. A better observation model is required that relates the HR image of the original scene to the acquired LR images. The observation model is presented in Fig1
Let the size of the desired HR image be IÃ—J that is sampled beyond the Nyquist sampling rate from continuous scene which is supposed to be bandlimited signal. The HR image can be represented in lexicographical order as the vector X=[x1, x2… xN]T where N=IÃ—J. Consider the LR image of size IÃ—J. Let the kth LR image in lexicographical notation be yk=[yk,1,yk,2,….,yk,M]T for k=1,2,……,p and M=IÃ—J. Now, it is assumed that X remains constant during the acquisition of the multiple LR images, except for any motion and degradation allowed by the model. Therefore, the observed LR images result from warping, blurring, and subsampling operators performed on the HR image X .The observation model can be formulated [3] as
yk =DBkMkX + nk for 1kp (1)
where Mk is the motion model,Bk is the blur model and D is the down sampling matrix and nk is the additive noise vector. It can be written [4] as
yk = WkX + nk. (2)
Wk is sometimes referred to as the warp matrix and provides a mapping from the highresolution image to the kth lowresolution image. The involved matrices Mk, Bk, D or Wk are very sparse, and this linear system is typically ill posed [9]. The matrices are unknown in real systems and it is necessary to estimate them from available LR images.

NON UNIFORM INTERPOLATION
The method for performing superresolution is to map pixels from the lowresolution images onto a common plane and then interpolate over a more nely sampled grid to obtain a higherresolution image [4].The super resolution process has three stages as shown in Fig 2. First
the estimation of relative motion is computed from one or more observed LR images (known as registration) second, the registered samples are aligned in a HR grid to increase the resolution. The non uniform second, the registered samples are aligned in a HR grid to increase the resolution. The non uniform interpolation is applied to the HR grid of nonuniformly spaced samples followed by removal of blur and noise as a final process for restoration.
The simple method is the bilinear interpolation. The nearest pixel is found first. Then the algorithm detects which LR image the pixel belongs to and then selects three neighbouring pixels from the same LR image. It is presented in Fig 3.
Gilman and Bailey [5] used near optimal non uniform interpolation that a synthetic image is used to derive the optimal weights to be used for the input mages. The weights are computed to minimize the mean square error. It is assumed that the weights are independent of the image content but depend only on the relative position of the available samples in the grid. Therefore the weights depend only weakly on the image content. The optimal weights derived on known image can be used for other images with the same offset. This method provides better results with the computational overhead of calculating optimal coefficients.
Panagiotopoulou and Anastassopoulos [6] presented a method of non uniform interpolation of SR reconstruction. Gradient based motion estimation is performed. The kriging interpolation method is carried out to estimate the value of each HR grid point with the information from non uniformly placed LR frames to construct uniformly positioned samples. This is a direct reconstruction procedure as opposed to iterative. For every pair of two LR pixels the distance is calculated. Then the semivariance matrix for each LR frame is created. A semivariance matrix is obtained from computing the differences between the two matrices. A semivariogram is constructed from the distance and semivariance pairs. A Gaussian model is employed to fit data on semivariogram. Wiener filter is used for restoration.
1 N1 1 N2 1
2 r
2 r
yk(r1,r2)= Xk
1 m1 ,
2 m2
(3)
T1,T2 m1 0 m2 0
T1 N1
T2 N2
Fig 2. Super Resolution process (Figure taken from [Park et al. 2003])
The disadvantage is that the number of known values should be more than 5 to calculate every HR grid point. In addition the computational difficulty is high in performing nonuniform interpolation.
Directional bicubic interpolation is [10] presented to preserve the sharpness of the image. First local variance is computed. If the variance is less than the threshold bilinear interpolation is performed. Otherwise gradient method is performed along the direction of the edges. The two biggest gradients are extracted and interpolated by bicubic interpolation. The bilinear interpolation is used based on local image variance to reduce the computational complexity.
In hybrid interpolation [11] linear interpolation for flat area and max relativity edge interpolation for edges are performed in parallel. The cubic Bspline linear interpolation is executed for low frequency region and edge interpolation algorithms are applied in the high frequency information. The method has low computation and higher PSNR value. The advantage of the nonuniform interpolation approach is that it takes relatively low computational load and makes realtime applications possible [3].

FREQUENCY DOMAIN APPROACH
The frequency domain approach by Tsai and Huang is based on the following three principles: i) the shifting property of the Fourier transform, ii) the aliasing relationship between the continuous Fourier transform (CFT) of an original HR image and the discrete Fourier transform(DFT)of observed LR images ,iii)and the assumption that an original HR image is band limited.[3].
Let x(t1,t2) be a continuous image and xk(t1,t2) be the kth shifted images where k=1,2….p. Then the continuous Fourier transform (CFT) of the shifted image will be xk(t1,t2) = x(t1+x, t2+y) where x, y are the arbitrary
The matrix form can be obtained as Y=X where Y is a pÃ—1 column vector with the kth element of the DFT coefficient yk(r1,r2),X is a N1N2 Ã— 1 column vector with the samples of the unknown CFT coefficients of x(t1,t2) and is a p Ã— N1N2 which relates the DFT of the LR images to the samples of the continuous HR image.
Fig 3. Bilinear nonuniform interpolation and near optimal nonuniform interpolation
(Figure taken from [15])

REGULARIZED SR RECONSTRUCTION APPROACH
Many of the SR reconstruction methods are inverse problem because of numerical instability. The inverse problem is often ill posed that generates high resolution content from the set of observed LR images whereas the forward model is a method of deriving LR images from the given HR image. The measures generated to stabilize the inverse nature of the system and to provide a unique solution, regularization is necessary .The methods are classified in to Deterministic and stochastic approaches.
The Deterministic method solves the inverse problem by means of prior information about the solution space to get a well posed problem. Many standard techniques exist for doing this, but perhaps the most common approach is to impose a smoothness prior via Tikhonov regularization on top of a leastsquares optimization function [4].The matrix Wk in equation (2) is estimated for each LR image yk. The HR image can be estimated by minimizing the function
known values. Then the continuous Fourier transform p 2
(CFT) of the shifted image x
(u , u ) can be written as x
yk Wk X CX
(4)
k 1 2 k
(u1, u2) =exp [j2(xu1+yu2)]x(u1, u2) where x(u1, u2) is the CFT of x(t1,t2).The shifted images xk(t1,t2) are sampled with the sampling period T1 and T2 and LR images are generated yk(n1,n2)=xk(n1t1+x,n2t2+y) .From assuming the band limitedness of xk (u1, u2) xk (u1, u2) =0 for u1 (N1) / T1, u2 (N2) / T2.The relationship between the CFT of the HR image and the DFT of the kth observed LR image can be written as [3].Let yk(r1,r2) be the DFT of the LR images
k 1
where C is a high pass filter , is the regularization parameter and controls how much weight is given to the regularization constraint. The larger value of will smooth the image.
Stochastic Approach relates the SR reconstruction steps stochastically toward optimal reconstruction. The HR image and motions among lowresolution inputs can be both regarded as stochastic variables [9].The Bayesian model is the commonly used approach for superresolution.
The PDF of the original image must be recognized. There are two techniques Maximum likelihood (ML) and Maximum a posterior (MAP).The MAP is considered over ML because of its prior distribution. Consider the equation
(2) in the observation model. The LR images yk, noise n, the HR image X are assumed to be stochastic and Wk is known. The estimator X maximizes the probability of the HR image of the observed LR images is expressed as
X=arg max P(Xy1,y2,……..yp) (5)
The Bayes theorem is applied to the conditional probability and by taking logarithm we get
X=arg max(log P(y1,y2,….yp)X)+logP(X)) (6)
The P(X) prior density of X and P(y1,y2,….yp)X) the conditional density has to be defined by the HR image X and the statistical information of noise.

PROJECTION ON TO CONVEX SETS
This is a settheoretic approach where each piece of a priori knowledge is formulated as a constraining convex set. Once the group of convex sets is formed, an iterative algorithm is employed to recover a point on the intersection of the convex sets [4].
g i+1 = PMPM1.P2P1gi (7)
Where g0 is the initial guess, Pj is a projection of a given point on to the jth convex sets and M is the number of convex sets.
The advantage of the POCS technique lies in its simplicity to include any kinds of restriction and prior knowledge. The disadvantage of the POCS is wellknown for its heavy computation, slow convergence. The solution is not unique.

ITERATIVE BACK PROJECTION
The SR reconstruction algorithm of iterative back projection (IBP) was proposed by Irani and Peleg [12].The simulated LR images of the subsequent observed LR images are produced from the HR image. The estimated SR image is iteratively refined by back projecting the error (i.e., the difference) between synthetically created LR images and observed LR images until the error is minimized. The IBP function is formulated as
1

CHALLENGES AND ISSUES IN SR
Many algorithms have been proposed for SR reconstruction works theoretically but have implementation limitation. Ther are challenges and issues for applications of SR techniques.

Image Registration
The registration step is crucial to SR reconstruction. Therefore accurate registration methods, based on robust motion models including multiple object motion, occlusions, transparency, etc., should be needed [13]. The SR techniques treat registration as separate step and the recovered HR image depends on the accuracy of the registration process. On one hand, accurate subpixel motion estimation benefits HR image estimation. On the other hand, high quality HR image can facilitate accurate motion estimation. Therefore, tailored to the SR reconstruction problem, the LR image registration can be addressed together with the HR image reconstruction, leading to joint ML or MAP framework for simultaneous estimation. These joint estimation algorithms capture the dependence between LR image registration and HR image estimation, and performance improvements are observed [9]. However estimation of registration parameters jointly results in over fitting. A simultaneous registration and reconstruction approach is also expected to reduce the effect of registration error in the SR estimates, since registration and reconstruction process are closely interdependent [3].

Computational Efficiency
The SR algorithms should be fast enough to apply in real time applications. The great number of unknown values leads to exhaustive computation thus reduces efficiency. The nonuniform interpolation is suitable for its low computational cost. Many of the algorithms proposed for efficiency requires precise registration. Ce Liu and Dequing Sun proposed a Bayesian approach that simultaneously estimates underlying motion, blur kernel and noise level while reconstruction the original HR images [15].It has huge computation.

Robustness
The SR technique suffers from noise, motion blur, motion errors etc. Robustness is important because the image degradation model parameters cannot be estimated perfectly, and sensitivity to outliers may results in visual artifacts, which are not suitable for many applications. Several algorithms are proposed to treat outliers but are of limited practical use.

Performance Limitations
X(n+1)=X(n)+c Mk
k
[ hbpf * s ( k – yk ) ] (8)The analysis of the performance limits for all SR techniques is difficult. The SR reconstruction is a complex
where c is constant, bpf is the backprojection kernel,
s is the up sampling operator and k is the simulated kth LR image from the current HR image estimate. The IBP is easily understood. Due to the illconditioned nature, the IBP solution is not unique. It depends on choosing the IBP kernel.
task which consists of many interdependent components. Second, it is still unknown what is the most informative prior given the SR task, especially for the examplebased approaches. Last, a good measure instead of simple MSE is still needed for performance evaluation [9].Also there is a lack of objective metrics for the measurement of the quality of the SR image.


CONCLUSION
This paper reviews the super resolution techniques. The observation model is covered. The existing methods of nonuniform interpolation are presented followed by frequency domain method. Furthermore some of the challenges and limitations of SR techniques is also presented.
REFERENCES

Sundaresh Ram and Jeffrey J. Rodriguez,Single Image Super
Resolution Using DictionaryBased Local Regression IEEE 2014

MILANFAR, Peyman. Superresolution Imaging, CRC Press, 2011. 472s PARK, S. C., PARK, M. K., AND KANG, M.G.2003. "Super resolution image reconstruction: A technical overview". IEEE Signal Processing Magazine 20, 3 (May), 2136.

Neil Alldrin , Â´Â´SuperResolution, University of California, San Diego

A. Gilman, D.G. Bailey, Near optimal nonuniform interpolation for image super resolution from multiple images

A. Panagiotopoulou and V. Anastassopoulos, Superresolution image reconstruction employing Kriging interpolation technique 2007 IWSSIP & ECSIPMCS, Slovenia

Sina Farsiu, Dirk Robinson,Michael Elad,Peyman Milanfar, Advances and Challenges in SuperResolution 2004 Wiley Periodicals, Inc

Marco Bevilacqua. Algorithms for superresolution of images and videos based on learning methods. Image Processing. UniversitÂ´e Rennes 1, 2014. English.

Jianchao Yang, Thomas Huang Image superresolution: Historical overview and future challengesJianchao Yang, Thomas Huang University of Illinois at UrbanaChampaign

Liu J ing, Gan Zongliang, Zhu Xiuchang Directional Bicubic Inter polation A New Method of Image SuperResolution3rd International Conference on Multimedia TechnologyICMT 2013)

Xiangguang Zhang, Yun Liu, A Computationally Efficient Super Resolution Reconstruction Algorithm Based On the Hybird Interpolation JOURNAL OF COMPUTERS, VOL. 5, NO. 6, JUNE 2010

S. Borman and R.L. Stevenson, Superresolution from image sequencesA review, inProc. 1998 Midwest Symp. Circuits and Systems, 1999, pp. 374378

LIU, C.; SUN, D., A Bayesian Approach to Adaptive Video Super Resolution, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 209 216, 2011

KOC, U.; LIU, K., Interpolationfree subpixel motion estimation techniques in dct domain. IEEE Trans. Circuits Syst. Video Technol., vol. 8 pp. 460487, Aug. 1998.