Study and Analysis of Image Inpainting Algorithms

Study and Analysis of Image Inpainting Algorithms

Kunti Patel

Computer Technology (M.Tech.) RCET Bhilai CG

Amit Yerpude

Computer Science & Engineering RCET Bhilai CG

Abstract Inpainting is an ancient art used for restoring removal and filling in removed region. It has been widely applied in restoring scratched old photos and old movies, text removal from images, digital zooming and especial effects in movies now it is also applied in add or remove object in photos. This paper deals with different algorithms in the field of image inpainting such as, PDE based image inpainting, Exemplar based image inpainting and texture synthesis based image inpainting and provide detail analysis of them. Purpose of this paper is to give better solution to identify best image inpainting algorithm without the need of further studies.

Keywords- Image Inpainting, PDE, Texture synthesis, Exemplar.

1. INTRODUCTION

   Removing objects and repairing damaged region are very interesting task, especially nowadays the technique development greatly advanced. People often keep their graphics, photos or the valuable films in their computer. By the image processing, these photos can delete or inpaint the interested object or other related applications to achieve visually plausible image and to create a new digital image for humans. Figure.1 shows an example of image inpainting, where the foreground object (manually selected as the target region) is automatically replaced by data sampled from the rest of the image. And it can use to apply to change of image foreground, the reconstruction of the historic interest on image inpainting, virtual reality, etc.

   The most fundamental inpainting approach is the diffusion based approach [1][3], in which the missing region is filled by diffusing the image information from the known region into the missing region at the pixel level. This is the first family of inpainting techniques relies on Partial Derivative Equations (PDE) and variational method. Bertalmio et al. [1] filled in holes by continuously propagating the isophote (i.e., lines of equal gray values) into the missing region. They further introduced the NavierStrokes equation in fluid dynamics into the task of inpainting [2]. Chan and Shen [3] proposed a variational framework based on total variation (TV) to recover the missing information. Then a curvature-driven diffusion equation was proposed to realize the connectivity principle which does not hold in the TV model [4]. A joint interpolation of isophote directions and gray-levels was also designed to incorporate the principle of continuity in a variational framework [5]. Recently, image statistics learned from the natural images are applied to the task of image inpainting [6][8]. Some other PDE models are also applied to the task of image inpainting [9-11]. The diffusion-based inpainting algorithms have achieved convincingly excellent results for filling the non-textured or relatively smaller missing region. However, they tend to

   introduce smooth effect in the textured region or larger missing region.

   All the aforementioned approaches suffered from a common problem of texture restoration. Images may have structure region along with textured regions. PDE based techniques inpainted the small missing regions but with blurriness, which results into loss of texture information, as depicted in Figure 2. Moreover, if area to be filled is larger than some extent, inpainted regions can easily be visually detected due to flattening.

   Therefore, for enhancement of texture details, several researchers have used texture synthesis methods, second family of inpainting techniques, which targets basically for generating pure-infinite texture using sampling from limited source of texture samples. Recreation in such a way aims that the created texture should have similar appearance and structural properties. They should maintain local structure as much as possible [12]. Texture restoring techniques methods can be single pixel based or patch based. Pixels based synthesis takes more time than patch based methods.

   Efros and Leung [15] proposed a method to synthesize single pixel at a time with the use of a neighborhood of pixels. For their proposed approach, Markov Random Field (MRF) is assumed in which probability distribution of pixel values of a given neighborhood is assumed to be independent of rest of image. It runs pixel by pixel through the use of template patch based on the source texture and then searched area for possible matches. Once a match is obtained, center pixel of template is copied to destination. Success of this method depended on size of the template window. Efros and Freeman

   [16] proposed a method of Image Quilting for texture synthesis. Quilting means stitching small patches together. From available texture, blocks were replaced with adjacent to each other with minimum boundary cut [16]. Minimum boundary cut between two texture blocks is the best match where the overlap error is low. This method performed well for texture synthesis. But for inpainting, it is difficult to find suitable block as the texture fills the hole towards the center and noticeable artifacts may be generated. Using ideas from [12, 14, 15], Drori et al. [17] proposed multi-resolution based approach. For each fragment in the damaged region, it finds the confidence term for each level. This confidence term determines the fill order. Then it searches for best match for each fragment. After finding matched patch from source, it is merged with existing data at that point based on the confidence term similar to [14]. This method had a drawback that sharp edges tend to become slightly blurred. Moreover, this approach could not handle dilemma when two perpendicular regions intersect each other.

   Third family of inpainting techniques, Exemplar-based techniques, combines advantage of simultaneously restoring structure as well as texture information. Criminisi et al. [18, 19] proposed an outstanding exemplar-based inpainting algorithm for repairing larger missing region with composite structures and textures. In the algorithm, priority function was defined to decide the filling order that encourages to fill structure part, and sum of squared difference (SSD) was applied to search the most similar patch in the source region for the filling missing region. The algorithm first formulated the procedures of exemplar-based inpainting algorithms and afterwards many approaches follow this route.

   Many approaches improve the performance of Criminisi's algorithm from two sides, the priority function and the patch inpainting. Regarding priority function, Cheng et al.

   [20] redefined the priority function. Sun et al. [21] repaired the structure part preferentially along the manually added auxiliary line. Wu and Ruan [22] used across-isophote diffusion item to decide the filling order. Structure sparsity was defined in Xu and Sun [23] to determine the filling order and later it was combined with a modified confidence term to compute priority in Hesabi and Mah-daviAmiri [24]. Jemi Florinabel et al. [25] developed a method using the DCT coefficients of a patch located at fill-front to decide the filling order. Le Meur et al. [26] investigated on using structure tensors to determine the filling order. Wang et al. [27] applied the DS evidence theory to compute priority. Zhang and Lin
   [28] designed a priority scheme based upon color distribution.

   S. Badgujar and N. M. Shahane [29] use examplar-based image inpainting and approaches to improve the performance.

   P. Dhabekar et al. [30] designed an exemplar based inpainting though patch propagation. Zhidan Li et al. [31] developed a color gradient patch sparsity based image inpainting algorithm with structure coherence and neighborhood consistency. P. Goswami et al. [32] designed an exemplar-based image inpainting using ISEF (Infinite size Symmetric Exponential Filter) for priority computation.

   The paper is organized as follows. Section II, define the problem statement. Section III , describe the details of the inpainting methodologies. The comparison and anatysis result of the previous algorithms are defined in the Section IV. Finally, we conclude this work in Section V.

   (a)

   (b)

   Figure1. Removing objects from images. (a) Original image. (b) The regions corresponding to the foreground object has been manually selected

   and then removed.

   Figure2. Texture restoration problem in CDD method which is one of the PDE-Based techniques.

2. PROBLEM STATEMENT

   Conventionally Inpainting is carried out by professional artist and usually its very time consuming process because it was the manual process. The main goal of this process is to reconstruct damaged parts or missing parts of image. And this process reconstructs image in such a way that the inpainted region cannot be detected by a casual observer. Inpainting technique has found widespread use in many applications such as restoration of old films, object removal in digital photos, red eye correction, super resolution, compression, image coding and transmission. Image Inpainting reconstruct the damaged region or missing parts in an image utilizing spatial information of neighbouring region. Image Inpainting could also be called as modification and manipulation of an image. In image inpainting we would like to create original image but this is completely unfeasible without the prior knowledge about the image. In case of digital images we only have the image we are working on available to us and thus we are filling in a hole that encompasses an entire object. It is impossible to replace that entire object based on the presents information. Considering this as the aim of the inpainting algorithm is not only reconstruct what used to be in that hole. But instead to create a visually pleasing continuation of the data around the hole in such a way that it is not detectable by ordinary observer..

3. METHODOLOGY

   1. PDE based Inpainting

      Partial Differential Equation (PDE) based algorithm is proposed by Bertalmio et.al [1-2]. This algorithm is the iterative algorithm. The main idea behind this algorithm is to continue geometric and photometric information that arrives at

      the border of the occluded area into area itself. This is done by propagating the information in the direction of minimal change using isophote lines. This algorithm will produce good results if missed regions are small one. But when the missed regions are large this algorithm will take so long time and it will not produce good results. Then inspired by this work, Chan and Shen [3] proposed the Total Variational (TV) Inpainting model. This model uses Euler-Lagrange equation and anisotropic diffusion based on thestrength of the isophotes. This model performs reasonably well for small regions and noise removal applications. But the drawback of this method is that this method neither connects broken edges nor greats texture patterns. The TV model then extended to CDD (Curvature Driven Diffusion) [4] model. In which it included the curvature information of the isophotes to handle the curved structures in a better manner. All of the above mentioned algorithms are very time consuming and have some problems with the damaged regions with a large size. PDE based technique has been widely used in number of applications such as image segmentation, restoration etc. These algorithms were focused on maintaining the structure of the Inpainting area. And hence these algorithms produce blurred resulting image. Another drawback of these algorithms is that the large textured regions are not well reproduced.

   2. Texture Synthesis Based Inpainting

      Texture synthesis based algorithms are one of the earliest methods of image Inpainting. And these algorithms are used to complete the missing regions using similar neighbourhoods of the damaged pixels. The texture synthesis algorithms synthesize the new image pixels from an initial seed. And then strives to preserve the local structure of the image. All the earlier Inpainting techniques utilized these methods to fill the missing region by sampling and copying pixels from the neighbouring area. For e. g, Markov Random Field (MRF) is used to model the local distribution of the pixel. And new texture is synthesized by querying existing texture and finding all similar neighbourhoods. Their differences exist mainly in how continuity is maintained between existing pixels and inpainting hole.

      These synthesis based techniques perform well only for a select set of images where completing the hole region with homogenous texture information would result in nature completion. Then later, this technique was extended to fast synthesizing algorithm. This technique works by stitching together small patches of existing images referred to as image quilting. Bergen and Heeger [12] developed a parametric texture synthesis algorithm which can synthesize a matching texture, given a target texture. This was done by matching first order statistics of a linear filter bank which roughly match to the texture discrimination capabilities of Human Visual System [HVS]. Recently, a fast multi-resolution based image completion based and on texture analysis and synthesis was introduced fang et al in [13]. In their method, the input image was analysed by a patch based method using principal component analysis (PCA) and a Vector Quantization (VQ) based techniques was used to speed up the matching process of the texture inside the whole region.

      The main objective of texture synthesis based inpainting is to generate texture patterns, which is similar to a given sample pattern, in such a way that the reproduced texture

      retains the statistical properties of its root texture. And it does not appear simply as a tiled rearrangement of the root-texture. Contrary to vibrational inpainting (PDE based inpainting), the texture synthesis based methods can fill large textured regions, but depends on user choices on sampling location and content. Texture synthesis approaches can be categorized into 3 categories: Statistical (parametric), pixel-based and patch- based (non-parametric). Statistical methods are more likely to succeed in reproducing stochastic / irregular textures, but usually fail to reproduce structured/regular textures. On the other hand, pixel-based methods build on the sample texture pixel-by-pixel instead of applying filters on it, and their final outputs are of better quality than those of statistical methods, but they usually fail to grow large structured textures. Finally, patch-based [15] methods build on a sample texture patch- by-patch as opposed to pixel-by-pixel, thus they yield faster and more plausible regular textures.

      Recently, a study for patch-based texture synthesis algorithms has shown that for handling special types of texture we have to develop the special purpose algorithms. Taking this aim and the variety of algorithms for texture synthesis into consideration, we can conclude that there is no universal texture synthesizer is present. Still it remains a goal to be desire. The texture synthesis based Inpainting perform well in approximating textures. These algorithms have difficulty in handling natural images as they are composed of structures in form of edges. Also they have complex interaction between structure and texture boundaries. In some cases, they also require the user to specify what texture to replace and the place to be replaced. Hence while appreciating the use of texture synthesis techniques in Inpainting, it is important to understand that these methods address only a small subset of Inpainting issues and these methods ar not suitable for a wide variety of applications.

   3. Exemplar based Inpainting

   The exemplar based approach is an important class of inpainting algorithms. And they have proved to be very effective. Basically it consists of two basic steps: in the first step priority assignment is done and the second step consists of the selection of the best matching patch. The exemplar based approach samples the best matching patches from the known region, whose similarity is measured by certain metrics, and pastes into the target patches in the missing region.

   Exemplar- based Inpainting iteratively synthesizes the unknown region i. e. target region, by the most similar patch in the source region. According to the filling order, the method fills structures in the missing regions using spatial information of neighbouring regions. This method is an efficient approach for reconstructing large target regions.

   Generally, an exemplar-based Inpainting algorithm includes the following four main steps:

   1. Initializing the Target Region, in which the initial missing areas are extracted and represented with appropriate data structures.

   2. Computing Filling Priorities, in this a predefined priority function is used to compute the filling order for all unfilled pixels p in the beginning of each filling iteration.

   3. Searching Example and Compositing, in which the most similar example is searched from the source region to compose the given patch, (of size N × N pixels) that centered on the given pixel p.

   4. Updating Image Information, in which the boundary of the target region and the required information for computing filling priorities are updated.

   Numbers of algorithms are developed for the exemplar based image Inpainting. Such as, Drori [17] proposed a fragment-based image Inpainting algorithm that iteratively approximated, searched, and added detail by compositing adaptive fragments. The computation time of this algorithm is intolerable. Then Criminisi [19] developed an efficient and simple approach to encourage fillingin from the boundary of the missing region where the strength of isophote nearby was strong, and then used the sum of squared difference (SSD) to select a best matching patch among the candidate source patches. In this algorithm of criminisi the region filling order is determined by the priority based mechanism. Cheng [20] generalized the priority function for the family of algorithms given in [19] to provide a more robust performance. Wong

   [14] developed a weighted similarity function. Sun [21] drew main curves manually to inpaint the missing structure first and then filled the unknown information using texture propagation. Hung [22] used the structure generation and Bezier curves to construct the missing edge information. Using the structure information and reconnecting contours by curve filling process, the damaged regions will be inpainted. Xu [23] proposed two novel concepts of sparsity at the patch level for modeling the patch priority and patch representation. Goswami [32] designed an exemplar-based image inpainting using ISEF (Infinite size Symmetric Exponential Filter) for priority computation. Compared with the diffusion-based approaches, the exemplar-based approaches achieve impressive results in recovering textures and repetitive structures no matter whether they are applied into the large regions or not.

4. RESULT

   TABLE I. MERITS AND DEMERITS

   Algorithm	Merits	Demerits
   Partial	This method	This method is not suitable for
   differential	produces good	large region because it produce
   equation	result for small	blurred image and cant be able to
   (PDE) based	missing region	connect broken edges and texture
   Inpainting	and noise	pattern.
   	removal.
   Texture	The texture	Method is not suitable for a wide
   synthesis	synthesis based	variety of applications and large
   based	inpainting	missing region.
   inpainting	perform well in
   	approximating
   	textures.
   Exemplar	This method is an	Exemplar based inpainting will
   based	efficient approach	produce good results only if the
   inpainting	for reconstructing	missing region consists of simple
   	large target	structure and texture. And if there
   	regions and also	are not enough samples in image
   	for texture and	then it is impossible to synthesize
   	structural pattern.	the desired image.

5. CONCLUSION

Image inpainting is recently very important research area in the field of image processing. Still many work on images is

done. There are different methods for inpainting such as PDE based inpainting, Texture synthesis based inpainting and Exemplar based inpainting. According to analysis exemplar based inpainting is better as compare to others because that is efficient approach for reconstructing large target regions and also for texture and structural pattern. In future we would try to improve exemplar based inpainting algorithm by using some best techniques those are useful for reducing computation complexity and produce better result.

REFERENCES

1. The M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, Image inpainting, in Proc. SIGGRAPH, 2000, pp. 417424.

2. M. Bertalmio, A. L. Bertozzi, and G. Sapiro, NavierStrokes, fluid dynamics, and image and video inpainting, in Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition, 2001, pp. 417424.

3. T. Chan and J. Shen, Local inpainting models and tv inpainting, SIAM J. Appl. Math., vol. 62, no. 3, pp. 10191043, 2001.

4. T. Chan and J. Shen, Non-texture inpainting by curvature-driven diffusions, J. Vis. Commun. Image Represent., vol. 4, no. 12, pp. 436449, 2001.

5. C. Bertalmio, M. Bertalmio, V. Caselles, G. Sapiro, and J. Verdera, Filling-in by joint interpolation of vector fields and gray levels, IEEE Trans. Image Process., vol. 10, pp. 12001211, 2001.

6. A. Levin, A. Zomet, and Y. Weiss, Learning how to inpaint from global image statistics, in Proc. Int. Conf. Comp. Vision, pp. 305313.

7. S. Roth and M. J. Black, Fields of experts: A framework for learning image priors, in Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition, 2005, pp. 860867.

8. S. Roth and M. J. Black, Steerable random fields, in Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition, 2007, pp. 18.

9. J.Hahn, X.C.Tai, S.Borok, A.M.Bruckstein, Orientation-matching minimization for image denoising and inpainting, Int.J.Comput.Vis. 92 (2011) 308324.

[10]J.Miyoun, X.Bresson, T.F.Chan, L.A.Vese, Nonlocal MumfordShah regularizers for color image restoration, IEEE Trans. Image Process. 20 (2011) 15831598.

[11]Y.W.Wen, R.H.Chan, A.M.Yip, A primaldual method for total variation-based wavelet domain inpainting, IEEE Trans. Image Process. 21(2012) 106114.

[12]David J. Heeger , James R. Bergen, Pyramid-based texture analysis/synthesis, Proceedings of the 22nd annual conference on Computer graphics and interactiv techniques, pp.229-238, September 1995.

[13]C.W. Fang and J.J.J. Lien. Fast Image Replacement Using Multi-Resolution Approach. Jiaya Jia and Chi Keung Tang. Inference of Segmented Color and Texture Description by Tensor Voting. IEEE.

[14]H. Igehy , L. Pereira, Image replacement through texture synthesis, Proceedings of the 1997 International Conference on Image Processing (ICIP '97), Volume 3, October 26-29, 1997.

1. Efros, A.A., Leung, T.K., "Texture synthesis by non-parametric sampling," The Proceedings of the Seventh IEEE International Conference on Computer Vision , Volume 2, pp.1033-1038, 1999.

2. Alexei A. Efros , William T. Freeman, Image quilting for texture synthesis and transfer, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pp.341-346, August 2001.

3. Iddo Drori , Daniel Cohen-Or , Hezy Yeshurun, Fragment-based image completion, ACM Transactions on Graphics (TOG), Volume 22, no. 3, July 2003.

[18]A.Criminisi, P.Perez, K.Toyama, Object removal by exemplar based inpainting, in:Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003, pp. 721728.

[19]A. Criminisi, P. Perez, and K. Toyama, Region filling and object removal by examplar- based image inpainting, IEEE Trans. Image Process., vol. 13, pp. 1200 1212, 2004.

[20]W. H. Cheng, C. W. Hsieh, S. K. Lin, C. W. Wang, J. L. Wu, Robust algorithm for exemplar-based image inpainting, in: Proceedings of International Conference on Computer Graphics, Imaging and Visua-lization, 2005, pp. 6469.

[21]J. Sun, L.Yuan, J.Jia, H.Y.Shum, Image completion with structure propagation, ACM Trans. Graph. 24 (2005) 861868.

[22]J.Y. Wu, Q.Q. Ruan, A novel exemplar-based image completion model, J. Inf. Sci.

Eng. 25 (2009) 481497.

[23]Z. Xu, J. Sun, Image inpainting by patch propagation using patch sparsity, IEEE Trans. Image Process. 19 (2010) 11531165.

[24]S. Hesabi, N. Mahdavi-Amiri, A modified patch propagation-based image inpainting using patch sparsity, in: Proceedings of International Symposium on Artificial Intelligence and Signal Processing, 2012, pp. 043048.

[25]D. Jemi Florinabel, S. Ebenezer Juliet, V. Sadasivam, Combined frequency and spatial domain-based patch propagation for image completion, Comput. Graph. 35 (2011) 10511062.

[26]O. Le Meur, J. Gautier, C. Guillemot, Examplar-based inpainting based on local geometry, in: Proceedings of IEEE International Conference on Image Processing, 2011, pp. 34013404.

[27]S. Wang, Y. Xu, X. Yang, "Joint D-S evidence theory and priority computation for image completion", in: Recent Advances in Computer Science and Information Engineering, Springer, Berlin, Heidelberg, 2012, pp. 529535.

[28]Q. Zhang, J. Lin, Exemplar-based image inpainting using color distribution analysis, J. Inf. Sci. Eng. 28 (2012) 641654.

[29]S. Badgujar, N. M. Shahane, 2013, Examplar based image inpainting and approaches to improve the performance, International journal of computer applications (0975 8887).

[30]P. Dhabekar, G. Salunke , M. Gupta, 2013, The examplar-based image inpainting algorithm through patch propagation, International Journal of Advanced Research in Computer Science and Software Engineering 3(7), pp. 1114-1121.

1. Zhidan Li, HongjieHe, ZhongkeYin, FanChen, 2013, A color gradient patch sparsity based image inpainting algorithm with structure coherence and neighborhood consistency, Elsevier Signal Processing 99(2014) pp. 116128.

2. Pratikgiri Goswami, and Chirag Paunwala, 2014, Exemplar-based image inpainting using ISEF for priority computation, IEEE International Conference on Circuits, Systems, Communication and Information Technology Applications(CSCITA), pp. 75-80.