A modified patch propagation-based image inpainting using patch sparsity

A Modified Patch Propagation-based Image Inpainting Using Patch Sparsity Somayeh Hesabi1, Nezam Mahdavi-Amiri2 Faculty of Mathematical Sciences Sharif University of Technology Tehran, Iran 1 [email protected], [email protected] Abstract—We present a modified examplar-based inpainting method in the framework of patch sparsity. In the examplar-based algorithms, the unknown blocks of target region are inpainted by the most similar blocks extracted from the source region, using the available information. Defining a priority term to decide the filling order of missing pixels ensures the connectivity of the object boundaries. In the exemplar-based patch sparsity approach, a sparse representation of missing pixels is considered to define a new priority term and the unknown pixels of the fill-front patch is inpainted by a sparse combination of the most similar patches. Here, we modify this representation of the priority term and take a measure to compute the similarities between fill-front and candidate patches. Also, a new definition is proposed for updating the confidence term to illustrate the amount of the reliable information surrounding pixels. Comparative reconstructed test images show the effectiveness of our proposed approach in providing high quality inpainted images. Keywords—Image inpainting; texture synthesis; patch sparsity. I. INTRODUCTION Restoring damaged regions of an image and removing undesired objects are termed as image inpainting. The basic idea is to fill in the lost or broken parts of an image using the surrounding information in such a way that the final restored result appears to be natural to a not familiar observer. The applications of image inpainting techniques include removal of scratches in old photographs, repairing of damaged regions of an image, removal of undesired objects, and restoration of missing blocks of transmitted images. The user is to identify the missing or damaged areas objectively, since these areas cannot be easily classified. These specified regions are called inpainting domain or target regions and the undamaged parts, whose information is used to repair the target region, are called source regions. Recently, several image inpainting approaches have been developed. They are roughly categorized into two main types: PDE (Partial Differential Equation)-based methods, and texture synthesis approaches. PDE-based methods use partial differential equations, which propagate edge information along isophote (i.e., a line with all the points having the same gray value) directions with diffusion techniques. Bertalmio et al. [1] were first to introduce an image inpainting method. Their PDE-based scheme propagated boundary information of the inpainting domain along the isophote directions. Inspired by the work in [1], two other PDEbased algorithms [2, 3] were proposed by Chan and Shen. The Total Variational (TV) model [2] uses an Euler-Lagrange equation coupled with an anisotropic diffusion to preserve the direction of isophotes. This method does not restore a single object well when its disconnected remaining elements are separated far apart within the target region. The Curvature Driven Diffusion (CDD) model [3] considers geometric information by defining the strength of isophotes. This extended version of the TV approach can inpaint larger damaged regions. A simple inpainting algorithm based on prior models was presented by Roth and Black [4]. They modified the diffusion technique of denoising approaches to learn image statistics from natural images and then applied it to the target region. Noori et al. [22] proposed a method based on bilateral filters which originally used for image denoising. They iteratively convolved the damaged image with a space variant kernel to restore the thin regions. In almost all PDE-based algorithms, blurring artifacts may be produced when the missing regions are large and textured. So, these methods perform well on images with pure structures or thin target regions. To fill in large regions with pure textures, the second class of approaches, texture synthesis techniques, were proposed. The common idea in these methods is to duplicate the information on the source region into the target region [5-10]; hence, the texture information is preserved. Texture synthesis approaches are classified into pixelbased sampling [5-7] and patch-based sampling [8-10] according to the sample texture size. Since the filling process in pixel-based scheme is performed pixel by pixels, the algorithm is very slow. However, the speed of patch-based sampling was greatly improved by even though filling in the target region is by blocks of pixels, but discontinuous flaws between neighboring patches still remains. As natural images usually contain both structure and texture components, Bertalmio et al. [11] decomposed the input image into its structure and texture components and then restored them separately. The final outcome was the sum of two reconstructed components. This method is not appropriate for repairing large and thick damaged regions, because the PDE-based approach being used to construct the structure component often admits blurring artifacts. Criminisi et al. [12] presented an examplar-based inpainting technique to propagate the known patches (i.e., examplars) into the missing ones by ordering the synthesizing process. They can remove large objects from the image according to the defined patch priority values assigned to the pixel. Constructing information on both the structure and the texture characteristics for large and thick damaged regions is a special feature of this method. Other examplar-based schemes were also proposed [13-15]. Comparing with the PDE-based approaches, the examplar-based inpainting algorithms have produced plausible results; however, they appear to fail on repairing other types of structures, such as curves, and are often faced with some artifacts in the output image. Some approaches based on image sparse representation were also introduced for the inpainting problem [16-19]. In these methods, an image is presented by a sparse combination of an overcomplete set of transformations (e.g., wavelet, contourlet, DCT, etc.), and then the missing pixels are inferred by adaptively updating the sparse representation. In [16], Elad et al. proposed an approach to separate the image into cartoon (structure) and texture components, and then represented the sparse combination of the two obtained components by two incoherent over-complete transformations. Although this approach can effectively fill in the regions with structure and texture, it may fail to repair the structure or might produce smoothing deffects similar to the PDE-based approaches. Fadili et al. [17] proposed a sparse representation-based iterative algorithm for image inpainting. They used the Expectation Maximization (EM) framework to consider recovering the missing samples based on representations. The proposed approach in [18] considered a simple exemplar-based model via global optimization. Hence, some problems associated with progressive fill-ins were avoided. Xu and Sun [19] presented an exemplar-based inpainting method using a patch sparsity representation. They introduced the idea of sparse representation under the assumption that the missing patch could be represented by sparse linear combinations of candidate patches. Then, a constrained optimization model was proposed for the patch inpainting. Nonetheless, at times the edges in the filled regions are not connected properly. Here, we intend to improve the patch sparsity image inpainting scheme based on the patch propagation scheme proposed in [19]. The remainder of our work is organized as follows. In Section 2, we explain the patch sparsity-based image inpainting. We present our proposed method in Section 3. Performance of our technique investigated in Section 4. Finally, Section 5 gives the concluding remarks. II. PATCH SPARSITY-BASED IMAGE INPAINTING We use the common notations being used in the inpainting literature. The target region, i.e., the region to be filled in, is shown by Ω, and its boundary is denoted by δΩ . The source region, ϕ , which remains fixed throughout the algorithm, supplies samples to fill in the missing regions (refer to Fig. 1). Figure 1. Notation diagram: The original image to be filled with target region , its boundary and the source region . The conventional examplar-based method [12] is as follows. Algorithm 1: A conventional method. Step 1: For each point construct a patch the patch. examplar-based on the boundary δΩ , , with in the center of Step 2: Compute the patch priority ( ) : ( ) is defined as the product of two terms: a confidence term ( ), and a data term ( ) (see Fig. 2): ( ) = ( ). ( ), ( )= ∈ ∩ ( ) , (1) ( )= ∇ ! ." # , (2) is the area of , $ is a where, normalization factor (e.g., $ = 255 for a typical grey-level image), % is a unit vector orthogonal to the boundary at the point , and ∇& is an isophote vector. ( ) lets linear structures to be synthesized first, and thus propagates securely into the target region, ( ) illustrates the amount of the reliable information surrounding the pixel and is initialized to be ( ) = 0, ∀ ∈ Ω, and ( ) = 1, ∀ ∉ Ω. Figure 2. Given the patch for image I, % is the normal to the boundary at , and ∇+& is the isophote vector. Step 3: Find the patch , with the highest priority being filled in with the information extracted from the source region ϕ (Fig. 3 (a)). Step 4: Make a global search on the whole image to having the most similarity find a patch with , . Formally, , = arg min ∈3 45 ,, 6, (3) where the distance 4 between two generic patches is simply defined as the sum of squared differences (SSDs) of the already known pixels in the two patches (Fig. 3 (b)). Step 5: Copy the value of each pixel to be filled in, 7 7 | ∈ ,∩9 , using its corresponding position inside , (Fig. 3 (c)). Step 6: Update the confidence term ( ) in the area encircled by , as follows: (:) = ( ; ), ∀: ∈ , ∩ Ω. (4) (a) (b) (c) Figure 3. Algorithm 1: (a) The patch , with the highest priority is found to be filled in, (b) The most similarity candidate patches with , are determined, e.g. 7 and 77 , (c) The best matching patch in the candidate set copied into the position occupied by , , and thus partial filling of Ω is achieved. Before discussing our improvement of the patch sparsity inpainting method proposed in [19], below we first explain the general steps of the algorithm. The method investigates the sparsity of image patches and measures the confidence of the patch located at the structure region by the sparseness of its nonzero similarities to the neighbouring patches. The patch with larger structure sparsity is assigned a higher priority for further inpainting. The algorithm can be presented in six steps. Steps 1, 3, 5, and 6 are quite similar to the conventional examplar-based method (Algorithm 1). The priority term in Step 2 is computed in a different way. A sparse linear combination of weighted candidate patches is also used to fill in the missing patch instead of using a single best match in Step 4. Therefore, the steps 2 and 4 of Algorithm 1 are modified as follows: Step 2: A new definition for patch priority, namely structure sparsity, is proposed. For any selected patch, a collection of neighbouring patches with highest similarities is also distributed in the same structure or texture. Therefore, the confidence of structure for a patch is measured by the sparseness of its nonzero similarities to the neighbouring patches. The patch with more sparsely distributed nonzero similarities is laid on the fill-front due to the high structure sparseness. For the patch , located at the fill-front δΩ, a neighbourhood window <( ) , with the center , is set (refer to Fig. 4 (a)). The sparseness of similarities for the patch is measured by =( ) = >? A CDE ( ) @ B, A F |DE ( )| |D( )| , (5) where the patch A is located in the known region centered at G , and @ , A refers to the similarity between and A (Fig. 4 (b)), as defined by: @ , A = H I( ) exp M− OP , RS A Q T, (6) with 4 measuring the mean squared distance of the already known pixels in the two patches, Z( ) being a normalization constant so that A CDE( ) @ , A = 1, and σ being set to 5. Finally, <W ( ) is defined to be <W ( ) = X G : G ∈ <( )and A ⊂ Ω\. (7) The patch priority (or structure sparsity) term is defined as the product of the transformed structure sparsity term and the patch confidence term that is, ( ) = ]^_,H` 5=( )6. ( ), (8) where ]^_,H` is a linear transformation taking =( ) into the interval ^a, 1` . This transformation scales the structure sparsity variations to be comparable with ( ). Step 4: Contrary to the traditional examplar-based method [19], in which the fill-front patch , is filled by the best match , , in the patch sparsity inpainting method, , is inpainted by the sparse combinations of multiple examplars in the framework of sparse representation. Indeed, from the source region, the top < most similar patches are selected as the set of candidates b c D dH . Therefore, the unknown pixels in patch , is approximated by linear combinations of D the b c dH , that is, , = D dH $ , where the coefficient vector $e = f$H , $B , … , $D h is obtained by solving a constrained optimization problem in the framework of a sparse representation (Fig. 4 (c)). (9) (a) (b) (c) Figure 4. The patch sparsity inpainting method proposed in [19]: (a) The neighbourhood window <( ), with the center , located at the fill-front ij, (b) To measure the sparseness of similarities for the is set for the patch and A , @ , A , is computed, (c) The unknown pixels in patch , , that has the highest similarity between priority, is filled by linear combinations of the top < most similar patches b b$ c D dH obtained by solving an optimization problem. This optimization problem minimizes the 0 norm of $e , i.e., the number of nonzero elements in the vector $e, with the linearity assumption of the combination. III. c THE PROPOSED METHOD Our proposed algorithm is given next. We modify steps 2, 4 and 6 to attain better results. Algorithm 2: A modified patch propagation-based image inpainting using patch sparsity. Step 1: For each point on the boundary δΩ , construct a patch is constructed, with as the center of the patch (as in [12] and [19]). Step 2: To compute the patch’s priority ( ), use a stable definition is used as follows: 7 ( ) = $]^_,H` 5=( )6 + l]^m,H` ( ( )), (10) where the terms ]^_,H` 5=( )6 and ( ) are the same as the ones defined in [19], and $ and l are the component weights with 0 ≤ $, l ≤ 1 and $ + l = 1. As illustrated in [20], the confidence value rapidly drops to zero as the filling process goes on. When the dropping effect occurs, error continually propagates to the central part of the reconstructed image, causing noticeable visual artifacts. Therefore, a regularizing transformation is used to control the decreasing rate of the confidence term. We propose using a linear transformation ] 7 to take ( ) into the interval ^o, 1`. Also, the priority term is changed to an additive form instead of a multiplicative form (because the D dH weighted by coefficients numerical multiplication is effectively sensitive to extreme values, while the additive form has shown to be more robust with respect to its input and hence more stable). Step 3: Find the patch , with the highest priority to be filled in with the information extracted from the source region ϕ (as in [12] and [19]). Step 4: Perform a global search on the whole image to find the < most similarity patches. Formally, 6 p = arg min ∈3 b45 , , + 4qrsO 5 + 4Otu 5 ,, ,, 6 6c, (11) where the distance 4 between two generic patches is simply defined as the sum of squared differences (SSDs) of the already known pixels in the two patches. As a patch may contain texture details, we also include the gradient and divergence differences ( 4qrsO and 4Otu , respectively) between the already known pixels in the two patches to attain more accuracy. Hence, these added terms cause the preservation of texture properties. Step 5: Once the < most similar patches are found, we use a linear combination of patches D b c dH to finally obtain the prediction of the missing pixels in , (as in [19]). Step 6: Let , = D dH $ . (12) Denote the unknown pixels in the patch , , by a matrix , filled by the corresponding pixels in , : , = ,. (13) As illustrated in [19], the coefficients $e = f$H , $B , … , $D h are obtained by solving the following constrained optimization problem: min v$evw B x. y. z { , − { , z | }, l~ , D t $t − A ∈DE ( ) = 1. @ , A A where, =€ • − { •l =ƒ •l •z B ‚, { (15) , A ∈DE ( ) @ , A A „. , D t $t − B • z | }, = 1, . (18) b ∈ • ∩Ž c • . ( ; ), ∀: ∈ , ∩ Ω, (19) where the numerator in the first term is considered to be the number of known pixels in the patch , . By this definition, we make a difference between the centered pixel, ; , and the pixels recently filled in, 7 | 7 ∈ ,∩Ž ; the new filled in pixels do not have information as reliable as the centered pixel. Hence, we can reduce the rate of error diffusion as the filling proceeds. Next, we present our experimental results. IV. IMPLEMENTATION AND EXPERIMENTAL RESULTS We have implemented our proposed method and compared the results with the ones obtained by a PDEbased technique [4], conventional approach [12], and the patch sparsity method [19] on several different images. To set up the same computing environment, we have implemented Algorithm 2, the conventional approach in [12], and the patch sparsity method in [19] using Matlab 7.10.0 instructions and executed the programs on a laptop with CPU specifications: Intel® Core™ i5, 2.4 GHz, and 4GB of RAM. The results for the method in [4] were obtained using the code provided by the first author of [4]. As proposed in [12], we applied the conventional method with the patch size (i.e., size of ) being equal to 9×9 pixels. The parameters used in the patch sparsity approach are the same as the ones given in [19]. In our proposed algorithm, the size of the patch window was set to 7×7 pixels. For most cases in Fig. 1, the size of the neighbourhood window, <( ), was set to 11×11 and the value of $ and l were set to 0.5 and 0.5, respectively. Also, the number of candidate patches, <, was fixed to be 21. (16) So, the optimization problem can be formulated as minv$evw x. y. z †Œ Š ‹ˆ † (:) = (14) | }, Š ‹ˆ † Step 7: Update the confidence term ( ) in the area encircled by , as follows: B The local patch consistency constraint can be rewritten in a compact form: , $e = ~ | }, The first and the second constraints concern the local patch consistency. The first constraint constrains the estimated patch , approximated by the target patch , over the already known pixels, and the second one forms the consistency between the newly filled pixels and the neighboring patches in appearance. It measures the similarity between the estimated patch and the weighted mean of the neighboring patches over the missing pixels. The last constraint imposes a normalization summation on the coefficients vector $e. This constraint is used to achieve invariancy while reconstructing the target patch from its neighboring candidate patches. The parameter } is to control the error tolerance, l balances the strength of the two first constraints, which is set to 0.25 in our implementation (as in [19]), and @ , A is defined to be as given in (6). z of ones. Then, we have a closed form solution as (a) (b) (17) which can be solved in the same way as in Locally Linear Embedding (LLE) for data reduction [21]. In our case, the Gramm matrix is … = ( • †• − ‡)• ( • †• − ‡) , where X is a matrix with columns b ,…, c, and 1 is a column vector ˆ ‰ (c) (d) Figure 5. The effect of new definition for the patch priority: (a) Original image, (b) Degraded image, (c) Inpainted image with the multiplicative form of ( ), and (d) Inpainted image with the additive form of ( ). Fig. 5 shows the constancy definition for the patch priority ( ) . As the numerical multiplication is effectively sensitive to extreme values, the additive form is more robust and more stable than the multiplicative form. So, the obtained result for this new definition is better than the result obtained by the conventional one. Fig. 6 shows some results obtained by our method in comparison with the proposed methods in [12] and [19] for the application of object removal. The first and second columns are the original and (a) (b) degraded images, respectively, the third column illustrates the results obtained from the conventional algorithm, the fourth column shows the results of the patch sparsity method, and the last two columns demonstrate the results obtained by our proposed algorithm using the conventional and the newly proposed similarity metric, respectively. As being observed in Fig. 6, the results of our proposed algorithm are superior over the two other ones, even using the conventional similarity metric. We visually PSNR = 37.23 (dB) SSIM = 0.994 PSNR = 42.22 (dB) SSIM = 0.996 PSNR = 41.31 (dB) SSIM = 0.991 PSNR = 43.0 (dB) SSIM = 0.997 PSNR = 31.89 (dB) SSIM = 0.967 PSNR = 34.94 (dB) SSIM = 0.985 PSNR = 35.26 (dB) SSIM = 0.979 PSNR = 35.56 (dB) SSIM = 0.986 PSNR = 31.09 (dB) SSIM = 0.980 PSNR = 42.59 (dB) SSIM = 0.996 PSNR = 42.64 (dB) SSIM = 0.992 PSNR = 43.11 (dB) SSIM = 0.997 PSNR = 35.89 (dB) SSIM = 0.980 PSNR = 35.15 (dB) SSIM = 0.974 PSNR = 35.86 (dB) SSIM = 0.976 PSNR = 36.52 (dB) SSIM = 0.982 PSNR = 26.50 (dB) SSIM = 0.933 PSNR = 33.02 (dB) SSIM = 0.957 PSNR = 27.88 (dB) SSIM = 0.941 PSNR = 26.07 (dB) SSIM = 0.896 (c) (d) (e) (f) Figure 6. Results obtained by Algorithm 2 in comparison with the ones obtained by the methods in [12] and [19]: (a) Original image, (b) Degraded image, (c) Inpainted image from [12], (d) Inpainted image from [19], (e) Inpainted image by Algorithm 2 using (3) as similarity metric, and (f) Inpainted image by Algorithm 2 using the proposed similarity metric, (11). similarity metric, i.e., (3), contain artifacts that are not well suited for the neighborhood, whereas our metric, i.e., (11), considering more properties between the patches, selects more visually pleasant patches and produces better results. find out that the presented results in column (e) are more acceptable than those illustrated in columns (c) and (d). Also, robustness of the newly proposed similarity metric is affirmed by comparing the fifth and sixth columns. We notice that patches chosen with just the conventional (a) (b) (c) (d) Figure 7. The effect of the new definition for confidence term, ( ): (a) Original images, (b) Degraded images, (c) Inpainted images with the conventional definition for ( ), and (d) Inpainted images with the new definition for ( ). The modification (19) for updating the confidence term after filling in the unknown pixels reduces the error propagation, since the rate of reliable information is controlled by the difference of the centered pixel and the newly filled ones. The effect of this new definition is shown in Fig. 7. The first and second rows show the original and degraded images, and the third and fourth rows show the images obtained by our proposed method using the conventional definition (4) and the newly proposed one (19) for the confidence term, respectively. As observed in Fig. 7, the result obtained by the new confidence term is more plausible. The original Images (a) (b) (c) The degraded Images Results of PDE-based algorithm [4] PSNR = 36.64 (dB) SSIM = 0.948 PSNR = 34.74 (dB) SSIM = 0.959 PSNR = 32.69 (dB) SSIM = 0.941 PSNR = 28.62 (dB) SSIM = 0.882 PSNR = 28.15 (dB) SSIM = 0.956 PSNR = 33.33 (dB) SSIM = 0.942 PSNR = 37.55 (dB) SSIM = 0.976 PSNR = 35.74 dB SSIM = 0.979 PSNR = 39.47 (dB) SSIM = 0.994 PSNR = 37.98 (dB) SSIM = 0.981 PSNR = 37.06 (dB) SSIM = 0.985 PSNR = 42.54 (dB) SSIM = 0.996 Results of conventional inpainting method [12] Results of patch sparsity approach [19] Results of our proposed algorithm Figure 8. Results obtained by Algorithm 2 in comparison with the ones obtained by the methods in [4], [12] and [19]: First and second rows are the original and degraded images, respectively. The third row shows the results of PDE-based technique [4], the results of conventional inpainting method [12] are presented in the forth row, the results obtained by [19] are illustrated in the fifth row, and the last row demonstrates the results of our proposed algorithm. Fig. 8 presents more examples for the application of text removal, scratch restoration and object removal. The first and second rows show the original and degraded images, and the third to the sixth rows respectively show the images obtained by a PDE-based technique [4], the conventional algorithm [12], the patch sparsity method [19] and our proposed algorithm. Roth and Black [4] used the prior models to restore the image; the models have shown to be useful when images have noise or uncertainty. They proposed a framework for learning image priors. The prior model used in their work was Markov random field (MRF), which assumes an image to be the result of a random process, described by an MRF. For images in Fig. 8, the size of the patch window was set to 7×7 pixels and the value of $ and l were set to 0.5 and 0.5, respectively. The size of the neighbourhood window, <( ), was set to 11×11, 19×19, and 51×51 pixels for images (a), (b) and (c), respectively. Also, the number of candidate patches, <, was fixed to be 11, 3 and 25, correspondingly. We have used the parameters of the approach in [4] unchanged for all the examples considered here. This scheme converted color image to YCbCr color model, and the algorithm was independently applied to all 3 channels. We used a FoE prior with 8 filters of 3×3 pixels. The algorithm inpainted the missing areas by iteratively propagating information. We set the number of iterations to 200. As seen in Fig. 8, the PDE-based method [4] performs well on piecewise smooth image, Fig. 8 (a), but fails to reconstruct areas containing texture with fine details and tends to blur the inpainted image, Fig. 8 (b) and (c). The conventional algorithm [12] cannot preserve the edge continuity and the texture consistency, and thus produce unpleasant artifacts. The patch sparsity algorithm [19] produces more pleasant results; however, it fails to reconstruct some edges properly. In contrast, the results obtained by our algorithm appear to be closer to the original image than the ones obtained by the other three methods. Because of the improvement in the priority term, high importance was given to the structures in a more robust way. Also, the proposed similarity metric encompasses more properties of the patches and selects the most visually pleasant ones. Furthermore, the modification in the confidence term after filling in pixels helps reduce the error propagation. For a quantitative comparison, we computed the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) values between the original and inpainted images, also observing overally better obtained PSNR and SSIM values by Algorithm 2. However, similar to [19], our algorithm cannot properly recover large missing areas consisting of structures while the known region doesn’t contain any structure cues. This is a limitation of our algorithm, which needs to be investigated in a future work. V. CONCLUSION We presented a modified patch sparsity scheme for inpainting degraded images. Addressing the patch sparsity approach as a robust inpainting method, the suggested modifications lead to an improvement in producing better results. We applied the proposed algorithm to several images and compared the obtained result with the ones obtained by three other methods. The high visual quality of the results obtained by our approach affirmed the effectiveness of the proposed algorithm. In future, we will also investigate the effects of varying patch sizes and try to select an optimal size. Moreover, the number of sufficient candidate patches would be explored since it has a direct impact on the computational time of inpainting. ACKNOWLEDGMENT The authors sincerely thank Stefan Roth for distributing the inpainting code and also Jian Sun for providing the original and degraded images as well as the results obtained by the patch sparsity method. The second author also thanks Sharif University of Technology for its support. REFERENCES [1] M. Bertalmio, G. Sapiro, V. Caselles and C. Ballester, “Image inpainting”, Computer Graphic (SIGGRAPH 2000), pp. 417– 424, July 2000. [2] T. F. Chan and J. Shen, “Mathematical models for local nontexture inpainting”, SIAM Journal of Applied Mathematics, vol. 62, no. 3, pp.1019–1043, 2002. [3] T. F. Chan and J. Shen, “Non-texture inpainting by curvaturedriven diffusions”, Journal of Visual Communication and Image Representation, vol. 12, issue 4, pp. 436-449, 2001. [4] S. Roth and M. J. Black, “Fields of Experts”, International Journal of Computer Vision Computer, vol. 82, no. 2, pp. 205229, 2009. [5] M. Ashikhmin, “Synthesizing natural textures”, ACM Symposium on Interactive 3D Graphics, pp. 217–226, 2001. [6] A. A. Efros and T.K. Leung, “Texture synthesis by nonparametric sampling”, IEEE International Conference on Computer Vision, vol. 2, pp. 1033–1038, 1999. [7] L. Y. Wei and M. Levoy, “Fast texture synthesis using treestructured vector quantization”, Proc. ACM Conf. Comp. Graphics, pp. 479–488, 2000. [8] A. A. Efros and W.T. Freeman, “Image quilting for texture synthesis and transfer”, Proc. ACM Conf. Comp. Graphics (SIGGRAPH), Eugene Fiume, pp. 341–346, 2001. [9] W.T. Freeman, T.R. Jones and E.C. Pasztor, “Example-based super-resolution”, IEEE Computer Graphics and Applications, vol. 22, issue 2, pp. 56–65, 2002. [10] L. Liang, C. Liu, Y. Xu, B. Guo and H.-Y. Shum, “Real-time texture synthesis using patch-based sampling”, ACM Trans. on Graphics, vol. 20, issue 3, pp. 127–150, 2001. [11] M. Bertalmio, L. Vese, G. Sapiro and S. Osher, “Simultaneous structure and texture image inpainting”, IEEE Trans. Image Process, vol. 12, pp. 882–889, 2003. [12] A. Criminisi, P. Perez and K. Toyama, “Object removal by exemplar based inpainting”, IEEE Computer Vision and Pattern Recognition (CVPR), vol. 2, pp. 721–728, 2003. [13] I. Drori, D. Cohen-Or and H. Yeshurun, “Fragment-based image completion”, ACM Trans. Graph., vol. 22, no. 2003, pp. 303– 312, 2005. [14] J. C. Hung, C. H. Hwang, Y. C. Liao, N. C. Tang and T. J. Chen, “Exemplar-based image inpainting based on structure construction”, Journal of Software, vol. 3, no. 8, pp. 57-64, 2008. [15] A. Wong and J. Orchard, “A nonlocal-means approach to exemplar-based inpainting”, Proceeding of IEEE International Conference on Image Processing, pp. 2600-2603, 2008. [16] M. Elad, J. L. Starck, P. Querre and D. L. Donoho, “Simultaneous cartoon and texture image inpainting using morphological component analysis”, Appl. Comput. Harmon. Anal., vol. 19, pp. 340–358, 2005. [17] M. J. Fadili, J. L. Starck and F. Murtagh, “Inpainting and zooming using sparse representations”, The Comput. J., vol. 52, no. 1, pp. 64–79, 2009. [18] B. Wohlberg, “Inpainting by joint optimization of linear combinations of examplars”, IEEE Signal Processing Letters, vol. 18, no. 1, pp. 75–78, Jan 2011. [19] Z. Xu and J. Sun, “Image inpainting by patch propagation using patch sparsity”, IEEE Transactions on Image Processing, vol. 19, no. 5, pp. 1153–1165, 2010. [20] W.-H. Cheng, C.-W. Hsieh, S.-K. Lin, C.-W. Wang and J.-L. Wu., “Robust algorithm for exemplar-based image inpainting”, CGIV, 2005. [21] S.T. Roweis, and L.K. Saul., “Nonlinear dimensionality reduction by locally linear embedding”, Scienc, vol. 290, pp. 2323–2326, 2000. [22] H. Noori, S. Saryazdi, and H. Nezamabadi-pour, “A bilateral image inpainting”, IJST Transactions of Electrical Engineering, vol. 35, no. E2, pp. 95–108, 2011.