Study on Creation of Video Inpainting Algorithm based on Texture Synthesis

Study on Creation of Video Inpainting Algorithm based on Texture Synthesis

S. Nithya

Assistant Professor, Department of ECE K.Ramakrishnan College of Technology Samayapuram, Trichy, Tamilnadu

J. Deepa

Assistant Professor, Department of ECE K.Ramakrishnan College of Technology Samayapuram, Trichy, Tamilnadu

Abstract For the broken video sequence in fixed background and approximate cyclical movement of prospect, this paper puts forward a digital video inpainting algorithm. It calculates the movement value of video in each frame and establishes panoramic picture. In the process of fixing background, according to movement value of current frame video and information of broken position, corresponding position in whole picture is directly discovered. During copying pixel value to fill broken and fix background, based on single-frame texture inpainting algorithm, this paper applies periodicity of foreground motion and expands the searching matching block scope to other frames of consistent corresponding foreground motion status in video sequence. The experiment results show whether broken region belongs to background or foreground, methods in this paper can more accurately restore the broken real scenario.

Keywords Inpainting; Texture; Frame; Priority; Confidence

can punch the holes correctly according to the size of the object. There are two widely alignment approaches, namely, the dense based approach and the sparse based approach. The dense approaches estimate the 2D or 3D motion vectors. The 2D methods compute the motion vectors between consecutive frames in the video. The 3D methods estimate the global camera motion by using all frames in the video. This generally provides more accurate results but at the expense of a higher computational cost. Sparse-based methods yield a fast and robust alignment. Sparse-based methods yield a fast and robust alignment. A single homography transformation is not sufficient to align a pair of images. The image inpainting problem can be formalized using either a local or global optimization framework. A well-known algorithm of this of this category is the exampler-based inpainting algorithm.

1. TEXTURE SYNTHESIS-BASED IMAGE RESTORATION ALGORITHM

   1. INTRODUCTION

      Digital inpainting technique mainly applies modern image and video processing technology to repair image and video problems such as scratching, spot, blink, etc. Its main working principle is to utilize neighborhood or indicated restoration region of inpainting region in digital image to fill broken region or given restoration region of image. It is broadly applied for image transmission, image compression, inpainting scratch and flaw picture, inpainting old video cinefilm. With the development of computer technology, it has become an important subject to semi-automatically or automatically complete image or video restoration realization in digital image processing field study. This paper applies image decomposition technology to digital image and video repair, and integrates partial differential equation-based restoration as well as texture synthesis advantages to study noisy damage image and video restoration. Then, in theory, we put forward a global optimization-based texture synthesis video restoration, and inputs structure information facto to consider texture, node and motion feature into video and image restoration. Simulation experiments show that the detection accuracy in this paper is high and repairing effect meets requirement. Furthermore, its calculating complexity is low and the processing speed is fast.

      Video inpainting is the process of removing the objects in

      a video. Videos are an important medium of communication and expression in todays world. Videos are widely used in a number of areas such as surveillance, movie industry, journalism, medical imaging, home videos and so on .current video inpainting techniques are generally complex in computation to find the similar patch to find the lost frames. Additionally, the holes also appear large. In our system, we

      1. Principle Idea

         We use the texture synthesis method based on sample block of Criminisi. Assuming the inpainting region of image I is , and its

         boundary is . The square module p of outline in object region, whose center point p is on the outline , its module contains part of synthesized pixels. Each pixel point in the module has a priority P and confidence C . The sequence of filling should be from outside to inside gradually. The procedures of algorithm are:

         Step 1: compute the priority of all the modules on and

         choose the block p waiting to be inpainted which has the maximum priority;

         Step 2: Search the optimal matching block e ;

         Step 3: Copy image data information of source sample

         matching block to target sample block, and mark p as inpainted;

         Figure 1. Schematic diagram of Criminisi algorithm

         The size of priority is determined by two factors: one is the data value of the module, which reflecting the weakness of structure information of module, to ensure that the linear

         structural part can be synthesized first; the other is the confidence value of module. It demands the modules containing more parts of filled pixels be filled, since such operation can depend on more known pixels.

         Give a point p , the equation of Pp is:

         Pp C ( p ) D( p) (1)

         C ( p)

         q p (1 )

         motion value between each frame is calculated at first and panoramic image of background is set up. During repairing background, according to motion value of current video player and broken location information, corresponding location is discovered in panoramic image to copy pixel value for filling broken part. During repairing foreground, based on single- frame texture restoration algorithm, periodicity of foreground movement is utilized to expand searching match block scope

         D ( p)

         C ( p)

         | I p np |

         | p |

         (3)

         (2)

         to other frames which is consistent to corresponding foreground movement in video sequence.

         Algorithm in this paper is adaptive to repair scene whose video sequence is made up by fixed background and motion foreground. Furthermore, foreground approximates to periodic

         C ( p) , D ( p) and C ( q) denote the confidence of

         module, data value of module, and confidence value of pixel inside the module.

         In initialization, the value of each point in target region is set as 0 and that of known region is set as 1. | p | is the area of module p and is a standardized parameter. For general grey graph 255 . np is unit normal vector of contour line at point p

         , and I p is the strength and direction at point p . According to equation 2 and 3, if the

         module contains more filled pixels, its confidence will be larger; The module with obvious structure and prominent line segment has bigger value. In addition, the priority of model with large value will tends to the part of prominent segments. The searching of optimal matching block can be based on the principle that corresponding pixels has the minimum variance.

2. VIDEO INPAINTING

   1. Basic Process

      Video restoration can be actually seen that time continuity adds to image restoration algorithm. Time is seen as the third dimension. If one video is seen as a very thick book, each frame image is one page in book. Smoothly mobile image data loss in video seems a twisted body worm to move forward to bite hole in book. Hole length is consistent time of broken image in video. Once the cave boundary is determined, program can restore three-dimensional cav which is bited by worm, it will restore each frame by two-dimension data destruction and create smooth video image quality.

      Patwardhan expanded single-frame texture restoration algorithm to the restored video sequence and respectively restores foreground and background according to different broken coverage scope. However, it can only process the recorded video under condition of silent video player with limitation. Based on the method in literature [5], Patwardhan restored the recorded video under video player motion status through setting up three mosaic images. In inpainting background, according to priority sequence, match block is only sought in background mosaic to be repeatedly filled. In inpainting foreground, foreground mosaic is used to search the sequence number which offers broken information of foreground in the whole sequence. Then, match blocks are sought in these frames to repair. However, foreground mosaic is very complex. In the process of background restoration, once one error match block is copied, it may possibly generate vicious reproduction so that repair results are not consistent in vision. For fixed background and broken video sequence whose foreground approximates to periodic movement, this chapter proposes a new repair algorithm. At first, video play

      movement. Background and foreground contain broken point and video player movement is not limited. There are four steps of algorithm process.

      Step 1: Calculate moving value of video player between each frame and roughly segment each frame scene in video sequence into foreground and background and artificially mark the position of broken frame

      Step 2: According to the calculated video player motion value between all frames and the segmented background between each frame, panoramic picture of background is established.

      Step 3: During repairing video frame only with broken background, according to moving value of all frame video player before current frame and the broken position information of current frame, corresponding position in panoramic image is directly discovered to copy color value to fill broken.

      Step 4: During repairing broken video frame in foreground field and background field, foreground field is repaired at first. The expanded single-frame texture restoration algorithm is adopted to search and copy match block of other frames consistent with foreground motion status in video sequence for filling. The repairing background of last-step algorithm is adopted after foreground restoration.

   2. Establish Panoramic Background

      When video is moving, video image will generate displacement in horizontal direction or vertical direction. We can calculate movement value of video between each frame. At first, one 3×3 background block whose one pixel point is not only broken but also covered by foreground in all video frames is taken as reference object. Various points position and corresponding color value of this background block is recorded in the first frame to render a 3x3x3 three- dimensional array, which is marked as a. Then, in each frame, 3×3 block b is sought according to formula 1 to find out the smallest point of record with its coordinate value. By means of one-by-one comparison, movement value of video player between frames is obtained. If background block of reference object is selected and there are many similar blocks in other part of background, it may result in error reference point. Therefore, it can be impCro. ved by two below aspects:

      According to above method, we select n small background

      reference blocks (al,a2,…,an) and determine position relationship of variously small-block center point in the first frame. In each one frame, the smallest point of value which is sought according to formula 2. Position relationship of various

      center points bl,b2…bn all correspond to position relationship of various block center point in al,a2,…,a n.

      Since reference object is artificially selected, we try to

      select blocks whose colors are largely different in image. For instance, border line is selected between different objects.

      After the movement value of video in each frame is calculated, through block match between each frame and its previous one frame, we mark the blocks with obvious displacement as foreground while other blocks are marked as background. During establishing whole image of background, we calculate panoramic size of background through mobile value of each frame video and fill each frame background in the panoramic image. When filling panoramic image, for some repeated filling fields, we get median of all pixel point color value in the same position as its color value.

   3. Broken Background Inpainting

      The broken background pixel point in current frame can directly use the aligned information in image neighboring frame for restoration. Since there are multi-reference image frame in time window, priority calculation is introduced to rank sequence in each neighboring frame image, utilize information of neighboring image frame in high priority value to restore broken pixel in current frame.

      Mark the damaged area of image frame I t at hour, The

      annular image area which expanded w pixels in width is tt k . Then corresponding image region to aligned image frame Itt k at hour t k is marked. Compute the MSN value among the three components of the known pixels in tt k and t and set the priority value from big to small for each frame, according to the sequence of MSN value. Assuming the frame which has the highest priority at hour t k is Itt k . First we

      search the corresponding I t k pixels in t , in the unknown background points of current frame I t . If the pixel information is known, complete the inpainting of unknown pixels of I t by copying its color information directly; otherwise, keep this pixel unknown. If all the processing of background pixels to be inpainted are completed and there are still unknown background pixels, we perform similar operation in the neighbor image frame which has higher priority, until the unknown background points of I t are inpainted, or the image frame in neighborhood of current time windows are visited, till the end.

   4. Broken Foreground Inpainting

      During repairing broken foreground, since the foreground is in constant motion status, unlike inpainting background, information can be copied in the established single panoramic image. Based on single frame texture restoration, we expand it to be applied to video restoration. We hope to obtain appropriate information from unbroken foreground in frame to fill. However, if foreground in the whole sequence all searches and matches, workload is huge inefficient. Considering the periodicity of foreground motion, we restrict the scope of searching match block into corresponding foreground motion status in video sequence and consistent as well as unbroken frame in current frame. This paper designs the following inpainting measures:

      1. Continuously read three frames and find out broken region. Because there is large difference between scratch lightness and its surrounding pixel, this can be utilized to find out broken region.

      2. Useful information before and after frame is used to restore. Because scratch cannot necessarily appear in each frame or same position of video, information before and after frame can be used to restore. If there is not broken part, it is over. If there is broken part, enter the third step.

      3. Pulse method is used to judge broken type. Small- scaled broken part is restored by partial differential equation- based restoration algorithm. Towards large scaled broken part, texture-based restoration algorithm is applied to restore.

      4. To judge whether this is broken in this frame, if there is broken, repeat (3); otherwise, go to the end.

3. EXAMPLER BASED METHOD

   Texture synthesis is used to fill huge image regions with texture pattern similar to given model. Texture is mainly the repetitive pattern of the image. So the large image regions are generated from sample texture. There are different techniques of Texture synthesis like Technique Statistical, pixel-based and patch-based. Statistical processes are important in reproducing unequal textures, but are a smaller amount useful for regular textures. Pixel-based methods repeat the sample texture pixel-by-pixel as a replacement for applying filters on it, and their outputs are better than those of statistical methods, but they usually be unsuccessful to grow large regular textures. Patch-based methods repeat the sample texture patch- by-patch as opposed to pixel-by-pixel, thus they give quicker and better results for regular textures. The hub of partial differential equation based algorithms is mainly on maintaining the formation of the inpainting area. It is good if lost area is small. But results blurred if lost area is large. Another drawback with these algorithms is that the large textured areas are not regenerated. PDE is a differential equation contains one or more variables, matching the values of the function itself and its derivatives of different orders. As a result, a PDE is a differential equation that uses partial derivatives.so, we choose another advance of exampler based method. The exemplar based image inpainting is a vital category of inpainting algorithms. The exemplar based image inpainting is a well-organized method of reinstallation of big target regions.

4. EXPERIMENTAL RESULTS ANALYSIS

   1. Simulation Settings

      This experiment processes a video with 48 frames in length and is marked from the first to the 48th frame according to priority order. The broken point is randomly constructed on the 16th frame. We respectively use texture to respectively restore synthesis-based single image restoration algorithm, TV model-based single image restoration algorithm and the algorithm in this paper. Figure 2 describes the result figure after these three algorithms are used to restore the fifth frame of this video sequence. Figure 2(a) is an image whose foreground region and background region has broken parts. Its original size is 240 x 320 pixel. Figure 2(b) shows the processed results that broken image of single- frame texture restoration. By means of observation, it can be discovered that foreground leg appears discontinuity and road appears obvious restoration. Figure 2(c) shows the result that TV model algorithm is applied to restore broken image. Foreground region appears large area of interruption. Figure 2(d) displays the result that algorithm in this chapter restores image. From this figure, whether foreground or background, there is no obvious restoration trace and inpainting effect is satisfactory.

      Through observing image after restoration and observing results after restoration to synthesize video for playing effect, the effect after restoration on algorithm in this paper is better than the result with single-frame texture and model

      restoration. However, the restoration effect of some images cannot only depend on peoples subjective judgment to evaluate, we compare through calculated value. Table 1 displays result figure of the proposed algorithm in this chapter, single-frame texture restoration algorithm and single frame model restoration algorithm after respective restoration and the compared calculation value of originally unbroken image. From this result, restoration effect of algorithm in this paper is better than single-frame image restoration effect.

      1. Broken image (b) Single frame texture inpainting

   (c) Single frame TV model inpainting (d) Improved algorithm Figure 2. Inpainting results of single

   frame

   REFERENCES

   1. S.Nithya, Study on Creation of Motion Pictures Inpainting Using Mean Shift Algorithm For Object Removal, IJEEE,2016,136-140

   2. Patwardhan Kedar A., Sapiro Guillermo, Bertalmío Marcelo. Video inpainting under constrained camera motion, IEEE Transactions on Image Processing, 2007, 16:545-553

   3. Koochari Abbas, Soryani Mohsen . Exemplar-based video inpainting with large patches,: Journal of Zhejiang University: Science C, 2010, 11:270-277

   4. Raimbault Félix, Kokaram Anil. Stereo-video inpainting, Journal of Electronic Imaging, 2012, 21:17-25

   5. Tang Nick C., Hsu Chiou-Ting, Su ChihWen. Video inpainting on digitized vintage films via maintaining spatiotemporal continuity, IEEE Transactions on Multimedia, 2011, 13:602-614

   6. Vijay Venkatesh M., Cheung, Senching Samson, Zhao Jian. Efficient object-based video inpainting, Pattern Recognition Letters, 2009, 30:168-179

   7. Jan Herling, Broll Wolfgang. High-quality real-time video inpainting with PixMix, IEEE Transactions on Visualization and Computer Graphics, 2014, 20:866-879

   [7] Newson Alasdair Almansa AndréS Fradet Matthie. Video inpainting of complex scenes, SIAM Journal on Imaging Sciences, 2014, 7:1993-2019

5. CONCLUSIONS

This paper generalizes related work of video inpaining and the research status of the newest domestic and foreign technological method. Then, it introduces a new thought and realization of video inpainting algorithm. Our proposed algorithm is adaptive to the scene of video sequence which is made up by fixed background and moving foreground. On this basis, foreground approximates to cyclical motion. At first, foreground and background are segmented and panoramic scenario of background is set up to restore background of broken frames. Then, when periodicity of foreground motion is applied to fix foreground of broken frame, matching block search is restricted in video frame consistent with motion state to keep overall consistence. The experiment results show that this algorithm is clear and natural to keep video image and overall visual effect is perfect.