Abstract—This paper proposes a way to enhance the
compression ratio of images by deleting some parts of the image before
transmission. The remaining data besides essential details for recovering the
removed regions are encoded to produce the output data. At the decoding side an
inpainting method is applied to retrieve the removed region. The Shearlet
Transform is used for the smoothing purpose of the recovered image. The
Shearlet Transform has the ability to provide a very precise geometrical
characterization of general discontinuity occurring in images. This transform
can identify the location of singularities of a function and also the
orientation of discontinuity curves.
Keywords- Image inpainting, image compression, compression ratio, peak
signal-to-noise ratio, structural similarity.
Image inpainting is a technique of filling in the missing
region of an image. It is the art of modifying an image in a form that is not
easily detectable by an ordinary observer. The main usage of inpainting is the
restoring of damaged part of the picture 1. Without transmitter sending
entire picture the receiver can recover the missing part of the image to come
up with the whole image in the end. Compression is acceptable for natural
images as a large amount of redundancy is included in such images. By not
sending significant portion of the image, that can later be restored from
remaining part, the amount of bits needed to transmit the image can be reduced
Each image is composed of discrete points called pixels.
The value relevant to each pixel is the result of sampling from light or color
intensity in the original image domain. Natural images consist of separate
areas indicating the object surfaces or sceneries. Since the light intensity
and color in such areas are approximately constant the relevant values for
pixels are highly correlated. Every pixels in such areas are likely to be of
the same or very close value compared with the adjacent pixels. Representing
the image by storing all pixel values results in a large amount of redundancy.
For the inpainting method to be successful it is important
to choose and erase the block that can be easily restored. There are two types
of regions that can be relatively easily reconstructed, structure and texture. It is important to properly
classify these blocks into either of these two. In image inpainting the only
information available for reconstruction is the average value of the erased
The most significant information within an image is
located in the boundary regions or edges. The boundary region not only
specifies its overall shape but also shows how pixel values change from
neighboring regions to the inner regions. So it is possible to retrieve the
inner areas using pixels located on the boundaries. Therefore boundaries or
edges are all the required information for displaying an image. The variation in values of pixels
orthogonal to the edges is significant. Hence, areas in the neighborhood of
edges may be considered as essential image information. While moving along the
edge direction, no significant changes in pixel values will be observed. Moving
further to the inner points of the boundaries will result in a considerable
correlation for pixel values. Edges also represent some other necessary
information including shapes. Redundancies related to the correlation along the
edge direction may also be exploited via extracting shape information from
pixel values in boundary regions.
Pixel values at the endpoints of an edge will be used for
recovering the entire edge and boundary region. In order to recover boundary
regions and pixels located perpendicular to the edge direction, samples of
source points should be provided. These samples should come from different
areas at each side of the edge.
inpainting is a method for recovering regions in images whose pixels are
distorted or removed in some way. Inpainting methods are commonly based on
partial differential equations. The method proposed in this paper is based on
eliminating the information of correlated regions and filling in the missing
areas using sample pixels. In this method, some regions are intentionally
removed at the encoder and recovered using an inpainting or interpolation
technique at the decoder. In partial differential equation techniques, pixel
values around the region to be inpainted are considered to be the boundary
condition for a boundary value problem. Then, a proper equation for
interpolating in that area will be solved. Image inpainting has a variety of
applications such as text and object removal 2,3, denoising, super
resolution, digital zooming 4,filling-in 5 and compression.
1 A general case of image inpainting
Image inpainting is aimed to fill in missing regions or to
modify damaged regions in a visually plausible and non-detectable way. In order
to well clarify the inpainting problem, assume u0 as the
intensity function of the image defined in domain D. As indicated in
Fig. 1, there is a hole ??D with unknown information. The objective is
to find the recovered version of u0, namely u, in such
a way that the intensity function in the area D?? remains equal
to u0 while meaningfully filling in other regions. In this
way, information on the boundary ? is diffused into ? via a 2D
general form, no information is available about the regions to be inpainted.
The resulting inpainted image is not necessarily similar to the original one. For
the application of compression, it is necessary for the inpainted image to be
similar to the original one with a sufficient degree of accuracy. As the
original image is in hand, it is possible to extract all of the information
required for compression with an acceptable quality. Here the only essential
information for retrieving an image includes source point pixels and edges.
Texture inpainting is to find best match from referable
surrounding blocks in statistical aspect. Texture synthesis process is like
below. At first, when we call texture synthesis function, we give referable
neighborhood block information. Secondly, we set the template which is 3×3 or
4×4 and next to the missed pixel we want to fill. We classify blocks whose
statistical properties are similar to those of surrounding blocks into texture. So, when we want to fill the texture block,
we exploit the statistical similarity with other blocks. In big picture (512×512),
8×8 block is not a big portion. And, in smooth area, we don’t use texture
synthesis because of peak signal-to-noise ratio (PSNR)
quality, even though texture synthesis is faster than structure inpainting. In
most cases, we use texture synthesis in very coarse area or pattern area. From
those reasons, we copy the whole block (8×8) to the missed texture block, after
finding the block which has the closest mean and variance to missed texture
block. In this case, we can get a good result in visual aspect, despite of
almost same result in PSNR.
Structure is the region that can be clearly divided into
two or more sectors through clear edges. Each sector is relatively devoid of
minute details and the missing block within the sector is easily predicted from
surrounding blocks. Structure inpainting is the process of gradually
propagating the information contained in the surrounding blocks into the
missing block. Basically, this process is very similar to that of diffusion.
When the heat source (surrounding block) is placed around closed area (missing
block), heat (information) gradually flows into the area.
It is now widely
acknowledged that traditional wavelet methods do not perform as well with
multidimensional data. Indeed wavelets are very efficient in dealing with point
wise singularities only. In higher dimensions, other types of singularities are
usually present or even dominant and wavelets are unable to handle them very
efficiently. Images, for example, typically contain sharp transitions such as
edges, and these interact extensively with the elements of the wavelet basis.
As a result, many terms in the wavelet representation are needed to accurately
represent these objects. In order to overcome this limitation of traditional
wavelets, In this paper, a new wavelet transform is introduced, namely shearlet.
Continuous Shearlet Transform
The continuous shearlet
transform is a non isotropic version of the continuous wavelet transform with a
superior directional sensitivity. In dimension n=2, this is defined as the
f(a,s,t) = ‹ f, ? a,s,t ›
elements ? a,s,t called shearlets
has a frequency support on a pair of trapezoids, at various scales, symmetric
with respect to the origin and oriented along a line of slope s. The support
becomes increasingly thin as a ? 0. As
a result, the shearlets form a collection of well localized waveforms at various scales, orientations and locations,
controlled by a, s, and t respectively. The frequency supports of some
representative shearlets are illustrated in Fig.2
Figure 2 Frequency support of shearlets for various
values of a and s
Discrete Shearlet Transform (DST)
By sampling the
continuous shearlet transform on appropriate discretizations of the scaling,
shear, and translation parameters a, s, t one obtains a discrete transform
which is associated to a Parseval (tight) frame for L2(R2).
The following procedure describes the construction of Shearlet transform and
the procedure is illustrated on Fig. 3.
(1) Apply the
laplacian pyramid scheme to decompose faj-1
into a low pass image faj and a high pass
(2) Compute P fdj on a pseudo polar grid.
(3) Apply a band pass filtering to the matrix P fdj.
re-assemble the Cartesian sampled values and apply the inverse two-dimensional Fast
Fourier Transform (FFT).
The steps of compression are depicted in Fig. 4. The
original image I is analyzed and the blocks to be removed are determined. The
assistant information R should be sent with masked image to decoder. R should
contain the locations of blocks removed and the algorithm to be used to fill in
the missing region. For missing region, fill in DC values to minimize the size
of JPEG-encoded image. R is compressed using lossless encoder while DC-filled
image D is encoded by JPEG. On the decoder side, R’ and D’ are decoded and R is
used to fill in the removed blocks of D. The bit rate is calculated by (size of
D’) + (size of R’ (entropy encoded R)) / (size of the image). The
different images are shown in Fig. 5.
For comparison purpose the PSNR values are computed and
the values are tabulated and shown in Table1.
Figure 3 Succession of laplacian pyramid and directional filtering
Figure 4 An algorithm for image compression
Performance Comparison Using Psnr Values
(a) (b) (c)
Figure 4 (a) Noisy image (b) masked image
(c) received image after inpainting.
A conventional image quality index is the PSNR, which is
the ratio between the maximum possible power of a signal and the power of the
corrupting noise that affects the fidelity of its representation. It is widely
used for the estimation of quality in lossy image compression algorithms. The
signal in this case is the original data and the noise is the error introduced
by compression. This index is popular for its simplicity; however, it loses its
advantages compared with natural human perception 6.
A better index for image quality measurement is the
structural similarity (SSIM), which is a method for measuring the similarity
between two images. The SSIM index is a full reference metric, the measuring of
image quality based on an initial uncompressed image as a reference and is calculated
where, the covariance of X and Y, the average of X, the average of Y, the variance of X, the variance of Y.
In this paper, a new sparse representation-Discrete
Shearlet Transform domain inpainting model is presented. In this
framework, some kinds of distinctive features are extracted from images at the
encoder side, and regions with high correlation values are intentionally
skipped during encoding. The remaining areas along with information associated
with the edges are encoded to form the compressed output data. Removed
information is to be recovered with the assistance of information sent to the
decoder side. At the decoder, using a PDE-based inpainting algorithm, the
removed areas are recovered
M. Bertalmio, G. Sapiro, V. Caselles, and C.
Ballester, “Image inpainting,” SIGGRAPH, pp. 1033–1038, January.
Criminisi, A., Perez, P., Toyama, K., 2004.
Region filling and object removal by exemplar-based image inpainting. IEEE
Trans. Image Process., 13(9):1200-1212. doi:10.
Patwardhan, K.A., Sapiro, G., Bertalmio, M.,
2005. Video Inpainting of Occluding and Occluded Objects. IEEE Int. Conf. on
Image Processing, 2:69-72. doi:10.1109/ICIP. 2005.1529993.
Chan, T.F., Shen, J., 2005a. Image Processing
and Analysis. Society for Industrial and Applied Mathematics (SIAM),
Rane, S.D., Sapiro, G., Bertalmio, M., 2003.
Structure and texture filling-in of missing image blocks in wireless
transmission and compression applications. IEEE Trans. Image Process., 12(3):296-303.
Bovik, A.C., 2009. Mean squared error: love it or leave it? A new look at
signal fidelity measures. IEEE Signal Process. Mag., 26(1):98-117.
R.Easley, Demetrio Labate, and Flavia Colonna, “Shearlet
based total variation diffusion for denoising”, IEEE Transactions on Image Processing”, Vol.18,
S.D. Rane, G.
Sapiro, and M. Bertalmio, “Structure and texture filing-in of missing image
blocks in wireless transmission and compression applications,” IEEE
Transactions on Image Processing, pp.
296–303, March 2003.