Multi-Imaging Sensor Data Fusion using 2D DST

Multi-Imaging Sensor Data Fusion using 2D DST

M. Divya,

Dept. of ECE,

JNTU College of Engineering, Ananthapuramu, India,

V P S Naidu

Multi Sensor Data Fusion Lab, CSIR – National Aerospace Laboratories, Bangalore,

India

Abstract:- Image fusion is a process of combining two or more

X (k)

2 N 1 x(n) sin (n 1)(k 1) , 0 n N 1

images into a single image without any loss of information. Now-a-days, image fusion is playing a major role in research areas. In this paper, discrete sine transform based multi imaging sensor data fusion algorithms are developed,

n0

N 1

N

0 k N 1

(1)

implemented and tested using fusion quality evaluation metrics. The proposed fusion algorithms are compared with discrete Cosine transform based fusion algorithms and it is observed from the results that they are almost similar and comparable. The proposed fusion algorithms are computationally simple and can be used in real time applications.

The DST-I is orthogonal and it is exactly equivalent to a

DFT of real sequence that is odd around the 0th and middle points, scaled by 0.5. The DST-I is its own inverse.

The two dimensional (2D) discrete sine transform

X (k1 , k2 ) o f an image x(n1 , n2 ) of size N1 N 2 is defined as

2 2 N1 1N2 1

(n 1)(k 1)

Keywords:- Discrete Sine Transform, Image Fusion, Discrete

X (k1 , k2 ) x(n1 , n2 )sin 1 1

Cosine Transform.

N1 N2 n1 0 n2 0

N1

(2)

(n2 1)(k2 1)

1. INTRODUCTION

   Of-late, many researchers developed different image fusion algorithms to combine multiple images into a single image.

   sin

   N2

   Where, 0 k1 , k2

   N1

   1, N 2 1

   Image fusion using corresponding pixel averaging is the simplest method of all those different fusion algorithms. In image fusion literature, no researchers use the simple and well proven discrete Sine transform (DST) for fusion application. In this paper, an attempt is made to use DST for developing image fusion algorithms. This paper introduces different DST based image fusion techniques, studied their performance. The goal of this paper is to check the performance of image fusion based on DST algorithm using different block sizes and to compare with DCT based image fusion algorithm [1].

   Here, 2D DST is a self inverse [2].

   3. IMAGE FUSION

   The architecture of 2D DST based image fusion is shown in Fig.1.The images to be fused are divided into non- overlapping blocks of size NxN. DST is performed on corresponding blocks of images to be fused and fusion rules are applied on DST coefficients to get the fused DST coefficients. This process is repeated for each block of an image. This gives the fused image as output.

   Fusion rules

   Image 1

2. DISCRETE SINE TRANSFORM

The discrete sine transform (DST) is a Fourier-related transform similar to the discrete Fourier transform (DFT) but using a purely real matrix. It is the imaginary part of a DFT and its length is approximately twice the length of DFT. It is operating on real data with odd symmetry, whereas the DFT of a real and odd function is imaginary and odd. DST expresses of finitely discrete sequence in terms of sine functions oscillating at different frequencies. There are eight standard DST variants of which four are common and widely used for signal analysis.DST is a linear and invertible function. In literature eight DST

Image 2

DST

DST

DST

Fused Image

equations are there in that DST-1 to DST-4 are widely used. In this paper DST-1 is using because it is simple and it has its own inverse so no need of applying inverse DST.

Fig-1: Framework of 2D DST based image fusion architecture.

The one dimensional (1D) discrete sine transform X (k ) of a signal x(n) of length N is defined as:

The fusion rules which are used in image fusion process 4. FUSION EVALUATION METRICS

are presented in this section. Let consider X1 be the DST coefficients of image block from image 1 and similarly X 2 be the DST coefficients of image block from image 2.

Image fusion process is used to get good quality image and to evaluate the fusion quality, many fusion quality evaluation metrics are proposed in the open literature.

When the ground truth image is available the following

Assume that the image block is of size the fused DST coefficients.

3.1 DSTav

N N

and

X f be

fusion evaluation metrics are used to evaluate the performance of the different image fusion algorithms [4]:

4.1 SPATIAL FREQUENCY (SF)

The frequency in spatial domain indicates the overall

In this fusion rule, all DST coefficients from both image blocks are averaged to get fused DST coefficients. It is a very simple and basic image fusion technique in DST

activity level in the fused image and it is computed as row and column frequency of the images.

domain.

Spatial frequency criterion is: SF

Where, row frequency of the image:

RF 2 CF 2 (7)

X f (k1, k2 ) 0.5*(X1(k1, k2 ) X 2 (k1, k2 ))

(3)

1 M N

f

f

RF [x (n1, n2) x (n1, n2 1)]2

Where, k1 , k2

= 0,1,2,… …,N -1

MN n11 n22

3.2 DSTmax

Column frequency of the image:

By collecting DC components from both frames the

CF

1 N

[x (n1, n2) x (n11, n2)]2

M

f

f

average can be done. From the AC components greatest magnitude AC coefficients are considered and then the detailed coefficients correspond to sharper brightness changes in the images such as edges, object boundaries etc. The DC and AC coefficients are fluctuating around zero

MN n21n12

SF indicates the overall activity level in the fused image. The fused image with high SF will be considered [2, 3].

X f (0,0) 0.5* (X1 (0,0) X 2 (0,0))

1

Where, k1 , k2 = 0,1,2,… …,N -1

(4)

4.2 Peak Signal to Noise Ratio (PSNR)

Its value will be high when the reconstructed and reference images are similar. Higher PSNR value implies better

f 1 2

X1 (k1, k 2 ) X (k ,k )

| X1 (k , k 2 ) | | X2 (k1 , k 2 ) |

reconstruction. The peak signal to noise ratio is computed as:

2 1 2

X (k , k )

3.3 DSTah

| X1 (k1 , k 2 ) | | X2 (k1 , k 2 ) |

PSNR 10log

L2

(8)

By collecting the lowest AC components and including DC

10 1 M N x (n1, n2) x

(n1, n2)2

coefficients averaging process can be performed and the

MN

r f

n21n11

AC coefficients which are remained, are chosen based on largest magnitude.

Where, L is the number of gray levels in the image and xr

X (k , k ) 0.5*(X (k , k ) X (k , k )) (5)

is the reference/ground truth image.

f 1 2

1 1 2

2 1 2

Where, k1 , k2

= 0,1,2,… …,0.5N-1

4.3 Entropy (H)

f 1 2

X1 (k1 , k 2 ) X (k ,k )

X (k , k )

| X1 (k , k 2 ) | | X2 (k1 , k 2 ) |

1

| X (k , k ) | | X (k , k ) |

Entropy is used to measure the information content of an image. Using entropy, the information content of a fused

image is:

2 1 2

1 1 2

2 1 2

L

H

Where, k1 , k2

= 0,1,2,… …,N -1

H hx (i) log 2hx (i)

(9)

3.4 DSTe

In this DC components are averaged together. And the AC

Where,

h

x f

f f

i0

(i) is the normalized histogram of the fused image

coefficients correspond to the frequency band having largest energy considered.

x f and

LH number of frequency bins in the histogram.

X f (0,0) 0.5*(X1(0,0) X 2 (0,0))

X k , k Ej2 Ej2

(6)

Entropy is sensitive to noise and other unwanted rapid

fluctuations. The information entropy measures the

X k , k

1 1 2

richness of information in an image. Hence, entropy is

f 1 2

X 2

k1 , k2

Ej2 Ej2

higher, performance is better [4].

Where, k1 , k2

= 0,1,2,… …,N -1 and

j k1 k2

1. RESULTS AND DISCUSSIONS

   The fused image will get by applying the DST on fused coefficients as: xf DST(X f )

   Note: DCT based image fusion algorithms are very similar to DST based image fusion algorithms (Section 3.1 to 3.4) by simply replacing DST by DCT.

   The main objective of this paper is to fuse multiple images using 2D DST. The ground truth and source images (images to be fused) are shown in Fig-2 and Fig-3 respectively. Fused images using different DST algorithms are shown in Figs-4 to 7. The computational time of the proposed fusion algorithms are shown in Table-1. It is

   observed that, in DST based fusion process at 16×16 block size it is taking less time for all fusion rules except for DSTe(at 32×32 it is taking less time). whereas, in DCT at 128×128 block size it is taking less time for all fusion rules except for DSTe(at 64×64 it is taking less time). Image fusion quality evaluation metrics for the proposed fusion algorithms are shown in Table-2&3. The metrics shown with bold indicates better results and corresponding algorithms will be the best among others. It is observed that SF is same irrespective of block size. The comparison of DST and DCT based image fusion using block size 8×8 is shown in Table-4.It is observed that, fusion algorithm based on DST and DCT are performing almost similar. In fact, DSTav provides better PSNR than DCTav as shown in Table-4. The fused images with different fusion algorithms are shown in Figs-4 to 7. It is observed that, DST av shows better fusion image compare to other DST based fusion algorithms followed by DSTe. These results are comparable with DCT based fusion algorithms.

2. CONCLUSION

   Multi imaging sensor data fusion algorithms using 2D-DST are presented and evaluated using fusion quality evaluation

   metrics. The results are compared with DCT based fusion algorithms available in open literature. It is observed that fusion algorithms based DST and DCT are performing almost similar. The proposed fusion algorithms are computationally simple and can be used in real-time image fusion applications.

3. REFERENCES

1. VPS Naidu, Discrete Cosine Transform based Image Fusion Techniques, Journal of Communication, Navigation and Signal Processing (January 2012) Vol. 1, No. 1, pp. 35-45.

2. https://en.wikipedia.org/wiki/Discrete_sine_transform.

3. K. Rao and P. Yip, Discrete Cosine Transform, Algorithm, Advantages, applications, Academic Press, 1990.

4. S.A. Martucci, Symmetric Convolution and the Discrete Sine and Cosine transforms, IEEE Trans. Sig. Processing, SP-42, pp.1038-1051, 1994.

Table (1): Computational Time (in sec)

Fusion rules	DST/DCT	Block Size(rows & columns)
		2×2	4×4	8×8	16×16	32×32	64×64	128×128	256×256	512×512
Av	DST	7.4232	3.8034	1.9731	1.8318	2.6496	4.7131	9.1133	17.9871	35.9240
	DCT	22.6578	5.9152	1.5946	0.4896	0.1859	0.1617	0.1085	0.2009	0.2738
max	DST	7.8969	3.4496	2.0177	1.8393	2.6509	4.7310	9.0845	18.1372	36.0051
	DCT	23.1537	6.0213	1.6209	0.4881	0.1886	0.1691	0.1164	0.2028	0.2831
ah	DST	8.7456	3.6002	2.0647	1.8703	2.6434	4.8103	9.0807	17.9239	35.8718
	DCT	23.7942	6.1585	1.6665	0.5042	0.1988	0.1665	0.1226	0.2065	0.2847
e	DST	27.0125	14.0739	7.4312	5.4210	5.3831	8.2326	14.9772	29.0308	58.1472
	DCT	34.4017	11.9302	4.8228	2.3368	1.4845	1.4178	1.8222	4.2950	9.7422

Table (2): Spatial Frequency

Fusion rules	DST/DCT	Block Size(rows & columns)
		2×2	4×4	8×8	16×16	32×32	64×64	128×128	256×256	512×512
Av	DST	0.0358	0.0358	0.0358	0.0358	0.0358	0.0358	0.0358	0.0358	0.0358
	DCT	0.0358	0.0358	0.0358	0.0358	0.0358	0.0358	0.0358	0.0358	0.0358
max	DST	0.0450	0.0489	0.0515	00531	0.0535	0.0546	0.0555	0.0548	0.0569
	DCT	0.0544	0.0643	0.0667	0.0669	0.0668	0.0668	0.0666	0.0667	0.0611
ah	DST	0.0557	0.0503	0.0497	0.0486	0.0474	0.0487	0.0493	0.0474	0.0508
	DCT	0.0544	0.0528	0.0510	0.0501	0.0496	0.0493	0.0491	0.0489	0.0463
e	DST	0.0556	0.0560	0.0580	0.0610	0.0631	0.0677	0.0676	0.0668	0.0644
	DCT	0.0544	0.0642	0.0666	0.0668	0.0668	0.0667	0.0667	0.0667	0.0520

Table (3): Entropy of DST/DCT

Fusion rules	DST/DCT	Block Size(rows & columns)
		2×2	4×4	8×8	16×16	32×32	64×64	128×128	256×256	512×512
Av	DST	4.0546	4.0232	4.0217	4.0152	4.0181	4.0182	4.0174	4.0182	4.0176
	DCT	3.9888	3.9888	3.9891	3.9889	3.9882	3.9937	3.9933	3.9978	3.9943
Max	DST	3.9478	3.9000	3.9165	3.9698	4.0570	4.1263	4.2360	4.5376	4.6268
	DCT	4.0179	4.0090	4.0099	4.0006	3.9488	3.9918	4.1587	3.7418	5.5020
Ah	DST	3.9478	3.9480	3.9783	4.0183	4.0775	4.1074	4.2049	4.3844	4.4422
	DCT	4.0179	4.0246	4.0253	4.0286	4.0412	4.0701	4.1477	4.1916	4.8325
E	DST	3.9464	3.8561	3.8397	3.8748	3.9539	4.0498	4.0651	4.0914	4.3567
	DCT	4.0173	4.0064	4.0057	3.9889	3.9385	3.9669	3.9820	3.7125	4.8561

Table (4): Comparison between DST and DCT

	Block Size(8×8)
	DSTav	DSTmax	DSTah	DSTe	DCTav	DCTmax	DCTah	DCTe
Computational Time(in sec)	1.9731	2.0177	2.0647	7.4312	1.5946	1.6209	1.6665	4.8228
Spatial	0.0358	0.0515	0.0497	0.0580	0.0358	0.0667	0.0510	0.0666
PSNR	46.7141	43.9522	45.0890	44.4066	38.4255	40.7588	38.6715	40.7857
Entropy	4.0217	3.9165	3.9783	3.8397	3.9891	4.0099	4.0253	4.0057

Fig-2: Ground Truth Image-SARAS

Fig-3: Source Images (images to be fused) -SARAS

Fig-4a: Fused image using DSTe fusion algorithm Fig-4b: Fused image using DCTe fusion algorithm

Fig-5a: Fused image using DSTah fusion algorithm Fig-5b: Fused image using DCTah fusion algorithm

Fig-6a: Fused image using DSTmax fusion algorithm Fig-6b: Fused image using DCTmax fusion algorithm

Fig-7a: Fused image using DSTavfusion algorithm Fig-7b: Fused image using DCTav fusion algorithm