Scene Segmentation-Based Luminance Adjustment for Multi-Exposure Image Fusion

TL;DR

This paper introduces a new luminance adjustment technique based on scene segmentation to enhance multi-exposure image fusion, resulting in clearer, more informative images even with suboptimal input exposures.

Contribution

It proposes two novel scene segmentation methods based on luminance distribution and demonstrates their effectiveness in improving fusion quality over existing methods.

Findings

Improved fused image clarity and scene representation.

Outperforms state-of-the-art methods in multiple quality metrics.

Enables high-quality fusion with less ideal input images.

Abstract

We propose a novel method for adjusting luminance for multi-exposure image fusion. For the adjustment, two novel scene segmentation approaches based on luminance distribution are also proposed. Multi-exposure image fusion is a method for producing images that are expected to be more informative and perceptually appealing than any of the input ones, by directly fusing photos taken with different exposures. However, existing fusion methods often produce unclear fused images when input images do not have a sufficient number of different exposure levels. In this paper, we point out that adjusting the luminance of input images makes it possible to improve the quality of the final fused images. This insight is the basis of the proposed method. The proposed method enables us to produce high-quality images, even when undesirable inputs are given. Visual comparison results show that the proposed method can produce images that clearly represent a whole scene. In addition, multi-exposure image fusion with the proposed method outperforms state-of-the-art fusion methods in terms of MEF-SSIM, discrete entropy, tone mapped image quality index, and statistical naturalness.

Tables

Table 1. TABLE I : Tunable parameters in proposed method

Parameter	Effect
${\sigma }_1$ and ${\sigma }_2$	Determine the degree of enhancing local contrast, but larger values often boost noise or ringing.
$L_m$	Determine the shape of tone mapping operators: $L_m\le 1$ increases contrast, $L_m=1$ keeps all luminance values, and $1\le L_m$ suppresses contrast.
$M$ and $K$	Determine the number of adjusted multi-exposure images. Larger $M$ improves the quality of fused images, but increases computational cost. In Approach 2, $M$ is automatically set such that $1\le M\le K$.

Table 2. TABLE II : Discrete entropy scores for static scenes. Boldface indicates higher score. “w/o” means that images 𝒚 𝒚 \boldsymbol{y} were produced without luminance adjustment. “Prop. 1” and “Prop. 2” indicate proposed method using Approach 1 and proposed method using Approach 2, respectively.

Scene	Mertens [9]			Sakai [14]			Nejati [15]			Li [17]			Ma [32]
	w/o	Prop. 1	Prop. 2	w/o	Prop. 1	Prop. 2	w/o	Prop. 1	Prop. 2	w/o	Prop. 1	Prop. 2	w/o	Prop. 1	Prop. 2
Arno	5.987	5.931	5.833	5.992	5.935	5.844	7.016	7.016	6.793	6.312	6.095	6.072	7.054	7.100	6.813
Cave	4.938	5.920	6.277	4.946	5.942	6.290	6.132	6.839	7.044	6.115	6.840	6.999	6.082	6.782	7.043
Chinese garden	6.142	6.185	6.241	6.155	6.205	6.258	6.946	7.215	6.844	7.244	7.271	7.229	6.998	7.143	6.955
Corridor 1	2.858	5.840	5.738	2.856	5.859	5.752	2.752	7.134	6.976	3.350	7.242	7.262	2.773	7.088	7.016
Corridor 2	5.706	5.922	5.613	5.714	5.933	5.618	6.543	6.925	6.625	6.682	5.548	5.670	6.533	6.930	6.586
Estate rsa	6.577	6.604	6.585	6.560	6.584	6.573	6.785	6.917	7.014	7.001	7.015	6.914	6.802	6.921	7.014
Kluki	7.120	6.982	6.942	7.136	7.003	6.964	7.489	7.178	7.151	7.669	7.540	7.548	7.535	7.266	7.207
Laurenziana	6.672	6.478	6.599	6.678	6.490	6.609	7.153	6.956	6.898	7.585	7.239	7.417	7.267	7.091	7.008
Lobby	4.172	5.648	5.737	4.177	5.672	5.743	4.929	7.118	6.836	5.274	6.773	7.177	4.893	7.060	6.783
Mountains	6.920	6.480	6.571	6.912	6.492	6.586	6.823	6.790	6.846	6.491	6.233	6.339	6.796	6.781	6.838
Ostrow tumski	5.861	5.607	6.003	5.874	5.629	6.033	6.812	6.475	7.005	6.607	6.253	6.794	6.857	6.622	6.958
Window	5.047	5.715	6.060	5.052	5.735	6.072	5.407	6.750	6.848	5.547	6.617	7.099	5.402	6.752	6.868
\hdashlineAverage	5.667	6.109	6.183	5.671	6.123	6.195	6.232	6.943	6.907	6.323	6.722	6.877	6.249	6.961	6.924

Table 3. TABLE III : Statistical naturalness scores for static scenes. Boldface indicates higher score. “w/o” means that images 𝒚 𝒚 \boldsymbol{y} were produced without luminance adjustment. “Prop. 1” and “Prop. 2” indicate proposed method using Approach 1 and proposed method using Approach 2, respectively.

Scene	Mertens [9]			Sakai [14]			Nejati [15]			Li [17]			Ma [32]
	w/o	Prop. 1	Prop. 2	w/o	Prop. 1	Prop. 2	w/o	Prop. 1	Prop. 2	w/o	Prop. 1	Prop. 2	w/o	Prop. 1	Prop. 2
Arno	0.1289	0.1382	0.1233	0.1506	0.1633	0.1473	0.5164	0.5372	0.5442	0.4079	0.2852	0.3543	0.5076	0.6845	0.5851
Cave	0.0758	0.2458	0.2808	0.0758	0.2236	0.2512	0.4223	0.3413	0.0628	0.2665	0.2230	0.0347	0.4122	0.3261	0.0758
Chinese garden	0.420	0.4353	0.4612	0.4059	0.4222	0.4441	0.4609	0.4843	0.4152	0.4801	0.5316	0.4477	0.4771	0.4468	0.3863
Corridor 1	0.0004	0.1454	0.1374	0.0004	0.1998	0.1713	0.0006	0.9111	0.8792	0.0012	0.9017	0.9129	0.0006	0.8908	0.8825
Corridor 2	0.0628	0.1025	0.0744	0.0763	0.1338	0.0911	0.3368	0.5052	0.4459	0.3807	0.0957	0.1293	0.3212	0.5209	0.4632
Estate rsa	0.7919	0.8047	0.7992	0.7624	0.7755	0.7840	0.8884	0.9504	0.9872	0.9773	0.9894	0.9941	0.9006	0.9536	0.9903
Kluki	0.9665	0.9580	0.9105	0.9596	0.9432	0.8889	0.9316	0.9844	0.9749	0.5148	0.7616	0.7226	0.8659	0.9849	0.9940
Laurenziana	0.8046	0.6541	0.7387	0.8058	0.6595	0.7366	0.9330	0.9722	0.9558	0.6125	0.9408	0.8303	0.8902	0.9662	0.9319
Lobby	0.0167	0.1947	0.2093	0.0190	0.2307	0.2274	0.1019	0.9260	0.7637	0.1350	0.6566	0.6936	0.1025	0.8888	0.7285
Mountains	0.3953	0.5582	0.6231	0.3562	0.5795	0.6353	0.4702	0.7383	0.7634	0.4852	0.5860	0.6898	0.4373	0.7459	0.8171
Ostrow tumski	0.1445	0.1347	0.2975	0.1580	0.1516	0.3274	0.5090	0.6194	0.7568	0.4874	0.3090	0.8185	0.5145	0.7585	0.8162
Window	0.0569	0.0912	0.2135	0.0615	0.1142	0.2650	0.1617	0.6187	0.7031	0.2145	0.5070	0.8770	0.1655	0.6413	0.7475
\hdashlineAverage	0.3220	0.3719	0.4057	0.3193	0.3831	0.4141	0.4777	0.7157	0.6877	0.4136	0.5657	0.6254	0.4663	0.734	0.7015

Table 4. TABLE IV : Discrete entropy scores for dynamic scenes. Boldface indicates higher score. “w/o” means that images 𝒚 𝒚 \boldsymbol{y} were produced without luminance adjustment. “Prop. 1” and “Prop. 2” indicate proposed method using Approach 1 and proposed method using Approach 2, respectively.

Scene	w/o	Prop. 1	Prop. 2	Prop. 2 w/o CE
Baby at window	7.032	6.351	7.160	6.770
Baby on grass	7.072	6.711	7.271	6.966
Christmas rider	7.098	6.366	7.048	6.945
Feeding time	6.498	6.853	6.643	6.621
High chair	7.272	6.861	7.163	7.149
Lady eating	6.917	6.653	7.157	7.000
Piano man	6.633	6.632	6.791	6.752
Santas little helper	7.186	6.429	7.246	7.285
\hdashlineAverage	6.964	6.607	7.060	6.936

Table 5. TABLE V : Statistical naturalness scores for dynamic scenes. Boldface indicates higher score. “w/o” means that images 𝒚 𝒚 \boldsymbol{y} were produced without luminance adjustment. “Prop. 1” and “Prop. 2” indicate proposed method using Approach 1 and proposed method using Approach 2, respectively.

Scene	w/o	Prop. 1	Prop. 2	Prop. 2 w/o CE
Baby at window	0.9912	0.5344	0.9830	0.9438
Baby on grass	0.9779	0.6881	0.8532	0.8629
Christmas rider	0.8797	0.5972	0.7192	0.9593
Feeding time	0.1699	0.7162	0.6393	0.2859
High chair	0.9549	0.6721	0.7856	0.8372
Lady eating	0.8262	0.6731	0.9701	0.7076
Piano man	0.6369	0.6016	0.2942	0.7637
Santas little helper	0.6302	0.3866	0.3271	0.9089
\hdashlineAverage	0.7584	0.6087	0.6965	0.7837

Table 6. TABLE VI : Machine spec used for evaluating executing time

Processor	Intel Core i7-7700K 4.20 GHz
Memory	64 GB
OS	Ubuntu 16.04 LTS
Software	MATLAB R2017b

Equations

$$l_n^{\prime }(\mathbf{p})=\frac{l_n(\mathbf{p}{)}^2}{{\bar{l}}_n(\mathbf{p})},$$

$${\bar{l}}_n(\mathbf{p})=\frac{{\sum }_{{\mathbf{p}}^{\prime }\in \mathrm{P}}{l}_n({\mathbf{p}}^{\prime })g_{{\sigma }_1}(\Vert {\mathbf{p}}^{\prime }-\mathbf{p}\Vert )g_{{\sigma }_2}(l_n({\mathbf{p}}^{\prime })-l_n(\mathbf{p}))}{{\sum }_{{\mathbf{p}}^{\prime }\in \mathrm{P}}{g}_{{\sigma }_1}(\Vert {\mathbf{p}}^{\prime }-\mathbf{p}\Vert )g_{{\sigma }_2}(l_n({\mathbf{p}}^{\prime })-l_n(\mathbf{p}))},$$

$$g_{\sigma }(t)=\frac{1}{2\pi {\sigma }^2}\exp \left(-\frac{t^2}{2{\sigma }^2}\right)\mathrm{for}t\in \mathbb{R}.$$

$${\hat{l}}_m(\mathbf{p})=f_m(l_m^{\prime \prime }(\mathbf{p})).$$

$$f_m(t)=\frac{t}{1+t}\left(1+\frac{t}{L_m^2}\right)\mathrm{for}t\in [0,\infty ),$$

$$\mathbf{y}=\mathcal{F}({\hat{\mathbf{x}}}_1,{\hat{\mathbf{x}}}_2,\cdots ,{\hat{\mathbf{x}}}_M),$$

$${\mathrm{P}}_m=\{\mathbf{p}|{\theta }_m\le l_{\mathrm{med}}^{\prime }(\mathbf{p})\le {\theta }_{m+1}\},$$

$${\theta }_m$$

$$+\min l_{\mathrm{med}}^{\prime }(\mathbf{p}).$$

$$p({\mathbf{l}}^{\prime }(\mathbf{p}))=\sum _{k=1}^K{\pi }_k\mathcal{N}({\mathbf{l}}^{\prime }(\mathbf{p})|{\mathbf{\mu }}_k,{\mathrm{\Sigma }}_k),$$

$$p(z_k=1)={\pi }_k.$$

$$0\le {\pi }_k\le 1$$

$$\sum _{k=1}^K{\pi }_k=1.$$

$$\gamma (z_k|{\mathbf{l}}^{\prime }(\mathbf{p}))$$

$$=\frac{{\pi }_k\mathcal{N}({\mathbf{l}}^{\prime }(\mathbf{p})|{\mathbf{\mu }}_k,{\mathrm{\Sigma }}_k)}{{\sum }_{j=1}^K{\pi }_j\mathcal{N}({\mathbf{l}}^{\prime }(\mathbf{p})|{\mathbf{\mu }}_j,{\mathrm{\Sigma }}_j)}.$$

$$m=\underset{k}{\mathrm{argmax}}\gamma (z_k|{\mathbf{l}}^{\prime }(\mathbf{p})),$$

$$l_m^{\prime \prime }(\mathbf{p})={\alpha }_m{l}_n^{\prime }(\mathbf{p}),$$

$$G(l|{\mathrm{P}}_m)=\exp \left(\frac{1}{|{\mathrm{P}}_m|}\sum _{\mathbf{p}\in {\mathrm{P}}_m}\log \left(\max \left(l(\mathbf{p}),ϵ\right)\right)\right),$$

$${\alpha }_m=\frac{0.18}{G(l_n^{\prime }|{\mathrm{P}}_m)}.$$

$$n=\psi (m)=\underset{j}{\mathrm{argmin}}(0.18-G(l_j^{\prime }|{\mathrm{P}}_m){)}^2.$$

$${\hat{\mathbf{x}}}_m(\mathbf{p})=\frac{{\hat{l}}_m(\mathbf{p})}{l_{\psi (m)}(\mathbf{p})}{\mathbf{x}}_{\psi (m)}(\mathbf{p}).$$

$$S=\frac{1}{K}\mathcal{N}(\overline{l}|115.94,27.99^2)\mathcal{B}(\overline{{\sigma }_l}/64.29|4.4,10.1),$$

$$X(\mathbf{p})=E(\mathbf{p})\Delta t.$$

$$x(\mathbf{p})=f(X(\mathbf{p})),$$

$$v_i={\log }_2\Delta t_i-{\log }_2\Delta t_0.$$

$$x_0(\mathbf{p})$$

$$x_i(\mathbf{p})$$

$$x_i(\mathbf{p})=2^{v_i}{x}_0(\mathbf{p}).$$

$$E_c={\int }_0^{\infty }P(\lambda )R(\lambda )H_c(\lambda )d\lambda .$$

$$x_c$$

$$=f\left(\Delta t{\int }_0^{\infty }P(\lambda )R(\lambda )H_c(\lambda )d\lambda \right).$$

$$x_c^{\prime }$$

Full text

I Introduction

The low dynamic range (LDR) of the imaging sensors used in modern digital cameras is a major factor preventing cameras from capturing images as good as human vision. This is due to the limited dynamic range that imaging sensors have, and a single shutter speed that is utilized when we take photos. The limitations result in low-contrast images taken by digital cameras. To obtain images with better quality, numerous imaging and enhancement techniques based on a single image have been proposed [1, 2, 3, 4, 5, 6]. However, these techniques cannot restore saturated pixel values in LDR images.

Because of this situation, interest in multi-image-based imaging techniques [7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17] has been increasing to overcome this constraint that single-image-based techniques have. Multi-exposure image fusion is a solution to the LDR problem of digital cameras. Fusion methods utilize a set of differently exposed images, called “multi-exposure images,” and fuse them to produce an image with high quality. Their development was inspired by high dynamic range (HDR) imaging techniques [18, 19, 20, 21, 22, 23, 24, 25, 26, 27]. The advantage of these methods, compared with HDR imaging techniques, is that they can eliminate three operations: generating HDR images, calibrating the camera response function (CRF), and preserving the exposure value of each photograph. In this paper, we focus on multi-exposure image fusion.

However, to work well, existing fusion methods require two conditions [28, 29]: input multi-exposure images include no moving artifacts such as blurring or misalignment among images, and input multi-exposure images clearly cover the dynamic range of a scene. Moreover, the conditions have a close relation each other. To capture multi-exposure images covering a high dynamic range, it is generally needed to shoot multiple times while changing the shutter speed. In this case, moving artifacts generally occur due to time lags of each shooting and long-shutter speeds to capture bright images. Therefore, the two conditions are not always satisfied at the same time in various practical situations.

For eliminating moving artifacts among images, specialized camera systems [30, 31] and a number of robust fusion methods [32] have been studied. For specialized camera systems, spatially varying pixel exposure methods [30, 31] using a single sensor and new optical systems [33, 34] having a number of sensors have been proposed. These systems enable us to capture scenes at the same time by changing the amount of light absorbed at each pixel in the imaging sensor, or by varying the exposure for each sensor. However, these camera systems do not always capture images having a sufficient number of different exposure levels, due to the hardware limitation of camera systems.

In contrast, for robust fusion methods against moving artifacts, input multi-exposure images taken with conventional cameras are first aligned, and then aligned images are fused. These methods effectively produce high-quality images even when scenes include moving objects, if well-exposed multi-exposure images are prepared. However, the quality of input multi-exposure images affects both the accuracy of image alignment and the performance of fusion. Hence, insufficient multi-exposure images, such as ones taken with fast-shutter-speed due to limitations of camera capability, make it difficult to produce high-quality results. Because of these situations, we aim to propose a scene segmentation-based luminance adjustment method, for improving the quality of insufficient multi-exposure images.

In this paper, we first point out that it is possible to improve the quality of multi-exposure images by adjusting the luminance of images after photographing. Moreover, a scene segmentation-based luminance adjustment method is proposed on the basis of this insight. The proposed method enables us to produce high-quality images even when unclear input images are given. For the adjustment, two scene segmentation approaches are also proposed in order to automatically adjust input multi-exposure images so that they become suitable for multi-exposure image fusion. The former separates a scene in multi-exposure images, according to the luminance values of an input image having middle brightness. This approach has a closed-form, so it has a lower computational cost than typical segmentation methods. In the latter, which provides high-quality results, a scene is separated by considering the luminance distribution of all input images. The quality of fused images generally increases by using a large number of multi-exposure images, but the use of these images requires a large computational cost for fusion. In addition, the suitable number depends on a scene. For these reasons, the latter segmentation method automatically determines the suitable number of adjusted multi-exposure images, although most segmentation ones including $k$-means cannot.

In simulations, we evaluate the effectiveness of the proposed method with various fusion methods in terms of visual and quantitative comparison. Visual comparison results show that the proposed method can produce images that clearly represent a whole scene. In addition, multi-exposure image fusion with the proposed method outperforms state-of-the-art fusion methods in terms of MEF-SSIM, discrete entropy, tone mapped image quality index, and statistical naturalness.

II Previous work

As mentioned, existing multi-exposure image fusion methods use differently exposed images, referred to as “multi-exposure images.” Here, we summarize typical fusion methods and problems with them.

Various research works on multi-exposure image fusion have so far been reported for capturing images of real scenes having a wide dynamic range without clipped blacks or whites [8, 9, 10, 12, 13, 14, 15]. Many of the fusion methods provide the final fused image as a weighted average of input multi-exposure images. Mertens et al.[9] proposed a multi-scale fusion scheme in which contrast, color saturation, and well-exposedness measures are used for computing fusion weights. In the work by Nejati et al. [15], base-detail decomposition is applied to each input image, and base and detail layers are then combined individually. An exposure fusion method based on sparse representation, which combines sparse coefficients with fused images, was also proposed in [12]. Furthermore, a method that combines weighted average and sparse representation is presented in [14] and is used to enhance image details. The existing methods work very well when the following two conditions are satisfied.

• •

The scene is static, and the camera is tripod-mounted.

• •

Input multi-exposure images clearly cover the dynamic range of the scene.

Especially, photographing moving objects with an unstable camera is the most common scenario in imaging. Hence, moving artifacts such as motion blur and misalignment generally occur when the long exposure time is used to capture sufficient multi-exposure images. These moving artifacts result in ghost-like artifacts in the final fused images.

Because of this, specialized camera systems and a lot of fusion methods have been proposed to eliminate ghost-like artifacts. Spatially varying pixel exposure methods [30, 31] and new optical systems [33, 34] have been studied to implement specialized camera systems. The former methods enable us to use conventional cameras for obtaining single-shot multi-exposure images by changing the amount of light absorbed at each pixel in the imaging sensor. The latter systems capture multi-exposure images with multiple sensors by splitting the light onto the sensors with different absorptive filters. However, the camera systems can capture a limited number of multi-exposure images due to the hardware limitation of the systems. For robust fusion methods against moving artifacts, Ma et al. [32] proposed a structural patch decomposition method for multi-exposure image fusion as one fusion method. By these research works, problems of the ghost-like artifacts are being solved, if well-exposed multi-exposure images are prepared. However, the quality of input multi-exposure images affects both the accuracy of image alignment and the performance of fusion.

Here, we demonstrate that adjusting the luminance of unclear multi-exposure images affects the quality of the final fused images. Figures 1 and 2 show examples of adjusted multi-exposure images and fused images from these multi-exposure ones, respectively. These results indicate that adjusting the luminance makes it possible to improve the quality of fused images. In other words, even when appropriate exposure values are unknown at the time of photographing, the multi-exposure images can be improved by adjusting unclear input images. The quality of multi-exposure images depends on the degree of adjustment.

Thus, we propose a novel luminance adjustment method, referred to as “scene segmentation-based luminance adjustment” (SSLA), for multi-exposure image fusion.

III Proposed scene-segmentation-based luminance adjustment

The proposed luminance adjustment method consists of three operations: local contrast enhancement, scene segmentation-based luminance scaling (SSLS), and tone mapping (see Fig. 3). An overview of the proposed method is first described, and SSLS is then explained.

III-A Notation

The following notations are used throughout this paper.

• •

Lower case bold italic letters, e.g., $\boldsymbol{x}$, denote vectors or vector-valued functions and are assumed to be column vectors.

• •

A superscript $\top$ denotes the transpose of a matrix or vector.

• •

The notation $(x_{1},x_{2},\cdots,x_{N})$ denotes a $N$-dimensional row vector.

• •

The notation $\{x_{1},x_{2},\cdots,x_{N}\}$ denotes a set with $N$ elements. In situations where there is no ambiguity as to their elements, the simplified notation $\{x_{n}\}$ is used to denote the same set.

• •

The notation $p(x)$ denotes a probability density function of $x$.

• •

$U$ and $V$ are used to denote the width and height of input images, respectively.

• •

$\mathrm{P}$ denotes a set of all pixels, namely, $\mathrm{P}=\{(u,v)^{\top}|u\in\{1,2,\cdots,U\}\land v\in\{1,2,\cdots,V\}\}$.

• •

A pixel $\boldsymbol{p}$ is written as $\boldsymbol{p}=(u,v)^{\top}\in\mathrm{P}$.

• •

The number of input images is denoted by $N$.

• •

An input image is denoted by a vector-valued function $\boldsymbol{x}:\mathrm{P}\to\mathbb{R}^{3}$, where its output means linear-RGB pixel values, namely, the RGB color space has a linear relationship with the CIE XYZ color space. Similarly, a fused image is denoted by $\boldsymbol{y}:\mathrm{P}\to\mathbb{R}^{3}$. For example, linear-RGB pixel values are obtained by applying the inverse gamma correction to sRGB pixel values [35]

• •

The luminance of an image is denoted by the function $l:\mathrm{P}\to\mathbb{R}$, where its output corresponds to Y components of the CIE XYZ color space.

III-B Overview

The use of the proposed method in multi-exposure image fusion is illustrated in Fig. 3, where the main contributions of this paper are SSLS and the way of producing adjusted images $\{\hat{\boldsymbol{x}}_{m}\}$. To enhance the quality of multi-exposure images, local contrast enhancement is applied to luminance $l_{n}$ calculated from the $n$-th input image $\boldsymbol{x}_{n}$. Next, SSLS separates pixel set $\mathrm{P}$ into $M$-areas $\mathrm{P}_{1},\mathrm{P}_{2},\cdots,\mathrm{P}_{M}$ according to luminance values, and luminance $l^{\prime\prime}_{m}$ for each area $\mathrm{P}_{m}$ is then obtained by scaling a set, $\{l^{\prime}_{n}\}$, of the enhanced luminance. Here, the number $M$ of the scaled luminance, which equals the number of adjusted multi-exposure images, is generally different from the number $N$ of input images. In addition, tone mapping is applied to each scaled luminance $l^{\prime\prime}_{m}$ in order to avoid the truncation of pixel values. The adjusted images $\{\hat{\boldsymbol{x}}_{m}\}$ are generated by combining the mapped luminance $\{\hat{l}_{m}\}$ and the input images $\{\boldsymbol{x}_{n}\}$, and the resulting image $\boldsymbol{y}$ is then fused from these adjusted ones $\{\hat{\boldsymbol{x}}_{m}\}$, by using a multi-exposure image fusion method $\mathscr{F}$ such as weighted average.

In Fig. 3, the proposed luminance adjustment should be applied before fusion. If the luminance adjustment is applied after fusion, an additional fusion process is needed because the luminance adjustment generates multi-exposure images even when a single image is given as an input.

III-B1 Local contrast enhancement

If the input images do not represent a scene clearly, their contrast is lower than that of ideally exposed images. Therefore, the dodging and burning algorithm is used to enhance the local contrast[36]. The luminance $l^{\prime}_{n}$ enhanced by the algorithm is given by

[TABLE]

where $\bar{l}_{n}(\boldsymbol{p})$ is the local average of luminance $l_{n}(\boldsymbol{p})$ around pixel $\boldsymbol{p}=(u,v)^{\top}$. It is obtained by applying a low-pass filter to $l_{n}(\boldsymbol{p})$. Here, a bilateral filter is used for this purpose.

$\bar{l}_{n}(\boldsymbol{p})$ is calculated by using the bilateral filter:

[TABLE]

where $g_{\sigma}(t)$ is a Gaussian function given by

[TABLE]

Parameters $\sigma_{1}=16$ and $\sigma_{2}=3/255$ are set in accordance with [36], and a real-time implementation of the bilateral filter [37] is utilized.

III-B2 Tone mapping

Because the scaled luminance value $l^{\prime\prime}_{m}(\boldsymbol{p})$ often exceeds the maximum value of the common image formats, pixel values might be lost due to truncation of the values. This problem is overcome by using a tone mapping operation to fit the adjusted luminance value into the interval $[0,1]$.

The luminance $\hat{l}_{m}(\boldsymbol{p})$ of an adjusted multi-exposure image is obtained by applying a tone mapping operator $f_{m}$ to $l^{\prime\prime}_{m}(\boldsymbol{p})$:

[TABLE]

Reinhard’s global operator is used here as the tone mapping operator $f_{m}$ [19].

Reinhard’s global operator is given by

[TABLE]

where parameter $L_{m}>0$ determines value $t$ to be $f_{m}(t)=1$. Note that Reinhard’s global operator $f_{m}$ is a monotonically increasing function. Here, setting $L_{m}$ as $L_{m}=\max l^{\prime\prime}_{m}(\boldsymbol{p})$ for each $m$, we obtain $\hat{l}_{m}(\boldsymbol{p})\leq 1$ for all $\boldsymbol{p}$. Therefore, truncation of the luminance values can be prevented. In contrast, if it is preferable not to change luminance by tone mapping, setting $L_{m}$ as $L_{m}=1$ is chosen. As described above, each of these two parameter settings has advantages and disadvantages. In this paper, parameter $L_{m}=\max l^{\prime\prime}_{m}(\boldsymbol{p})$ is utilized.

III-B3 Fusion of enhanced multi-exposure images

Adjusted multi-exposure images ${\hat{\boldsymbol{x}}_{m}}$ can be used as input for any existing multi-exposure image fusion methods [9, 15]. A final image $\boldsymbol{y}$ is produced as

[TABLE]

where $\mathscr{F}(\boldsymbol{x}_{1},\boldsymbol{x}_{2},\cdots,\boldsymbol{x}_{M})$ indicates a function for fusing $M$ images $\boldsymbol{x}_{1},\boldsymbol{x}_{2},\cdots,\boldsymbol{x}_{M}$ into a single image.

III-C Scene segmentation-based luminance scaling

The proposed SSLS is applied to the set $\{l^{\prime}_{n}\}$ after local contrast enhancement as shown in Fig. 3. The purpose of SSLS is to produce luminance $l^{\prime\prime}_{m}$, which clearly represents area $\mathrm{P}_{m}\subset\mathrm{P}$ having a specific brightness range in a scene. For this purpose, we introduce two processes: scene segmentation and luminance scaling (see Fig. 4). In the former, the set $\mathrm{P}$ of all pixels is divided into $M$ subsets $\{\mathrm{P}_{1},\mathrm{P}_{2},\cdots,\mathrm{P}_{M}\}$. A set $\{l^{\prime\prime}_{m}\}$ of luminance is produced in the latter process, so that luminance $l^{\prime\prime}_{m}$ clearly represents area $\mathrm{P}_{m}$. In the latter process, luminance scaling is applied to all luminance values $l^{\prime}_{n}(p)$ on $\mathrm{P}$, not on $\mathrm{P}_{m}$. This enables us to have the same image structure as before the scaling.

III-C1 Scene segmentation

Here, we discuss how to separate a scene in multi-exposure images into areas $\mathrm{P}_{1},\cdots,\mathrm{P}_{M}$, where each has a specific brightness and $\mathrm{P}_{1}\cup\mathrm{P}_{2}\cup\cdots\cup\mathrm{P}_{M}=\mathrm{P}$. This operation is regarded as image segmentation based on luminance values. This segmentation for multi-exposure image fusion differs from typical segmentation problems such as semantic segmentation [38, 39] in two ways.

• •

Multiple input images are given.

• •

Attention to the structure of images, e.g., edges, is not needed.

We propose two approaches to segmentation. The first approach (Approach 1), which has low computational cost, separates a scene in multi-exposure images, according to the luminance values of an input image having middle brightness. In the second approach (Approach 2), which provides high-quality results, the scene is separated by considering the luminance distribution of all input images.

Approach 1: Let $l^{\prime}_{\mathrm{med}}$ be the luminance having the middle brightness in the set $\{l^{\prime}_{n}\}$. The overexposed (or underexposed) areas in $l^{\prime}_{\mathrm{med}}$ are smaller than those in the other luminance images in $\{l^{\prime}_{n}\}$. Namely, $l^{\prime}_{\mathrm{med}}$ has the best quality in $\{l^{\prime}_{n}\}$. We thus utilize $l^{\prime}_{\mathrm{med}}$ to separate a scene.

In Approach 1, $\mathrm{P}_{1},\cdots,\mathrm{P}_{M}$ are given by dividing the luminance range of $l^{\prime}_{\mathrm{med}}$ into $M$ equal parts:

[TABLE]

where $\theta_{m}$ is calculated as

[TABLE]

Approach 1 is very simple but is effective for most of the images shown later.

Approach 2: Approach 2 takes into account the luminance distribution of all input images, while Approach 1 only uses one of the input images for segmentation. This enables us to produce a high-quality fused image, $\boldsymbol{y}$, even when Approach 1 does not work well. A Gaussian mixture distribution is utilized to model the luminance distribution of the input images. After that, pixels are classified by using a clustering algorithm based on a Gaussian mixture model (GMM) [40].

To obtain a model considering the luminance values of all input images, we regard luminance values at a pixel $\boldsymbol{p}$ as an $N$-dimensional vector $\boldsymbol{l}^{\prime}(\boldsymbol{p})=(l^{\prime}_{1}(\boldsymbol{p}),l^{\prime}_{2}(\boldsymbol{p}),\cdots,l^{\prime}_{N}(\boldsymbol{p}))^{\top}$. Then the distribution of $\boldsymbol{l}^{\prime}(\boldsymbol{p})$ is modeled by using a GMM as

[TABLE]

where $K$ is the number of mixture components, $\pi_{k}$ is a mixing coefficient, and $\mathcal{N}(\boldsymbol{l}^{\prime}(\boldsymbol{p})|\boldsymbol{\mu}_{k},\boldsymbol{\Sigma}_{k}))$ is an $N$-dimensional Gaussian distribution with mean $\boldsymbol{\mu}_{k}$ and variance-covariance matrix $\boldsymbol{\mathrm{\Sigma}}_{k}$.

To fit the GMM into a given $\boldsymbol{l}^{\prime}(\boldsymbol{p})$, variational Bayesian inference [40] is utilized. Compared with the traditional maximum likelihood estimation, one of the advantages is that variational Bayesian inference can avoid overfitting even when we choose a large $K$. For this reason, unnecessary mixture components are automatically removed by using the inference together with a large $K$. $K=10$ is used in this paper as the maximum of the partition number $M$.

Let us introduce a $K$-dimensional binary random variable, $\boldsymbol{z}$, having a 1-of-$K$ representation in which a particular element $z_{k}$ is equal to 1 and all other elements are equal to 0. The values of $z_{k}$ therefore satisfy $z_{k}\in\{0,1\}$ and $\sum_{k}z_{k}=1$. The marginal distribution over $\boldsymbol{z}$ is specified in terms of a mixing coefficient, $\pi_{k}$, such that

[TABLE]

In order for $p(z_{k}=1)$ to be a valid probability, $\{\pi_{k}\}$ must satisfy

[TABLE]

together with

[TABLE]

An area $\mathrm{P}_{m}$ containing a pixel $\boldsymbol{p}$ is determined by the responsibility $\gamma(z_{k}|\boldsymbol{l}^{\prime}(\boldsymbol{p}))$, which is given as the conditional probability:

[TABLE]

When a pixel $\boldsymbol{p}\in\mathrm{P}$ is given and $m$ satisfies

[TABLE]

the pixel $\boldsymbol{p}$ is assigned to a subset $\mathrm{P}_{m}$ of $\mathrm{P}$.

III-C2 Luminance scaling

The scaled luminance $l^{\prime\prime}_{m}$ is simply obtained by

[TABLE]

where parameter $\alpha_{m}>0$ indicates the degree of adjustment (see Appendix A).

Given $\mathrm{P}_{m}$ as a subset of $\mathrm{P}$, the approximate brightness of $\mathrm{P}_{m}$ is calculated as the geometric mean of luminance values on $\mathrm{P}_{m}$. We thus estimate an adjusted multi-exposure image $l^{\prime\prime}_{m}(\boldsymbol{p})$ so that the geometric mean of its luminance equals the middle-gray of the displayed image, or 0.18 on a scale from zero to one, as in [19].

The geometric mean $G(l|\mathrm{P}_{m})$ of luminance $l$ on pixel set $\mathrm{P}_{m}$ is calculated by using

[TABLE]

where $\epsilon$ is set to a small value to avoid singularities at $l(\boldsymbol{p})=0$.

By using eq. (16), parameter $\alpha_{m}$ is calculated as

[TABLE]

Since a smaller value for parameter $\alpha_{m}$ is better, $n$ is chosen as

[TABLE]

The scaled luminance $l^{\prime\prime}_{m}$, calculated by using eq. (15) with parameters $\alpha_{m}$ and $n$, is used as input of the tone mapping operation described in III-B2.

III-D Combining adjusted luminance and input images

Adjusted images $\{\hat{\boldsymbol{x}}_{m}\}$ are obtained by combining a set $\{\hat{l}_{m}\}$ of luminance adjusted by the SSLA and input multi-exposure images $\{\boldsymbol{x}_{n}\}$. To associate each $\hat{l}_{m}$ with an input image $\boldsymbol{x}_{n}$, eq. (18) is utilized. As a result, combining $\hat{l}_{m}$, the $\psi(m)$-th input image $\boldsymbol{x}_{\psi(m)}$, and its luminance $l_{\psi(m)}$, we obtain adjusted multi-exposure images $\hat{\boldsymbol{x}}_{m}$:

[TABLE]

This transformation can preserve original colors since the RGB color space has a linear relationship with the XYZ color space (see Appendix B).

III-E Proposed procedure

The procedure for generating an image $\boldsymbol{y}$ from $N$ input images $\{\boldsymbol{x}_{n}\}$ with the proposed method is summarized as follows (see Fig. 3).

1. i

Calculate the luminance $l_{n}$ from each input image $x_{n}$. 2. ii

Enhance the local contrast of ${l_{n}}$ by using eq. (1) to eq. (3) and then obtain the enhanced luminance $\{l^{\prime}_{n}\}$. 3. iii

Separate $\mathrm{P}$ into $M$ areas $\{\mathrm{P}_{m}\}$

1. Approach 1:

use eqs. (7) and (8). 2. Approach 2:

use eq. (9) to eq. (14). 4. iv

Calculate $\{l^{\prime\prime}_{m}\}$ by using eq. (15) to eq. (18). 5. v

Map $\{l^{\prime\prime}_{m}\}$ to $\{\hat{l}_{m}\}$ according to eqs. (4) and (5). 6. vi

Generate $\{\hat{\boldsymbol{x}}_{m}\}$ according to eq. (19). 7. vii

Obtain an image $\boldsymbol{y}$ with a multi-exposure image fusion method $\mathscr{F}$ as in eq. (6).

Note that the difference between Approach 1 and Approach 2 is only in step iii, and the number $M$ of $\mathrm{P}_{m}$ satisfies $1\leq M\leq K$. Tunable parameters in the proposed method are shown in Table I.

IV Simulation

We evaluated the effectiveness of the proposed SSLA in terms of the quality of fused images $\boldsymbol{y}$.

IV-A Simulation conditions

In the simulation, 20 sets of photographs and 550 sets of tone-mapped images were used as input images $\{\boldsymbol{x}_{n}\}$. Among the sets of photographs, four were taken with a Canon EOS 5D Mark II camera, and eight were selected from an available online database [41], where each set contained three multi-exposure images that were exposed with negative, zero, and positive exposure values. (see Figs. 5 and 6). The other sets were collected from database [42], where each set included moving objects. The sets of tone-mapped images were generated by tone-mapping 50 HDR images selected from online databases [43, 44]. From each HDR image, 11 sets of images were generated by linear response functions, where the number of images in ten sets was randomly decided in the range of $[2,5]$ and their exposure values were also randomly determined in the range of $[-7,0]$, to produce unclear multi-exposure images. The other set contained $15$ multi-exposure images whose exposure values were $-7,-6,\cdots,6$, and $7$.

The following procedure was carried out to evaluate the effectiveness of the proposed method.

1. i

Produce $\{\hat{\boldsymbol{x}}_{m}\}$ from $\{\boldsymbol{x}_{n}\}$ by using the proposed method. 2. ii

Obtain $\boldsymbol{y}$ fused from $\{\hat{\boldsymbol{x}}_{m}\}$ with $\mathscr{F}$. 3. iii

Compute scores of three objective quality metrics: MEF-SSIM, discrete entropy and TMQI, for $\boldsymbol{y}$, as described later.

Here, we used five fusion methods for $\mathscr{F}$: Mertens’s method [9], Sakai’s method [14], Nejati’s method [15], Li’s method [17], and Ma’s method [32]. In Approach 1, $M$ was set so as to be the same as the number $N$ of input images. In Approach 2, $M$ was a number that satisfies $1\leq M\leq K=10$, and the number $M$ was determined by the resulting GMM for each scene. The fitting of a GMM was cut off when the number of iterations reached 100, regardless of parameter convergence. In addition, Approach 2 was applied to downsized versions of $\{l^{\prime}_{n}\}$ for fast calculation. In particular, the resized width and height ($U^{\prime}$ and $V^{\prime}$, respectively) of $l^{\prime}_{n}$ were determined so that $\max(U^{\prime},V^{\prime})=256$.

IV-B Visual evaluation

Figures 7 and 8 show images fused with/without the proposed SSLA from the input multi-exposure ones $\{\boldsymbol{x}_{n}\}$ illustrated in Fig. 5. Figures. 7(7(a)) to (7(c)), Fig. 8(8(a)), and Fig. 8(8(b)) show that the effects of all of the fusion methods without luminance adjustment were not sufficient enough to visualize shadow areas when unclear input images were used as inputs. In contrast, images fused by using the proposed SSLA clearly represent the shadow areas while maintaining the quality of bright areas [see Figs. 7(7(d)) to (7(i)) and Figs. 8(8(c)) to (8(f))]. However, combining the proposed SSLA with Li’s method caused luminance inversion between bright and dark areas. Note that the inversion did not occur when Nejati’s and Ma’s methods, which are state-of-the-arts, were combined. The results for another image set in Fig. 6 are also displayed in Fig. 9. Figure 9 shows a similar trend as in Fig. 7. Therefore, the proposed SSLA makes it possible to improve the quality of fused images even when input multi-exposure images are unclear.

By comparing Approach 1 with Approach 2, Approach 2 generated images representing dark areas more distinctly, as shown in Figs. 7 and 9. The difference between the two approaches originates in segmented areas $\{\mathrm{P}_{m}\}$. Figures 10 and 11 show areas $\{\mathrm{P}_{m}\}$ separated by Approaches 1 and 2 and adjusted multi-exposure images based on each $\{\mathrm{P}_{m}\}$. From Fig. 10(10(a)), it is shown that Approach 1 was not able to separate the dark areas on the right, e.g., the door, and the areas on the left with middle-brightness, e.g., the floor, while it does low computational cost. For this reason, the brightest image in Fig. 10(10(c)) does not have sufficient brightness to clearly represent the darkest areas in the scene. In comparison, Approach 2 was able to separate scenes into appropriate areas, each having a specific luminance range, even though a large $K$ was given. Figure 11 shows almost the same result as Fig. 10. Hence, it is verified that Approach 2 can separate scenes with higher accuracy than Approach 1, while Approach 1 can be performed with closed-form calculation.

Figures 10 and 11 also illustrate that both Approach 1 and 2 maintained the structure of images even though the two approaches did not pay attention to the structure. This result represents that the segmentation method in the proposed SSLA does not require the information of image structure such as edges. Figure 12 shows the effect of local contrast enhancement in the proposed method. This enhancement can reduce blurring and make the structure in images clear, but it often causes noise and ringing to be boosted. In this case, the proposed method without local contrast enhancement is effective to suppress such noise boosting.

Fused images from input multi-exposure ones containing moving objects are shown in Fig. 13. Even when moving objects are included in input images, the proposed method can still provide high-quality fused images without ghost-like artifacts by using a robust method to deal with dynamic scenes as fusion method $\mathscr{F}$. In contrast, the image generated without the proposed method contained a few ghost-like artifacts. This result indicates that the proposed method is able not only to clarify fused images, but also to remove ghost-like artifacts.

IV-C Quantitative evaluation

To quantitatively evaluate the quality of fused images, objective quality metrics are needed. Typical quality metrics such as the peak signal to noise ratio (PSNR) and structural similarity index (SSIM) are not suitable for this purpose because they use a target image with the highest quality as a reference. Recently, some full-reference metrics for multi-exposure image fusion have been proposed [45, 46]. However, because high-quality multi-exposure images are needed for these metrics, they are also not suitable for evaluating the proposed method when the multi-exposure ones are unavailable. We therefore used the metric [45], referred to as MEF-SSIM, only when high-quality multi-exposure ones are available. In other cases, the discrete entropy of luminance values and tone mapped image quality index (TMQI) [47] were used as quality metrics.

MEF-SSIM is based on a multi-scale SSIM framework and a patch consistency measure. It keeps a good balance between local structure preservation and global luminance consistency. Discrete entropy represents the amount of information in an image, where the entropy is calculated by using the luminance values of a fused image $\boldsymbol{y}$. Discrete entropy of luminance also shows the global contrast of an image. TMQI represents the quality of an image tone-mapped from an HDR image; the metric incorporates structural fidelity and statistical naturalness. Statistical naturalness [47] is defined as

[TABLE]

where $K$ is a normalization factor, $\mathcal{B}(\cdot|\alpha,\beta)$ is a beta probability density function with parameters $\alpha$ and $\beta$, $\overline{l}$ is the average luminance of an image on a scale $[0,255]$, and $\overline{\sigma_{l}}$ is the average of local standard deviation of luminance on the scale. The Gaussian and beta distributions reflect naturalness of the global brightness and local contrast, respectively. Statistical naturalness is calculated without any reference images, although structural fidelity needs an HDR image as a reference. Since the processes of tone mapping and photographing are similar, TMQI is also useful for evaluating photographs. In this simulation, MEF-SSIM and TMQI were used only for evaluation with the sets of tone-mapped images, while discrete entropy and statistical naturalness were used for evaluation with all of the 570 sets of images.

Tables II and III summarize scores for the 12 sets of photographs of static scenes in terms of discrete entropy and statistical naturalness. For each score (discrete entropy $\in[0,8]$ and statistical naturalness $\in[0,1]$), a larger value means higher quality. Table II shows that the proposed SSLA provided high entropy scores for all 12 image sets, although the typical fusion methods provided low entropy scores for “Corridor 1” and “Lobby.” In addition, the proposed SSLA provided higher average scores than the typical fusion methods. Table III denotes that the proposed SSLA enables us to produce high-quality images in terms of statistical naturalness in almost all cases, compared with the typical methods.

Tables IV and V denote scores for the eight sets of photographs of dynamic scenes in terms of discrete entropy and statistical naturalness. Here, Ma’s method was used as fusion method $\mathscr{F}$. Since the sets contained enough number of multi-exposure images, the scores did not have a large difference between with/without the proposed method.

Figure 14 summarizes quantitative evaluation results for the 500 sets of unclear tone-mapped images in terms of discrete entropy, TMQI, statistical naturalness, and MEF-SSIM as box plots. The boxes span from the first to the third quartile, referred to as $Q_{1}$ and $Q_{3}$, and the whiskers show the maximum and minimum values in the range of $[Q_{1}-1.5(Q_{3}-Q_{1}),Q_{3}+1.5(Q_{3}-Q_{1})]$. The band inside the boxes indicates the median, i.e., the second quartile $Q_{2}$, and the crosses denote the average value. Similarly to statistical naturalness, larger TMQI and MEF-SSIM scores $\in[0,1]$ indicate higher quality. From this figure, it is shown that both of the approaches with the proposed method enable us to obtain images with higher-quality than images without any adjustment with any fusion method. In particular, Approach 2 provided the highest score under the use of the same fusion method in terms of all metrics. Therefore, the proposed SSLA is effective for multi-exposure image fusion when input multi-exposure images do not have a sufficient number of different exposure levels.

Figure 15 shows the performance of the proposed method when a sufficient number of multi-exposure images were given as inputs. Scores provided by the proposed method were almost the same as those of Ma’s method, but the scores slightly less than those of Ma’s method, Hence, it is confirmed that the proposed method does not generate a harmful effect even when well-exposed multi-exposure images are given.

For these reasons, it was confirmed that the proposed SSLA is effective for multi-exposure image fusion. It is also useful for producing high quality images that clearly represent an entire scene. Comparing Approaches 1 and 2, Approach 2 generated clear images, while Approach 1 can be performed by using a closed-form.

IV-D Computational complexity

To evaluate the computational complexity of the proposed method, we measured the executing time of the proposed one. Eight image sets selected from an available online database [41] were utilized for the evaluation. Here, the minimum and maximum number of pixels in the image sets were $425,430$ and $918,400$, where their average was $533,606.3$.

The simulation was run on a PC, with a 4.2 GHz processor and a main memory of 64 Gbytes (see Table VI). In the simulation, timeit() function in MATLAB was used for measuring the executing time, and every method was carried out on a single thread of execution.

Figure 16 shows average executing time for the eight image sets. From the figure, we can see that the overhead of Approach 1 was about one second. In contrast, Approach 2 needs additional computational cost for adjustment in addition to the cost of Approach 1. Compared with Approach 1, the computational cost for fusion also increased because the number of adjusted multi-exposure images is generally larger than that of input multi-exposure images. Therefore, Approach 1 has a lower computational cost than Approach 2, while Approach 2 can generate better images than Approach 1.

V Conclusion

In this paper, we proposed a novel luminance adjustment method based on scene segmentation for multi-exposure image fusion. We first pointed out that adjusting the luminance of input images makes it possible to improve the quality of final fused images, although existing fusion methods directly fuse input multi-exposure images without any adjustment. The proposed method enables us to produce high-quality images even when undesirable inputs are given. In the method, we utilize scene segmentation in order to automatically adjust input multi-exposure images so that they become suitable for multi-exposure image fusion. For the scene segmentation, two approaches, 1 and 2, were proposed. Approach 1 separates a scene in multi-exposure images into a number of areas with closed-form calculation. In Approach 2, a scene is separated by a GMM of luminance distribution. In simulations, the proposed SSLA was applied to five fusion methods, Mertens’s one, Sakai’s one, Nejati’s one, Li’s one, and Ma’s one. The results showed that the proposed SSLA is effective with any of the fusion methods in terms of entropy and TMQI. Moreover, visual comparison results showed that the proposed SSLA makes it possible to clearly represent shadow areas in images while maintaining the quality of bright areas. In particular, Approach 2 can generate clearer images than Approach 1. The proposed SSLA was also shown to be still effective for scenes including moving objects, and ghost-like artifacts were eliminated by using the proposed SSLA with Ma’s fusion method. This indicates that the proposed SSLA is expected to improve the performance of image alignment in various image fusion problems, such as multi-exposure image fusion, HDR imaging, and panoramic image stitching. Furthermore, combining the proposed SSLA with the burst photography would enable us to improve the noise robustness.

Appendix A Relationship between exposure values and pixel values

Figure 17 shows a typical imaging pipeline for a digital camera[48]. Here, we focus on grayscale images for simplicity. The radiant power density at a sensor, i.e., irradiance $E$, is integrated over the time $\Delta t\in[0,\infty)$ that the shutter is open, producing an energy density, commonly referred to as “exposure $X$.” If the scene is static during this integration, exposure $X$ can be written simply as the product of irradiance $E$ and integration time $\Delta t$ (referred to as “shutter speed”):

[TABLE]

The photographed image $x$ is given by

[TABLE]

where $f$ is a function combining sensor saturation and a camera response function (CRF). The CRF represents the processing in each camera that makes the final image $x$ look better.

Camera parameters, such as shutter speed and lens aperture, are usually calibrated in terms of exposure value (EV) units, and the proper exposure for a scene is automatically decided by the camera. The exposure value is commonly controlled by changing the shutter speed although it can also be controlled by adjusting various camera parameters. Here we assume that the camera parameters except for the shutter speed are fixed. Let $v_{0}=0\mathrm{[EV]}$ and $\Delta t_{0}$ be the proper exposure value and shutter speed under the given conditions, respectively. The exposure value $v_{i}\mathrm{[EV]}$ of an image taken at shutter speed $\Delta t_{i}$ is given by

[TABLE]

From (21) to (23), images $x_{0}$ and $x_{i}$ exposed at $0\mathrm{[EV]}$ and $v_{i}\mathrm{[EV]}$, respectively, are written as

[TABLE]

Assuming function $f$ is linear, we obtain a relationship between $x_{0}$ and $x_{i}$:

[TABLE]

Therefore, the exposure can be varied artificially by multiplying $x_{0}$ by a constant.

Appendix B Physical meaning of scaling RGB values

In the proposed SSLA, RGB pixel values of an adjusted multi-exposure image are given by eq. (19). Here, we prove that scaling RGB pixel values does not change colors of objects.

Let spectral distribution of illumination light striking an object, spectral reflectance of the object, and spectral response of RGB color filters on an imaging sensor as $P(\lambda),R(\lambda),$ and $H_{c}(\lambda)(c\in{r,g,b})$, where $\lambda$ means wavelength of light. Irradiance $E_{c}$ at the sensor is given as

[TABLE]

From eqs. (21) and (22), RGB pixel value $x_{c}$ is written as

[TABLE]

If the radiance power of $P(\lambda)$ is scaled with $\alpha$, RGB pixel value $x^{\prime}_{c}$ is calculated by

[TABLE]

Assuming function $f$ is linear, namely, $x_{c}$ is a linear-RGB pixel value, we obtain a relationship between $x_{c}$ and $x^{\prime}_{c}$:

[TABLE]

Therefore, scaling RGB pixel value $x_{c}$ with $\alpha$ corresponds to scaling the power of illumination light with $\alpha$. For this reason, scaling RGB pixel values does not change colors of objects.

Acknowledgment

This work was suppoeted by JSPS KAKENHI Grant Number JP18J20326.