Recently, the use of online shopping continues to increase due to the influence of the contactless culture from the prolonged COVID-19. As an alternative to this, individual skin condition diagnosis services and non-face-to-face makeup cosmetic color evaluation are emerging. However, it was possible to use an accurate service only by using a specific sensor until now. Therefore, a service that consumers can use conveniently in any environment is required. In this study, information of ambient light sources and individual skin spectral reflectance are estimated using a smartphone. It uses the individual skin reflection spectrum to determine the skin condition and predicts the spectral reflectance and color that could occur when makeup cosmetics are applied. Also, based on the accurate color reproduction method on the display of the smartphone, the same color as the actual color could be observed on the smartphone.

Firstly, images taken with a smartphone are analyzed for non-face-to-face diagnosis of individual skin and makeup simulation. An individual's skin spectral reflectance is inferred from the image through our algorithm.
Secondly, the values of parameters of the skin description model are estimated through the process of matching the value of the individual skin reflection spectrum with the ‘(epidermal/dermis) two-layer skin description model’ based on the Kubelka-Munk model. At this time, parameters such as volume fractions of melanin and hemoglobin, dermis/epidermal thickness, etc., could be used as personal skin information. This showed a reliable correlation between the skin information measured by the skin measuring device and the skin information value estimated by spectral matching in a clinical experiment on men and women in their 20s.
Thirdly, the spectral reflectance of skin applied with the color cosmetics was estimated based on the '(cosmetics/epidermis/dermis) 3-layer optical model' in which the color cosmetics spectral reflectance is added to the above transmission/reflection spectrum of the two-layered skin description model. The advantage of this non-face-to-face color test method is that it can be applied by changing the light source constant even if the light source is changed by obtaining the color spectrum. To compare predicted the spectral reflectance of skin applied with the cosmetics and real of that, we compared the spectral reflectance values measured with a spectrophotometer after applying cosmetics to the skin of men and women in their 20s and the predicted spectrum results.
Finally, the color of skin covered with the cosmetics on the smartphone display, the effect of the display backlight and light source was expressed as an equation. In general. when color information is input into the display, an output color on the display is changed by the kind of the display and the light source, so it is necessary to correct the color information. The equation was obtained in the following way. The information of the colors appearing on the display is obtained using a spectroradiometer under various lighting conditions. The relationship between the input color and the output color on the display was expressed as an equation. Also, each light source has a different correlation coefficient in the equation. It corrects the previously calculated color information of skin covered with the cosmetics using the equation.

It was possible to represent the spectral reflectance of individual skin similar to that measured by spectrophotometer based on the Kubelka-Munk model. By analyzing the spectral reflectance spectrum through optical simulation, we established a system that extracts the main parameters, such as melanin fraction, hemoglobin fraction. It is expected that this information can be applied to the healthcare field or linked with customized cosmetic recommendations. In addition, the spectral reflection spectrum of skin covered with cosmetics by applying a three-layer optical model was obtained. And, it was confirmed that the color matching between the actual color and display color was successively implemented by using the obtained formula. This implies that if color information is corrected with the obtained formula, consumers will observe the same color as when the cosmetic was applied on their skin in any lighting environment.

Virtually, our study provides a confident approach to represent individual health conditions and the color of real skin covered with cosmetics. Moreover, this method will assist to boost the growth rate of the beauty market, which is evolving into personalization and digitalization.