Difference between revisions of "Hyperspectral Imaging"

Latest revision as of 11:54, 27 September 2023

A hyperspectral camera captures a scene in many frequency bands across the spectrum, providing rich information and facilitating numerous applications in industrial, military, and commercial domains. Example applications of hyperspectral imaging include medical diagnosis (e.g., early detection of skin cancer), food-quality inspection, artwork authentication, forest monitoring, material identification, and remote sensing. The potential of hyperspectral imaging has been established for decades now. However, to date, hyperspectral imaging has only seen success in specialized and large-scale industrial and military applications. This is because of three main challenges: (i) the sheer volume of hyperspectral data which makes it hard to transmit such data in real-time and thus limiting its usefulness for many applications, (ii) the negative impact of the environmental conditions (e.g., rain, fog, and snow) which reduces the utility of the captured hyper-spectral data, and (iii) the high cost of hyperspectral cameras (upwards of $20K USD) which makes the technology out of reach for many commercial and end-user applications. The goal of this project is to address these challenges to enable wide adoption of hyperspectral imaging in many applications.

MobiSpectral: Hyperspectral Imaging on Mobile Devices

Hyperspectral imaging systems capture information in multiple wavelength bands across the electromagnetic spectrum. These bands provide substantial details based on the optical properties of the materials present in the captured scene. The high cost of hyperspectral cameras and their strict illumination requirements make the technology out of reach for end-user and small-scale commercial applications. We propose MobiSpectral, which turns a low-cost phone into a simple hyperspectral imaging system, without any changes in the hardware. We design deep learning models that take regular RGB images and near-infrared (NIR) signals (which are used for face identification on recent phones) and reconstruct multiple hyperspectral bands in the visible and NIR ranges of the spectrum. Our experimental results show that MobiSpectral produces accurate bands that are comparable to ones captured by actual hyperspectral cameras. The availability of hyperspectral bands that reveal hidden information enables the development of novel mobile applications that are not currently possible. To demonstrate the potential of MobiSpectral, we use it to identify organic solid foods, which is a challenging food fraud problem that is currently partially addressed by laborious, unscalable, and expensive processes. We collect large datasets in real environments under diverse illumination conditions to evaluate MobiSpectral. Our results show that MobiSpectral can identify organic foods, e.g., apples, tomatoes, kiwis, strawberries, and blueberries, with an accuracy of up to 94% from images taken by phones.

Overview of MobiSpectral

People

Neha Sharma
Muhammad Shahzaib Waseem
Shahrzad Mirzaei
Mariam Bebawy

Code and Datasets

Publications

N. Sharma, M. Waseem, S. Mirzaei, and M. Hefeeda, MobiSpectral: Hyperspectral Imaging on Mobile Devices, In Proc. of ACM Conference on Mobile Computing and Networking (MobiCom'23), Madrid, Spain, October 2023. (Artifacts Evaluated and Functional)
P. Moghadam, N. Sharma, M. Hefeeda, Enabling Hyperspectral Imaging in Diverse Illumination Conditions for Indoor Applications, In Proc. of ACM Multimedia Systems Conference (MMSys’21), Istanbul, Turkey, September 2021. (Artifacts Evaluated and Functional)

N. Sharma, M. Hefeeda, Hyperspectral Reconstruction from RGB Images for Vein Visualization, In Proc. of ACM Multimedia Systems Conference (MMSys'20), Istanbul, Turkey, June 2020. (Artifacts Evaluated and Functional)

M. Arab, K. Calagari, M. Hefeeda, Band and Quality Selection for Efficient Transmission of Hyperspectral Images, In Proc. of ACM Conference on Multimedia (MM'19), Nice, France. October 2019.

@@ Line 1: / Line 1: @@
-Hyperspectral imaging (HSI) is a powerful tool that can provide substantial information about a scene through remote sensing. This project addresses different challenges in the pipeline of capturing, transmitting, and identifying remote scenes using hyperspectral images. One of the main significant challenges in hyperspectral imaging is analyzing the data and extracting the required information. This is mainly because of the extremely high dimensionality of hyperspectral images, which limits our ability to identify spectral signatures fast and accurately. In addition, noise in hyperspectral images makes matters even more complicated.
+A hyperspectral camera captures a scene in many frequency bands across the spectrum, providing rich information and facilitating numerous applications in industrial, military, and commercial domains. Example applications of hyperspectral imaging include medical diagnosis (e.g., early detection of skin cancer), food-quality inspection, artwork authentication, forest monitoring, material identification, and remote sensing. The potential of hyperspectral imaging has been established for decades now. However, to date, hyperspectral imaging has only seen success in specialized and large-scale industrial and military applications. This is because of three main challenges: (i) the sheer volume of hyperspectral data which makes it hard to transmit such data in real-time and thus limiting its usefulness for many applications, (ii) the negative impact of the environmental conditions (e.g., rain, fog, and snow) which reduces the utility of the captured hyper-spectral data, and (iii) the high cost of hyperspectral cameras (upwards of $20K USD) which makes the technology out of reach for many commercial and end-user applications. The goal of this project is to address these challenges to enable wide adoption of hyperspectral imaging in many applications.
-== People ==
-* [https://maarab-sfu.github.io/ Mohammad Amin Arab]
-* Kiana Calagari
-* [https://www.cs.sfu.ca/~mhefeeda/ Mohamed Hefeeda]
-== Band and Quality Selection for Efficient Transmission of Hyperspectral Images ==
+'''MobiSpectral: Hyperspectral Imaging on Mobile Devices'''
-''' Abstract '''
+Hyperspectral imaging systems capture information in multiple wavelength bands across the electromagnetic spectrum. These bands provide substantial details based on the optical properties of the materials present in the captured scene. The high cost of hyperspectral cameras and their strict illumination requirements make the technology out of reach for end-user and small-scale commercial applications. We propose MobiSpectral, which turns a low-cost phone into a simple hyperspectral imaging system, without any changes in the hardware. We design deep learning models that take regular RGB images and near-infrared (NIR) signals (which are used for face identification on recent phones) and reconstruct multiple hyperspectral bands in the visible and NIR ranges of the spectrum. Our experimental results show that MobiSpectral produces accurate bands that are comparable to ones captured by actual hyperspectral cameras. The availability of hyperspectral bands that reveal hidden information enables the development of novel mobile applications that are not currently possible. To demonstrate the potential of MobiSpectral, we use it to identify organic solid foods, which is a challenging food fraud problem that is currently partially addressed by laborious, unscalable, and expensive processes. We collect large datasets in real environments under diverse illumination conditions to evaluate MobiSpectral. Our results show that MobiSpectral can identify organic foods, e.g., apples, tomatoes, kiwis, strawberries, and blueberries, with an accuracy of up to 94% from images taken by phones.
-Due to recent technological advances in capturing and processing devices, hyperspectral imaging is becoming available for many commercial and military applications such as remote sensing, surveillance, and forest fire detection. Hyperspectral cameras provide rich information, as they capture each pixel along many frequency bands in the spectrum. The large volume of hyperspectral images as well as their high dimensionality make transmitting them over limited-bandwidth channels a challenge. To address this challenge, we present a method to prioritize the transmission of various components of hyperspectral data based on the application needs, the level of details required, and available bandwidth. This is unlike current works that mostly assume offline processing and the availability of all data beforehand. Our method jointly and optimally selects the spectral bands and their qualities to maximize the utility of the transmitted data. It also enables progressive transmission of hyperspectral data, in which approximate results are obtained with small amount of data and can be refined with additional data. This is a desirable feature for large-scale hyperspectral imaging applications. We have implemented the proposed method and compared it against the state-of-the-art in the literature using hyperspectral imaging datasets. Our experimental results show that the proposed method achieves high accuracy, transmits a small fraction of the hyperspectral data, and significantly outperforms the state-of-the-art; up to 35% improvements in accuracy was achieved.
+[[File:Mobispectral.png|thumb|center|700px|Overview of MobiSpectral]]
-[[Band and Quality Selection for Efficient Transmission of Hyperspectral Images|More info [login required]]] ...
+== People ==
+* [https://www.sfu.ca/~nsa84/ Neha Sharma]
+* Muhammad Shahzaib Waseem
+* Shahrzad Mirzaei
+* Mariam Bebawy
-== Hyperspectral Reconstruction from RGB Images for Vein Visualization ==
+== Code and Datasets ==
+* [https://github.com/mobispectral/mobicom23_mobispectral MobiSpectral: Hyperspectral Imaging on Mobile Devices]
+* [https://github.com/pazadimo/HS_In_Diverse_Illuminations Enabling Hyperspectral Imaging in Diverse Illumination Conditions for Indoor Applications]
+* [https://github.com/nehasharma512/vein-visualization Hyperspectral Reconstruction from RGB Images for Vein Visualization]
+* [https://github.com/maarab-sfu/BQSETHI Band and Quality Selection for Efficient Transmission of Hyperspectral Images]
-''' Abstract '''
+== Publications ==
+* N. Sharma, M. Waseem, S. Mirzaei, and M. Hefeeda, [https://www2.cs.sfu.ca/~mhefeeda/Papers/mobiCom23_MobiSpectral.pdf MobiSpectral: Hyperspectral Imaging on Mobile Devices], In Proc. of ACM Conference on Mobile Computing and Networking (MobiCom'23), Madrid, Spain, October 2023. (Artifacts Evaluated and Functional)
+* P.  Moghadam, N. Sharma, M. Hefeeda, [https://www2.cs.sfu.ca/~mhefeeda/Papers/mmsys21_HS_illumination.pdf Enabling Hyperspectral Imaging in Diverse Illumination Conditions for Indoor Applications], In Proc. of ACM Multimedia Systems Conference (MMSys’21), Istanbul, Turkey, September 2021. (Artifacts Evaluated and Functional)
-A hyperspectral camera captures a scene in many frequency bands across the spectrum, providing rich information and facilitating numerous applications. The potential of hyperspectral imaging has been established for decades. However, to date hyperspectral imaging has only seen success in specialized and large-scale industrial and military applications. This is mainly due to the high cost of hyperspectral cameras (upwards of $20K) and the complexity of the acquisition system which makes the technology out of reach for many commercial and end-user applications. In this paper, we propose a deep learning based approach to convert RGB image sequences taken by regular cameras to (partial) hyperspectral images. This can enable, for example, low-cost mobile phones to leverage the characteristics of hyperspectral images in implementing novel applications. We show the benefits of the conversion model by designing a vein localization and visualization application that traditionally uses hyperspectral images. Our application uses only RGB images and produces accurate results. Vein visualization is important for point-of-care medical applications. We collected hyperspectral data to validate the proposed conversion model. Experimental results demonstrate that the proposed method is promising and can bring some of the benefits of expensive hyperspectral cameras to the low-cost and pervasive RGB cameras, enabling many new applications and enhancing the performance of others. We also evaluate the vein visualization application and show its accuracy.
+* N. Sharma, M. Hefeeda, [https://www2.cs.sfu.ca/~mhefeeda/Papers/mmsys20_vein.pdf Hyperspectral Reconstruction from RGB Images for Vein Visualization], In Proc. of ACM Multimedia Systems Conference (MMSys'20), Istanbul, Turkey, June 2020. (Artifacts Evaluated and Functional)
-[[Hyperspectral Reconstruction from RGB Images for Vein Visualization|More info [login required]]] ...
+* M. Arab, K. Calagari, M. Hefeeda, [https://www2.cs.sfu.ca/~mhefeeda/Papers/mm19_bandSelection.pdf Band and Quality Selection for Efficient Transmission of Hyperspectral Images], In Proc. of ACM Conference on Multimedia (MM'19),  Nice, France. October 2019.