THE INTEGRATION OF C OMPUTATIONAL METHODS AND NONLINEAR MULTIPHOTO N MULTIMODAL MICROSCOP Y IMAGING FOR THE ANALYSIS OF UNSTAINED HUMAN AND ANIMAL TISSUES By Gabrielle Alyse Murashova A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Chemistry Ð Doctor of Philosophy Computational Mathematics , Science and Engineering ÑDual Major 2019 ABSTRACT THE INTEGRATION OF C OMPUTATIONAL METHODS AND NONLINEAR MULTI PHOTON MULTIMODAL MICROSCOP Y IMAGING FOR THE AN ALYSIS OF UNSTAINED HUMAN AND ANIMAL TISSUES By Gabrielle Alyse Murashova Nonlinear multiphoton multimodal microscopy (NMMM) used in bio logical imaging is a technique that explores the combinatorial use of different multiphoton signals, or modalities, to achieve contrast in stained and unstained biological tissues. NMMM is a nonlinear laser -matter interaction (LMI), which utilizes multiple photons at once (multiphoton processes, MP). The statistical probability of multiple photons arriving at a focal point at the same time is dependent on the two -photon absorption (TPA) cross -section of the molecule being studied and is incredibly difficult to satisfy using typical incoherent light, say from a light bulb [1] . Therefore, the stimulated emission of coherent photons [2] by pulsed lasers are used for NMMM applications in biomedical imaging and diagnos tics. In this dissertation, I hypothesized that due to the near -IR wavelength of the Ytterbium(Yb) -fiber laser (1070 nm), the four MP Ñ two -photon excited fluorescence (2PEF), second harmonic generation (SHG), three -photon excited fluorescence (3PEF) and third harmonic generation (THG) , generated by focusing this ultrafast laser, will provide contrast to unstained tissues sufficient for a ugmenting current histological staining methods used in disease diagnostics. Additionally, I hypothesized that these NMMM images (NMMMIs) can benefit from computational methods to accurately separate their overlapping endogenous MP signals, as well as trai n a neural network for image classification to detect neoplastic, inflammatory, and healthy regions in the human oral mucosa. Chapter II of this dissertation explores the use of NMMM to study the effects of storage on donated red blood cells (RBCs) using non-invasive 2PEF and THG without breaching the blood storage bag [3] . Unlike the lack of RBC fluorescence previously reported [4Ð7], we show that with two -photon ( 2P) excitation from an 800 nm source, and three -photon (3P) excitation from a 1060 nm source, there was sufficient fluorescent signal from hemoglobin as well as other endogenous fluorophores . Chapter III employs NMMM to establish the endogenous MP signals present in healthy excised and unstained mouse and Cynomolgus monkey retinas using 2PEF, 3PEF, SHG, and THG. We show t he first epi -direction detected cross -section and depth -resolved images of unstained isolated retinas obtained using NMMM with an ultrafast fiber laser centered at 1070 nm and a ~38 fs pulse [8] . Two spectrally and temporally distinct regions were shown ; one from the nerve fiber layer (NFL) to the inner receptor layer (IRL), and one from the retinal pigmented epithelium (RPE) and choroid. Chapter IV focuses on the use of minimal NMMM signals from a 1070 nm Yb -fiber laser to match and augment H&E -like contrast in human oral squamous cell carcinoma (OSCC) biopsies. In addition to performing depth -resolved (DR) imaging directly from the paraffin block and matching H&E -like contrast, we showed how the combination of characteristic inflammatory 2PEF signals undetectable in H&E stained tissues and SHG signals from stromal collagen can be used to analytical distinguish healthy, mild and severe inflammatory, and neoplastic regions and determine neoplastic margins in a three -dimensional (3D) manner . Chapter V focuses on the use of computational methods to solve an inverse problem of the overlapping endogenous fluorescen t and harmonic signals within mouse retinas. The least -squares fitting algorithm was most effective at accurately assigning photons from the NMMMIs to their source . This work, unlike commercial software, permits using custom signal source reference spectra from endogenous molecules , not from fluorescen t tags and stains . Finally, Chapter VI explores the use of the OSCC images to train a neural network image classifier to achieve the overall goal of classifying the NMMMIs into three categories Ñhealthy, inflammatory, and neoplastic. This work determined th at even with a small dataset (<215 images), the features present in NMMMIs in combination with tiling, transfer learning can train an image classifier to classify healthy, inflammatory, and neoplastic OSCC regions with 70% accuracy. My research successful ly shows the potential of using NMMM in tandem with computational methods to augment current diagnostic protocols used by the health care system with the potential to improve patient outcomes as well as decrease pathology departmental costs. These results should facilitate the continued study and development of NMMM so that in the future, NMMM can be used for clinical applications. iv ÒThe brick walls are there for a reason. The brick walls are not there to keep us out. The brick walls are there to give us a chance to show how badly we want something. Because the brick walls are there to stop the people who donÕt want it badly enough. TheyÕre there to stop the other people.Ó ! Randy Pausch, The Last Lectu re v ACKNOWLEDGMENTS I would first like to extend my greatest gratitude to the various professors : Dr, Dana Spence, Dr. Dirk Colbry, Dr. Marcos Dantus, and Dr. Zhen Qiu, within the Departments of Chemistry , Computational Mathematics, Science and Engineering , as well as The Department of Biomedical Engineering, who have advised and mentored me throughout my time here at Michigan State University. Without your guidance, I would not have been able to complete this work. Secondly, I would like to than k my advisor (Dr. Dana Spence) and fellow committee members (Dr. Gary Blanchard, Dr. Johnathan McCracken, Dr. David Weliky, Dr. Dirk Colbry, and Dr. Yang Yang) for your continued support, particularly throughout this past year. Next, I would like to thank my former group members : Dr. Ilyas Saytashev, Dr. Muath Nairat, Dr. Nagitha Ekanayake, Dr. Christopher Mancuso, Dr. Gennady Rasskazov, Dr. Anton Ryabtsev, Dr. Vadim Lozovoy, Jurick Lahiri, and Matthew Michie , for your experimental advice, kindness , and friendship. I would also like to thank my former advisor (Dr. Marcos Dantus) for the valuable life and scholarly lessons. Furthermore, I would like to thank my family for your endless support throughout this journey. Thank you, mom, for never failing to as k me if I have eaten because I looked thinner and for always letting me take up space in your home, rent free. You have always supported me, never doubted me, and always told me to follow my dreams. You and dad always stood by me, and you continuously told me how proud you were of me. If dad were here to see me now, I know he would be very proud. I want to thank my amazing husband, Misha. Thank you for keeping me going, for grounding me, for driving me to the airport and back at 3 a.m., for checking up on me throughout the day to make sure I remembered to eat and drink water, and more than anything, thank you for your unconditional and unsurpassed love and support. I could not have done this without you. To my fellow cohort and friends within both departmen ts, thank you for the countless jokes, memes, and shared grievances about the non -trivial task of juggling course work, research, and teaching. vi Special thanks go to Bri and Brit; You both helped and provided me with the self -care, laughter, and reality che cks that I needed . Last, but certainly not least, in addition to my human family and friends, IÕd like to thank my two cats, Leo and Mittens (their cameo appearance in this dissertation can be seen in Chapter 6, Figure 51 ). Additionally, I would like to t hank my car (2005 Chevy Cobalt), even though you are an absolute rust bucket with 260k miles, you have never failed to get me through my 80 -150 mile round -trip to campus and back. I could go on forever, but I hope that all of you know the impact you have m ade on me throughout my life and my time at Michigan State University. I am very thankful for having you all in my life. vii PREFACE Medical diagnostic techniques and resources have improved exponentially over the past century, from the discovery of the X-ray by Wilhelm Conrad Roentgen in 1895, and the first use of the X -ray for medical imaging a year later by doctors in New Zealand, to the most recent advances in Magnetic Resonance Imaging (MRI) that can provide multi -contrast images from a single acquis ition [9,10] . On a tangential path, the development of digital computation and the methods thereof have also grown trem endously, from the development of the first mathematical proofs behind the least squares method to solving matrix math, and the development of Neocognition and convolutional neural networks in 1979 and 1980, through applying predictive algorithms for facia l recognition and biomedical image reconstruction [11 Ð14]. These two seemingly independent fields have evolved into a highly intertwined cross discipline where the predictive algorithms used in artificial intelligence are used as a te chnique for reconstructing medical images to improve image quality, reduce image acquisition times, and even diagnose illnesses and medical abnormalities. Beginning with the history and science behind multiphoton processes and concluding with machine learn ing architecture, in this dissertation I will present the path of my doctoral research and how it began with a focus on biomedical imaging using ultrafast lasers and turned into a cross -over between imaging and using digital computational methods for inver se-problem solving and image classification to expand the limits of nonlinear multiphoton multimodal images. viii TABLE OF CONTENTS LIST OF TABLES ................................................................................................................................. xi LIST OF FIGURES ............................................................................................................................... xii KEY TO ABBREVIATIONS ................................................................................................................ xx Chapter I Introduction .............................................................................................................................. 1 1.1 !On the relevance of NMMM imaging in biomedical settings ............................................................ 2 1.2 !Imaging Modalities of NMMM ........................................................................................................ 4 1.2.1 !Harmonic Generation ................................................................................................................ 7 1.2.2 !Multiphoton Excited Fluorescence ........................................................................................... 11 1.2.3 !Dependency of nonlinear multiphoton multimodal signals on pulse duration ............................ 11 1.3 ! Limitations of NMMM of unstained tissues .................................................................................. 16 1.4 ! Computational History and Inverse Problems ................................................................................ 17 1.4.1 !Inverse problem and solutions .............................................................................................. 17 1.5 !Convolutional Neural Networks and Machine Learning ................................................................. 18 1.5.1 !Architecture ......................................................................................................................... 19 1.5.2 !Caveats of Machine Learning and Their Solutions ................................................................ 21 Chapter II Multiphoton excited hemoglobin fluorescence and third harmonic generation for non-invasive microscopy of stored blood ......................................................................................... 23 Abstract ............................................................................................................................................... 23 2.1 !Introduction ................................................................................................................................... 23 2.2 !Materials and Methods ................................................................................................................... 25 2.3 !Results ........................................................................................................................................... 28 2.3.1 !2PEF microscopy imaging of RBCs on a coverslip and through PVC bag using Ti:Sapphire laser .................................................................................................................................... 28 2.3.2 !THG microscopy imaging of RBCs on a coverslip and through PVC bag using Yb -fiber laser ........................................................................................................................................... 30 2.3.3 !The source of fluorescence in RBCs .................................................................................... 32 2.4 Conclusion s.................................................................................................................................... 37 Chapter III Multimodal nonlinear optical imaging of unstained retinas in the epi -direction with a sub -40 fs Yb-fiber laser ........................................................................................................................................ 39 Abstract .............................................................................................................................................. 39 3.1 !Introd uction ................................................................................................................................. 39 3.2 !Materials and methods .................................................................................................................. 41 3.2.1 !Signal source, acquisition, and processing ........................................................................... 41 3.2.2 !Preparation of fixed and unfixed thin sliced mouse retinas ................................................... 42 3.2.3 !Preparation of fixed flat mount monkey retina ..................................................................... 44 3.2.4 !Preparation of reference solutions ....................................................................................... 44 3.3 !Results ......................................................................................................................................... 45 3.4 !Discussion .................................................................................................................................... 51 3.5 Conclusions .................................................................................................................................. 54 Chapter IV Tetra -modal multiphoton microscopy: A non -invasive technique for augmented histopathological analysis of oral squamous cell carcinoma biopsies ...................................................... 56 ix Abstract ............................................................................................................................................... 56 4.1 !Introduction ................................................................................................................................... 56 4.2 !Materials and Methods ................................................................................................................... 59 4.2.1 !Human Tissue excision, processing, and sample preparation ................................................. 59 4.2.2 !Canine Tissue excision, processing, and sample preparation .................................................. 60 4.2.3 !Laser setup ........................................................................................................................... 61 4.2.4 !Imaging, detection, and signal processing ............................................................................. 62 4.2.5 !Data processing .................................................................................................................... 64 4.2.6 !Collagen orientation calculations .......................................................................................... 64 4.2.7 !Statistical analysis ................................................................................................................ 65 4.3 !Results and Discussion .................................................................................................................. 65 4.3.1 !Rapid NMMMI of unstained canine and human OSCC tissues mimic H&E contrast ................. ............................................................................................................................................. 65 4.3.2 !Additional histological contrast is achieved using time -correlated single photon counting to explore concealed multiphoton signals in human and canine OSCC biopsies ......................... 67 4.3.3 !Probing the inflammatory microenvironment of HOSCC using multiphoton excitation shows spectral signal distinction ...................................................................................................... 69 4.3.4 !Examination of collagen fibril orientation in HOSCC stroma shows distinct patterns in normal, neoplastic, and inflammatory regions of H OSCC biopsied tissues ......................................... 75 4.3.5 !H&E -like contrast achieved at depths imaged directly from the paraffin block ...................... 77 4.4 Conclusions .................................................................................................................................... 81 Chapter V Towards spectral unmixing of multiphoton multimodal images of unstained retinas using user -defined signal sources and inverse problem solving algorithms .............................................................. 83 Abstract ............................................................................................................................................... 83 5.1 Introduction .................................................................................................................................... 84 5.2 Materials and Methods ................................................................................................................... 86 5.2.1 Laser setup, detection, imaging, signal processing ................................................................ 86 5.2.2 Detection .............................................................................................................................. 87 5.2.3 Tissue ................................................................................................................................... 88 5.2.4 Preparation of reference solutions ......................................................................................... 88 5.2.5 Preparation of Reference Signal Matrices, NMMMIs, and Phantoms for LSqF algorithm ...... 89 5.3 Results and Discussion ................................................................................................................... 90 5.4 Conclusions .................................................................................................................................. 103 Chapter VI A supervised machine learning approach to diagnose human and canine oral squamous cell carcinoma fr om a very small dataset of nonlinear multiphoton multimodal microscopy images of unstained biopsies ................................................................................................................................ 106 Abstract ............................................................................................................................................ 106 6.1 Introduction ................................................................................................................................ 106 6.2 Materials and Methods ................................................................................................................ 111 6.2.1 Excitation source, Sample preparation, and image acquisition ........................................... 111 6.2.2 Ima ge pre -processing and class distributions ..................................................................... 112 6.2.3 Computational time ........................................................................................................... 114 6.3 Results ........................................................................................................................................ 116 6.3.1 Convolutional network training model .............................................................................. 116 6.3.2 Evaluation of a modelÕs performance and customization of training model parameters and hyperparameters on the Kaggle dataset ............................................................................ 118 6.3.3 Baseline results of Kaggle model architecture without with augmentation ......................... 123 6.3.4 Baseline results of original resolution 64x64 -pixel NMMMI using Kaggle model architecture without and with augmentation ........................................................................................ 125 x 6.3.5 Updating the Adam optimizer parameters to better suit the NMMMIs and implementing the reduce learning rate callback from Keras ......................................................................... 127 6.3.6 Dimension scaling study for original and resized NMMMIs for Kaggle image resolution equivalency ..................................................................................................................... 130 6.3. 7 Improving testing (validation) accuracy of NMMMI IC without increasing input data using transfer learning on the pre -trained Kaggle model ............................................................ 135 6.3.8 Following Keras documentation for the number of frozen and retrainable layers ...................... ....................................................................................................................................... 136 6.3.9 Determining the optimal Finesse chunk width for transfer learning on the NMMMI data se t ....................................................................................................................................... 138 6.4 Conclusion ................................................................................................................................. 141 Chapter VII Summary and Outlook ...................................................................................................... 143 7.1 Multiphoton excited hemoglobin fluorescence and third harmonic generation for non -invasive microscopy of stored blood ........................................................................................................ 143 7.2 Multimodal nonlinear optical imaging of unstained retinas in the epi -direction with a sub -40 fs Yb -fiber laser ................................................................................................................................... 144 7.3 Tetra -modal multiphoton microscopy: A non -invasive technique for augmented histopathological analysis of oral squamous cell carcinoma biopsies ...................................................................... 145 7.4 Towards spectral unmixing of multiphoton multimodal images of unstained retinas using user -defined signal sources and inverse problem solving algorithms ................................................... 146 7.5 A supervised machine learning approach to diagnose hum an and canine oral squamous cell carcinoma from a very small dataset of nonlinear multiphoton multimodal microscopy images of unstained biopsies ...................................................................................................................... 147 APPENDICES ..................................................................................................................................... 149 APPENDIX -I Extra information on the fits from Chapter III ............................................................ 150 APPENDIX -II A2E solution meas urements from Chapter III ............................................................ 151 APPENDIX -III Publication List ....................................................................................................... 152 BIBLIOGRAPHY ............................................................................................................................... 153 xi LIST OF TABLES Table 1. Fluorescence lifetime decays obtained from fitting one -photon excitation (355 nm) curves using single and double exponential models. ................................................................................................... 34 Table 2. Fluorescence lifetime decays obtained from fitting two -photon excitation (800 nm) curves using single and double exponential models. ................................................................................................... 34 Table 3. Pearson correlation coefficients of 2PEF spectra. ...................................................................... 35 Table 4. Lifetime fitting of data shown in Fig ure 28. The parenthetical values correspond to one standard deviation. R 2 is the coefficient of determination, a dimensionless quantity. ............................................. 49 Table 5. Fitting with fixed parameters for lifetime decays shown in Figures 28(a) and 28(b). The parenthetical valu es are the one standard deviation errors and have units of ns. R 2 is the coefficient of determination (R 2 error) and is a dimensionless quantity. ....................................................................... 53 Table 6. Error values from the output of the four spectral un -mixing solution algorithms. .................... 102 Table 7. I mageDataGenerator augmentation parameters and values applied to the training images. ..... 124 Table 8. Adam optimizer default and new parameter values used for the NMMMI model .................... 128 Table 9. NMMMI image sizes and corresponding training and testing dataset sizes for the dimensions scaling study ........................................................................................................................................ 130 Table 10. New dimensions of the input and output NMMMIs, the total NMMMIs and the corresponding training and testing allocations for the new dimensions are also shown .. .............................................. 132 Table 11. Total, trainable, and non-trainable parameters with respect to number of frozen layers for transfer learning the Kaggle model on the scaled 64x64 -pixel NMMMIs. ......................................................... 139 xii LIST OF FIGURES Figure 1. Diagnostic sample collection and preparation process for pathology. a) typical sites of suspicion in the oral cavity. b) dehydrated excised biopsies. c) cassette preparation for biopsy mounting. d) mounting of biopsy in paraffin wax. e) microtome slicing of FFPE tissues for slide preparation ............................... 3 Figure 2. Typical laser scanning micros copy setup for nonlinear multiphoton multimodal microscopy. .... 5 Figure 3. Effect of index -matching fluid on multiphoton signals ............................................................... 6 Figure 4. Comparison of multiphoton processes detected from biological tissues and the corresponding transmission curves of select optical filters used to isolate said multiphoton signals. ................................ 7 Figure 5. Potential energy diagrams of 2PEF, 3PEF, SHG, and THG. ...................................................... 8 Figure 6.The effect of a strong electric field on the polarization of a centrosymmetric material (top left) and a noncen trosymmetric material (top right) and the resulting frequencies (bottom), when the waveforms are Fourier transformed. ................................................................................................................................ 9 Figure 7.Time scales of molecular processes. Adapted from Figure 2 from Brinks, et.al . [15] ................ 10 Figure 8. Figure obtained from CLEO2016, SC352: Introduction to ultrafast pulse shaping Ñprinciples and applications. First -, second -, and third -order dispersion in the spectral phase domain (top and their effect on the frequency domain (middle) and t he time domain (bottom) of the laser pulse. .............................. 14 Figure 9. The process of measuring and characterizing, the unknown from an incoming laser pulse and correcting it to a TLim pulse. Figure obtained from Lozovoy, V.V, et al. [16] . ............................... 15 Figure 10. Emission spectra of common sources of endogenous fluorescence in biological tissues . ........ 16 Figure 11. General workflow for the two types of machine learning, supervised and unsupervised. ........ 18 Figure 12. Architecture of a typical neural network . ............................................................................... 19 Figure 13. The processes taking place in active nodes of a neural network. ............................................ 20 Figure 14. Comparison of two functions , the sigmoid activation function and step function . ................... 21 Figure 15. General overview of the transfer learning topic . .................................................................... 21 Figure 16. Schematic diagram of the microscopy setup for multi -photon imaging using different lasers. Ti:Sapphire or Yb -fiber laser osc illators can be used one at a time. ........................................................ 26 Figure 17. 2PEF image of unstained human RBCs on a coverslip imaged by 15fs pulses with 10mW average power from the Ti:Sapphire laser tuned to 800 nm. Scale bar is 20 " m. ................................................... 29 Figure 18. 2PEF image of RBCs detected thr ough the PVC storage bag in the epi direction obtained with 10mW average power 15fs pulses from the Ti:Sapphire laser tuned to 800 nm. Scale bar is 20 " m. Video of flowing RBCs ( Visualization 1 ). ............................................................................................................ 29 ''() !" xiii Figure 19. THG microscopy imaging of human RBCs on the glass cover slip obtained using a 1060 nm Yb -fiber laser emitting 45fs pulses with 8mW average power. a) Static image; b) Video of flowing RBCs (Visualization 2 ). Scale bar is 20 " m. ...................................................................................................... 31 Figure 20. THG images of RBCs detected through the PVC storage bag in trans direction (a) and in epi direction (b) excited by 7mW average powe r from an Yb -fiber laser. Scale bar is 10 " m. ........................ 32 Figure 21. (a) One -photon excitation (355nm, 12 ps) fluorescence decay curves of hemoglobin, NADH, biliverdin, and riboflavin compared to (b) 2PEF (800nm, 15fs) decay curves for RBCs, their membranes and rea gent -grade hemoglobin, biliverdin, bilirubin, riboflavin, and NADH. .......................................... 34 Figure 22. Hemoglobin fluorescence signal versus average excitation power from Ti:Sapphire laser (14 fs pulse duration FWHM) plotted in a logarithmic scale. The experimental points are fit by a linear function with slope equal to 1.89 ± 0.03, which is consistent with two -photon excited fluorescence. .................... 35 Figure 23. (a) 2PEF (800nm, 15 fs) emission spectra for RBCs, erythrocyte ghosts and reagent - grade fluorophores. (b) 2PEF peak wavelen gth vs decay lifetimes for RBCs, their membranes and reagent -grade hemoglobin, biliverdin, bilirubin, and NADH. ....................................................................................... 36 Figure 24. a) Transient absorption measurements of PBS, hemoglobin solution and purified RBCs following 1040nm pump and 735nm probe. b) Absorption spectra of PBS solution with ghosts washed 1 to 4 times. Inset shows absorbance of ghosts after varying number of washes, probed at 414nm. Spectra were corrected for background and Rayleigh scattering. ................................................................................................ 37 Figure 25. The experimental apparatus c onsisted of an ultrafast fiber laser operating at 1.07 µm, a MIIPS pulse shaper, a laser scanning inverted microscope, and a photon detector, mounted in the epi -direction. The TCSPC was used to collect fluorescence spectra and lifetime, whereas the single PMT was used to acquire the depth resolved images. ......................................................................................................... 45 Figure 26. Three colored (red, green, blue) composite multimodal images of the retinal layers from a 7 " m slice of a mouse retina taken with the TCSPC at 6.9 mW of power depths using a 1.07 " m Yb -fiber laser with 35.0 fs pulse durations. Image acquisition was done at 30 -second intervals for a total of 4.5 minutes. (a) Here the blue, green, and red channels represent emission centered at 535 nm, 575 nm, and 629 nm, respectively. (b) The blue, green, and red channels represent emission centered at 355 nm, 535 nm, and 629 nm, respectively. The bandwidth of each channel is ~37.5 nm. ............................................................... 46 Figure 27. Spectral emission from 480 to 680 nm detected from a mouse retina. a) The non - normalized emission spectra revea l that the receptor layers (ORL and IRL) have the strongest fluorescence emission. b) The normalized spectra more readily compare the different spectral shapes across retinal layers. Layers from the NFL thru the OPL all have nearly identical spectral shape s. The sclera has a unique spectral shape, where the peak at 535 nm is attributed to SHG (see Discussion). ............................................................ 47 Figure 28. Fluorescence lifetimes across the retinal layers from a mouse over a spectral range of a) 556 -594 nm, b) 610 -648 nm, and c) short: 556 -594 nm and long: 610 -648 nm. The plots reveal that for a given spectral band, all the layers from the NFL through the ORL have nearly identical lifetimes, whereas the choroid and RPE have nearly identical lifetimes. Additionally, the lifetimes in the chor oid and RPE become shorter for longer detection wavelengths (i.e. from panel a to panel b). This trend is shown directly in panel c, where the lifetimes for the choroid, RPE, IRL, and IPL are fitted for both the short wavelength range and the long waveleng th range. Also in panel c is the lifetime measured at 535 nm from the sclera. SHG from collagen in the sclera is the source of this emission and coincides with the IRF of our system. ............... 48 xiv Figure 29. Depth -resolved imaging of an unstained, fixed, Cynomolgus monkey retina flat mount. The total extent of the Cynomolgus monkey retina was 220 µm. The depth of each image is indicated in yellow. The scale bar is 15 µm and can be seen in the NFL panel. Each 2D image was an average of 5 scans, averaged for 5 seconds. Each layer, beginning with the NFL and ending with the ORL and RPE retinosomes, from left to right, was an average of 182 images (9.1 " m), 190 images (9.5 " m), 182 images (9.1 " m), 139 images (7.0 " m), 51 images (2.63 " m), 233 images (11.7 " m), 211 images (10.6 " m), and 33 images (1.7 " m), respectively. The depth resolved stack was obtained with 7 mW of average power at all depths using a 1.07 "m Yb -fiber laser with 34.8 fs pulse durations. In the IRL, inner segments of cone photoreceptors (large white fea tures) and rod photoreceptor inner segments (smaller round features between the cone inner segments) are easily distinguishable. We believe that the bright particles in the bottom right panel are perhaps melanin or RPE retinosomes attached to the tips of p hotoreceptors. At each depth, the characteristic morphology of the retina layers is clear, indicating that the Yb -fiber laser is effective at achieving the cellular resolution needed for depth resolved imaging. Video of depth resolved imaging of retina use d to obtain the images presented in this figure (Visualization 3). ................................................................... 50 Figure 3 0. Excised human oral cancer biopsy tissue sample preparation. a) Human oral cavity with typical sites of OSCC. b) Freshly excised biopsy tissue after OCT clearing and dehydrat ion. c) Paraffin embedding of the dehydrated biopsy tissue. d) Set and treated biopsy tissue paraffin blocks are sliced at 4 µm thickness. One slice is left unstained for NMMMI and the other is stained with H&E . ............................................ 60 Figure 3 1. Schematic of the beam path an d optical setup for NMMMI. An Yb -fiber oscillator excitation beam path leading into the MIIPS pulse characterizing system, and then into the Nikon TE2000 inverted microscope, where the paraffin block and tissue slides are imaged using NMMMI. ................................ 61 Figure 3 2. Optical table and NMMMI detection setup. a) 3D animated optical table arrangement of the NMMMI experimental setup. Photons are detected using two methods, a single photo -multiplier tube (PMT) or redirected via flip mirror for detection by, the time -correl ated single photon counter (TCSPC). b) Simplified diagram to show how epi -direction photons are coupled into the TCSPC optical fiber bundle. c) Schematic of how photons are detected and recorded by the TCSPC software to build up the waveform data. ...................................................................................................................................................... 62 Figu re 3 3. H&E and NMMMIs of normal and neoplastic regions of HOSCC biopsies showing comparative contrast to H&E staining achieved with two optical filters. NMMMIs of the unstained normal (a) and neoplastic (b) HOSCC biopsy tissue slices, corresponding to the H&E image, are presented as individual channels (green and blue), containing photons from their respective optical filter bandpass ranges, 515 nm -540 nm (SHG) and 605 nm -785 nm (2PEF), respectively, as well as an overlay (SHG/2PEF). Zoomed in regions of th e white, dashed boxes in the 2PEF images show a zoomed -in image of where we identify the nuclei, pointed out by red arrows. H&E and NMMMI scale bar = 50 " m, gray -scale image scale bar = 10 "m. ........................................................................................................................................................ 66 Figure 3 4. NMMMI of human and canine OSCC biopsies acquired using TCSPC. Brightfield images and single -channel NMMMIs of the corresponding H&E -stained tissue slices from normal and neoplastic OSCC biopsies from human (a,b) and canine (c,d). Each row contains the H&E image of where each NMMMI was obtained, a split -image of the H&E and RGB overlay NMMMI showing the histological accuracy achieved with multiphoton multimodal (MM) excited imaging, and the individual channels of the NMMMIs for each signal type. Wavelength ranges for SHG is centered at 535 nm, for the red channel 2PEF, 550 nm - 580 nm, and for the blue 2PEF channel, 610 nm -660 nm. White scale bar = 50 µm. ....... 68 Figure 3 5. NMMMIs of excised unstained and H&E stained human oral biopsy samples. NMMMI images of normal squamous epithelium and connective tissue of the stroma (a), mild inflammation (b), severe xv inflammation (c), and neoplastic region (d) with their corresponding H&E brightfield images. White arrows show collagen of the stroma. Pink arrows represent the highly fluorescent 3PEF cells of (b). Yellow ar rows show the 2PEF signals from inflammatory cells. The white box in the RGB image of (b) shows where the 3PEF and 2PEF cells are present and are independent of one another. White and black scale bars represent 100 µm. ................................................................................................................................................. 70 Figure 3 6. Histogram plots of 6 R OIs from the NMMMIs (a) normal, (b) MI, (c) SI, and (d) neoplastic regions in each pseudo -colored channel of Figure 37. ............................................................................ 73 Figure 3 7. Collagen orientation in normal, MI, SI, and neoplastic regions of HOSCC biopsied tissues. a) CCD images of brightfield H& E stained slices of normal, MI, SI, and neoplastic regions of HOSCC biopsied tissues. Colored boxes represent ROIs where NMMMI was obtained from, n ROI =3 for each condition. b) NMMMI of unstained HOSCC biopsy tissues (515 nm -535 nm). Scale bar = 50 µm. c) Collagen orientation polar plots for each tissue classification. The colors represent the appropriate ROIs from (a). ................................................................................................................................................ 75 Figure 38. Representation of 2D images at different depths of a 3D depth -resolved stack from a human tumor embedded in a paraffin block. A sampling of images at a range of depths are shown. The rows of panel images are shown at depths ranges with the increasing dep th; 1 -12.9 µm, 24.8 -27.4 µm, 44.6 -47.8 µm, 94.5 -98.8 µm, and 121.8 -135.5 µm. The wavelength ranges corresponding to the pseudo -colors used shown at the base of each column in the panel arrangement. The magenta images contain photons from all multiphoton si gnals detected. 3PEF and 2PEF images show the fluorescence from inflammatory response cells. The green channel (SHG) shows strong signals from collagen. Scale bar = 100 µm. ...................... 77 Figure 39. Volumetric rendering of depth -resolved NMMMI of HOSCC biopsy obtaine d directly from the paraffin block. a) (top) Amira 3D rendering of the 2D image stacks of the human OSCC tumor as a YZ -plane flat face, Normalized intensity plots for each MM signal are plotted along the y -axis of (a). The color of each plot corresponds to the pseudo -colored channel for each signal type. (bottom) angled YZ -plane to show the joint of two different YZ -planes. Yellow dashed lines show where the hyalinization can be identified indirectly. White scale bars indicate 50 µm. b) Collagen orientatio n angle polar plot for the depth -resolved images in the green channel of Figure 40b. .............................................................................. 79 Figure 4 0. Wavelength spectra of pure fluorophores and harmonophores found within human retinas. Solutions of pure fluorophores are excited with the Yb -fiber laser at the focal point of the microscope objective. Photons are detected using the TCSPC, where each PMT detector collects wavelength data over a 12.5 nm range, These signals are reconstructed into 2D array using a Python script. Each spectra is normalized from 0 to 1 to show the relative overlap and distance between neighboring peaks. Notice that many of the compounds overlap significantly, making difficult to determine their individual contributions. .............................................................................................................................................................. 84 Figure 4 1. a) Yb -fiber oscillator and beam path. b) Nikon TE2000 inverted microscope and single PMT. c) Time -correlated single photon counter (TCSPC). Showing the orientation of the retina slice (green circle). The laser beam propagates perpendicular to the retina slice. ................................................................... 88 Figure 4 2. Single PMT NMMMIs from the TCSPC. Each image contains photons over a 12.5 nm wavelength range. Photons within the top left image are centered at 302 nm. The wavelength ranges for each image increase towards the right and down, where the final image of the panel (bottom right) has photons centered at 683 nm. Images are 512x512 -pixels (~320x320 -m) in size. White scale bars are 50 µm......................................................................................................................................................... 91 xvi Figure 4 3. NMMMIs with distinct fluorophore signals characteristic to healthy retinas. The wavelength range covered in each image is shown above each tile. Images are 512x512 -pixels (~320x320 -m) in size. Note the distinct formations identified in each image by their characteristic fluorescent and harmonic signals. White scale bars are 50 µm....................................................................................................... 92 Figure 4 4. Three colored (red, green, blue) composite m ultimodal images of the retinal layers from a 7 " m slice of a mouse retina taken with the TCSPC at 6.9 mW of power depths using a 1.07 " m Yb -fiber laser with 35.0 fs pulse durations. Image acquisition was done at 30 -second intervals for a total of 4.5 minu tes. (a) Here the blue, green, and red channels represent emission centered at 535 nm, 575 nm, and 629 nm, respectively. (b) The blue, green, and red channels represent emission centered at 355 nm, 535 nm, and 629 nm, respectively. The bandwidth of each ch annel is ~37.5 nm. ............................................................... 94 Figure 4 5. (top) Normalized emission curves of the pure fluorophore and harmonic standards obtained from the TCSPC. (bottom) Normalized fluorescence and harmonic emission spectra acquired from the MP signals detected within excised r etinas using the TCSPC. ....................................................................... 95 Figure 4 6. Generated image matrices to act as virtual phantoms. Each matrix is 512x512 -pixels to match the NMMMIs. The contributing signals within each phantom are shown above each image. In the case of the single -source phantom s, only one type of fluorescent or harmonic signal occupies the ROI, noted by yellow pixels. The phantoms are normalized on a scale of 0 to 1, where a value of 1 is represented by yellow coloring. The heterogenous phantom contains fluorescent and harmonic signals from all of the sources, organized into vertical ROIs. White scale bars are 50 µm. ...................................................................... 96 Figure 4 7. Images showing the output and the error in the photon assignments for each type of input phantom. The top row shows the input phantoms that were fed into the LSqF algorithm. The second row, titled ÒOutput ImagesÓ shows the resulting image from after the LSqF algorithm, we see that in the case of single -source phantoms, most of the signals are properly assigned, i.e. most of the pixels in each ROI are equal to 1 (yellow). The bottom row presents the error, or photons that the LSqF algorithm incorrectly ass igned. Here we show that in all cases, photons were incorrectly assigned to the FAD source, where the FAD photons were all incorrectly assigned to the NADH source. White scale bars are 50 µm. ............... 97 Figure 48. Output images of the LSqF algorithm with the FNMA 1A10SE virtual phantom image (Figure 48) as input. The top and bottom rows show the output of the LSqF algorithm for the heterogenous source phantom, where each image shows the photons assigned to a single source, identified by the source name above each i mage. FAD was incorrectly assigned as the source for A2E 1mM and 10mM as well as Melanin photons, and all of the FAD photons were incorrectly sourced to NADH. White scale bars are 50 µm. ... 98 Figure 49. Results of applying the single -vector LsqF algorithm o n a NMMMI of unstained excised healthy retina. Each image contains photons corresponding to a single fluorescent or harmonic source. The signal source corresponding to each image is shown in the title above each output tile. Additionally, we present the er ror for each signal source in photons. Each image contains a total of nearly ~15.36x10 3 photons, therefore, the number of incorrectly assigned photons is less than 3% of the total photons observed. We note the accurate assignment of the SHG photons to the sclera which is comprised of collagen, as well as the main contribution of Melanin to be in the IRL, where the majority of the melanin disks are present, and the assignment of A2E (in both concentrations) to the ORL, where the catabolites of the visual cyc leÑbreakdown of melanin into A2E, occurs. White scale bars are 50 µm. .................................................. 100 Figure 5 0. Compressed sensing image solutions from each of the four algorithms. Each column of images is the output from one algorithm, the rows of the figure correspond to which fl uorescent or harmonic signal xvii sourceÕs photons are contained within that image. All images are 512x512 -pixels (~320x320 µm) and were acquired using the TCSPC. White scale bars are 50 µm. ...................................................................... 101 Figure 5 1. Input image (left) and the resulting augmented images (3x3 panel on right), from using the Keras ImageDataGenerator. The operations performed on each image are identical to those used in this work and are shown in Table 7 [17,18] . ............................................................................................................... 110 Figure 5 2. Schematic showing the process of tiling the large -scale (x,y -dimensions > 1000 µm) NMMMIs into multiple 64x64 -pixel (~40x40 µm) images to increase the size of the NMMMI dataset. White scale bar on large image is 200 µm. White scale bar on tiled image is 32 µm. ..................................................... 112 Figure 5 3. Class distributions for the original (a) and rescaled (b) NMMMIs. The color of each bar plot represents each of the three classes, where red is for the Cancer class, orange is for Inflammation, and the yellow bars are for the Healthy class. Each plot has the class size distributions for each of the image dimension sizes used throughout this body of work, organized within groups based on the image dimensions along the x -axis. Annotated black brackets, surrounding each of the bar plots within the image dimension groups, shows the sum, or total number of tiled images within each group. .......................................... 114 Figure 5 4. Typical run times for training the models on the Kaggle (left), original resolution NMMMIs (middle), and new resolution NMMMIs (right), for the different dataset sizes. Training times are represented in seconds form. Each color represents specific image dimensions. Diagonal patterned bar plots show for which image dimensions training was not performed. Note the decrease in training time neces sary for the NMMMIs that are rescaled to match the resolution of the Kaggle images. ................................. 115 Figure 5 5. Typical training times for performing TL with a specified number of FL for the original and new resolution NMMMIs. The training times for the Keras recommen ded number of FL for the original and new resolution NMMMIs are shows by the solid red bar plot and the white bar plot with a red border, respectively. These times were obtained by training on 64x64 -pixel images. Times are represented in seconds form. We n ote the relationship between training time and the number of FL, as the number of FL increases, training time decreases. ........................................................................................................ 116 Figure 5 6. Model parameters and architecture with 14 trainable layers and the appropriate softmax layer to reflect the accurate nu mber of classes for the images. ........................................................................... 117 Figure 5 7. Characteristics of learning curves that provide valuable information regarding a modelÕs performance and about the datasets. From left to right and top to bottom: (top) learning curves showing a good fit, und erfitting, underfitting, (bottom) overfitting, unrepresentative training dataset, high variance in training set, and high bias. ................................................................................................................... 119 Figure 58. Training and testing (validation) accuracy plots for determining the ideal combination of optimizer algorithm a nd learning rate for the Kaggle dataset. ............................................................... 122 Figure 59. Sample of 25 non -augmented, labeled, and normalized input images from the Kaggle dataset which were fed into the ML model. Each image is 64x64 -pixels or ~32x32 µm in size, with a resolution of 0.49 µm/pixel. White scale bars are 10 µm. .......................................................................................... 123 Figure 6 0. Comparison between non -augmenting (left) and augmenting (right) the Kaggle images prior to being fed into the model. Training accuracy and loss curves are green and blue, respectively. Testing (validatio n) accuracy and loss are red and orange, respectively. Steady -state training and testing (validation) xviii accuracy and loss values are shown above all curves. Inset plots of sample images from each experiment are presented within each of the two plots. White s cale bars are 10 µm. ............................................... 124 Figure 6 1. Twenty -five sample images of the NMMMIs that are used in the image classifier model. Each image is 64x64 -pixels, or ~40x40 µm in size, with 0.624 µm/pixel resolution, and normalized on a scale of 0 to 1. The class labels for each image are along the bottom of each image, where the one -hot encoded labels are along the y -axis of each image. White scale bars are 10 µm. ................................................. 125 Figure 6 2. Comparison between non -augmenting (left) and augmenting (right) the NMMMIs prior to being fed into the un -trained model. Training accuracy and loss curves are green and blue, respectively. Testing (validation) accuracy and loss are red and orange, respectively. Steady -state training and testing (validation) accuracy and loss values are shown above all curves. Inset plots of sample images from each experiment are presented within each of the two plots. Key observation in this figure is that the loss curve is growing quickly i n the non -augmented data while the loss is held under control with the augmented data. White scale bars are 10 µm. ........................................................................................................................... 126 Figure 6 3. Accuracy results from the model trained on the NMMMI data with adjusted Adam optimizer parameter settings and implementing th e Keras ReduceLROnPlateau callback. Testing (validation) accuracy and loss are red and orange, respectively. Steady -state training and testing (validation) accuracy and loss values are shown above all curves. Inset plots of sample images from each experime nt are presented within each of the two plots. White scale bars are 10 µm...................................................................... 129 Figure 6 4. (left) 85x85 -pixel (~53x53 µm), (middle) 128x128 -pixel (~80x80 µm), and (right) 256x256 -pixel (~160x160 µm) tiled normalized and labeled NMMMIs. Image class label is shown on the x -axis of each image whereas the one -hot encoded label corresponding to the class label is on the y -axis of each image. White scale bars are 10 µm. ...................................................................................................... 130 Figure 6 5. Accuracy curves for dimension study of NMMMIs. The models trained and tested ( validated) on 64x64 -pixel, 128x128 -pixel, and 256x256 -pixel training NMMMIs are shown from left to right, respectively. Testing (validation) accuracy and loss are red and orange, respectively. Steady -state training and testing (validation) accuracy and los s values are shown above all curves. Inset plots of sample images from each experiment corresponding to each dimension used are presented within each of the two plots. White scale bars are 10 µm. ................................................................................................................. 131 Figure 6 6. Rescaled NMMMIs (left) 64x64 -pixel (~40x40 µm), (middle) 128x128 -pixel (~80x80 µm), and (right) 192x192 -pixel (~120x120 µm) tiled normalized and labeled NMMMIs. Image class label is shown on the x -axis of each image whereas the one -hot encoded label corresponding to the class label is on the y -axis of each image. White scale bars are 10 µm.................................................................................... 132 Figure 6 7. Accuracy curves for scaling study of rescaled NMMMIs. The models trained and tested (validated) on 64x64 -pixels, 128x128 -pixels, and 192x192 -pixels training NMMMIs are shown from left to right, respectively. Testing (validation) accuracy and loss are re d and orange, respectively. Steady -state training and testing (validation) accuracy and loss values are shown above all curves. Inset plots of sample images from each experiment corresponding to each dimension used are presented within each of the two plot s. White scale bars are 10 µm. ........................................................................................................ 133 Figure 68. Workflow schematic of transfer learning experiments with the pretrained Kaggle model on the rescaled 64x64 -pixel NMMMI dataset. The key in the top -right corner of the figure shows the meaning for the abbreviat ions used within the figure. The workflow is split into three parts, 1. The initial training of the Kaggle and NMMMI models, independently, 2. The FC study to determine the optimal ratio of FL to xix retrainable layers, and 3. Appending the optimized number of retrainable layers determined in phase II to the last layer of the full Kaggle model. ................................................................................................. 135 Figure 69. Transfer learning model parameters and architecture with 9 FL (non -retrainable) Kaggle layers, 4 TL, and an updated softmax layer to reflect the accurate number of classes for the NMMMIs. .......... 137 Figure 7 0. Steady -state accuracy values for recommended number of frozen layers for the models partially trained and then validated Ñusing a portion of the testing dataset, on both the (left) original resolution 64x64 pixel NMMMIs and the (right) Kaggle -matched resolution 64x64 pixel NMMMIs. Matching the resolution of the NMMMIs to the Kaggle images shows improves accuracy from 64.1% to 65.8%. ...................... 137 Figure 7 1. Steady -state accuracy values for the finesse chunk width stu dy on the Kaggle -matched resolution 64x64 pixel NMMMIs. Testing (Validation) accuracy improves with fewer frozen layers than what was recommended by Keras (Figure 72). .................................................................................................... 139 Figure 7 2. Selected accuracy curves for transfer learning of the Kaggle model on th e NMMMIs with 2(a), 4(b), 9(c), and 13(d) frozen layers. Steady -state (last 25 epochs) metric values for each of the experiments are in the textbox located in the top -right corner of each plot. Observe that as the number of frozen layers increase the varia bility in the results stabilizes and a general improvement on training and testing of the validation accuracy. ............................................................................................................................. 140 Figure 73 (Left). Lifetime decay fits and the corresponding residual plots for the RPE and choroid for the wavelength range of 556 -594 nm. T his figure shows the RPE and choroid fit poorly to a mono -exponential function. (Right). Lifetime decay fits and the corresponding residual plots for the ORL through NFL for the wavelength range of 610 -648 nm. This figure demonstrates that the ORL -NFL laye rs fit poorly to a tri -exponential function ............................................................................................................................ 150 Figure 74. The spectra (Left) and lifetime (Right) from a 1 mM solution of A2E. The spectra peaks near 625 nm, which is the same peak seen in the choroid, RPE, and receptor layers. The lifetime of ~173 ps agrees with the literature values [29]. ................................................................................................... 151 xx KEY TO ABBREVIATIONS LASER Light amplification by stimulated emission of radiation UV Ultra -Violet LMI Light -matter interactions TPA Two -photon absorption NMMM Nonlinear multiphoton multimodal microscopy H&E Hematoxylin and eosin FFPE Formalin -fixed paraffin embedded 3D Three -dimensional LSM Laser scanning microscopy NA Numerical aperture 2PEF Two -photon excited fluorescence 3PFE Three -photon excited fluorescence PMT Photo -multiplier tube SHG Second harmonic generation THG Third harmonic generation SONLS Second -order nonlinear susceptibility AMD Age -related macular degeneration IRF Instrument response function NADH Nicotinamide adenine dinucleotide FAD Flavin adenine dinucleotide A2E Di-retinoid -pyridinium -ethanolamine TPE Two -photon emission GDD Group -delay dispersion TLim Transform -limited xxi TOD Third -order dispersion MIIPS Multiphoton intrapulse interference phase scan SLM Spatial light modu lator Yb Ytterbium DAPI 4# ,6 -diamidino -2-phenylindole DNA Deoxyribonucleic acid STEM Science, technology, engineering, and mathematics ENIAC Electronic Numerical Integrator and Computer ML Machine learning SML Supervised machine learning USML Unsupervised machine learning NN Neural network TrL Transfer learning MNIST Modified National Institute of Standards and Technology VGG Visual Geometry Group RBC(s) Red blood cell(s) Ti:Sapphire Titanium Sapphire laser BBO Barium Borate TCSPC Time -correlated single photon counter/counting NIH National Institute of Health TA Transient absorption IACUC Institutional Animal Care and Use Committee of the Massachusetts General Hospital BIRB Biomedical and Health Institutional Review B oard PVC Polyvinyl chloride AS-1 Additive solution -1 PBS Phosphate buffered saline xxii Tris -HCl Hydroxymethyl aminomethane hydrochloride EDTA Ethylenediaminetetraacetic acid OCoT Optical coherence tomography ANSI American National Standards Instit ute 2D Two -dimensional ADC Analogue to digital converter PND Post -natal day OCT optimal cutting temperature RPE Retinal pigmented epithelium NFL Nerve fiber layer GCL Ganglion cell layer IPL Inner plexiform layer INL Inner nuclear layer OPL Outer plexiform layer ONL Outer nuclear layer IRL Inner receptor layer ORL Outer receptor layer OSCC Oral squamous cell carcinoma NMMMI(s) Nonlinear multiphoton multimodal microscopy imaging/images ROI Region of interest DR Depth -resolved 4D Four -dimensional PIL Python Imaging Library GIMP GNU Image Manipulation Program HOSCC Human oral squamous cell carcinoma xxiii ANOVA Analysis of variance MI Mild inflammatory SI Severe inflammatory MM Multiphoton multimodal RGB Red -green -blue MMP Matrix metalloproteinase IHC Immunohistochemical 2P Two -photon MP Multiphoton process 3P Three -photon LSqF Least -squares fit/fitting LASSO/lasso Least Absolute Shrinkage and Selection Operator AI Artificial Intelligence FL Frozen layers TL Trainable/retrainable layers HPCC High Performance Computing Center TrPs Trainable parameters NTrPs Non -trainable parameters ToTrPs Total trainable parameters FC Finesse chunk width MMI Multiphoton Multimodal Imaging/ Mul tiphoton Multimodal Imaging Model SNR Signal -to-noise Ratio RMSProp Root mean square propagation ToPrs Total parameters 1 Chapter I Introduction Multiphoton microscopy, once thought of as impossible concept, has more recently begun its migration into the biomedical imaging and diagnosis regime, owing its applicable uses to the development of the LASE R [1,19] . Interactions between light and matter are naturally occurring phenomena. These observable light and matter interactions are occurring all around us and are essential in our day -to-day lives. Whether it be the reflected photons that show a reflection in the mirror, the chemical isomerization reaction of 11-cis -retinal to all -trans -retinal in the retina , or the ability to sanitize the bacteria on surfaces by using ultra violet (UV) light , interactions between light an matter are extremely use ful and necessary [20 Ð22]. Most of these interactions are linear processes that involve incoherent light , hence the reason for the abundance in nature . This incoherence, or the random polarization of photons, is a result of spontaneous emission. Linear light -matter interactions (LMI) are not the only possible or useful LMIÕs, in fact, nonlinear interactions , utilizing multiple photons at once (multiphoton processes) , have sho wn over the last century how they can enhance and even revolutionize the field of biomedical imaging. Prior to the development of the laser Ñpreviously referred to in this dissertation as LASER, in 1960, experimental evidence of multiphoton processes, spec ifically two -photon absorption (TPA), was lacking. TPA, first theorized and predicted by Maria Gıppert -Mayer in 193 1 [1] , is the probability of two photons of incoming light being absorbed simultaneously. The TPA was calculated to be on the order of 10 -50 cm4 s photon -1 (1 G M), a very small cross -section that would require a light pulse with power density of ~1010W/cm 2, to achieve. This is a nearly unsurmountable amount of power density to obtain from an incoherent light source. Stimulated emission of coherent (a constant phase difference between the waves of neighboring photons [2] ) photons by lasers facilitated experimental evidence for nonlinear LMIs . This led to exploration of nonlinear optical effects in media that transformed optical applications and forged the way for nonlinear spectroscopy. Nonlinear multiphoton multimodal microscopy (NMMM) used in biological imaging is a technique that explores the combinatorial use of different multiphoton signals, or modalities, to achieve contrast in stained and unstained biological tissues. A focus of this dissertation is the methodical application of 2 NMMM imaging in unstained biological tissues to study the quantifiable properties that can be used to distinguish normal from abnormal tissues , in attempts to develop universal techniques to enhance current neoplastic ( cancer ) diagnostic procedures . Additionally, the work in the dissertation explores the use of computational methodologies to augment the findings from the NMMM images in the form of machine learning and inverse problem -solving techniques. 1.1 !On the relevance of NMMM imaging in biomedical settings Early diagnosis of abnormal, or potentially cancerous, tissues are pivotal to successful treatment and survival. If post -treatment neoplastic cells go undetected and reach the blood stream , there is a likelihood that a secondary tumor may develop from the migrating primary tumor cells . Determining if a tumor is benign or malignant can be a tedious process for the patient, physician, and pathologist . When a physician detects a suspicious area of tissue, a biopsy of that tissue region is taken and sent for further examination. Depending on the tumor, the physician resect s the diseased or suspicious lesion , small sections at a time, and send s each section for pathology staining and examination . In more prominent tumors, as much of the area as possible will be excised without putting the patient at risk, under the notion that it is better to take more tissue than less. In both scenarios, the excised tissues go through a series of specific fixation steps, such as dehydration, embedding, sectioning , i.e. formalin -fixed and paraffin embedded (FFPE), prior to antigen retrieval in order to preserve the natural structure and prevent degradation (Figure 10) [23] . 3 Figure 1. Diagnostic sample collection and preparation process for pathology . a) typical sites of suspicion in the oral cavity. b) dehydrated excised biopsies. c) cassette preparation for biopsy mounting. d) mounting of biopsy in paraffin wax. e) microtome slicing of FFPE tissues for slide preparation. Unstained micron -thick tissues resemble plastic wrap and lack the necessary contrast to distinguish neighboring cells from one another. Pathologists utilize various histological stains to add cell -, structure -, and molecular -specific contrast to biopsied tissues. One of the most well -known and commonly used combination stains is the hematoxylin and eosin (H&E) stain, where the basophilic hematoxylin attaches to acidic structures and stains them purple or blue, whereas the acidophilic eosin stains basic structures pink or red. This system of stains help d istinguish structural components of tissues such as collagen and cytoplasm from cellular components, such as nuclei [24] . In addition to the H&E stain, there are other staining methods that utilize fluorescent antibodies, which can provide increased contrast of proteins in the cells. Furthermore, the combination of more than one staining technique is often necessary to distinguish cell types within certain structures, such as distinguishing collagen from elastin in connective tissue using VerhoeffÕs stain which is a combination of 7 different stains [25] . The use of NMMM provides contrast to multiple cell and tissue types without histological staining. While this technique does not eliminate the costly process of preparing and staining biopsies for diagnosis purposes, the combination of NMMM with current techniques can augment the diagnostic process. 4 Rapid detection of cancerous tissue by non -invasive NMMM can promote timely and accurate diagnosis, increasing the likelihood of successful treatment and decreasing the amount of unnecessary or erroneous biopsy procedures on patients. NMMM provides visual information with sub -micrometer resolution, making it a great candidate for non -invasive biopsies . Over the nearly three -decades following the first demonstration of laser scanning two photon excited fluorescence ( 2PEF ) microscopy by Denk, et.al [26] , the interest in and advancement of nonlinear optical microscopy and NMMM imaging has increased exponentially. More importantly, NMMM imaging has moved from feasibility experiments to demonstrating the benefits of using multimodal contrast in imaging living tissues [27] and to the use of NMMM in clinical research centers as an increasingly valuable tool [28] . Sub -micrometer resolution, chemical specificity, and precise three -dimensional ( 3D) reconstruction of imaged volume at depths down to 1.2 mm [29] are becoming a necessity in tissue examinations [30] for high specif icity and accuracy in diagnosing cancers. 1.2 !Imaging Modalities of NMMM A typical laser scanning microscopy (LSM) experimental setup is shown in Figure 2. A femtosecond excitation source is used to generate ultrashort pulses. The position of the columnated ex citation beam is controlled by a scanning system, such as a set of galvanometer mirrors that dictate the position of the excitation beam on the sample. By controlling the voltage applied to the mirrors, the dwell time, or how long the laser beam stays at a certain position, can be controlled. The scanned beam is sent through a set of lenses, a scan lens and a tube lens. The combination of these two lenses creates an afocal telescope system that will project an enlarged image of the laser beam on the pupil o r base of the objective [31] . 5 Figure 2. Typical laser scanning microscopy setup for nonlinear multiphoton multimodal microscopy. A high numerical aperture (NA) objective is used to minimize the excitation volume of the incoming laser beam on the sample tissue. In most cases, due to the omnidirectional scattering of fluorescence processes, immersio n fluid (oil or liquid depending on the sample) is placed in between the lens of the objective and the glass base of the mounted tissue slide (Figure 3). 6 Figure 3. Effect of index -matching fluid on multiphoton signals In Figure 2, the objective is mounted on a motorized stage to control the distance (z) of the objective from the sample, and is one of the necessities for performing depth -resolved imaging. Emitted photons are detected in the epi - (backwards) direction after being reflected off of a d ichroic element at the base of the objective. The dichroic mirror prevents any excitation photons from the laser beam from being detected. Typically, wavelength - or wavelength range -specific optical filters are placed in front of the detector to collect ph otons within a specific wavelength range; this is a method of obtaining spectrally -resolved MMIs (Figure 4). Figure 4 shows a transmission spectra of multiple different band -pass filters used to separate the multiphoton signals (two -photon excited fluores cence (2PEF) and three -photon excited fluorescence (3PEF)) detected from unstained tissues. 7 Figure 4. Comparison of multiphoton processes detected from biological tissues and the corresponding transmission curves of select optical filters used to isolate said multiphoton signals. The filtered photons are then focused by a lens onto a detector, typicall y a photo -multiplier tube (PMT), where the detect ion of a photon is transduced into an electrical signal. Image acquisition software will take these electrical signals and render them into images. Multiple data points are acquired at each position and thes e points are averaged to reduce noise in the final rendered image. Specific experimental setup, acquisition parameters, and instrumentation are discussed in detail in Chapters 2 -6. 1.2.1 !Harmonic Generation Harmonic processes are parametric multiphotonic proce sses ( Figure 5) [32,33] . Like multiphoton fluorescence, two or more photons interact w ith the molecule simultaneously, however, because these processes are parametric, no transition to an excited state occurs, and therefore, no energy is transferred prior to emission [32 Ð34]. 8 Figure 5. Potential energy diagrams of 2PEF, 3PEF, SHG, and THG. Parametric processes are polarization dependent. When the strong electric field of the laser interacts with matter, the electric field drives the electrons causing them to oscillate at the same frequency of the incident light, in other words it induces a polarization that is linearly dependent on the electric field strength: (1) Where is the polarization, is the permittivity of free space, is the nonlinear susceptibility term, and is the electric field. The polarization is e xpressed in a Taylor series expansion to express the polarization under intense laser fields leading to nonlinear processes. (2) Here , the second - and third -order nonlinear susceptibility terms are the dominating terms that lead to second harmonic generation ( SHG ) and third harmonic generation ( THG ), respectively. The polarization of the electrons depends on structural characteristics s uch as local organization and symmetry of the dipole moments within the molecule or crystal ( Figure 6). For materials with inversion symmetry, for example table salt, the second -order nonlinear susceptibility (SONLS) vanishes. Materials without inversion symmetry have non -zero SONLS and will exhibit SHG [33] . THG is not dependent on the local symmetry of the material and is observed when a change in the index of refraction occurs at the focal plane, for 0~~(1) ()() PtEt !"=~()Pt0!(1) !~()Et0~~~~ (1)(2)*(3)* ()[()()()...] PtEtEEtEEEt !""" =+++ 9 example in the presence of an interface [32 Ð34]. This interface is required because the sign of the propagation k vector of laser light changes through the focal plane and is known as the Gouy phase -shift [33] . In the absence of an interface, odd -numbered nonlinear optical processes (relevant to this discussion being THG) build up prior to the focal plane and then destructively interfere with those after the interface (because of their opposite sign k vector). This is why THG is not observed in bulk media and is observed when an interface is found at the focal plane. The above discussion can be illustrated in the model below. We begin by introducing a planar wave that causes a linear oscillation, , of a dipole moment, and then restrict this local response or polarization by limiting the maximum excursion of the electron like it is shown in Figure 6(a). Centrosymmetric and noncentrosymmetric media are both considered. When we Fourier transform the polarization to obtain the field emitted by the polarization in the frequency domain, as shown in Figure 6(b), even -ordered frequencies vanish for centr osymmetric media, but not for non -centrosymmetric media. Figure 6.The effect of a strong electric field on the polarization of a centrosymmetric material (top left) and a noncentrosymmetric material (top right) and the resulting frequencies (bo ttom), when the waveforms are Fourier transformed. cos() t!b. a. 10 Given that the main structural component of most biological tissues is collagen, a known second harmonic generator, the SHG signal at 535 nm is expected. Additionally, THG is expected whenever lipid depos its are found at the focal plane, such as in skin as well as in the retin as of patients with age -related macular degeneration (AMD) , where drusen, i.e. yellow fatty deposits, form [35,36] . In addition to spectral differences in the emission properties of multiphoton fluorescence versus harmonic generation, the temporal profiles of these processes differ as well. Mentioned earlier was that harmonic generation is a parametric process, meaning that generation of a harmonic signal is solely dependent on the interactions between the laser and the medium. Therefore, there is no lifetime decay of a harmonic process because the electron never reaches an excited state for radiative decay to occur and is only polarized while the laser pulse is present. Harmonic emission is therefore limited by the pulse duration of the laser, in this case ~40 fs [37]. Therefore, we use harmonic emission to determine the instrument response function (IRF) for our system which is ~150 ps [38,39] . Conversely , fluorescence, being a spontaneous process, can take from tens of picoseconds to tens of nanoseconds to occur ( Figure 7) [37,40,41] . Fluorescence lifetimes are also dependent on the molecular environment, such as solvent, temperature, concentration of the fluorophore, and whether that fluorophore is bound to a protein, such as the case for nicotinamide adenine dinucleotide ( NADH) and flavin adenine dinucleotide ( FAD ) [42 Ð48]. Figure 7. Time scales of molecular processes. Ad apted from Figure 2 from Brinks, et.al . [15] . 11 1.2.2 !Multiphoton Excited Fluorescence The development of ultrafast lasers and the discovery of multiphoton multimodal processes have enabled imaging transparent media that was only possible with the use of histological staining. Fluorescence is a spontaneous emission process that follows excitation of a molecule to its excited stat e [34,37] . In other words, as the excited state relaxes to its equilibrium geometry, energy is released as heat; emission occurs, and the electronic configuration relaxes back to the ground state by releasing a photon at a longer wavelength (lower in energy) than the excitation wavelength. Multiphoton excited fluorescence follows a similar process, however instead of a single photon, two, three, or more photons arriving simultaneously are absorbed by the molecule to induc e a transition to the excited state. In these cases, the wavelength of the emitted photon is dependent on how many photons were absorbed ( Figure 5) [34] . Therefore, a two -photon excited fluorescence (2PEF) process would have a resulting photon with a wave length equal to longer than half of the excitation wavelength. A three photon excited fluorescence (3PEF) process would result in a photon with a wavelength longer than one third of the excitation wavelength [34] . We have observed these multiphoton fluores cence processes from a number of endogenous fluorophores within the retina such as from NADH, FAD, rhodopsin, all trans -retinol, lipofuscin, and di-retinoid -pyridinium -ethanolamine (A2E ) Ðthe major lipofuscin fluorophore [49] . 1.2.3 !Dependen cy of nonlinear multiphoton multimodal signal s on pulse duration While single photon fluorescence is a linear process, multiphoton fluorescence and harmonic generation are nonlinear; therefore, high peak intensities from ultrashort laser pulses are essenti al for these processes to occur [42,50,51] . The likelihood a molecule will be excited by two photons of the same energy is determined by the TPA cross -section [33,51,52] . The probability of two -photon emission (TPE) is proportional to the absorption rate, hence, the higher the TPA, the greater the amount of fluorescence signal is expected for said laser intensity. The absorption rate for an n-photon process via pulsed laser irradiation can be ex pressed as 12 (3) Where is the n-photon absorption rate, is the n-photon cross -section, and denotes the intensity of the laser pulse elevated to the nth power. Equation 3 can be expressed as the n-photon absorption rate per second, over the time period defined by the repetition rate of the laser. (4) Where corresponds to the repetition rate of the laser, is PlanckÕs constant, is the speed of light, and is the central wavelength of the laser. For the case of a Gaussian pulse the a bsorption rate becomes (5) Where is a constant, is the full -width at half maximum (FWHM) duration of the pulse, and is the pulse energy. It can therefore be shown that the nonlinear signal intensity to be expected is proportional to: (6) Here, ignoring constants for concise comparison, the non -linear signal is proportional to , which is proportional to the nth-order exponential of the pulse energy, divided by the pulse duration elevated to the ( n-1) power. Laser pulses that are short in the temporal domain are broad in the spectral domai n [53,54] . Due to the mathematical dependency of nonlinear signal intensity on pulse duration (Eq. (6)), as well as the experimental evidence for two -photon microscopy, shorter pulses result in higher probability of achieving multiphoton multimodal signal generation ( Figure 8). [55] A caveat of broadband pulses is the sensitivity !"=!"!"!!! ""#$%!"#!!"!"!"#()!"#=$!"#$%!"$!"!#!!"$%&'()$ !"!!!!!!!"=== !"#$$$$$ !!""!!##"## !!$%%&'('' !"!!"#!"#!!"!"!#$!"#$!"#$ 13 to group -delay dispersion (GDD), where the spectral phase function controlling the duration of the pulse has some curvature. In order to understand the dependence of pulse duration on spectral phase it is important to define spectral phase. The spectral phase of the pulse, which can be arbitrar ily complicated, can be expressed by the Taylor series (7) The shortest pulse possible corresponds to the case when and is known as transform -limited (TL im), i.e., all the frequency components within the pulse have the same phase ( Figure 8) [54,56] . GDD or , also known as chirp is not the only type of dispersion that is encountered when performing multiphoton microscopy with a high -NA microscope third -order dispersion (TOD) or can be significant when there is a microscope objective along the beam path. The effect of spectral phase on a pulse is illustrated in Figure 8. Here, the first -order dispersion function, has a constant, linear, positive slope , and results in all the frequencies being delayed (see middle left figure) and arriving after . When applying the parabolic spectral phase (top center figure) lower frequencies get advanced whereas highe r frequencies get delayed linearly with respect to the central frequency, (middle center figure). When pulse has TOD, the higher and lower frequencies are delayed or advanced in the same manner with respect to . In a pulse with TOD, some of the frequencies destructively interfere with one another, splitting up the pulse with respect to the central frequency. Additionally, a pulse with TOD, the frequencies within the pulse are dispersed in time and either arrives before or after depending on the sign of [56] . Corresponding laser pulse profiles are shown in Figure 8(bottom row). !!""#=!$+!%""$#&'""#"$#+!%%""$#('""#"$#(+!%%%""$#)'""#"$#)+!%%%%""$#*'""#"$#*++++!(")=0 !''("0) !'''("0) !'("0)!!"!!""#"$%!!"!!"0! !'''("0) 14 Figure 8. Figure obtained from CLEO2016, SC352: Introduction to ultrafast pulse shaping Ñprinciples and applications. First -, second -, and third -order dispersion in the spectral phase domain (top and their effect on the frequency domain (middle) and the time domain (bottom) of the laser pulse. Multiphoton intrapulse in terference phase scan (MIIPS) is used [16,56 Ð58] to characterize a pulse and correct for dispersion. The spectral amplit ude, and the spectral phase, can be defined when the laser pulse is characterized,. When presented with a laser pulse with phase unknown, , MIIPS applies a set of calibrated reference functions, , to the input pulse using the spatial light modulator (SLM) shown in Figure 9(a) [16,56 Ð58]. MIIPS uses the SHG spectra to chara cterize the unknown phase [16,56 Ð58]. Mentioned earlier was the effect of on pulse broadening as well as how nonlinear signals decrease with longer pulses [51] . MIIPS uses this relationship in order to determine when the spectral phase of the p ulse [16,56 Ð58]. Once each reference phase is applied, it cancels the local curvature of the unknown spectral phase, and the nonlinear optical signal at that frequency is enhanced [16] . This is shown in Figure 9(b), where the maximum SHG signal corresponds to the matching reference function. As soon as , the respective phase of the input laser pulse has been characterized [16] . MIIPS compresses the pulses once characterization of the pulse is achieved for all frequencies within the laser ''() !" f(!)=a(!"!0)20''!()''() ''f!"! = 15 pulse bandwidth. In order to compress the laser pulse, MIIPS performs a double integration on determined during the pulse characterization to obtain , and then applies the negative counterparts of these functions, to the SLM, thus canceling the phase of the laser pulse [16] . The process of characterization and compensation can be applied iteratively to arrive at pulses that are close to the theoretical transform limit [16,56 Ð58]. Figure 9. The process of measuring and characterizing, the unknown from an incoming laser pulse and correcting it to a TLim pulse. Figure obtained from Lozovoy, V.V, et al. [16] . By correcting the spe ctral phase, as well as all the terms in the Taylor series, we are able to correct the temporal profile of the pulse to have all of the frequencies within the pulse arrive at the focus simultaneously. Doing so achieves maximum peak intensity at the focal plane of the microscope objective, ()''!"()!"()!"#''() !" 16 and enhancing all nonlinear optical processes to yield the ability to enhance a multiphoton event at the focal point of the laser on the tissue sample [16] . 1.3 Limitations of NMMM of unstained tissues The combination of fluorescence and harmonic signals resulting from the interaction between biological tissues and an ultrafast excitation source, such as the Ytterbium -fiber (Yb -fiber) laser oscillator, provides a minimum of 4 levels of contrast ÑTHG, 3PEF, SHG, 2PEF, to unstained biopsies. However, resolving the individual multiphoton signals is highly dependent on th e molecular and structural source in addition to the detector. The compounds in biological tissues that are multiphoton excited by the Yb -fiber laser, central wavelength of 1070 nm, emit within the visible range of ~300 nm Ð 700 nm. In any region of biolog ical tissue there can be numerous peptides, amino acids, and soft tissue structures that emit fluorescence within this range [59 Ð62]. The peak and the wi dth of the detected emission spectra are dependent on the molecular structure, as well as the solvent environment surrounding the molecule. These traits add difficulty to identifying the single source of the emission. An example of some common sources of e ndogenous fluorescence detected in biological tissues is shown in Figure 10. Figure 10. Emission spectra of common sources of endogenous fluorescence in biological tissues . In typical laboratory settings, fluorescent staining will be used to provide contrast to biological tissues. In these cases, biological cells or tissues are added to a solution of a fluorescent molecule, such as 4# ,6 -diamidino -2-phenylindole (DAPI) [63,64] . After incubation, these cells can be imaged , and in the case of a DAPI -stained cell, the deoxyribonucleic acid ( DNA) will be sta ined blue. How this pertains to the limitations of NMMM for unstained tissues is not towards data acquisition but more towards data 17 processing. Unlike unstained imaging with NMMM, the contrast in fluorescently stained tissues comes from the emission wavele ngth of the fluorescent dye. The emission of these dyes also has a high quantum efficiency, meaning they have strong illuminance compared to endogenous fluorescence, causing the lower intensity autofluorescence to be more of background noise than contrast. Additionally, there are many software programs that will spectrally -separate the signals from different fluorescent dyes in images. These programs have the emission spectra of each commercial fluorescent dye that are used to un -mix the fluorescent signals within the tissue image on a pixel -by-pixel basis. Unfortunately, these types of software are not readily available for users who wish to upload their own source emission spectra and apply those to the spectral signals from images of unstained tissues . Th erefore, additional methods for un -mixing the spectral signatures of NMMM images must be explored. 1.4 Computational History and Inverse Problems Upon the development of the first programmable and Turing -complete electronic computer in 1941 by Konrad Zuse, nearly all people in the science, technology, engineering, and mathematics (STEM) fields in addition to the military were excited with the capability of programming a machine to perform arithmetic in scales that would be too vast for the mind of a single person or group. Following the release of the Electronic Numerical Integrator and Computer (ENIAC) in 1945 and the continued improvements in mathematics, computer programming, and hardware engineering, the use of computational methods in research (biology ~1980Õs), military operations, and meteorology had exploded. 1.4.1 Inverse problem and solutions Computational algorithms have the ability to perform arithmetic at speeds well beyond what a human could process in a reasonable amount of time . Approximating the solution for the problem surrounding the lack of spectral resolution in NMMM images is one that can be done by appl ying a known, least -squares fitting algorithm [65 Ð69]. This type of problem, previously referred to in this dissertation as Òspectral un -mixingÓ is equivalent to a specific type of mathematical problem termed a n inverse problem. An inverse problem is the case where one would desire to determine the model or source parameters that produce the data we observe [70 Ð72]. 18 !"#$ (8) Where b is our observed data, A is the model or source, and x are the weight(s) on each of the sources. Least -squares minimizes the sum of the squared residuals Ñthe difference between the observed value, b, and the fitted value. We plan to employ this method to approximate the weights of each endogenous multiphoton signal detected from the NMMMI s of unstained normal retinas which is discussed further in Chapter 5 of this dissertation. 1.5 Convolutional Neural Networks and Machine Learning A rapidly growing field using computational methods to solve everyday problems is machine learning. Machine learning (ML) is a computational method aimed to achieve generalized learning of a specific problem [73] . This is done by training a model to extract features, or characteristic data points, from images a nd other datatypes presented to it , and applying what it ÒlearnedÓ in order to properly classify an new and previously unseen image or input [73,74] . There are two types of ML, supervised (S) and unsupervised (US) . Figure 11. General workflow for the two types of machine learning, supervised and unsupervised. 19 USML is used to separate data into groups where the ground truth, or the correct value is unknown and there is no training data , whereas the data used in supervised ML (SML) contains the ground truth labels corresponding to each data point, or image. In the case of SML on biological images, a pathologist will examine the images and label regions of the tissue these labels are term ed classes. A general overview schematic of the two ML subtypes is shown in Figure 11. A successful SML model will have a generalize d knowledge of the features corresponding to the class of each image, so that when the model is presented with a new image t hat it has never ÒseenÓ before, it will accurately assign it to the appropriate class. 1.5.1 Architecture ML models are comprised of an arrangement of connected neurons or nodes to form multiple layers , called a neural network (NN) , similar to the NN architecture shown in Figure 12 [75] . NNs are modeled in a way to mimic the architecture of the human brain, where neurons pass information in the form of electri cal pulses to other neurons. Inputs are received by the nodes of a NN, in the case of passive nodes, such as those that comprise the input layer, copies of the input data is made and then sent to all of the next nodes in the following layer. The layers fol lowing the input layer are active, meaning that the data is multiplied by weights before the values are duplicated and summed with the weighted values from the other nodes . Figure 12. Architecture of a typical neural network 20 Similarly, to the action potentials of the brain, in the active nodes of a NN, before new information is sent to the next layer, the values are passed through an activation function which emits the nodes output . The framework of the events occurring at the active no des is depicted in Figure 13. There are many activation functions that can be specified depending on the type of problem being addressed, however, the importance lies in the threshold of the activation function . Figure 13. The processes taking place in active nodes of a neural network. Just as not all human decisions have a distinct yes or no, and your decision may come from ÒweighingÓ pros and cons, when developing a NN to tackle everyday problems, we want a NN to have this similar ÒthoughtÓ process. Th is is where the threshold of the activation function plays an important role, if we chose an activation function, such as a step function , plotted on the left panel of Figure 14 , whe n y is below some threshold value , there will be no activa tion and everything above the threshold will be activated, even some values that may be close to being true will be labeled as false . The sigmoid function is the most basic example used in NN s due to its smooth threshold and the fact that it is differentiabl e and is plotted on the right panel of Figure 14 . Newer activation functions, such as rectified exponential linear unit (ReLu) have been shown to perform better than the sigmoid function [76] . Nevertheless, t he smooth threshold of the sigmoid function allows one to best approximate the o ptimal weights and serves as a sufficient example showing the importance of using nonlinear activation functions . 21 Figure 14. Comparison of two functions, the sigmoid activation function and step function The process of receiving data, a pplying a predetermined weight, duplicating the data, and outputting the sum of weighted data to the next layer is repeated until the final layer is reached. One single pass through the layers for all of the data is termed an epoch. ML models repeat this process fo r many epochs, and after each epoch , the weights at each node are evaluated and updated, typically through a process called gradient descent until the global minimum is reached [77,78] . 1.5.2 Caveats of Machine Learning and Their Solutions Currently, the largest setback of utilizing ML methods for research and other problems is the need for large amounts of data. Typically, a general rule of thumb for the minimum number of images necessary in order to properly train a ML model for image clas sification, is 10x the amount of trainable parameters or features in a model Õs NN. Obtaining this amount of data is expensive, however there are methods aimed at trying to combat this large data necessity, such as transfer learning (TrL) [79,80] . Figure 15. General overview of the transfer learning topic 22 TrL is a process by which new data is fed into a select amount of NN layers from a previously trained model. An assumption made by T rL is that the low -level (less specific) features are extracted by the beginning layers of the NN, and the more high -level or s ample -specific features are extracted by the final layers of the NN [80,81] . Knowing this, one can import a previously -trained model , such as one by Visual Geometry Group (VGG) net, or one trained on ImageNe t or Modified National Institute of Standards and Technology ( MNIST ) datasets, which are typically trained o r contain upwards of 10k curated labeled images , and only retrain the final layers on their data to have a successful classifier [82,83] . Figure 15 shows an overview schematic of the T rL process, where a set number of NN layers are not retrained on the new data and the remaining are retrained to learn the high -level features of t he new data. ML is utilized in many fields, such as patient records, image reconstruction, and wearable health trackers in healthcare, facial recognition in security, and many others, however the work done in this dissertation focuses on applying SML and T rL to my dataset of oral cancer NMMMI s. This work , as well as more specific terms and parameters involved in image classifiers, i.e. overfitting and augmentation, are discussed further in Chapter 6. Following more proof -of-concept research that explores the biomedical relevance and applications of NMMM imaging (Chapters 2 -4), this dissertation explores the novel use of NMMM images of unstained tissues in a of couple computational models to achieve max imal spectral -tissue assignment accuracy (Chapter 5) as well as using a ML image classifier in attempts to determine the health status of oral cancer biopsies (Chapter 6) . The combination of computational methods and NMMM shows promise towards augmenting c urrent diagnostic protocols used by the health care system. 23 Chapter II Multiphoton excited hemoglobin fluorescence and third harmonic generation for non -invasive microscopy of stored blood Abstract Red blood cells (RBC) in two -photon excited fluorescence (2PEF) microscopy usually appear as dark disks because of their low fluorescent signal. Here we use 15fs 800nm pulses for 2PEF, 45fs 1060nm pulses for three -photon excited fluorescence, and third harmonic generation (THG) imaging. In this chapter, we find sufficient fluorescent signal that we attribute to hemoglobin fluorescence after comparing time and wavelength resolved spectra of other expected RBC endogenous fluorophores: NADH, FAD, biliverdin, and bilirubin. We find that both 2PEF and THG microscopy can be used to examine erythrocyte morphology non - invasively without breaching a blood storage bag. 2.1 Introduction Two -photon excitation fluorescence ( 2PEF ) imaging of unstained red blood cells (RBCs) for non -invasive label -free blood analysis and defor mability has been deemed undetectable at 800nm [4,6,7] . This assessment is based upon the fact that spontaneous emission is dominated by fast non -radiative decay [6,7] . RBCs exhibit strong absorption and are known to cast dark shadows in nonlinear fluorescence imaging of capillaries in vivo [5] . While increasing laser intensity may yield a fluorescent signal that is strong enough for label -free analysis, the high intensity would likely cause both linear and nonlinear photo -thermal damage to the RBCs, especially when using pulse widths greater th an 150fs. 2PEF intensity is highly dependent on the characteristics of the laser source, with shorter pulse durations leading to higher fluorescence emission yields [51,84] . It follows that short pulse durations may lead to appropriately high levels of 2PEF signal, while limiting nonlinear photo -thermal damage for optimal non -destructive imaging. The long -term storage of RBCs leads to known changes in their health status. Current protocols call for the destructio n of these stored blood components after 42 days, based on guidelines from the Committee for Standardization in Haematolog y [85] . As RBCs age, they lose the important flexibility and deformability that enables them to squeeze through small capillaries to deliver oxygen to tissue; this 24 capability cannot be regained after the transfusion occurs [86,87] . Indeed, the effe cts of older (>14 days) transfused blood on mortality have been the subject of numerous studies [88 Ð91]. Recent work analyzing RBC cell membrane deformability before and three days following surgery found that storage times longer than three weeks led to irreversible damage to RBCs, which are then removed by the liver [86,87] . If blood could be imaged quickly and non -invasively prior to transfusion, it may be possible to assess the health of RBCs and thus reduce the risk of postoperative complications. Similarly, in emergencies, it may be possible to find healthy RBCs in blood beyond the 42 days storage, thus extending the availability of limited blood supplies. Non -invasive monitoring of RBC health via changes in cellular morphology can be accomplished, in principle, by imaging RBCs through the blood bag. Previous optical imaging studies of RBC morphology have required breaching the storage bag; these efforts, li ke those described above, found irreversible changes to the morphology with increasing storage duration [85,92,93] . Nonlinear optical imaging of RBCs has been accomplished via several different methods including TPA [7,94] , 2PEF [95,96] , and THG [97,98] . For TPA imaging, an intensity modulated pump pulse train at 775nm and delayed probe at 650nm were employed based upon the different excited state dynamics of oxyhemoglobin and deoxyhemoglobin [94] . 2PEF imaging has been accomplished via two - photon excitation of the Soret band in hemoglobin with ~250 fs pulses in the 600 -750nm wavelength range [95,96] . The fluorescence signal severely diminished when the excitation wa velength exceeded 750nm [4] . Spectroscopic measurements were made on a solution of stabilized human lyophilized ferrous hemoglobin powder. Imaging of fresh mice blood was accomplished with 600nm excitation wavelength [4] , a wavelength that produced the strongest signal. Previous studies have shown that 2PEF signal increases by decreasing pulse duration [51,84] . This suggests that short < 20fs pulses at 800 nm might be useful for imaging RBCs by enhancing 2PEF while keeping the number of laser photons (thermal energy) to a minimum. THG images of RBCs, on the other hand, have been efficiently generat ed by tuning the excitation wavelength to achieve resonant enhancement via the Soret band [97,98] . 25 We propose a method to image RBC morphology that does not require breaching the sterile environment of the blood storage bag. This consideration distinguishes the present study from prior work in that it p rovides a solid foundation for assessing RBC status non -destructively in a clinical setting. We explore 2PEF and THG modalities and compare these different contrast mechanisms to determine guidelines for imaging RBCs in storage while maintaining sterility. Nonlinear imaging with pulses shorter than 50fs from a Yb -fiber laser produce bright THG images of tissues [99] ; here, we used a short -pulse Yb -fiber oscillator [100] and a short -pulse Ti:Sapphire laser to image RBCs. Additional time and frequency resolved measurements were carried out in order to assign the emission signals. 2.2 !Materials and methods Two different lasers were used for this work. We used an 86 MHz repetition rate Titanium Sapphire (Ti:Sapphire ) laser (KM labs, Boulder, CO), with an external pulse shaper (MIIPS Box 640, Biophotonic Solutions Inc., East Lansing, MI), producing sub -15fs pulses; and a 42 MHz repetition rate Yb -fiber laser with a built -in pulse shaper (MIIPS HD, Biophotonic Solutions Inc., East Lansing, MI) [100] producing sub -45 fs pulses. The laser output is scanned by a pair of galvanometer mirrors (QuantumDrive 1500, Nutfield Technology, Inc., Hudson, NH) as i llustrated in Fig ure 16 . Dispersion correction, including high -order terms accumulated in the beam path, was accomplished using MIIPS [51] using an ultra -thin barium bora te ( BBO) crystal located at the focal plane (Microscope Detection Unit, Biophotonic Solutions Inc., East Lansing, MI). For imaging we used a 40x water immersion objective with a working distance of 0.5mm (Zeiss LD -C APOCHROMAT 1.1 NA, Jena, Germany), mount ed on a TE200 inverted microscope (Nikon, Tokyo, Japan), modified for multi -photon microscopy. 2PEF spectra and fluorescence -lifetime decay measurements were carried out in the epi direction using a 16 -channel time -correlated single photon counting (TCSPC) system (SPC - 830, Becker & Hickl GmbH, Berlin, Germany). Images were obtained in the epi direction using a PMT (HC20 -05MOD, Hamamatsu, Japan) after de -scanning and separation of signals using a 635nm long -pass dichroic mirror (Di02 -R635 -25x36, Semrock Inc ., NY) and a 680nm short -pass emission filter (ET680 -SP-2P8, Chroma 26 Technology Corp., VT). The microscope objective and filter combination in the epi direction resulted in poor THG detection efficiency at ~353nm. In the forward direction, THG was collected by a 15x objective (ReflX for UV, NT59 - 886, NA 0.28, Edmund Optics Inc., NJ) using a 410nm short -pass filter (410SP, Chroma Technology Corp., VT). This filter prevents detection of three -photon excited fluorescence expected at ~480nm. The forward signal was detected by a different PMT detector (H10720 - 210, Hamamatsu, Japan). Signals were digitized by a PC data acquisition board for further image reconstruction. Ten to thirty 512x512 16 -bit grayscale raw images were combined into a stack. An ImageJ ( Natio nal Institute of Health ÑNIH, MD, USA) software function for averaging images in a stack was performed resulting in a single 16 -bit grayscale image. Brightness and contrast levels were adjusted to increase the visibility of RBCs and the image was converted to 8 -bit grayscale. False coloring from grayscale to shades of red was performed for Fig ure 17 and 19(b). Images were cropped to exclude scanning aberrations at the edge of the field of view. No editing was performed within the images. Figure 16 . Schematic diagram of the microscopy setup for multi -photon imaging using different lasers. Ti:Sapphire or Yb -fiber laser oscillators can be used one at a time. A separate system was also used to measure the transient absorption (TA) properties of hemoglob in, which relies on the sequential stepwise absorption of two photons from the ground state to a final excited 27 state via an intermediate excited state. TA measurements were obtained from both human hemoglobin (Sigma -Aldrich H7379, St. Louis, MO) and red bl ood cells obtained from mice in accordance with the Institutional Animal Care and Use Committee of the Massachusetts General Hospital (IACUC protocol #2016N000078). Imaging was performed with a tunable dual -output pulsed femtosecond laser source (Spectra - Physics Insight DeepSee, Santa Clara, CA), using the fixed 1040nm output as the pump beam and the tunable output set to 735nm as the probe beam. This configuration allows for the stepwise absorption of 1040nm and 735nm photons by hemoglobin, which roughly equates to the absorption of a single 430nm photon. Similar multiphoton -based absorption techniques have been used and validated in the past to visualize heme proteins, such as in the case of two -photon excited photothermal lens microscopy [101] . Intensity modulation of the 1040nm beam was achieved using an electro -optic modul ator (Thorlabs EO -AM-R-20-C2, Newton, NJ) with 20 MHz modulation. Imaging was carried out on a modified confocal microscope (Olympus FV1000, Center Valley, PA) using a 1.20 NA 60x water immersion objective (Olympus UPLSAPO 60XW, Center Valley, PA). Forward detection was achieved using a photodiode coupled to a lock -in amplifier (APE Lock -in Amplifier, Berlin, Germany) placed downstream of a 710nm longpass filter (Chroma E710LP, Bellows Falls, VT) and a 950nm shortpass filter (Thorlabs FES0950, Newton, NJ). This configuration allows the transmission of the 735nm probe beam to the photodiode while blocking the 1040nm pump beam, where the lock -in amplifier can detect any intensity modulation transfer from the pump beam to the probe beam at the 20MHz modulation frequency. The output of the lock -in amplifier is then fed into an Olympus input -output box system and digitized for acquisition by the Olympus Fluoview confocal microscopy control software. All procedures involving human subjects, including consent forms, were approved by the Biomedical and Health Institutional Review Board (BIRB) at Michigan State University. Whole blood was obtained from consented healthy human donors by venipuncture and collected into heparinized tubes [102] . Upon collection in a citrate p hosphate dextrose buffer solution, the blood was immediately centrifuged for 10min at 500 g and 4¡C. The plasma and leukocytes were removed by filtration and the RBCs were added to an AS -1 storage solution. RBCs were subsequently diluted from ~70% to 0.4% i n additive solution -1 28 (AS-1) solution for imaging. RBCs were introduced either in a chamber containing prepared RBCs or directly inside a sealed polyvinyl chloride ( PVC ) blood storage bag (200 gauge PVC, 50 " m thick film) (Uline, WI) - the same as used for commercial storage - and hermetically enclosed by thermal splicing [103] . Erythrocyte ghosts (the resulting RBC membrane with all other intracellular components removed) were prepared according to a wash protocol based on published work [104] . RBCs were suspended in phosphate buffered saline (PBS ), and then washed 3x at 500 g for 10 minutes with the supernatant aspirated off after each wash. Four 40µL aliquots of compact RBCs s uspended in 1mL lysis buffer (described below) were then centrifuged at 22,000 g for 15 minutes. After discarding the supernatant, the remaining membranes were washed in lysis buffer 3x at 22,000 g for 5 minutes. Finally, the supernatant was discarded, and t he lysates were pooled. Lysis buffer was prepared by mixing 10 mM hydroxymethyl aminomethane hydrochloride ( Tris -HCl ) with 0.2 mM ethylenediaminetetraacetic acid (EDTA) at pH 7.2. The linear absorbance of erythrocyte ghosts was measured using a Unicam UV -2 spectrophotometer (ATi Unicam, Cambridge, UK) in a 1 mm quartz cuvette. 2.3 !Results 2.3.1 2PEF microscopy imaging of RBCs on a coverslip and through PVC bag using Ti:Sapphire laser 2PEF images of human RBCs were obtained in the epi direction with the 800nm Ti:Sapphire laser as shown in Fig ure 17 and through the PVC storage bag in Fig ure 18 . Individual RBCs and their central pallor can clearly be seen. 29 Fig ure 17 . 2PEF image of unstained human RBCs on a coverslip imaged by 15fs pulses with 10mW average power from the Ti:Sapphire laser tuned to 800 nm. Scale bar is 20 " m. Fig ure 18. 2PEF image of RBCs detected through the PVC storage bag in the epi direction obtained with 10mW average power 15fs pulses from the Ti:Sapphire laser tuned to 800 nm. Scale bar is 20 " m. Video of flowing RBCs ( Visualization 1 ). The possibility of photodamage was carefully considered. At lower excitation power (<5mW) the 2PEF signals are very weak, but no signs of photodamage were observed in RBC appearance after > 2min of exposure. With an increase of excitation power to 10mW or 22mW using 800nm or 1060nm lasers, respectively, we observed both cell shrinkage and an increase o f the fluorescence signal after tens of seconds of exposure. Similar photodamage effects on RBCs have been reported during optical trapping of human erythrocytes [105] . 30 RBC morphology can be clearly seen in Fig ure 18 and is important for determination of hematologic diseases. In fact, many diseases have normal blood counts but abnormal membrane morphology [106] . Under normal circumstances, mature RBCs are round biconcave disc -shaped cells measuring 7 -8 microns in diameter. Both THG and 2PEF modalities allow measurements of the average cell diameter and thickness. From our results we obtain from 2PEF a mean diameter and one standard deviation 6.9 ± 0.5 " m and from THG a mean diameter of 8.1 ± 0.5 " m. We do not consider the different diameters to be significant given that these were two different blood samples. Taking into account a measured thickness of 2.2 ± 0.2 " m we are able to estimate the mean volume at 82 ± 11 " m3 and the surface area at 123 ± 18 " m2. Previous studies using holographic microscopy of RBCs reported a variation of the diameter [107] ranging from 6µm -7.8µm, corpuscular volume [85,107] ranging from 88 to 102 CV [µm 3], and surface area [85] ranging from 107 to 131 µm 2 with increasing storage times from 8 to 57 days. It is well known that preparation of fresh blood between coverslips can affect RBC appearance , where they can become echinocytes (star shaped RBCs) [108] . Moreover, the blood collection tube is internally coated with EDTA. While its role is to prevent coagulation of collected blood, there is a possibility that echinocytosis may occur upon contact of RBCs with the EDTA coating [109] . In the blood storage bag, however, the concentration of EDTA is low enough that echinocytosis is not likely to occur [110] . Nevertheless, we expect that other morphological deformities such as elliptocytosis, cigar cells, schistocytosis, and sickle cells can indeed occur, and can be determined by non -destructive 2PEF imaging. 2.3.2 !THG micr oscopy imaging of RBCs on a coverslip and through PVC bag using Yb -fiber laser THG microscopy does not require fluorescence from the molecule; THG signal generation only requires a change in the index of refraction at the focus [111] . While THG typically requires high peak intensities for imaging, this limitation is easily overcome by using shorter pulses and a lower average power. In Fig ure 19, the images were generated with less than 8mW of average laser power at the objective focus. THG images of RBCs on a glass cover slip detected in the trans direction are shown in Fig ure 19. Compared 31 to 2PEF , the RBC membrane boundaries are clearly seen on the THG image. The non -zero back ground in Fig ure 19 is a direct result of the out -of-focus THG signal generated from the glass -liquid interface. Fig ure 19. THG microscopy imaging of human RBCs on the glass cover slip obtained using a 1060 nm Yb-fiber laser emitting 45fs pulses with 8m W average power. a) Static image; b) Video of flowing RBCs (Visualization 2 ). Scale bar is 20 " m. Precise morphology measurements such as RBC size can be performed with or without the blood bag. We used an Yb -fiber laser with a central wavelength of 1060nm and 45fs duration pulses to image RBCs through the PVC storage bag, as shown in Fig ure 20. The nonlinear optical signal was detected in both trans and epi directio ns, as shown in Fig ure 20(a) and Fig ure 20(b), respectively. For the trans direction acquisition, images were obtained near the edge of the PVC storage bag where absorption of the THG signal was minimized. In the epi direction, on the other hand, imaging c an be performed anywhere in the bag. There is a difference with the images taken in the epi and trans direction, which is due to the emission being directional and phase matching favoring trans detection. The dependence of epi versus trans detection of THG signal has been quantified, with trans detection being best for thin samples and epi detection being strongly favored for thick samples where the signal corresponds to backscatter [112] . The shape of RBCs can be clearly seen in both images. It is worth noting that the average excitat ion power was maintained below 20mW in order to avoid damaging the PVC bag, which occurs above 25mW. 32 Fig ure 20 . THG images of RBCs detected through the PVC storage bag in trans direction (a) and in epi direction (b) excited by 7mW average power from an Yb-fiber laser. Scale bar is 10 " m. 2.3.3 !The source of fluorescence in RBCs 2PEF fluorescence from RBCs and erythrocyte ghosts has a broad emission spectrum from 400 to 570nm with a peak around 480nm that can be excited by two 800nm photons. This weak fluorescence emission from blood has been attributed to a number of sources in past studies, including flavin -containing molecules such as flavin adenine dinucleotide (FAD) and riboflavin, nicotinamide adenine dinucleotide (NADH) [113] , hemoglobin [95,114] and its fluorescent catabolites Ð biliverdin and bilirubin. Tryptophan, a common residue to most proteins, has an absorption band in the 260 -290 nm range that can be reached via three - photon excitation by the Ti:Sapphire laser, but not the Yb -fiber [115] . Excitation produces a broad fluorescence centere d at 340 nm that extends from 310 to 370nm. Fluorescence from RBCs was centered near 480nm. While it is possible that tryptophan was excited by the Ti:Sapphire laser, little or no fluorescence would be detected in our experiment given the 370nm long pass f ilter in our setup. The fluorescence lifetime reported for tryptophan in proteins shows a small amplitude ~0.2 for the 0.5ns component and the two equally weighted major components with ~2ns and ~5ns lifetime, respectively [116] . The difference in emission wavelength and fluorescence lifetime allows us to rule out tryptophan as the source of RBC signal. We investigated the fluorescence lifetime following one - (Fig ure 2 1(a)) and two -photon (Fig ure 21(b)) excitation for RBCs, erythrocyte ghosts, NADH, biliverdin, bilirubin, riboflavin and hemoglobin; 33 all in physiological salt solution (PSS) containing 4.7mM KCl, 2.0mM CaCl 2, 1.2mM MgSO 4, 140.5mM NaCl, 21.0mM Tris -hydroxym ethyl aminomethane, 5.5mM glucose, and 5% bovine serum albumin at pH = 7.4; all reagents were from Sigma Aldrich. The 2PEF lifetime decays were measured and compared with one-photon excited (355nm centered 12ps laser pulses) fluorescence lifetime decays for the same samples. Bilirubin, RBCs, and erythrocyte ghosts did not exhibit detectable fluorescence upon one -photon UV exc itation and are thus not present in Fig ure 21 (a). Table 1 summarizes fluorescence lifetimes for one -photon excitation measurements, and Table 2 summarizes two -photon excitation fluorescence lifetimes obtained by fitting decay curves using single and double exponential decay models. The system response time for the two -photon excitation measurements is ~130 ps, while the system response time for the single -photon TCSPC setup is ~45 ps (full -width at half maximum). We confirmed the two -photon dependence of the Hb 2PEF detected as a function of laser intensity (reported as average laser power ) and show those results in Fig ure 22. Note that previous studies measured a 2PEF lifetime for Hb excited at 600nm to be 230ps, a value indistinguishable from their system response time [4] , whereas we measured a lifetime of 280 ± 20ps. One explanation for the difference in lifetime i s the intersystem crossing [4] and charge transfer states near 630nm [117] , these pathways are not accessible when exciting at longer wavelengths. The NADH lifetime is dependent upon solvent pH, and whether it is bound or unbound. Bound and free forms of NADH are known to have lifetimes [113] corre sponding to 1 -2 ns and 450 -600ps, respectively. We measured free NADH and found its lifetime in the 450 -600ps range. Bound NADH has a lifetime that is too long to correspond to the 2PEF signal from RBCs. 34 Fig ure 21 . (a) One -photon excitation (355nm, 12 ps) fluorescence decay curves of hemoglobin, NADH, biliverdin, and riboflavin compared to (b) 2PEF (800nm, 15fs) decay curves for RBCs, their membranes and reagent -grade hemoglobin, biliverdin, bilirubin, riboflavin, and NADH. Table 1. Fluorescence lifetime decays obtained from fitting one -photon excitation (355 nm) curves using single and double exponential models. Table 2. Fluorescence lifetime decays obtained from fitting two -photon excitation (800 nm) curves us ing single and double exponential models. 35 Fig ure 22. Hemoglobin fluorescence signal versus average excitation power from Ti:Sapphire laser (14 fs pulse duration FWHM) plotted in a logarithmic scale. The experimental points are fit by a linear function with slope equal to 1.89 ± 0.03, which is consistent with two -photon excited fluorescence. 2PEF emission spectra of RBCs, their membranes and commercially obtained fluorophores are shown in Fig ure 23 (a). To measure quantitatively the similarity between 2PEF spectra, we calculated the Pearson correlation coefficients between the fluorescence spectrum of RBCs and that of other samples, as summarized in Table 3. Both bilirubin and riboflavinÕs weak Pearson correlation coefficients of 0.552 and 0.208, respec tively, suggest that they are not responsible for RBC fluorescence. Furthermore, riboflavin was omitted from the comparative plot of 2PEF peak wavelength vs. fluorescence lifetime (Fig ure 23 (b)), because its lifetime is over an order of magnitude longer th an that of RBCs and ghosts. We therefore conclude that the observed 2PEF emission from RBCs and ghosts indeed originates from hemoglobin. Table 3. Pearson correlation coefficients of 2PEF spectra . RBC Ghosts Hemoglobin NADH Biliverdin Bilirubin Riboflavin 1.000 0.995 0.975 0.939 0.883 0.552 0.208 36 Fig ure 23 . (a) 2PEF (800nm, 15 fs) emission spectra for RBCs, erythrocyte ghosts and reagent - grade fluorophores. (b) 2PEF peak wavelength vs decay lifetimes for RBCs, their membranes and reagent -grade hemoglobin, biliverdin, bilirubin, and NADH. It is well known that RBCs are densely packed with large amounts of hemoglobin. Hemoglobin is also bound to the membrane, as has be en determined after several washes [118] . The absorption spectrum of hemoglobin originates from heme, having an intense Soret or B -band (~400 -430 nm, depending on oxidation s tate) and weak transition to the Q -band (~550nm). It is known that the fluorescence emission of hemoglobin is undetectable with one - photon excitation; however, 2PEF imaging of hemoglobin has recently been demonstrated using two -photon excitation wavelengt hs ranging from 550nm to 750nm [113] . 2PEF of hemoglobin excited at 800nm has not been reported in any p rior work, despite hemoglobinÕs large two -photon absorption at longer wavelengths, with a maximum around 825nm [114] . Further confirmation of the participation of hemoglobin was obtained by comparing transient absorption decay curves for pure hemoglobin and purified RBCs in Fi gure 24(a). In these experiments, we monitor transmission of 735 nm photons as a function of time following excitation with 1040nm photons. Absorption at 1040 nm is likely associated with the absorption of oxyhemoglobin at that wavelength. The similarity between the two sugge sts that the fluorescent signal from RBCs is the result of an excited state of hemoglobin, as opposed to other potential fluorophores. Background signal from the PBS solution appeared strictly when the two pulse trains were overlapped, likely due to the op tical Kerr effect from the water solvent [119] . To first approximation, we assumed that hemoglobin was contained only within the volume of the RBC and not the membrane. Washing of the ghost cell s was done to remove all hemoglobin, and we expected to find no more 2PEF signal. We found that after three washes, the signal from hemoglobin 37 remained constant, indicating that some of the hemoglobin was bound to the membrane and could not be removed. The inset in Fig ure 24(b) tracks absorption at 414nm with increasing washes and shows that a certain percentage of hemoglobin remained. Previous methods with subsequent washing of RBCs also found a small percentage of hemoglobin in the membrane that cannot be removed via washing [118] . Fig ure 24. a) Transient absorption measurements of PBS, hemoglobin solution and purified RBCs following 1040nm pump and 735nm probe. b) Absorption spectra of PBS solution with ghosts washed 1 to 4 times. Inset shows absorbance of ghosts after varying number of washes, probed at 414nm. Spectra were corrected for background and Rayleigh scattering. 2.4 Conclusion s We have investigated 2PEF and THG for label -free non -invasive RBC imaging. Unlike conventional laser microscopy systems (>100fs), the laser systems employed here produce very short pulses (15fs for the Ti:Sapphire and <45fs for the Yb -fiber lasers). Therefore, these short -pulse so urces deposit less thermal energy and reduce photo -thermal damage to the RBCs. 2PEF signal increases as the inverse of pulse duration, while THG signals increase as the inverse of the pulse duration squared [113] . Following successful 2PEF imaging of RBCs, we explored the source of the fluorescence and concluded it originated from two -photon excitation of t he Soret band in hemoglobin based on fluorescence spectra, fluorescence lifetimes, as well as both linear and transient absorption data. The images are sufficiently detailed to assess morphological anomalies of RBCs non -destructively without breaching ster ility using commercially available compact femtosecond laser oscillators. Multi -photon microscopy modalities such as THG and 2PEF can be used for non -invasive imaging of blood cells through the storage bag. Moreover, it was shown here that THG imaging prov ided the best resolution and image sensitivity for noninvasive imaging of stored RBCs without photodamage. We 38 conclude that using compact and reliable ultrafast laser oscillators may lead to improvements in non -invasive blood analysis, including point -of-care assessment of RBC morphology. 39 Chapter III Multimodal nonlinear optical imaging of unstained retinas in the epi -direction with a sub -40 fs Yb -fiber laser Abstract Ultrafast lasers have potential use in ophthalmology for diagnoses through non -invasive imaging as well as for surgical therapies or for evaluating pharmacological therapies. New ultrafast laser sources, operating at 1.07 µm and sub -40 fs pulse durations, offer exciting possibilities in multiphoton imagining of the retina as the bulk of the eye is relatively transparent to this wavelength, 3P excitation is not absorbed by DNA, and this wavelength has a greater penetration depth compared to the commonly used 800 nm Ti:Sapphire laser. In this chapter, we present the first epi -direction detected cross -section and depth -resolved images of unstained isolated retinas obtained using multiphoton microscopy with an ultrafast fiber laser centered at 1070 nm and a ~38 fs pulse duration. Spectral and temporal characterization of the autofluorescence signals show two distinct regions; the first one from the nerve fiber layer to the inner receptor layer, and the second being the retinal pigmented epithelium and choroid. 3.1 Introduction With the goal of exploring the feasibility of novel retinal diagnostics and therapies, we evaluate the use of unstained multimodal nonlinear optical imaging techniques [8,26,120 Ð122] with a compact femtosecond fiber laser [123] . While advance s in optical coherence tomography (OC oT) have made retinal imaging widely accessible in the clinic [124,125] , there is room for improvement with respect to providing che mical as well as subcellular resolution. Moreover, the sub -40 fs Yb -fiber laser centered at 1.07 µm being considered here for multimodal imaging could in principle be used for diagnostic imaging as well as for performing therapeutic treatments [126] . Here, we explore the use of this laser for retin al imaging. When compared to a 800 nm Ti:Sapphire laser, the longer central wavelength is advantageous because the absorption of DNA at 355 nm (3 -photon of 1.07 " m) is at least 5 orders of magnitude weaker than at 266 nm (3 -photon of 800 nm) [127,128] . Furthermore, scattering decreases for longer wavelengths, which for 40 retinal tissues can be estimated to be 28% less for the longer wavelength [129,130] , allowing greater imaging depth. In multiphoton biomedical imaging contrast arises from several different nonlinear optical processes, such as 2PEF, 3 PEF, SHG, and THG, which are greatly enhanced at the focal plane [120 Ð122,131,132] . Multiphoton autofluorescence from NADH and FAD are conventionally used in biomedical imaging for diagnosis and monitoring the metabolic activity of cells [45,133] . Label -free nonlinear optical imaging of retina ha s been reported using a Ti:Sapphire [132,134 Ð137] and also with a Yb -fiber laser detecting THG in the forward direction and 2PEF in the epi direction [138] . Specifically , from the retina, multiphoton emission has been found from lipofuscin and A2E [139] . This is significant because lipofuscin and A2E emissions have been used to diagnose retinal diseases such as AMD [44,140 Ð150]. While this work presents the feasibility for an in vivo imaging system for retinal disease diagnosis, many additional studies will be needed to address technical limitations. Among the different limitations to be addressed, is the need for adaptive compression of the femtosecond laser pulses to account for the dispersion introduced by the cornea, lens, and vitreous humor. Fortunately, the dispersion introduced by those tissues has been measured and adaptively compensated [57,151] . Similarly, there may be the need for adaptive optics to adjust the focus of the laser in order to compensate for diffractive imperfections amongst different eyes, as well as for the limited NA of the eye [135,152] . In terms of safety, the American National Standards Institute ( ANSI ) standards limit eye exposure from a laser at ~1.07 " m to 4 mW, assuming a dilated pupil [153] . This value does not take into account the use of adaptive optics which would be used to improve focusing at the retina. Once these technical issues are addressed, validating the diagnosis of hea lthy retina animal models as well as diseased retina animal models, using this system, will need to be done. Finally, human trials would be required once all of the preliminary studies are completed successfully. In this work, we present the first all epi -direction images of unstained mouse retinas and a Cynomolgus monkey retina using a 1.07 µm Yb -fiber ultrafast laser with a ~38 fs pulse duration. Epi - as opposed to forward -direction detection was chosen as it is the only collection geometry that would permit 41 in vivo imaging. In addition, we measure both the spectra and lifetimes across the retinal layers from 7 " m-thick cross -sections of mouse retinas. The spectroscopic and lifetime information that becomes access ible through multimodal imaging, as performed here, may in the future provide the functional imaging at sub -cellular resolution that is presently missing from retinal diagnosis. Furthermore, the laser being used may at some point be evaluated for its abili ty to perform therapeutic interventions such as cauterization with greatly enhanced three -dimensional accuracy, as compared to present continuous wave lasers. 3.2 !Materials and methods 3.2.1 !Signal source, acquisition, and processing The custom designed laser sourc e for our multiphoton microscope is a Yb -fiber laser oscillator producing pulses with sub -40 fs pulse durations (full -width half -maximum) (1.07 µm, 42 MHz) [123] (Fig ure 25). The microscope used was a Nikon TE2000 in the inverted configuration. For imaging, a 40x water immersion objective was employed with a working distance of 0.5 mm (Zeiss LD -C APOCHROMAT 1.1NA, Jena, Germany) to focus the beam on the retina to a beam w aist (diameter of the beam at the focus) of ~0.5 µm, allowing the generation of peak intensities high enough to induce multiphoton processes with less than 7 mW of average power and pulse durations of 38 ± 1 fs. Thus, minimizing the effects of photobleachi ng and thermal damage. Laser scanning was done with galvanometer mirrors. The peak intensity was maximized through the use of a pulse -shaper (MIIPS HD, BioPhotonic Solutions Inc., East Lansing, MI, USA) to compensate for the high -order dispersion along the beam path [51,57,154] . Signal detection was accomplished in the epi -direction with two separate detection systems. To obtain frequency and time -resolved data a TCSPC system with a compact spectrometer and a 16 -photomultiplier tube (PMT) array were used (SPC -830 TCSPC, Becker -Hickl, GmBH). The spectral resolution of the TCSPC system is ~12.5 nm, limit ed by the physical size of each PMT in the array and confirmed with a mercury lamp. The grating in the spectrometer was rotated to select different spectral regions. Here two grating positions were used to collect the fluorescence spectra; one with a colle ction 42 range from ~300 nm to ~500 nm and the other from ~480 nm to ~680 nm. The images of the sliced mouse retinas obtained using the TCSPC were imaged with 6.9 mW of power or less for a total exposure time of 4.5 minutes (90 seconds per grating position). The lifetime curves had a collection window of 12.5 ns, chosen because it is four times longer than the longest observed emission lifetime and not limited by the repetition rate of the laser. The analogue to digital converter (ADC) resolution was set to 25 6 time bins. The first two nanoseconds before the rise of the lifetime signal peak were used for background subtraction. The IRF was measured to be ~110 ps. Fits for the lifetimes were done in Python using the curve -fit function in module scipy.optimize [155] . No efforts were made to deconvolve the IRF in the fits. However, this should only slightly affect fit values for lifetimes that are close to the IRF. Depth resolved imaging of the unstained Cynomolgus monkey flat mount was done with the same system as the sliced mouse retinas, however the emission was collected with a single PMT ( HC2005MOD, Hamamatsu) instead of the TCSPC. To achieve depth imaging, the focal plane depth was scanned by a motorized stage (Focus Drive with Integrated Controller by TOFRA, Inc. Palo Alto, California) which was controlled by a home -built data acquisition program in LABVIEW (Dr. Peng Xi, Dantus Research Group, Michigan State University). The depth resolved images were acquired for a total depth of 220 µm, with a 0.05 -µm step in between -512x512 pixel image. Depth resolution is estimated to be ~1 µm, howeve r, oversampling in depth allowed us room to average a few images at each depth. Each two -dimensional ( 2D) image was averaged for 5 seconds. Depth resolved imaging was done using a constant laser power of 7 mW through all the retinal layers. 3.2.2 !Preparation of fixed and unfixed thin sliced mouse retinas The C57BL/6 mice were purchased from the Jackson Laboratory (Bar Harbor, ME, USA) and were raised in a 12 -hour light/12 -hour dark cyclic environment and maintained on a standard diet at the Case Western Reserve University, School of Medicine animal resources facility. It is known that when B6 mice are raised under cyclic light conditions (12 hours of light and 12 hours of darkness), A2E Ñ the major 43 component of lipofuscin, increases only moderately up to the age o f 18 weeks [156] . The fixed thin retinal sections were prepared as follows: eyeballs were obtained from post -natal day (PND) 14 male pup of C57BL/6 mice immediately after euthanasia. Eyeballs were fixed with 4% paraformaldehyde in phosphate buffered saline (PBS, 136 mM Na Cl, 2 mM KCl, 8 mM Na2HPO4, and 1 mM KH2PO4, pH 7.4) containing 5% sucrose for short time. Then, their cornea and lens were removed in PBS solution. Resulting eyecups were fixed for 2.5 hours at 4 ¡C with gentle agitation. After fixation, eyecups were dehy drated with increasing series of sucrose in PBS and then infiltrated with a 2:1 mixture of 20% sucrose /PBS and optimal cutting temperature (OCT) compound (Sakura) [157] . Eyecups were then frozen using 2 -Methylbutane cooled with liquid N 2. The unfixed thin sliced retina was prepared from a P ND 80 ± 3 female mouse. The procedure of sample preparation is similar to that described for fixed retina preparation with the following modifications: 1) their cornea and lens were removed in Hank's Balanced Salt Solution (Thermo Fisher Scientific, Waltha m, MA, cat. no. 14175095), 2) immediately after the dissection, resulting eyecups were transferred into a 2:1 mixture of 20% sucrose/PBS and OCT compound, and 3) eyecups were incubated for 5 minutes and then frozen. Cryo -sections (7 µm) of the retinas were prepared with cryostat Ðmicrotome (CM1850, Leica, Bannockburn, IL). While cryo -preservation methods have been known to cause deleterious effects on endogenous fluorescence, it was reported [158] that endogenous fluorescence of FAD and NAD(P)H were preserved during fixation, para ffin -embedding, and subsequent slide preparation. In this work, we used less destructive cryo -preservation methods. Thus, unlike the paraffin -embedding method, tissue was not dehydrated with 100% ethanol and Xylene. Therefore, greater expected preservation of endogenous fluorescence is expected when compared to paraffin -embedding method Ñ which is already known not to affect coenzyme fluorescence. Sections were dried at room temperature for ~2 hours for unfixed conditions and overnight at 37¡C for fixed cond itions. Then, sections were rehydrated with PBS for 20 minutes. Rehydrated sections were mounted in an imaging medium {glycerol 85% w/vol (Fisher Scientific, Pittsburgh, PA, cat. no. G -31), Mowiol4 -88 15% w/vol (Millipore Sigma, Billerica, MA, cat. no. 475904) in Mammalian Ringer's solution [159] } and covered with #1.5 coverslips. Slides were kept at 4 ¡C. Unfixed retinal sections were imaged within 2 days. It is important to note that fixation with 44 paraformaldehyde has been known to shift the emission peaks of molecules found in the retina [160] , however, these shifts are usually small (~5 nm) and do not impact the findings of this manuscript. All procedures and experiments were approved by the Institutional Animal Care an d Use Committee (IACUC, Case Western Reserve University) and conformed to the recommendations of both the American Veterinary Medical Association Panel on Euthanasia and the Association of Research for Vision and Ophthalmology 3.2.3 !Preparation of fixed flat mou nt monkey retina The Cynomolgus monkey was raised in a 12 -hour light/12 -hour dark cyclic environment and maintained on a standard diet at Ricerca Biosciences LLC, Painesville, OH. Ricerca Biosciences collected a fresh eyeball from a 4 -year -old male Cynomo lgus monkey after euthanasia. The eyeball was washed with PBS three times. Then, the cornea, lens, and vitreous were removed in PBS solution. Next, retina was separated from the retinal pigmented epithelium ( RPE ) and four slits were made to the retina. The retina was then flattened in a cell culture dish with ganglion cell -side up. Following so, the flattened retina was fixed with 4% paraformaldehyde in PBS for 6 hours at 4 ¡C. After the fixation, the retina was washed with PBS and transferred onto a large glass slide with the ganglion cell side up (EMS, Hatfield, PA, cat. no. 71862-01). The flattened retina was mounted in the imaging medium and covered with a #1 coverslip (EMS, cat.no. 63774 -01). The retina was kept at 4 ¡C until it was imaged. 3.2.4 !Preparation of reference solutions Reference solutions of FAD and NADH were prepared by performing a 1:10 dilution of 10X PBS (Dot Scientific Inc.). The dilution was performed using MilliQ water. Thirty milligrams (mg) of each compound was added to separate solutions in PBS (136 mM NaCl, 2 mM KCl, 8 mM Na 2HPO 4, 1mMKH2PO4, pH7.4) yielding concentrations of approximately 350 µM for NADH (Sigma Aldrich) and 300 " M for FAD (Sigma Aldrich). An A2E reference solution was prepared using a 10 mM stock solution diluted in dime thyl sulfoxide to yield a final 1 mM concentration. A drop of each solution was placed on a microscope slide (Corning) followed by a coverslip (Corning) and then placed coverslip -side down on the 45 microscope objective. Water based immersion fluid (Zeiss, Im mersolTM, n e = 1.334 (23¡C), v e = 85) was used to increase the numerical aperture (NA) between the coverslip and the microscope objective. Fig ure 25. The experimental apparatus consisted of an ultrafast fiber laser operating at 1.07 µm, a MIIPS pulse shaper, a laser scanning inverted microscope, and a photon detector, mounted in the epi -direction. The TCSPC was used to collect fluorescence spectra and lifetime, whereas the single PMT was used to acquire the depth resolved images. 3.3 !Results The multimodal images in Fig ure 26 present all the retinal layers, including the nerve fiber layer (NFL), ganglion cell layer (GCL), inner plexiform layer (IPL), inner nuclear layer (INL), outer plexiform layer (OPL), outer nuclear layer (ONL), inner receptor layer (IRL ), outer receptor layer (ORL), and RPE, as well as the choroid and sclera. In Fig ure 26(a), the blue, green, and red channels represent emission centered at 535 nm, 575 nm, and 629 nm, respectively. Subsequently, the blue, green, and red channels in Fig ure 26(b) represent emission centered at 355 nm, 535 nm, and 629 nm. Each channel in both Fig ure 26(a) and 2(b) have a spectral bandwidth of ~37.5 nm. Here, 355 nm and 535 nm correspond to THG and SHG, respectively. Further information for how the other wave lengths were chosen can be found in the Discussions section. 46 Fig ure 26. Three colored (red, green, blue) composite multimodal images of the retinal layers from a 7 " m slice of a mouse retina taken with the TCSPC at 6.9 mW of power depths using a 1.07 " m Yb-fiber laser with 35.0 fs pulse durations. Image acquisition was done at 30 -second intervals for a total of 4.5 minutes. (a) Here the blue, green, and red channels represent emission centered at 535 nm, 575 nm, and 629 nm, respectively. (b) The blue, gre en, and red channels represent emission centered at 355 nm, 535 nm, and 629 nm, respectively. The bandwidth of each channel is ~37.5 nm. In Fig ures 27(a) -27(b) we plotted the lineouts of the spectral components from the multimodal images (Fig ure 26). In each of these figures the data are comprised of the sum of all pixels in a slice of each layer. The same sized slice was taken for each layer (~5 " m x 85 " m) to most accurately compare the emission spectra and emission yield across the layers. Although thi s prevents measuring the inhomogeneity in each retinal layer, the data were analyzed as such to achieve an acceptable signal -to-noise ratio. The emission spectra (Fig ure 27(a)) show that the strongest fluorescence is seen in the receptor layers (ORL and IR L). When the emission spectra are normalized to the maximum value for each layer (Fig ure 27(b)), it can be seen that the spectral shape of the emission is nearly identical in the most anterior layers of the retina (NFL through the OPL), with a maximum near 560 nm. The more proximal retinal layers, from the ONL to the choroid bear a greater signal at wavelengths > 600 nm. At the most posterior portion of the ocular tissue, the sclera has the most unique spectrum, displaying a peak at ~535 nm, that we attribu te to SHG (see Discussion). Here, only a plot for the spectral range of 480 -680 nm is shown, as the only 47 emission below 480 nm that could be characterized was a small amount of THG from the choroid, RPE, and sclera. No attempts were made at spectral unmixi ng as not all fluorophores contributing to emission spectrum were identified, and we were unable to obtain pure lipofuscin, which we believe to be a major component of the emission spectrum. Fig ure 27. Spectral emission from 480 to 680 nm detected from a mouse retina. a) The non - normalized emission spectra reveal that the receptor layers (ORL and IRL) have the strongest fluorescence emission. b) The normalized spectra more readily compare the different spectral shapes across retinal layers. Layers from the NFL thru the OPL all have nearly identical spectral shapes. The sclera has a unique spectral shape, where the peak at 535 nm is attributed to SHG (see Discussion). The lifetimes across the retinal layers are presented for two spectral regions, 556 -594 nm (Fig ures 28(a), 28(c)) and 610 -648 nm (Fig ures 28(b), 28(c)). Both spectral regions show two characteristic lifetimes across the retinal layers. These two lifetimes can be grouped as a ÒlongerÓ lifetime seen in the layers from the NFL through the ORL and the sclera, and a ÒshorterÓ lifetime seen in the RPE and choroid. Interestingly, the lifetimes for the RPE and choroid become shorter at longer detection wavelengths (i.e. from Fig ure 28(a) to Fig ure 28(b)). This trend is shown directly in panel c, whe re the lifetimes for the choroid through the IPL are fitted for both the short wavelength range and the long wavelength range. In addition, we have also plotted in Fig ure 28(c) the lifetime measured from the sclera at 535 nm. The emission is dominated by SHG from collagen and coincides with the measured IRF of our system (~110 ps). 48 Fig ure 28. Fluorescence lifetimes across the retinal layers from a mouse over a spectral range of a) 556 -594 nm, b) 610 -648 nm, and c) short: 556 -594 nm and long: 610 -648 nm. The plots reveal that for a given spectral band, all the layers from the NFL through the ORL have nearly identical lifetimes, whereas the choroid and RPE have nearly identical lifetimes. Additionally, the lifetimes in the choroid and RPE become shorter for longer detection wavelengths (i.e. from panel a to panel b). This trend is shown directly in panel c, where the lifetimes for the choroid, RPE, IRL, and IPL are fitted for both the short wavelength range and the long wavelength range. Also in panel c is t he lifetime measured at 535 nm from the sclera. SHG from collagen in the sclera is the source of this emission and coincides with the IRF of our system. The copious lifetime curves, and their accompanying parameters necessary to measure the fits for Fig ure 28, could not be effectively presented alongside each curve in the figure. Therefore, the fitting parameters and their calculated values for the lifetime decays shown in Fig ures 28(a) -28(b) are presented in Table 4 and Table 5. Given the similarities in t he measured lifetimes from the ORL through the NFL layers (within 5%), we summed them together. The same was done for signals from the choroid and RPE, prior to fitting. Two different sets of fitting methods were used: mono - and bi -exponential fits for the ORL through the NFL and bi - and tri -exponential fits for the choroid and RPE. The fits were performed over two wavelength ranges: 556 -594 nm and 610 -648 nm. Tri -exponential fit values were excluded from Table 4 and Table 5 for the ORL -NFL considering that the t 2 and t 3 values were equal. Likewise, mono -exponential 49 fit values were excluded from Table 4 and Table 5 for the RPE and choroid, as all of the fits had an R 2 value of less than 0.91. Additional information on the fits, representative lifetime decay data, including why certain fits were excluded, can be found in Appendix I. Table 4. Lifetime fitting of data shown in Fi gure 28. The parenthetical values corresp ond to one standard deviation. R 2 is the coefficient of determination, a dimensionless quantity. Having established the nonlinear optical signals from the different retinal layers from thin - sliced mouse retinas, we performed depth -resolved imaging on an unstained fixed Cynomolgus monkey retina flat mount with a thickness of 220 µm. We were fortunate this retina became available as it most closely resembles the histology of a human retina. Single PMT signal detection allowed the collection of multiple photons per laser pulse (unlike TCSPC, for which fewer than one photon per pulse is the standard). Single PMT signals, however, lack spectral and temporal information. Figure 29 shows an 8 -panel image of 2D slices of the retina at different depths. To assess the potential for in vivo imaging of human retinas with 1.07 µm and sub -40 fs (34.8 fs) laser pulses, we used the flat mount configuration, as flat mounts are more representative of the geometry (the orientation of the plane of the retina to the incoming l aser pulse) that is need in vivo . In addition to being the first epi -direction detected multimodal images of a retina, the quality of the images is excellent. We attribute the quality of the images to the use of sub -50 fs pulses, a high NA objective, and the use of a longer wavelength 1.07 µm laser with longer scattering length [129,130,161] . In addition, we imaged in 0.05 " m z-steps so that averaging could be done for each retinal layer. The depth -resolved image in Fig ure 29 shows sub -cellular resolution at each layer, with morphological characteristics 50 of specific layers being easily distinguishable. For example, nerve fibers in the NFL, ganglion cells in the GCL, cell nuclei in the INL, receptor cell nuclei in the ONL, inner segments of cone photoreceptors (large white features) and rod photoreceptor inner segments (smaller round features between the cone inner segments) in the IRL. Additionally, the ability to visualize details of the IPL and OPL layers is fascinating considering these layers are quite thin. This level of resolution, which could include additional wavelength channels, could be of importance for the diagnosis of reti nal and ocular disorders, especially when observing the structural organization of the rods and cones in instances of retinitis pigmentosa, cone -rod dystrophy, as well as other retinal diseases associated with the sclera and composition of the retinosomes present in the RPE [162 Ð165] . Fig ure 29. Depth -resolved imaging of an unstained, fixed, Cynomolgus monkey retina flat mount. The total extent of the Cynomolgus monkey retina was 220 µm. The depth of each image is indicated in yellow. The sca le bar is 15 µm and can be seen in the NFL panel. Each 2D image was an average of 5 scans, averaged for 5 seconds. Each layer, beginning with the NFL and ending with the ORL and RPE retinosomes, from left to right, was an average of 182 images (9.1 " m), 19 0 images (9.5 " m), 182 images (9.1 " m), 139 images (7.0 " m), 51 images (2.63 " m), 233 images (11.7 " m), 211 images (10.6 " m), and 33 images (1.7 " m), respectively. The depth resolved stack was obtained with 7 mW of average power at all depths using a 1.07 "m Yb -fiber laser with 34.8 fs pulse durations. In the IRL, inner segments of cone photoreceptors (large white features) and rod photoreceptor inner segments (smaller round features between the cone inner segments) are easily distinguishable. We believe th at the bright particles in the bottom right panel are perhaps melanin or RPE retinosomes attached to the tips of photoreceptors. At each depth, the characteristic morphology of the retina layers is clear, indicating that the Yb -fiber laser is effective at achieving the cellular resolution needed for depth resolved imaging. Video of depth resolved imaging of retina used to obtain the images presented in this figure ( Visualization 3). 51 3.4 !Discussion With the work presented here, we have begun the initial studies required to propose use of the 1.07 "m Yb -fiber laser for label free non -invasive retinal imaging and diagnosis in the clinical setting. As opposed to the conventional 800 nm Ti:Sapphire laser, fiber lasers exciting at 1.07 " m offer potential adva ntages over the latter. Some of said advantages have been evaluated or determined by this initial study. In particular, we have shown that contrast can be achieved in unstained thin -sliced mouse retinas and we have also provided initial spectral and tempor al characterization of retinal native autofluorescence. Additionally, depth - resolved imaging of a monkey retina was done to show feasibility of our system to image in a near -in vivo orientation using retinas that have similar characteristics to those in humans. The multimodal images of mouse retina in Fig ure 26 show clear delineation of the different retinal layers and their different spectral signatures. The emission spectra from each of the layers are given in Fig ures 27 (a) and 27(b), where we are able t o distinguish three main emissions. First, and best defined is the emission near 535 nm corresponding to SHG. The strongest SHG emission comes from the sclera which the protective layer of the eye and is primarily composed of collagen and elastin, both st rong SHG emitters [33,121,166] . Not shown in Fig ures 27(a) and 27(b) but also observed was emission at 353 nm corresponding to THG, primarily from the choroid, however, smaller amounts can be detected in the sclera and RPE. THG emission is possible from interfaces where the index of refraction changes, such as in lipid and tissu e interfaces, cellular membrane boundaries, as well as highly absorptive pigments like those in the RPE [33,136 Ð138] . THG microscopy of thin retina sections, imaged in the forward -direction configuration with a Yb -fiber laser, centered at 1044 nm, found bright signals from the ONL, INL as well as the GCL [138] . They did not image the choroid where we found the brightest THG signal in the epi direction. In terms of 2PEF, we observe two unresolved broadband emissions one centered at ~575 nm and the other at ~6 29 nm. We attribute the first to lipofuscin and the latter to A2E. The emission spectra of lipofuscin and its many fluorophores are highly dependent on the excitation wavelength and tissue layer 52 location within the retina [167,168] , but generally, this emission peaks at wavelengths > 550 nm. Unfortunately, little is known regarding the isolation and characterization of all the fluorophores of lipofuscin. A2E Ñ fluorophore of lipofuscin, is one of the f ew that has been characterized and is known to be a degradation product of the visual cycle [140,141,144,146 Ð148,162,169 Ð171]. Specifically, A2E is derived from phagocytosis of photoreceptor disks in the RPE that comprise the outer segments of photoreceptor cells within the ORL. Due to the nature of their functions, both the choroid and RPE have a more diverse metabolic environment [140,143,147,149,150] . Therefore, detection of a greater concentration of A2E in these regions agrees with previous studies and is expected. In relatio n to the detection of FAD and NADH, the relative absence of fluorescence from FAD and NADH in our images was highly unanticipated. However, after obtaining the emission spectra and fluorescent lifetimes of pure FAD (300 µM) and NADH (350 µM) solutions with the TCSPC following direct excitation from our laser (5 -minute acquisition time), the signal yield was significantly lower than any other fluorescent emissions seen in the retina. We found the expected fluorescence from NADH and FAD was present only as a result from three -photon excitation. The intrinsic low fluorescence yield of these chromophores, together with the required three -photon excitation explains their absence in our retinal images. We thus conclude FAD and NADH have negligible contributions to the emission spectra of unstained retinas with a 1.07 µm laser. The ability to isolate emission from lipofuscin and A2E from that of FAD and NADH may in fact be a significant advantage over other imaging techniques, as it is known that the levels of lipof uscin and A2E have been used to diagnose retinal disease [45,133,145] . The lifetimes plotted in Fig ures 28(a) -28(c) more readily allow for comparison from one re tinal layer to the next. From the NFL thru the ORL a relatively long lifetime can be seen. In contrast, a shorter lifetime is seen in the RPE and choroid. Table 4 shows fit values for these two different curves. However, these fits do not yield lifetime values that are readily comparable to literature values of known chromophores. However, from the spectral data, we find nearly identical emission spectrum can be se en from the OPL through the NFL, with a peak emission around 560 nm (approximated from the plotted spectra). In addition, a peak around 640 nm becomes prominent from the ONL to the choroid. Clearly, we 53 are dealing primarily with two compounds, lipofuscin a nd A2E. Table 5. Fitting with fixed parameters for lifetime decays shown in Fi gures 28(a) and 28(b). The parenthetical values are the one standard deviation errors and have units of ns. R 2 is the coefficient of determination (R 2 error) and is a dimensionle ss quantity. We repeated the fits presented in Table 4, this time holding the lifetimes constant t wo values found in the literature for lipofuscin and A2E [44] . In Table 5, for the ORL to NFL, a bi -exponential fit works well when t 1 = 0.39 ns and t 2 = 2.24 ns are fixed, which corresponds to the bi -exponential values for lipofuscin in the literature [44] . In the RPE and choroid curve, a tri -exponential fit works well when t 1 = 0.17 ns, t 2 = 0.39 ns and t 3 = 2.24 ns are fixed, where t 1 corresponds to the literature value for the lifetime of A2E, and t 2 and t 3 are the lifetimes associated with lipofuscin [44] . Additionally, in the RPE and choroid curve, a bi - exponential fit is performed where t 1 = 0.17 ns and t 2 = 1.3 ns. Here, t 1 is again the lifetime of A2E and t 2 is approximately the value of the mono -exponential fit for the ORL to NFL curve. Note that we find a greater contribution from A2E in the RPE and choroid. We also attribute the signals detected in the NFL to the ORL to either lipofuscin or one of its degradation products. The presence of degradation products could explain why an emission peak can be seen at 629 nm emission in the ONL, ORL and IRL, yet these layers do not show a short lif etime found for those signals in the RPE and choroid. To confirm that A2E can be readily excited with a 1.07 µm laser, we measured the spectrum and lifetime of a solution of A2E. The emission was very strong and the spectral shape and lifetime agreed with the literature and the emission from the retina (See Appendix II). Lipofuscin is very hard to isolate; therefore, we were unable to measure it in solution. In addition to lipofuscin and A2E, other known fluorescent compounds in the retina, 54 including rhodopsin, all -trans -retinal, and melanin, could contribute to the signals observed. Gathering our spectral and lifetime findings we are able to conclude that multimodal signals correspond to THG, SHG and 2PEF from lipofuscin and A2E. We find that the deep er retinal layers, where the receptors are concentrated, contain greater amounts of A2E. Our results include the first depth resolved multimodal images of monkey retina obtained in the ep -direction. The images from the stack show excellent sub -cellular res olution, making it easy to distinguish among the different cell types in each layer. The robust signals detected bode well for the use of multimodal microscopy with a femtosecond Yb -fiber laser source for future retinal diagnostics. The composite images in Fig ure 26, their corresponding emission spectra (Fig ures 27(a) and 27(b)), as well as the lifetime measurements (Fig ures 28(a) -28(c)) come from one retina. However, similar measurements were performed on 9 total retinas (5 fixed, 4 unfixed). All measureme nts were performed with identical image acquisition times to ensure consistency amongst the samples. The results from those measurements are consistent with the findings of this work. We acknowledge that these are very preliminary studies and more studies will need to be completed prior to claiming this system as ready for clinical use. Furthermore, future studies should address differences in the retina from different animals (both sexes) for both healthy and for a few of the most common diseases. However, these initial findings presented here establish the feasibility for the overall goal. 3.5 Conclusions In summary, we presented the first epi -direction multimodal imaging of unstained isolated mouse and Cynomolgus monkey retinas with an ultrafast fiber las er centered at 1.07 µm. Measurements of the fluorescence spectra and lifetime from a thin cross -section of a mouse retina showed that emission from the ORL to the NFL have similar spectra, including a relatively long lifetime. The RPE and choroid have simi lar spectra, including a relatively short lifetime. We attribute a majority of the short lifetime signal to A2E, and a majority of the long lifetime signal to lipofuscin or other lipofuscin degradation products. Interestingly, we show that FAD and NADH do not significantly contribute to the fluorescence emission from a 1.07 µm 55 laser. This is different from most multiphoton microscopy studies where FAD and NADH are usually the strongest autofluorescent signals. In addition, depth resolved imaging of an unsta ined Cynomolgus monkey retina is also presented using the same laser and experimental setup. The depth resolved images from the Cynomolgus monkey show that it is feasible to use our collection system to image the retina of live -animal subjects, and in the future of humans. 56 Chapter IV Tetra -modal multiphoton microscopy: A non -invasive technique for augmented histopathological analysis of oral squamous cell carcinoma biopsies Abstract Hematoxylin and eosin (H&E) staining is the typical test used by pathologists to differentiate among cell types in biopsies. Currently, it is widely used in oncology diagnostics (i.e. oral squamous cell carcinoma, OSCC), even though it has some disadvantag es in costs and labor intensity. In this chapter , we developed a method of nonlinear multiphoton multimodal microscopy imaging (NMMMI ) that uses second and third harmonic generation (SHG and THG) in tandem with two - and three -photon excited fluorescence (2- and 3PEF) for histopathological analysis in oncology diagnostics. To validate our method, we analyzed human and canine OSCC biopsies with varying stages of OSCC. As the result, we mimicked H&E contrast and achiev ed further differentiability that is not attained through H&E staining. Altogether, we were able to differentiate amongst eosinophilic structures such as cytoplasm, collagen, and elastin of the stroma in the oral mucosa that are not distinct in an H&E stai n. In addition, we spectrally resolved inflammatory response cells (i.e. plasma cells and lymphocytes) using their 2PEF and 3PEF signals. Furthermore, our 1070 nm ultrafast excitation source permits depth -resolved imaging with 2 - and 3PE modalities resulti ng in images that can be used in quantitative analysis, such as comparing mean signal intensities amongst tissue classifications and collagen fibril orientation angles. With further optimization, we believe this technique can aid pathologists in a more tim ely, cost -effective, and efficient method in oncology diagnostics. 4.1 !Introduction Oral squamous cell carcinoma (OSCC) is a malignant neoplasm of the oral cavity which falls under the category of head and neck cancers. Major contributing factors to OSCC inc lude: environmental (e.g. alcohol consumption, tobacco use, and recurrent ulceration), genetic (e.g. mutations in p53 - responsible for 25 -69% of all oral cancers), and Human Papilloma Virus 16 (HPV16) [172 Ð177]. One of the most 57 impactful dangers of OSCC is that in the earliest stages, it can be completely painless and show no obvious changes, either physical or symptomatic, a nd without treatment can be fatal [178] . The inferior labium (bottom lip), the front two -thirds of the tongue, the floor of the mouth, gingivae, roof of the mouth, the cheeks of the oral cavity, the pharyngeal regions, and salivary glands are affected by OSCC [178] . OSCC contributes to roughly 2 -4% of all cancer c ases and is more prevalent amongst men in developing countries as well as in countries where smoking and chewing tobacco are common, such as India, South -Central Asia, and Pakistan [175,178 Ð180] . Survival rates for OSCC are typically above 60% in developed countries and near 40% for developing countries, which is heavily dependent on the awareness of the disease the patient has, initial wrong diagnosis, and lack of early detection [146,148] . Currently, the methods for diagnosis begin with a physical exam, where a physician or dentist examines the oral cavity for abnormalities, such as irritation in the form of sores or leukoplakia (i.e. white patches) [179] . In cases of suspicion, a biopsy will be sent to a pathologist for further analysis. The typical test for histology diagnostics is to perform H&E staining on the sliced samples. Unfortunately, this test has high human variance in diagnosis and is very costly. Therefore, research on exploring alternative techniques that can augment H&E staining for increased confidence in diagnosis is needed. For example, recent work on spectrum - and time -resolved endogenous multiphoton signals f rom premalignant and malignant gastric mucosa by Wei Zheng et al. showed that using multiphoton microscopy 2D and 3D images can be useful for characterizing subcellular morphological changes in the gastric mucosa and providing quantifiable identifiers of g astric disorders [181] . As demonstrated by Zheng W., Boppart S., Tromberg B., Li X., a nd many others in the field, multiphoton microscopy has shown to be an incredibly valuable tool in detecting abnormalities in the cellular microenvironment of biological tissues [181 Ð186]. Unlike staining, the contrast in imaging unstained tissues arises from the endogenous fluorescence and other multiphoton nonlinear signals present in all biological tissues [8,181,187,188] . Other nonlinear multiphoton signals than fluorescence can provide a further specified identification of biological molecules and tissues. SHG , a two -photon process that is a result of ultrafast laser light interacting with an arrangement of molecules lacking a center of inversion, is a well -documented method of identifying certain biological 58 structures such as collagen fibrils and bundles [8,33,121,189 Ð191]. Three -photon harmonic proce sses, such as THG can provide additional contrast to multiphoton images of unstained biological tissues. THG signals arise from a shift in the Gouy phase at boundaries where there is a change in the index of refraction [3,8,32,33,52,120,137] . This modality is ideal for the heterogeneous environment of biological tissues, as THG distinguishes fat globules, muscle fibers, and cell m embranes from the surrounding cells and has shown its potential in detecting margins in esophageal squamous cell carcinoma [57,123,137,190] . While 800 nm excitation sources are conventionally used in multiphoton imaging, excitation using near -IR radiation is advantageous due to the decreased scattering properties of this wavelength in biological heterogenous tissues [42,50] . Contrary to excitation with a Titanium -Sapphire laser ( $center = 800nm, $3p= 265nm), the longer wavelength (1070 nm) permits three -photon methods ( $3p= 353nm, e.g. THG and 3PEF), above the DNA -damaging UV range. Therefore, in this study, we developed an imaging method that yields high -fidelity to H&E 2D slices and 3D volumetric imaging capabilities using an ultrafast Yb -laser with a central wavelength of 1070 nm (near -IR radiation). As the result, this method obta ined multiphoton multimodal excited images of unstained FFPE oral cancer biopsy tissues mounted on slides (2D) as well as directly from the paraffin block (3D). Using two pseudo -color channels and our most primitive detection method Ð a single PMT and two optical filters Ð we matched H&E contrast and provided additional spectral distinction between the connective tissue of the stroma and the cytoplasm of the squamous cell layer. Furthermore, addition of a third filter to our detection method provided contr ast and rapid detection of 2PEF signals from inflammatory cells in the stroma of unstained tumors. The contrast in these images permit collagen orientation angle measurements, which were directly related to the state of health of the tissue region. Overall , the ability of our technique to provide enhanced contrast over traditional H&E staining with a more primitive detection scheme shows great potential for oncology diagnostics. In fact, our nonlinear multiphoton multimodal microscopy imaging (NMMMI) techni que can potentially be used in tandem with H&E staining to provide additional diagnostically relevant information that can lead to a more accurate diagnosis. In addition, the portability of this excitation source and its ability to penetrate biological tis sues at depth in a non -destructive 59 form (i.e. directly from the paraffin block) can further reduce the cost of pathology sample processing in a clinical and research settings. 4.2 !Materials and Methods 4.2.1 Human Tissue excision, processing, and sample prepa ration Head and neck squamous cell carcinoma biopsy samples were collected from six patients (50% male). All patients presented with biopsy confirmed squamous cell carcinoma and were clinically staged T2 (n=5) or T4 (n=1). Following examination and diagnosis, patients were scheduled for primary tumor removal ( Figure 30 a). During surgery, from the centered area of the primary tumor, a biopsy sample, representative of the primary tumor, was collected ( Figure 30 b). Biopsy samples were kept on ice, prior to slow -freezing in OCT. Hereafter biopsy samples were stored in the -80¡C until further use. Tissue samples fixed in 10% Neutral Buffered Formalin are processed and vacuum infiltrated with paraffin on the Sakura VIP 2000 tissue processor; followed by emb edding using a Thermo Fisher HistoCentre III embedding station ( Figure 30 c). Once blocks cool, they are placed on a Reichert Jung 2030 rotary microtome and trimmed. Once the block is trimmed to expose tissue, it is cooled and finely sectioned at 4 -5 micro ns ( Figure 30 d). Sections are dried at 56¡C (not exceeding) in slide incubator to ensure adherence for 2 Ð 24 hrs. Then, slides removed from the incubator and stained with a routine H&E method as follows: Two changes of Xylene Ð 5 minutes each, two chan ges of absolute ethanol Ð 2 minutes each, two changes of 95% ethanol Ð 2 minutes each, running tap water rinse for 2 minutes, endure Hematoxylin (Cancer Diagnostics Ð Durham, NC) for 1 % minutes followed directly by a 10 Ð 15 second differentiation in 1% a queous glacial acetic acid and running tap water for 2 minutes to enhance nuclear detail. Upon completion of running tap water slides are placed in one change of 95% ethanol Ð 2 minutes, 1% Alcoholic Eosin -Phloxine B Ð 2 minutes to stain cytoplasm, one ch ange of 95% ethanol for 2 minutes, four changes of 100% ethanol Ð 2 minutes each, four changes of Xylene Ð 2 minutes each followed by cover slipping with synthetic mounting media for permanent retention and visualization with light microscopy. Slides for multiphoton laser microscopy are deparaffinized in the same manner described above using slight 60 differences. The differences are: only the first xylenes and ethanols to running tap water and then blocked for auto fluorescence utilizing ammonia/ethanol pret reatment the cover -slipped with ProLong Gold anti Ð fade media for imaging. 4.2.2 !Canine Tissue excision, processing, and sample preparation Squamous cell carcinoma cases within the oral cavity were identified from the archives of the Michigan State Universit y Veterinary Diagnostic Laboratory. All cases were surgical biopsies submitted by practicing veterinarians for histologic examination and diagnosis. The regions of tissue with normal and neoplastic histology were chosen by a board -certified veterinary pat hologist using the adopted and accepted methods of cancerous cell classification and diagnosis [192,193] . Tissues are fixed in 10% neutral buffered formalin for 24 -48 hours and processed routinely into paraffin embedded blocks. Blocks are sectioned at 5 " m and stained with eosin an d hematoxylin for diagnosis by a board -certified veterinary pathologist (Figure s 30c and 30d). The database was queried for canine and feline squamous cell carcinoma within the period of 2014 -2018 and 27 cases were arbitrarily selected. An additional 5 " m-thick section from each case was examined and all diagnoses were independently confirmed by Dalen Agnew, DVM, PhD, DACVP, Veterinary Diagnostic Laboratory, Michigan State University. Demographic data including species, breed, age, and sex were also collected (supp. Material Table 1). Figure 30 . Excised human oral cancer biopsy tissue sampl e preparation. a) Human oral cavity with typical sites of OSCC. b) Freshly excised biopsy tissue after OCT clearing and dehydration. c) Paraffin embedding of the dehydrated biopsy tissue. d) Set and treated biopsy tissue paraffin blocks are sliced at 4 µm thickness. One slice is left unstained for NMMMI and the other is stained with H&E. 61 4.2.3 !Laser setup The laser source for our multiphoton microscope is a custom designed Yb -fiber laser oscillator generating pulses with sub -40 fs pulse durations (full -width half -maximum) (1.07 µm, 42 MHz) [123] (Figure 31 ). Imaging was done on a Nikon TE2000 multiphoton inverted microscope ( Figure 32). A 40x water immersion (Carl Zeiss !, Immersol ! W, n e = 1.334 (23 ¡C), v e = 72) objective was employed with a working distance of 0.5 mm (Ze iss LD -C APOCHROMAT 1.1NA, Jena, Germany) to focus the beam on the tissue to a beam waist (beam diameter at the focus) of ~0.5 µm, favoring the generation of peak intensities high enough to induce multiphoton processes with less than 4 mW of average power and pulse durations of 36 ± 1 fs. High peak intensities and low average power minimize the effects of photobleaching and thermal damage. The peak intensity was maximized through the use of a pulse -shaper (MIIPS HD, BioPhotonic Solutions Inc., East Lansing, MI, USA) to compensate for the high -order dispersion along the beam path [194 Ð196]. Laser scanning on the tissue was done with galvanometer mirrors. Figure 31 . Schematic of the beam path an d optical setup for NMMMI . An Yb -fiber oscillator excitation beam path leading into the MIIPS pulse characterizing system, and then into the Nikon TE2000 inverted microscope, where the paraffin block and tissue slides are imaged using NMMMI . 62 4.2.4 !Imaging, detection, and signal processing All signal detection was accomplished in the epi -direction. Two detection methods were used, the first being with a time -correlated single photon counter (TCSPC, SPC -830, Becker -Hickl, GmBH). The TCSPC is able to collect frequency and time -resolved data with a compact spectrometer and a bialkali 16 -PMT [38,193] (Figure 32 ). The spectral resolution of the TCSPC system is ~12.5 nm, limited by the physical size of each PMT in the array. The epi -directional photons are reflected off of a mirror and coupled into an optical fiber bundle. The input of the fiber bundle is circular (d " 4 mm) while the output of the bundle, before the polychromator, is rectangular (lxw " 7 mm x 1 mm). At each pixel in the imaging region (512 pixels x 512 pixels in this work), for every photon, a timing pulse of the photon and the number of the channel (PMT) that detected the photon, is recorded. Over the signal period, the TCSPC builds up a photon distribution for e ach channel, resulting in emission waveform and lifetime decay data of the detected photons for each channel [38] . In this work, the lifetime decay data was not obtained as the acquisition method needed to achieve sufficient temporal resolution would require such an increase acquisition times that they would not be considered practical in a clinical setting. For instance, the acquisition time for a single 512x512 pixel image is t image =90s/ #range whereas the acquisition time for obtaining a well -resolved (~256-time binds) decay measurement is t lifetime = 90s/16x64 pixel region of interest ( ROI). Figure 32 . Optical table and NMMMI detection setup. a) 3D animated optical table arrangement of the NMMMI experimental setup. Photons are detected using two methods, a single photo -mul tiplier tube (PMT) or redirected via flip mirror for detection by, the time -correlated single photon counter (TCSPC). b) Simplified diagram to show how epi -direction photons are coupled into the TCSPC optical fiber bundle. c) Schematic of how photons are d etected and recorded by the TCSPC software to build up the waveform data. 63 Each 512x512 TCSPC image was obtained at two polychromator angles (~200 nm wavelengths range per each angle) permitting a total wavelength range of 400 nm (327 nm Ð 685 nm), collec ted at each pixel. 2 sets of images were obtained for each 512x512 (X -pixel by pixel) region of interest on the tissue area. Each image was acquired for 45 s. Therefore, the tissue was imaged for 90 seconds at each of the two polychromator position angles for a total acquisition time of 180 seconds. The images were obtained at an average power of ~3.6 mW. For each type of tissue region (normal or cancerous), 3 -10 512x512 images were obtained to collect the data from multiple layers of the tissue (epithelium , squamous cell, and stroma layer) as well as to provide a broader view image of the tissue region. The second method of detection was using a single PMT and optical bandpass filters. In all figures with NMMMI s acquired using the single PMT, 1 -3 optical filters were used. The three bandpass filters that were used to restrict the detected wavelengths had the following transmission wavelength bands: 340 nm Ð 410 nm, 515 nm Ð 545 nm, and 605 nm Ð 785 nm. All images were imaged at 3.2 mW -6.7 mW for 4 -10 secon ds. The depth -resolved (DR) images obtained directly from the paraffin blocks of canine, feline, and human tissues were done so using the same microscope and objective as the TCSPC 2D images, however, the DR image data was detected using a single photo -mul tiplier tube (PMT, HC2005MOD, Hamamatsu). To achieve depth imaging, the focal plane depth was regulated by a motorized stage (Focus Drive with Integrated Controller, no. 101 -18, TOFRA, Inc. Palo Alto, California) mounted directly to the base of the microsc ope, which was operated by a home -built data acquisition program in LABVIEW (Dantus Research Group, Michigan State University) [197] . The DR images were acquired for a total of 170 µm, with $z= 0.1 µm step in between each 2D 512x512 pixel image scan. Each 2D image was averaged for 5 seconds. DR imaging was completed using 12 mW of average power. To achieve spectrally separated DR images, the same optical filters were used as those in the single PMT NMMMI s. All images of the H &E stained tissues were obtained using a Hamamatsu slide scanner. The images were then cropped using the Hamamatsu software, NDP NanoZoomer Digital Pathology. 64 4.2.5 !Data processing The TCSPC data files are converted to asci format for processing. The asci -form at data was reshaped into a four -dimensional (4D) array (16 PMTs, 512 pixels in X, 512 pixels in Y, 4 timing bins). The input 4D array is summed across the timing axis to result in a 3D array where a 512x512 image can be rendered for each of the 16 PMTs. All data processing was completed using python codes constructed in -house. For image processing and RGB rendering, the Python Imaging Library (PIL) was used [194] . Single PMT images are processed using ImageJ software (National Institute of Health) [195,196,198] . For 2D images, all of the raw images are up loaded to ImageJ. The Òz -stackÓ function is used to ge t a single, averaged image from the raw image stack. For images larger than ~320 µm (1-512x512 pixel image), the BigStitcher plug -in for ImageJ was used to stitch together images into a larger mosaic an d reduce brightness artifacts around the border of each image [195,196,198] . The only additional image -processing that was done on the 2D images was done using GNU Image Manipulation Program (GIMP) to stitch the multi -image regions together. For DR images, the ImageJ Import.jlm macro was used to compile the raw DR imaging folders into an image sequence of averaged 2D image scans [195,196,198] . This DR image sequence was uploaded to Image J where it can be viewed as gif or rendered into a 3D image. The scan ranges for each image in the DR image panel figures were chosen from the uploaded image sequence. 4.2.6 !Collagen orientation calculations The angle of collagen fibril orientation was performed using the OrientationJ plugin for ImageJ (NIH). The OrientationJ plugin has multiple functions, however, we are using the distribution function of the plugin to calculate the orientations of the collagen fibrils on a pixel -by-pixel basis over a 5 12x512 pixel ROI in normal (healthy), inflammatory, and neoplastic regions within human oral squamous cell carcinoma (HOSCC ) biopsied tissues [199] . The parameter s used in this work had been previously published [200] . Briefly, we used a cubic spine gradient, with a minimum coherency and maximum energy level set to 10%. These thresholding levels were chosen to reduce the likelihood of smaller structures, such as elastin, 65 interfere with the collagen measurements, as well as to limit the number of less isotropic structures dominating the histogram, respectively. The histograms f or each of the ROIÕs were saved and plotted as a circular histogram to enhance the ability to compare one to the other. Circular histograms were plotted using Plotly plotting software (Plotly Technologies Inc., Polar Charts in Python, Montr”al, QC, 2019 UR L: https://plot.ly/python/polar -chart/#new -to-plotly ). Each histogram was repeated in triplicate, averaged, and normalized on a scale from 0 to 1 to better compare the relative shape of the distribution of angles from one classification of tissue to the next rather than comparing the frequency of each present angle in the distribution. 4.2.7 !Statistical analysis The total data size of the human contains n = 55 for normal and n = 101 for neop lastic tissues. We consider each 512x512 pixel image as an individual sample for the statistical analysis performed from the ROIs and multiple ROIs are obtained from each 512x512 pixel image. There are 6 total human biopsies used in this study, some of whi ch have only neoplastic tissues, while others have normal and inflammatory regions in addition to neoplastic regions. One -way analysis of variance ( ANOVA) test was performed on the dataset of the images presented in Figure 35 , with n = 21. We compared four different tissue classifications for this study: normal (n = 3), mild inflammatory (MI) regions (n = 6), severe inflammatory (SI) regions (n = 6) , and neoplastic regions (n = 6). 4.3 !Results and Discussion 4.3.1 !Rapid NMMMI of unstained canine and human OSCC tissues mimic H&E contrast Typical H&E staining achieves distinction between eosinophilic structures, such as the cytoplasm and structural components of the stroma and basophilic nuclei. A hallmark of neoplasia is identifying the distribution, size, and m orphology of nuclei [192,193] . The nuclei in neoplastic cells go through many changes, such as enlarged size, increased mitotic activity, pyknosis (shrunken and dense nuclei), and karyolysis (nuclear clearing) [201] . Some of these abnormal nuclear states can be diagnosed via H&E 66 stai ning. To evaluate if our novel imaging technique can achieve similar contrast as H&E staining, we imaged normal and neoplastic HOSCC biopsies. Figure 33 . H&E and NMMMI s of normal and neoplastic regions of HOSCC biopsies showing comparative contrast to H&E staining achieved with two optical filters. NMMMI s of the unstained normal (a) and neoplastic (b) HOSCC biopsy tissue slices, corresponding to the H&E image, are prese nted as individual channels (green and blue), containing photons from their respective optical filter bandpass ranges, 515 nm -540 nm (SHG) and 605 nm -785 nm (2PEF), respectively, as well as an overlay (SHG/2PEF). Zoomed in regions of the white, dashed boxe s in the 2PEF images show a zoomed -in image of where we identify the nuclei, pointed out by red arrows. H&E and NMMMI scale bar = 50 " m, gray -scale image scale bar = 10 " m. Figure 33 shows NMMMI s of normal and neoplastic HOSCC biopsy tissues with H&E -lik e contrast that were acquired using two different optical filters with a single PMT detector. The wavelength ranges of the two filters are 515 nm - 540 nm (green channel) and 605 nm - 785 nm (blue channel). The combination of these two filters allowed us t o isolate the SHG signals from collagen in the stroma (green channel) and the 2PEF signals from the surrounding cytoplasm (blue channel), which is unattainable via H&E staining. This is due to the structure of the amino acids that make up each collagen fib ril. The non -centrosymmetric molecules yield a process called SHG, which is detectable by photons at exactly 535 nm 67 (the second harmonic wavelength of our excitation source is 1070 nm) [33] . These collagen specific SHG signals allow us to identify the boundary between the stroma and the squamous cell layer, which is useful when invasive neoplastic fronts begin to invade the stroma, as any disruption in the basement membrane facilitates cell migration. While the nuclei are identified by the hematoxylin in an H&E stain (appear blue), our method gives us the ability to detect the nuclei indirectly. We are unable to excite the nuclei with any of our harmonic wavelengths ( #3photon = 353 nm, #2photon = 535 nm); therefore, they appear dark and show a lack of signal in our NMMMI s (red arrows, Figure 33 ). Nevertheless, the combination of signals in the blue channel from the cytoplasm and the lack of signal from the nuclei permit us to detect where t he nuclei are in both normal and neoplastic tissues ( Figure 33 ). Thus, we can quantify and track the ratio of nuclear to cytoplasmic diameter, which is a thoroughly explored method in neoplasia diagnostics. 4.3.2 !Additional histological contrast is achieved using time -correlated single photon counting to explore concealed multiphoton signals in human and canine OSCC biopsies To improve current methods in diagnosis, we also explored two methods by which we increased the spectral differentiation of our NMMMI s. The first method utilizes a TCSPC, which provides spectral differentiability by using a dispersive element to reflect the detected signals across an array of 16 PMTs [38] . Contrary to optical filters, which have broad transmission curves, the TCSPC can be thought of as a Òfine -toot hed combÓ, where each detector has a range of 12.5 nm. The narrow resolution of the TCSPC can separate obscure signals that may be hidden due to a broad band filter. Thus, we used the TCSPC data to identify the wavelength ranges which corresponded to the c ritical changes in local signals of normal and neoplastic HOSCC tissues. 68 Figure 34 . NMMMI of human and canine OSCC biopsies acquired using TCSPC. Brightfield images and single -channel NMMMI s of the corresponding H&E -stained tissue slices from normal and neoplastic OSCC biopsies from human (a,b) and canine (c,d). Each row contains the H&E image of where each NMMMI was obtained, a split -image of the H&E and RGB overlay NMMMI showing the histol ogical accuracy achieved with multiphoton multimodal (MM) excited imaging, and the individual channels of the NMMMI s for each signal type. Wavelength ranges for SHG is centered at 535 nm, for the red channel 2PEF, 550 nm - 580 nm, and for the blue 2PEF chan nel, 610 nm -660 nm. White scale bar = 50 µm. Figure 34 shows the multiphoton images acquired using the TCSPC. Using this set up, we detected a secondary emission in the stroma of the oral mucosa that was not differentiable with our previous set up (Figure 33) or via H&E staining. The secondary signals in the stroma arise from different structures than collagen; therefore, the combination of these new signals with SHG increased the contrast and texture of our NMMMI s. 69 The secondary signals in the stroma of both normal human and canine tissues, u nlike the surrounding collagen fibrils, lack SHG capabilities (centrosymmetric) and have a 2PEF (550 nm - 580 nm) emission . In the healthy tissues, these fibrils are thinner than th eir collagenous neighbors and more dispersed than those in neoplastic regions (2PEF Red, Figure 34 ). Due to the location, morphology, and emission wavelength of the secondary structures of the stroma, we concluded that these structures were elastin fibrils . The differences in multiphoton multimodal (MM) signals and location between the collagen and the elastin fibrils were most apparent in the red -green -blue ( RGB ) split -images in Figure 34 . Furthermore, in both human and canine neoplastic tissues, the coll agen fibrils (SHG, Figure 34 ) tend to retain a longer and more continuous shape whereas the elastin fibrils (2PEF Red, Figure 34 ) are very short and tend to be less grouped than in healthy tissues. This discontinuous pattern observed in elastin fibrils cou ld be due to a time -dependent relationship between elastin -specific matrix metalloproteinase (MMP) activities and progression from a hyperplastic state towards neoplasia. In fact, a tactic that neoplastic cells utilize to promote cell migration and metasta sis involves the use of these MMPs to breakdown the structural components of the stroma [192,202,203] . For example, MMP -12 is an elastin -specific MMP that has been shown to promote elastolysis, creating elastin fragments [204 Ð206]. These fragments further stimulate inflammatory responses, proliferation, and angiogenesis [204 Ð206]. In addition, MMP -12 showed a significant expression change between cases of neoplasia and healthy patients [207] [207] . Thus, based on our results, further studies on MMP -12 as detection tool of the subtle changes in the stroma, particularly in regions adjacent to the basement membrane, is needed. This could aid an early detect ion of neoplastic activity. 4.3.3 !Probing the inflammatory microenvironment of HOSCC using multiphoton excitation shows spectral signal distinction The immune systemÕs response to neoplastic cells is very similar to that of wound healing [193,208 Ð211]. In wound healing, an influx of fibrinogen due to activ ated platelets begin to form a clot at the site of action [193] . Chemokine signals initiate growth factor activation, MMPs, and fibroblasts 70 during tissue granulation for extracellul ar matrix remodeling [193] . Similarly, in the case of invasive carcinomas, neoplastic -associated chemotactic signaling recruits lymphogenic and angiogenic proteins. These mitogenic fa ctors aid in the migration, proliferation, and promotion of neoplastic cells [212] . Contrary to the cellular environment in an area undergoing wound healing, the cellular environment of neoplastic invasive fronts lacks organization [212] . To examine these chemical and microenvironmental changes in tissues, specialized and immunohistochemi cal (IHC) stains are used. To evaluate our NMMMI s ability to reveal the MM and morphological changes in different OSCC cellular environments, we imaged regions of normal, inflammatory (mild and severe), and neoplastic biopsies. Figure 35 . NMMMI s of excised unstained and H&E stained human oral biopsy samples. NMMMI images of normal squamous epithelium and connective tissue of the stroma (a), mild inflammation (b), severe inflammation (c), and neoplastic region (d) with their corresponding H&E bri ghtfield images. White arrows show collagen of the stroma. Pink arrows represent the highly fluorescent 3PEF cells of (b). Yellow arrows show the 2PEF signals from inflammatory cells. The white box in the RGB image of (b) shows where the 3PEF and 2PEF cel ls are present and are independent of one another. White and black scale bars represent 100 µm. 71 In this method, we increased the spectral differentiation of our NMMMI s by utilizing a third optical filter (i.e. 3PEF) with a single PMT detector ( Figure 35 ). While the TCSPC gave higher spectral resolution, the optical elements (optical fiber bundle) used in the setup were less efficient, leading to longer acquisition times. Therefore, we used the information we obtained from our TCSPC NMMMI s in combination w ith the original two -filter NMMMI s to determine which wavelength range would be ideal for adding a third color channel to our single PMT regime. We detected 2PEF signals, similar to those seen in Figure 33 b, from non -cytoplasmic cells and 3PEF signals from larger tissue structures in inflammatory regions (Figure s 34b and 34c). These two signals, in addition to the SHG signals from collagen, were determined to be the defining signals between healthy, inflammatory, and neoplastic tissues. The optical filters separated signals into three wavelength ranges: 340 nm -415 nm (3PEF), 515 nm -540 nm (SHG), and 605 nm -785 nm (2PEF). Figure 35 shows NMMMI images obtained at a regions of normal, mild inflammation (MI), severe inflammation (SI), and neoplasia. We refer to MI regions as those that are furthest away from the neoplastic area, whereas SI regions are those that are neighboring the diseased tissue. Comparable to histology ( Figure 35a), distinct separation of the squamous cell layer and stroma, the uniform nuclear size, and the typical density of nuclei throughout the squamous cel l layer are also detected in the NMMMI images of the normal regions of the oral mucosa. The distinct separation of the stroma and squamous cell layer can be seen when examining SHG NMMMI s of the normal region, where SHG from collagen is only present in the stroma (white arrows), signifying an intact basal membrane. There is a homogeneous 2PEF signal detected in the squamous cell layer and in the stroma surrounding the collagen structures which enabled detection of individual cells. We used the combination o f SHG signals from collagen and 2PEF signals from the cytoplasm to observe the amount and distribution of nuclei from the basal membrane to the superficial region of the layer. The MI, SI, and neoplastic regions of Figure 35 present heterogeneous environm ents, highly contrasting the NMMMI s of the normal oral mucosa. In all three of the non -normal cases in Figure 35 , we see a lack of organization and separation of the connective tissue network from the other cell types and 72 layers (SHG, 2PEF, and RGB NMMMI s). Additionally, the 3PEF signals in areas of MI and SI were prominent, however, these signals were not detected in the areas of normal and neoplastic oral mucosa (Figure 35 ). Using the H&E stained image as a reference, we inspected the nuclei of the 3PEF c ells in Figure 35 b (pink arrows) to determine their identity. We concluded that these cells were lymphocytes due to the large nuclei that crowded the majority of the cytoplasm and the diagnostic state of that region [213] . Moreover, we observed increased 2PEF signals (yellow arrows) from the cells that are not present in the normal tissues. We confirmed that the two - (yellow arrows) and three - (pink arr ows) photon excited fluorescent signals arose from two different cell types and show their distinct locations in the white box in Figure 35 b. We believe that these cells are plasma cells due to their morphology (H&E images, Figure s 35b-35d) as well as previously reported findings that state the presence of plasma cells in the humoral response of the neoplastic infiltrate [214] . Furthermore, in both cases of the proposed lymphocytes and plasma cells, there is a noticeable trend in the density of the MM signals ass ociated with them. The peak quantity of these cells appeared to be in MI regions and declined with disease progression towards neoplasia. We explored this hypothesis by calculating the ratio of SHG to 2PEF for all tissue classifications in Figure 35 . Mean intensity measurements were taken of the 2PEF and SHG signals from all ROIs of each NMMMI (n = 6), and SHG to 2PEF intensity ratios were calculated. The mean SHG/2PEF ratios for normal, MI, SI, and neoplastic NMMMI s were determined to be 0.88 ± 0.44, 0.63 ± 0.20, 0.73 ± 0.17, and 0.93 ± 0.24, respectively. While there was an anticipated numerical trend (SHG/2PEF ratio in MI regions should be lowest due to the high prevalence of plasma cells), there was no statistically significant difference in the ratios a mong the different tissue classifications (one -way ANOVA, p -value = 0.18). 73 Figure 36 . Histogram plots of 6 ROIs from the NMMMI s (a) normal, (b) MI, (c) SI, and (d) neoplastic regions in each pseudo -colored channel of Figure 35 . Typically, diagnosis of neoplasia is done by examining images of tissues or stained slides; however, this method can become complicated in unusual cases. A numerical value or quantifiable pattern used to generalize multiple different stages of neoplasia wo uld optimize this diagnostic process. To further evaluate the ability of our NMMMI s to facilitate neoplastic diagnostics, we examined the histogram plots of the SHG, 2PEF, and 3PEF signals from the NMMMI s of normal, MI, SI, and neoplastic tissue classific ations (Figure 36 ). In normal and MI tissues (Figure s 36a and 36b), we observed 3 -4 distinct histogram modes. Among the 6 histogram plots within each MM signal group, there is little variation. On the contrary, in both cases 74 of SI ( Figure 36 c) and neoplas tic regions ( Figure 36 d), the histograms are tetra -modal and have dissimilarities within each MM signal group. The histogram curves of the SHG signals broadened as health status declined: the averaged histogram width for normal, MI, SI, and neoplastic tiss ues were 11 ± 2 bins, 92 ± 12 bins, 120 ± 7 bins, and 134 ± 8 bins, respectively ( Figure 36 ). We attributed the trend in the shape of the SHG histogram curves to be representative of the increased heterogeneity and disorganization that was observed in the stroma (SHG, Figure 35 ) with disease progression. The 2PEF histogram profile of the MI regions was unimodal w ith a right -skewed gaussian shape. We predict that this mode belongs to the 2PEF signals from the plasma cells within the MI region (2PEF, Figure 35 ) due to the absence of this mode in Figure 36 a and the lack of plasma cells in the NMMMI s of normal tissues . In contrast to normal and MI tissues, both of the 2PEF histograms from the SI and the neoplastic regions were bi -modal with gaussian profiles, with the median bin for each being 49 and 52, and 58 and 64, respectively. Additionally, we observed that these two modes became more distinct and had a greater distance between their peaks (3 and 6 bins respectively) as the health of the tissue continued towards a neoplastic state. We attributed the increased intra -modal distance in the 2PEF histograms of SI and neoplastic tissues to the increased prevalence of neoplastic cells in those regions (2PEF, Figure s 35c and 35d). While the 3PEF histograms for normal, MI, and neoplastic tissue regions were all unimodal with a relatively gaussian profile (41 ± 3, 73 ± 3, 17 2 ± 38 bins, respectively), the 3PEF histograms for SI regions were bimodal with narrow distributions (median bins = 42 and 46, histogram widths = 69 bins and 73 bins). We related the bimodality and narrow distribution of these histograms to the combinatio n of inflammatory and humoral responses, wound healing patterns (myofibril regeneration and stroma hyalinization), and neoplastic cell presence taking place in the SI area ( Figure 35 c). 75 4.3.4 !Examination of collagen fibril orientation in HOSCC stroma shows dis tinct patterns in normal, neoplastic, and inflammatory regions of HOSCC biopsied tissues Figure 37 . Collagen orientation in normal, MI, SI, and neoplastic regions of HOSCC biopsied tissues. a) CCD images of brightfield H&E stained slices of normal, MI, SI, and neoplastic regions of HOSCC biopsied tissues. Colored boxes represent ROIs where NMMMI was ob tained from, n ROI =3 for each condition. b) NMMMI of unstained HOSCC biopsy tissues (515 nm -535 nm). Scale bar = 50 µm. c) Collagen orientation polar plots for each tissue classification. The colors represent the appropriate ROIs from (a). Changes in the macroenvironment of the tissue structure is an additional hallmark of carcinoma diagnosis. A known identifier of neoplastic activity is the integrity of the basement membrane, which acts as the boundary between the vascular environment of the stroma and the squamous cell layer [193] . To 76 evaluate our imaging technique as a method to track changes in the structure of collagen with disease progression, we employed collagen angle measur ements on our SHG NMMMI s. In contrast to neoplastic regions, we have shown how the matrix network of the stroma in healthy tissues is highly ordered, where collagen and elastin fibrils form continuous and conjoint groupings. We measured the orientation of the collagen fibrils from normal (healthy), MI, SI, and neopl astic tissues to provide additional merit to these qualitative findings. Collagen orientation measurements were performed on three ROIs from each tissue classification category using the OrientationJ plug -in for ImageJ (NIH) (Figure 37 b) [195,196,200] . The green, magenta, purple, and red squares in Figure 37 a correspond to one of the locations where the NMMMI s were acquired for the normal, MI, SI, and neoplastic NMMMI s of Figure 37 b. Figure 37 c shows the results of these measurements with the intensities of the histogram plots normalized on a scale from 0 to 1. The color of each row of orientation plots corresponds to the tissue classification NMMMI they were obtained from - the green, magenta, purple, and red orientation plots correspond to normal, MI, SI, and neoplastic regions. Each tissue classification shows a characteristic collagen orientation angle pattern, dependent on the health status of the tissue. We observed a trend toward s a more negative orientation angle as the tissues diverge from a healthy state. The majority of the collagen fibrils in healthy tissues (green polar plots) had an orientation between 0 ¡ and 90 ¡ and neoplastic collagen fibrils (red polar plots) had an orie ntation between 0 ¡ and -90¡. Depending on the severity or proximity to the neoplastic regions, the distribution of collagen orientation angle differs drastically. In cases of mild inflammation (magenta plots), there is a near 50% distribution of collagen o rientations between -90¡ and 90¡; whereas, inflammatory regions adjacent to neoplastic cells - SI regions (purple polar plots) - show orientation angle distributions that are very similar to those of neoplastic areas (red polar plots) ( Figure 37c). We calc ulated the ratio of negative to positive orientation angles for each of the tissue classification to be 52% ± 29%, 104% ± 9%, 105% ± 18%, and 216% ± 123% for normal, MI, SI, and neoplastic tissues, 77 respectively. Although our measurements of the ratio of neg ative to positive angle measurements matched the trend we observed, where we see an increase in negative collagen orientation angles with a decline in the health status of the tissue, the results showed there was no statistical significant difference among the groups (one -way ANOVA, p -value = 0.07). 4.3.5 H&E -like contrast achieved at depths imaged directly from the paraffin block Figure 38 . Representation of 2D images at different depths of a 3D depth -resolved stack from a human tumor embedded in a paraffin block. A sampling of images at a range of depths are shown. The rows of panel images are shown at depths ranges with the increasing depth; 1 -12.9 µm, 24.8 -27.4 µm, 44.6 -47.8 µm, 94.5 -98.8 µm, and 121.8 -135.5 µm. The wavelength ranges corresponding to the pseudo -colors used shown at the base of each column in the panel arrangement. The magenta images contain photons from all multiphoton signals detected. 3PEF a nd 2PEF images show the fluorescence from inflammatory response cells. The green channel (SHG) shows strong signals from collagen. Scale bar = 100 µm. 78 Traditional histopathological preparation techniques have had very few changes over the past century and continue to be a costly process [215] . Unfortunately, the lack of reliable and diverse alternative options prevents any portion this process from being replaced. While there are additional options for unstained imaging, the increased scattering properties of other excitation wavelengths (800 nm) make them less ideal for volumetric imaging of minimally prepped biopsies or for in-vivo situations. To investigate our excitation source (1070 nm) as a candidate for future in -clinic imaging, we performed volumetric imaging of FFPE biopsied tissues directly from the paraffin block. Figure 38 shows a representative view of the depth -resolved images achieved using the 1070 nm excitation source with single PMT detection. The volumetric stack was imaged directly from the paraffin block. The full depth -resolved panel arrangement can be seen in Figure SX. Each volumetric image was obtained as a stack of multiple, contiguous 2D slices to reduce noise, with the depth range for each 2D image shown along the y -axis. The location where the DR NMMMI s were obtained is shown by the purple square on the H&E stained SI tissue slice in Figure 3 7a. To directly compare the different multiphoton signals detected from the biopsy, the summed volumetric image stacks are arranged in rows for each of the multiphoton signal sources detected. The magenta pseudo -colored images in Figure 38 contain the imag es detected using the optical filter with the broadest transmission spectrum (465 nm) (All signals, Figure 38 ). Additionally, the 3PEF NMMMI s are pseudo -colored as blue and second harmonic generation (535 nm) signals from collagen are pseudo -colored green. Furthermore, the remaining signals that result from 2PEF are pseudo -colored as red. The final column in Figure 38 contains the RGB overlay of the 2D images acquired with the 3PEF, SHG, and 2PEF optical filters. Importantly, we show that we are able to ret ain contrast at increasing depths using our imaging technique. We observed the individual collagen bundles at depths approaching 130 µm due to their strong SHG signal. Additionally, we were able to resolve individual plasma cells at greater depths (>150 µm) than individual collagen bundles due to the 2PEF signals from 79 these cells. Moreover, the RGB overlay images ( Figure 38 ) nearly matched the 2D depth images obtained using our broad bandpass optical filter (All signals, Figure 38 ). Figure 39 . Volumetric rendering of depth -resolved NMMMI of HOSCC biopsy obtained directly from the paraffin block. a) (top) Amira 3D rendering of the 2D image stacks of the human OSCC tumor as a YZ-plane flat face, Normalized intensity plots for each MM signal are plotted along the y -axis of (a). The color of each plot corresponds to the pseudo -colored channel for each signal type. (bottom) angled YZ-plane to show the joint of two different YZ -planes. Yellow dashed lines show where the hyalinization can be identifi ed indirectly. White scale bars indicate 50 µm. b) Collagen orientation angle polar plot for the depth -resolved images in the green channel of Figure 38 b. In Figure 39 a we show the 3D rendering of an OSCC tumor imaged directly from a paraffin block. We utilized only the SHG and 2PEF channels from Figure 38 in the 3D rendering due to the low intensity of 3PEF signals that were present at greater imaging depths. The substantial decrease in 3PEF that we observed was due to the mathematical relationship b etween MM signal intensity and imaging depth; where the signal strength of multiphoton signals that is detected decreases exponentially with the distance from the focal point. This exponential relationship is highly dependent on the order of the multiphoto n process 80 as well as the imaged media [26,33,216 Ð219]. Nevertheless, the contrast that was achieved by the strong 2PEF signals from the plasma cells and the SHG signals from the surrounding collagen allowed us to identify the hyalinization bu ndles which were previously identified by 3PEF (3PEF, Figure 35 ). The borders of these bundles are outlined with a yellow -dashed curve that is shown in Figure 39 a. The intensity profiles for each MM signal which were used in the rendering ( Figure 39 a) are shown in the plot of Figure 39 b. The color of each of the curves in Figure 39 b corresponds to the pseudo -colored channel for each MM signal - the green and red curves belong to SHG and 2PEF, respectively. Although SHG is a multiphoton process and th e signal intensity decreases exponentially, we are able to detect these signals from collagen at our most inferior depths (approaching 200 µm). This allowed us to perform the collagen fibril orientation angle measurements and track the trend of the collage n fibril orientations in a volumetric regime. The polar histogram plots of Figure 39 c contain the collagen orientation angles that were calculated at each of the imaging depths presented in the NMMMI s of the SHG channel of Figure 38 . The histogram traces are all plotted on a single plot for a direct comparison of orientation angle distributions at depths farther into the bulk of the biopsy. Each of the histogram measurements were taken in triplicate, averaged, and normalized from 0 to 1 . The color gradient of the polar plots in Figure 39 c corresponds to a specific imaging depth; the lightest magenta line plot corresponds to the most superficial image of Figure 38 , and the dark purple polar plot belongs to the most distal image of Figure 38 . We observed pol ar plot patterns in Figure 39 c that were comparable to those that were obtained at more shallow depths in similar inflammatory regions in Figure 37 c. Interestingly, the measurements that were obtained at depths farther into the bulk of the tumor, resembled those of healthy tissue regions. This finding, in combination with the decrease in plasma cells and lymphocytes present in the 2 - and 3PEF channels of Figure 38 , lead us to the conclusion that this can be attributed to the lack of inflammatory events taki ng place at this depth i.e. the inflammatory response is more superficial. This may be potentially useful in determining tumor margins on a more precise scale in a volumetric capacity. 81 4.4 Conclusions Oral squamous cell carcinoma (OSCC), a form of head a nd neck cancer, is responsible for roughly 2-4% of cancer cases and can show little to no symptoms. If left untreated, OSCC can be fatal. Current methods of diagnosis involve time and cost consuming process that not only involves an uncomfortable biopsy pr ocedure but also costs histology departments resources to prepare, stain, and mount these biopsied tissues on slides for a pathologist to manually inspect and interpret. By using the human eye as a diagnostic technique, the risk of an improper diagnosis i s high. Here, we have shown how the use of NMMMI can be a useful tool to augment the current gold -standard of neoplasia diagnosis, H&E staining. In addition to matching H&E -like contrast in unstained excised tissues, we show our ability to distinguish elastin fibrils from collagen fibrils in the stroma and note changes in their structure and distribution in neoplastic tissues when compared to healthy tissues. Using these MM signals, our imaging technique can serve as a more direct way to evaluate the status of the basement membrane. Furthermore, we exci te and detect additional 2PEF and 3PEF signals from cells in mild and severe inflammatory regions we attribute to plasma cells and lymphocytes, respectively. Our inability to excite the nuclei of the squamous cell layer warrants our ability to distinguish the boundaries between nuclei and the surrounding cytoplasm. The combination of direct and indirect detection makes it plausible to employ existing statistical measurements used in diagnostics, on images acquired with our new technique. Furthermore, our 10 70 nm excitation source permits the ability to image FFPE HOSCC biopsies directly from the paraffin block and maintain the same the multi -spectral contrast in depth -resolved images as in 2D NMMMI s obtained from slide -mounted unstained tissue slices. Our NMMMIs of unstained tissues can be used for qualitative and quantitative measurements, including the orientation angle measurements of collagen fibrils in 2D and depth -resolved images of healthy, MI, SI, and neoplastic regions. We show a pattern in the chang e in collagen angle orientation distinct to the state of the tissue. We note that these angles trend towards negative values as the region of interest is in closer proximity to neoplastic regions. We conclude that with continued studies and improved instru mentation, the combination of endogenous 2PEF and 3PEF signal classification in inflammatory regions, evaluation of elastin fibril changes, and collagen orientation angle measurements show promise as a method of 82 augmenting current gold standard HOSCC diagn ostic protocols as well as increasing the likelihood of early detection. 83 Chapter V Towards spectral unmixing of multiphoton multimodal images of unstained retinas using user -defined signal sources and inverse problem solving algorithms Abstract In this chapter, we begin by evaluating the most commonly used method of solving the inverse problem of the NMMMIs of unstained mouse retinas, least squares fit. The initial performance of the algorithm was evaluated on virtual phantoms that mimic the fluo rescent profiles and heterogenous histological patterns observed in the NMMMIs where it was determined that the algorithm was less effective on distinguishing signals when there was significant overlap in the emission spectra of the source compounds, parti cularly in the case of FAD and NADH. We applied the algorithm to the NMMMIs of mouse retinas, where it was shown that the LSqF algorithm successfully resolved the photon sources for collagen in the sclera as SHG, melanin disks in the IRL as melanin, and v isual cycle catabolites in the ORL as both concentrations of A2E. We evaluated two other inverse problem solution algorithms, lasso and linear regression. Lasso was more effective at separating signals with significant overlapping spectra, e.g. NADH and FA D, however it was not accurate in assigning photons in the IRL and ORL, where nearly 13% (~2,000/15,360) of the photons were assigned to FAD and less than 7% (~1000/15,360) of photons within these regions were assigned to melanin and A2E Ñthe main molecular components of those layers. The linear regression algorithm only assigned ~7% of the photons within the IRL to melanin, additionally, it did not incorrectly assign the photons in the ORL to melanin as was done by lasso, and accurately assigned them to A2E . Furthermore, linear regression model incorrectly assigned ~13% of the melanin photons to the sclera, in regions surrounding the collagen bundles as well as within them. Furthermore, we conclude that the LSqF algorithm is best suited for solving the inve rse problem of the NMMMIs from mouse retinas. With additional source signals provided, we suggest that using LSqF algorithm will become valuable technique to accurately assign and spectrally un -mix all endogenous photons on the pixel -level from unstained t issues. Moreover, this technique can be used as a baseline for endogenous signals in healthy tissues with the potential being used to diagnose abnormalities on a molecular level. 84 5.1 Introduction In microscopy, fluorescent staining is used to segment types of cells and proteins within biological tissues. Achieving this type of segmentation is possible in unstained tissues using ultrafast lasers and NMMMI . In this type of microscopy, endogenous fluo rescence and harmonic processes within the tissues produce a signal that can be used to segment and identify different tissues within the image [8,220 Ð222]. Biological compounds often have broad emission spectra that occur within the visible range of the electromagnetic spectra (Figure 40 ). While unstained imaging is often desired because it is less resource intensive, difficulties with accurately distinguishing adjacent emission peaks can arise due to the overlap of emission spect ra, which in some cases, only a few nanometers separate one peak from the other. This can be seen in Figure 40 , where the wavelength spectra of pure fluorophores in solution and harmonophores are plotted. There is a significant distance between the elastin (light pink) and the A2E 1mM (red) emission peaks, whereas the emission peaks of NADH (orange), FAD (green), and melanin (magenta) are almost completely overlapping. Figure 40 . Wavelength spectra of pure fluorophores and harmonophores found within huma n retinas. Solutions of pure fluorophores are excited with the Yb -fiber laser at the focal point of the microscope objective. Photons are detected using the TCSPC, where each PMT detector collects wavelength data over a 12.5 nm range, These signals are rec onstructed into 2D array using a Python script. Each spectra is normalized from 0 to 1 to show the relative overlap and distance between neighboring peaks. Notice that many of the compounds overlap significantly, making difficult to determine their individ ual contributions. 85 Currently, a data acquisition method that can help with this issue is a TCSPC. The TCSPC disperses the collected emission signals across an array of PMTs using a grating & [38] . While this technique provides the user with the wavelength of the detected photons at each pixel (B), differentiation of the emission peaks is limited by the physical capabilities of the detector , and i n the case of the TCSPC, th e restricting element is the ~12.5 nm distance between cathodes. Here, we use a generic mathematical relationship to represent our scenario ; where the emission wavelength of the collected photons is known (via TCSPC collection) and most of the sources of those signals can be assumed and used as a reference for fluorescence (A), the remaining unknown is the contributi on from each of these reference signals (X) at each pixel within the image matrix. #%"& (9) This type of problem is commonly known as an inverse problem & [70 Ð72]. Because there are many possible solutions to the above equation, it can be difficult know ing which solution is correct or best for the specific situation. However, in this case, it can be assumed that there are a finite number of fluorescent sources at each pixel and therefore the solution to X is sparse. If we consider the following equatio ns: #%"& (10) #'()*+ ,-."/ %'()*0 ,-1232. &,'(4*+ Where A is a square matrix containing the pixel data of the multiphoton multimodal image of an unstained retina, with dimensions, m and n, equal to the pixels in x and y, or 512x512 pixels. The component matrix, X, contains the source signals present on the map of A, and has dimensions that are dependent on the amount of source signal vectors known. Though the MP signal processes we are detecting are nonlinear in nature, the fact that the signals in A are known and only their parameter variables (X) are unknown, this problem is linear in a statistical sense [223] . B is the resulting image matrix where each component of B can be written as the linear combination of multiphoton signal sources and their appropriate weights of 86 unstained retina images determined by the product of A and X. The linear combination can be wr itten as follows: !5"#65",7$8598)8:4 (11) We use these concepts in attempts to spectrally decompose the endogenous signals from unstained retinas. Our approach uses least -squares fit in order to determine the weighted linear combination of signals at each pixel within the image matrix, that results in the lowest error [65,68,69] . Because biological t issues contain numerous molecules and structures with fluorescent and harmonic properties, the component matrix is difficult to complete, resulting in a case where {x i} is not a basis for C l x n , and therefore, only maps a subset of B. The remaining unmap ped signals of A are denoted as r. !";<,7=$>$>?@9)8:4 (12) Using these assumptions, in this work, we applied compressed sensing techniques to our reconstructed NMMMI s as well as virtually stained phantom images and used the results to evaluate which inverse problem -solving algorithm is ideal for this specific case. While our component matrix of endogenous signals is not complete, we show partially unmixed images of unstained retinas. Additionally, we show the remain ing image that is based on r, or the remainder of unmixed multiphoton signals. 5.2 Materials and Methods 5.2.1 Laser setup, detection, imaging, signal processing The laser source for our multiphoton microscope is a custom designed Yb -fiber laser oscillato r generating pulses with sub -40 fs pulse durations (full -width half -maximum) (1.07 µm, 42 MHz) [224] (Figure 41 a). Imaging was done on a Nikon TE2000 multiphoton inverted microscope ( Figure 41 b). A 40x water immersion (Carl Zeiss !, Immersol ! W, n e = 1.334 (23 ¡C), v e = 72) objective was employed with a working distance of 0.5 mm (Zeiss LD -C APOCHROMAT 1.1NA, Jena, Germany) to focus the beam on the tissue to a beam waist (beam diameter at the focus) of ~0.5 µm, favoring the generation of peak intensities high enough to induce multiphoton processes with less than 3.2 mW of average power and pulse 87 durations of 38 ± 1 fs. High peak intensities and low average power minimize the effects of photobleaching and thermal damage. The peak intensity was maximized through the use o f a pulse -shaper (MIIPS HD, BioPhotonic Solutions Inc., East Lansing, MI, USA) to compensate for the high -order dispersion along the beam path [225 Ð227]. Laser scanning on the tissue was done with galvanometer mirrors. 5.2.2 Detection Signal detection was accomplished in the epi -direction with a time -correlated single photon counter (TCSPC, SPC -830, Becker -Hickl, GmBH). The TCSPC is able to collect frequency and time -resolved data with a compact spectrometer and a 16 - PMT [228] (Figure 41 c). The spectral resolution of the TCSPC system is ~12.5 nm, limited by the physical size of each PMT in the array. For each pixel in the imaging region (512 pixels x 512 pixels in this work), the emission wavele ngth and lifetime decay data of the detected photon are collected. The grating in the spectrometer was rotated to select different spectral regions. Here, two grating positions were used to collect the fluorescence spectra; one with a collection range from ~300 nm to ~500 nm and the other from ~480 nm to ~680 nm. The images of the sliced mouse retinas obtained using the TCSPC were imaged with 6.9 mW of power or less for a total exposure time of 4.5 minutes (90 seconds per grating position). 88 Figure 41 . a) Yb -fiber oscillator and beam path. b) Nikon TE2000 inverted microscope and single PMT. c) Time -correlated single photon counter (TCSPC). Showing the orientation of the retina slice (green circle). The laser beam propagates perpendicular to the retina slice. 5.2. 3 Tissue Transverse slices of mouse retinas and a Cynomolgus monkey retina flat mount were mounted on a microscopy slide (Corning) and was positioned perpendicular to the propagation of the laser beam. Information regarding how the slides were prepared and how the retinas were harvested can be seen in the original publication on the nonlinear multiphoton multimodal microscopy imaging of these retinas by Murashova et.al [8] as well as in Chapter 3 of this dissertation . 5.2. 4 Preparation of reference solutions Preparation of reference solutions Reference solutions of FAD and NADH were prepared by performing a 1:10 dilution of 10X PBS (Dot Scientific Inc.). The dilution was performed using MilliQ water. Thirty milligrams (mg) of each compound was added to separate solutions in PBS (136 mM NaCl, 2 mM KCl, 8 mM Na2HPO4, 1mMKH2PO4, pH7.4) yielding concentrations of approximately 350 µM for NADH (Sigma Aldrich) and 300 " M for FAD (Sigma Aldrich). An A2E reference solution was prepared using a 10 mM stock solution diluted in dimethyl sulfoxide to yield a final 1 mM concentration. A drop of each solution was placed on a microscope slide (Corning) followed by a coverslip (Corning) and then placed 89 coverslip -side down on the microscope objective. Water b ased immersion fluid (Zeiss, ImmersolTM, ne = 1.334 (23¡C), ve = 85) was used to increase the numerical aperture (NA) between the coverslip and the microscope objective. 5.2.5 Preparation of Reference Signal Matrices, NMMMIs , and Phantoms for LSqF algorithm Prior to being fed into the least -squares fitting (LSqF) algorithm, the spectral wavelength plots for each of the fluorescent or harmonic sources was transformed into a nxm matrix, where n was equal to the amount of total source spectra and m was equal to the length of each array. All data, including the spectra for the source signals are obtained from the TCSPC, where the detected wavelength profile is built up at each pixel in the 512x512 -pixel array, for the entire wavel ength range. Therefore, in order to construct the reference signal matrix, the 512x512 -pixel image was converted into a 1x 262,144 arra y using the ravel() function from the Python Numpy library [229] . The resulting signal source matrix was of shape 6x262,144. The NMMMIs were obtained at two different grating positions, which expanded the wavelength range of photons detected for each image. Therefore, we initially collected photon data over 32 PMTs, where there were only 4 PMTs with overlapping wavelength ranges, once the overlapping data was emitted, we resulted in data from 28 continuous PMTs , and a wavelength range of nearly 400 nm . In addition to the reference signal spectra, the NMMMIs were also transformed from a 512x512 -pixel image into a 262,144 -length array , for the 28 PMTs, resulting in a 28x262,144 -length array. After being fed the NMMMIs, t he algorithm returned a 6x 262,144 -length array, where each column (n) had the photons assigned over the entire wavelength range for one of the source signals ÑNADH, FAD, melanin, SHG, A2E1mM, A2E10mM, or elastin. Each 262,144 -length array was reshaped, usin g the reshape() Python Numpy function, into a 512x512 -pixel image [230] , resulting in one 512x512 -pixel NMMMI for each of the signal sour ces. In order to construct phantoms, select pixel ROIs from a 512x512 -pixel matrix of zeros, were assigned values from the linear arrays of the pure compound spectra. The values were normalized on a scale of 0 to 1, so that the concentration of the compou nd, and thus the quantum yield of the signals, would not interfere with evaluating the baseline performance of the LSqF algorithm. ROIs were chosen to best 90 represent the histological shapes and organization seen in biological tissues, where there were vert ical and horizontal bar ROIs of various widths as well as square ROIs. In addition to having single -source phantoms, where the ROIs contained data from only one type of spectral source, a heterogenous signal phantom was also constructed. The heterogenous p hantom, containing signals from all sources ÑNADH, FAD, melanin, SHG, A2E1mM, A2E10mM, and elastin, had a vertical bar ROI for each of the signals. The width of these vertical bars were also not homogenous. Just as with the NMMMIs, the phantoms were first t ransformed into a 1D array prior to being fed into the LSqF algorithm. The resulting output from the algorithm was reshaped into a 512x512 -pixel matrix to show the photon assignment for the ROIs for each of the single signal source phantoms as well as for the heterogenous phantom. 5.3 Results and Discussion The method by which the photons are organized in the images obtained by the TCSPC can be thought of as a binning process, where each PMT is a bin, with a width of 12.5 nm. The wavelength range of the photons detected at each PMT is dependent on the angle of the dispersing grating, where the total wavelength range dispersed across the array of 16 PMTs, at one grating position, is approximately 200 nm. By acquiring images at two grating positions, we cover close to the entire visible range (300 nm Ð 700 nm) of the electromagnetic spectrum. Images obtained using the TCSPC are rendered on a single PMT basis and showed in the 7x5 (row x column) image plot of Figure 42 . The wavelength range of the 35 images begins at 302 nm and ends at 684 nm. 91 Figure 42 . Single PMT NMMMI s from the TCSPC . Each image contains photons over a 12.5 nm wavelength range. Photons within t he top left image are centered at 302 nm. The wavelength ranges for each image increase towards the right and down, where the final image of the panel (bottom right) has photons centered at 683 nm. Images are 512x512 -pixels (~320x320 -µm) in size. White scale bars are 50 µm. In addition to providing histological contrast, plotting all of the individual PMT images ( Figure 41 ) allows us to identify over which wavelength ranges there are distinct multiphoton processes. In Figure 42 , we show where these distinct multiphoton signal s are detected within the single PMTs from Figure 41 . We 92 see two independent three -photon processes at 353 nm and 475 nm within the area of the choroid and RPE as well as the IRL and ORL. Interestingly at 535 nm, which is the second harmonic wavelength of our excitation source (1070 nm), we see a very diverse multiphoton environment. In addition to the SHG detected in the sclera of the eye, we detect a two -photon (2P) processes in the inner receptor layer ( IRL ) and outer receptor layer ( ORL ). Sum -frequency excitation with 1070 nm (535 nm) can result in second harmonic processes, which occur in non -centrosymmetric structures, and 2PEF processes, however, there can also be three -photon excited fluorescence detected at this wavelength. The stokes -shift of a flu orescence process is dependent on the molecular structure of the compound, which is one of the reasons for the varying width of emission spectra of fluorescent compounds. Therefore, the multiphoton (MP) processes detected in the IRL and ORL can be the comb ination of new 2PEF signals with the 3PEF processes detected at 475 nm or these MP processes may be from a single fluorescent compound with the two emissions, however, for the scope of this work we refer to them as two independent MP processes. At 660 nm w e detect a strong 2PEF process in the retinal pigmented epithelium ( RPE ) and choroid, as well as a secondary MP process in the sclera distinct from the SHG of collagen. Figure 43 . NMMMI s with distinct fluorophore signals characteristic to healthy retinas. The wavelength range covered in each image is shown above each tile. Images are 512x512 -pixels (~320x320 -µm) in size. Note the distinct formations identified in each image by their characteristic fluorescent and harmonic signals. White scale bars are 50 µm. To improve on the qualitative methods that show the distinction between the chosen characteristic MP signals from Figure 42 , we assign a specific color to each wavelength range. In Figure 43 , the three -photon ( 3P) process signals, centered at 353 nm, are assigned as blue, the 2P signals centered at 535 nm as 93 green, and the additional 2P processes centered around 660 nm as red. Areas where two or more of the signals are present results in a pixel color that is the combination of the RGB false -colored channels. Interpreting these pixel colors allows us to qualitatively determine the dominating MP signal in that layer of the retina. In Figure 44 , we see that the MP processes from the IRL are the combination of signals in the wavelength ranges of 4 75 nm Ð 550 nm and 625 nm Ð 675 nm. Unlike the yellow pixel color representing the combination of signals detected in the more distal layers of the retina (ganglion cell layer Ð nerve fiber layer), the color of the pixels containing the IRL data have a hi gh contribution of signals over the wavelength range of 625 nm Ð 675 nm (green). The locations in the retina where the combination of different MP processes are detected is related to the function of the cells or structures within that layer. In structura l locations of the eye, such as the sclera, we detect discrete MP signals that are separated by histological barriers, such as the SHG signals from collagen (green) and the 2PEF processes detected surrounding individual collagen bundles (red). In regions w here there are multiple cellular processes taking place simultaneously, such as in the receptor layers (RL), where incoming light is focused on melanin disks in the outer segment of the RL (ORL), and cellular metabolism takes place in the inner segment of the RL (IRL), as part of the visual cycle, we detect a heterogeneous MP environment due to the multiple fluorescent compounds present. 94 Figure 44 . Three colored (red, green, blue) composite multimodal images of the retinal layers from a 7 " m slice of a mouse retina taken with the TCSPC at 6.9 mW of power depths using a 1.07 " m Yb -fiber laser with 35.0 fs pulse durations. Image acquisition was done at 30 -second intervals for a total of 4.5 minutes. (a) Here the blue, green, and red channels represent emis sion centered at 535 nm, 575 nm, and 629 nm, respectively. (b) The blue, green, and red channels represent emission centered at 355 nm, 535 nm, and 629 nm, respectively. The bandwidth of each channel is ~37.5 nm. We have shown how the spectral components of the retina that are detected by the TCSPC can be used to provide contrast to these unstained tissues in MP images. However, the capabilities of the TCSPC are limited by the resolution and physical size of the detectors. In combination with the broad em ission spectra of fluorescent compounds, accurately identifying the individual MP signal contribution to each pixel is challenging. The computational techniques we use in efforts to determine the signal contribution of each fluorophore within our MP images are modeled as a system of linear equations. We aim to solve for the weights of each fluorophore on a pixel -by-pixel basis using the signals detected in healthy retinas in combination with the normalized emission spectra of pure compounds found within the retina. We use the pure compound signals as standards in this problem and are shown in the top plot of Figure 45 . The emission spectra, corresponding to the fluorescent and harmonic signals detected in the healthy retinas from the MP images (Fig ures 42-44), are normalized and plotted in the bottom plot of Figure 45 . In both plots, all of the signals are normalized from 0 to 1, while this type of normalization prevents us from accurately taking into 95 account the relative intensity of one fluorophore to the n ext, the overlap of multiple signals at each wavelength becomes more apparent. Additionally, the emission plots for the MP images of the retina are plotted so that each curve corresponds to a single layer or region of the retina, showing how the MP signal contributions differ depending on the layer. Figure 45 . (top) Normalized emission curves of the pure fluorophore and harmonic standards obtained from the TCSPC. (bottom) Normalized fluorescence and harmonic emission spectra acquired from the MP signals detected within excised retinas using the TCSPC . In biological imaging techniques where fluorescent tags are used to target specific cells, phantoms are used to establish a baseline for the fluorescent signals detected. We adopted a similar approach to establish the baseline of the least -squares fitting (LSqF) algorithm used in our study. We used the fluorescent and harmonic profiles of the pure compounds ( Figure 45 top) to create virtual phantoms that incorporate a variety of different patterns and that mimic the dimensions of our MP images and are shown in Figure 46 . We created virtual phantoms that only contain one MP signal within a specified region of interest (ROI), such as the FAD, NADH, Melanin, A2E (1mM), A2E (10mM), SHG, and Elastin Phantoms, 96 as well as virtual phantoms that contain all of the MP signal processes with different sized ROIs (FNMA1A10SE Phantom) ( Figure 46 ). Figure 46 . Generated image matrices to act as virtual phantoms . Each matrix is 512x512 -pixels to match the NMMMIs. The contr ibuting signals within each phantom are shown above each image. In the case of the single -source phantoms, only one type of fluorescent or harmonic signal occupies the ROI, noted by yellow pixels. The phantoms are normalized on a scale of 0 to 1, where a value of 1 is represented by yellow color ing. The heterogenous phantom contains fluorescent and harmonic signals from all of the sources, organized into vertical ROIs. White scale bars are 50 µm. FAD is the dominating signal source in all cases where the LSqF algorithm is applied to the virtual phantoms. In Figure 47 we explore the error in photon source assignment in cases where both single MP source virtual phantoms and the heterogenous phantom were used. The top row of Figure 47 shows the virtual phantoms used as the input for the LSqF algor ithm, the output images are the returned results from the algorithm, or, in the case of the heterogeneous input phantom, the output image is just one of the resulting images, and the bottom row shows the resulting error images where incorrect photons were assigned. In the case of mono -MP source virtual phantoms, photons were incorrectly sourced to FAD, however, the magnitude of the incorrectly identified photons was directly related to the intensity of the true source emission spectra where it overlapped wi th FAD, i.e. the emission spectra of melanin is more similar in shape and intensity to FAD than A2E (1mM), hence the larger percentage of melanin photons assigned to FAD than A2E (1mM) photons assigned to FAD. The heterogeneous virtual phantom resulted in a 97 multitude of incorrectly assigned photons, we further examine this virtual phantom more in depth in the following section. Figure 47 . Images showing the output and the error in the photon assignments for each type of input phantom. The top row shows the input phantoms that were fed into the LSqF algorithm. The second row, titled ÒOutput ImagesÓ shows the resulting image from after the LSqF algorithm, we see that in the case of single -source phantoms, most of the signals are properly assigned, i.e. mos t of the pixels in each ROI are equal to 1 (yellow). The bottom row presents the error, or photons that the LSqF algorithm incorrectly assigned . Here we show that in all cases, photons were incorrectly assigned to the FAD source, where the FAD photons were all incorrectly assigned to the NADH source. White scale bars are 50 µm. For virtual phantoms with only one fluorophore or harmonophore present, the algorithm was able to correctly identify the MP signal source in most cases. Interestingly, this was not the case for the virtual phantom containing all of the fluorophores and har monophores. Figure 48 shows the output image results from using the least squares fitting algorithm on the heterogeneous virtual phantom. Each image in Figure 47 shows the prediction of which ROIs had photon signals belonging to the MP signal source being examined (examined source is shown in the title above each image). The final image of Figure 47 shows the summation of all the photons from the resulting image outputs. Each ROI has a total value of 1, confirming that each photon has been accounted for, th ough some are incorrectly assigned. All photons belonging to FAD were incorrectly assigned to the NADH source, this could be due to the significant overlap between the peaks of the FAD and NADH spectra or due to the 10-pixel distance between the ROIs 98 for e ach fluorophore. Interestingly, a combination of <20% of photons from Melanin and A2E (1 and 10mM) were wrongly assigned to FAD. Furthermore, we see partial incorrect photon source assignment in the Melanin and A2E (10mM) phantoms, where <10% of photons be longing to SHG were assigned to Melanin. Additionally, we see <5% of Melanin and A2E (1mM) photons were incorrectly assigned to SHG and <5% of Melanin, A2E (1mM), and SHG signals assigned to A2E (10mM) sources. We believe that the incorrect photon assignme nt is due to the significant overlap between the source emission profiles as well as the relative intensity of the profile peaks where those overlaps occur, such as in the case of the SHG, Melanin, and both A2E emission profiles in Figure 45 , where each si gnal has a peak that overlaps with the bulk of the SHG spectral profile. Figure 48 . Output images of the LSqF algorithm with the FNMA1A10SE virtual phantom image ( Figure 46) as input. The top and bottom rows show the output of the LSqF algorithm for t he heterogenous source phantom, where each image shows the photons assigned to a single source, identified by the source name above each image. FAD was incorrectly assigned as the source for A2E 1mM and 10mM as well as Melanin photons, and all of the FAD p hotons were incorrectly sourced to NADH. White scale bars are 50 µm. When the LSqF algorithm was applied to the MP images of the retina, we see similar MP patterns as in the manually -segmented MP images in Figures 43 and 44. Similar to the combination of 2PEF and SHG signals detected in the sclera of Figures 43 and 44, the resulting single -source MP images of Figure 49 show signal 99 contributions of SHG and A2E (1mM and 10mM) in the sclera. Collagen is a known second harmonic generating structure and the main structural component of the sclera; therefore, we believe that the photon assignment of SHG in the scleral region is accurate. The strong 2PEF signals detected in the IRL and ORL of the retina (Fig ures 43 and 44) are attri buted the melanin, A2E (1mM) and A2E (10mM) by the LSqF algorithm which aligns well with the function of these layers. The outer segment of the receptor layer is comprised of stacks of melanin disks, with the purpose of absorbing electromagnetic radiation that is focused by the lens of the eye. As part of the visual cycle, the melanin within these disks are catabolized into A2E, packaged in vesicles in the RPE and transported out of the retina via the choroid. We see the majority of the signals in the ORL t o be attributed to A2E (1mM) where only a portion of the signals are attributed to melanin and higher concentrations of A2E (A2E 10mM), we believe this is due to the fact that the tissues imaged here are not freshly -excised, and the majority of the melanin present in the ORL has degraded. In addition to this, we also attribute the presence of A2E around the collagen bundles of the sclera to the fact that these are not freshly excised tissues, as it has been shown that fluorescent components, including A2E, aggregate increase with time in FFPE tissues [158] . The ratios of FAD and NADH are commonly used to asses cellular metabolism and respiration rates in tissues, therefore, the presence of one without the other is unlikely. Nevertheless, in Figure 49 , <5 photons/pixel were sourced to FAD and were present in the RPE, whereas NADH was present in the choroid. While the LSqF algorithm struggled to resolve photons from FAD and NADH independently from one another on the virtual phantoms, we see that the error for each of the MP sources is within 2 -3% (~15.36x10 3 photons per image) of the total photons in the input MP image. We attribute these errors to multiple sources, including electronic noise from the detectors, light leak noise from the imaging location, as well as the fact that we do not have fluorophore standards for every MP signal present in healthy retinas. 100 Figure 49 . Results of applying the single -vector LsqF algorithm on a NMMMI of unstained excised healthy retina. Each image contains photons corresponding to a single fluorescent or harmonic source. The signal source corresponding to each image is shown in the title above each output tile. Additionally, we present the error for each signal source in photons. Each ima ge contains a total of nearly ~15.36x10 3 photons , therefore, the number of incorrectly assigned photons is less than 3% of the total photons observed. We note the accurate assignment of the SHG photons to the sclera which is comprised of collagen, as well as the main contribution of Melanin to be in the IRL, where the majority of the melanin disks are present, and the assignment of A2E (in both concentrations) to the ORL, where the catabolites of the visual cycle Ñbreakdown of melanin into A2E, occurs. White scale bars are 50 µm. While least -squares algorithms are the most commonly employed algorithms for solving an overdetermined inverse problem, there are other methods, such as linear regression and Least Absolute Shrinkage and Selection Operator (LASSO/lasso) . These two additional methods both have the same objective as LSqF Ñto minimize the error in the prediction of parameter values, however, they go about this in a different manner, such as having different normalization functions that can be us eful in cases where the variables are highly correlated [223,231 Ð234]. Following the initial experiments of using the LSqF algorithm on the NMMMIs, we evaluated the performance of using a linear regression model and a lasso model to spectrally un -mix the endogenous signals within the retina NMMMIs. 101 Figur e 50 . Compressed sensing image solutions from each of the four algorithms. Each column of images is the output from one algorithm, the rows of the figure correspond to which fluorescent or harmonic signal sourceÕs photons are contained within that image. A ll images are 512x512 -pixels (~320x320 µm) and were acquired using the TCSPC. White scale bars are 50 µm. 102 Table 6 . Error values from the output of the four spectral un -mixing solution algorithms. Figure 50 and Table 6 show the output images and the corresponding error resulting from each of the four methods used to spectrally un -mix the NMMMIs of mouse retinas. In Figure 50 , each column of the panel figure corresponds to the algorithm that was used on the NMMMIs, where as the rows of the figure correspond to the photons belonging to the fluorophore or harmonophore source signal within that image. The first two columns are the results from using the LSqF algorithm, while the signal sources were fed into the algorithm as a 1xm vector for each signal in column II (and in the previous work for this chapter), the signal sources were fed into the LSqF algorithm as an nxm matrix for column I. In column I of Figure 50 , the LSqF algorithm assigned the majority of the photons withi n the IRL and ORL to NADH and A2E 1mM, however, it was shown in previous work that the quantum yield of NADH when excited by a 1070 nm source is very low, therefore, having a high percentage of the photons assigned to NADH is unlikely to be accurate. Moreo ver, the histological environment of the IRL is comprised of melanin disks, therefore, we would expect to see the majority of photons within this region to be assigned to having melanin as their source , and a portion of those photons to be assigned to the melanin catabolites, A2E, however, we see that there are very few photons assigned to melanin in any of the images in column I . Interestingly, we show that in addition to some of the photons being incorrectly assigned, feeding the signal source spectra int o the LSqF algorithm as an nxm matrix had an impact on the error associated with each of the signal sources, particularly in the case of FAD and A2E 10mM , where we see a 95% increase in error for FAD and a 125% increase in error for A2E 10mM when compared to the error from the LSqF results in column II. Unlike the LSqF algorithm, the lasso algorithm ( Figure 50 , column III), assigned ~2000 photons/pixel to the RL regions and did not assign any photons to belong to NADH. Due to the fact that the peak absorban ce 103 wavelength for NADH fluorescence is not within any of our excitation wavelength ranges ( #3P=353 nm, #2P=535 nm, #NADH=475 nm) , and the fact that the peak absorbance wavelength of FAD is near the SH wavelength of our laser ( #2P=535 nm, #FAD =530 nm), we believe that the photon assignment of FAD in column III is accurate . Therefore, we suggest that the lasso algorithm performed best on accurately distinguishing photons from highly -overlapped emission spectra, such as the case of NADH and FAD, however, it w as less accurate at correctly assigning photons to melanin within the RL regions than the LSqF algorithm of column II, where by the histological makeup of the retina, the majority of the photons from melanin would be found in the IRL, not the ORL [8,171,235 Ð237]. While the linear regression algorithm of column IV was able to distinguish a portion of the FAD photons from the NADH photons , it failed to accurately assign melanin and SHG photons. Due to the constraints on SHG, we should only be detecting SHG photons from the collagen bundles within the sclera [8,52,121,238] , while the LSqF and lasso algorithms were able to solely assign photons within these bundles to SHG, the linear regression model assigned a portion of the photons from collagen to melanin. While melanin i s found in other areas of human tissue, it is unlikely that there would be melanin within the collagen bundles. Within the retina, the main location of melanin is within the stacks of melanin disks, comprising the IRL , however, there are photons surrounding the collagen bundles in the LSqF and lasso results of Figure 50 , we suspect that these photons may actually belong to A2E due to the fact that this tissue is FFPE . 5.4 Conclusions Due to the overlapping fluorescenc e and harmonic emission and wavelength spectra, separating the pixel -level contribution of each photon to its source can be difficult. Unfortunately, commercial algorithms that are able to spectrally un -mix signals rely on the using pure fluorescence spect ra from fluorescently -labeled tissues as one of the input parameters. These commercial un -mixing software programs are not customizable, and therefore, users cannot load pure emission spectra for fluorophores and harmonophores endogenous to unstained tissu es, and apply those to the spectral un -mixing algorithm. 104 In this chapter, we begin by evaluating the most commonly used method of solving the inverse problem of the NMMMIs of unstained mouse retinas, least squares fit. The initial performance of the algor ithm was evaluated on virtual phantoms that mimic the fluorescent profiles and heterogenous histological patterns observed in the NMMMIs where it was determined that the algorithm was less effective on distinguishing signals when there was significant over lap in the emission spectra of the source compounds, particularly in the case of FAD and NADH. Following the establishment of the baseline performance of the LSqF algorithm, we applied the algorithm to the NMMMIs of mouse retinas, where it was shown tha t the LSqF algorithm successfully resolved the photon sources for collagen in the sclera as SHG, melanin disks in the IRL as melanin, and visual cycle catabolites in the ORL as both concentrations of A2E , and very few photons assigned to the metabolites, F AD and NADH. We evaluated two other inverse problem solution algorithms, lasso and linear regression. Lasso was more effective at separating signals with significant overlapping spectra, e.g. NADH and FAD, however it was not accurate in assigning photons i n the IRL and ORL, where nearly 13% (~2,000/15,360) of the photons were assigned to FAD and less than 7% (~1000/15,360) of photons within these regions were assigned to melanin and A2E Ñthe main molecular components of those layers. Unlike the lasso algorit hm, the linear regression algorithm was less effective at resolving the FAD and NADH photons within the retina regions, and assigned nearly all of the photons within the metabolically -active area of the RL to NADH and FAD to the photons within the choroid. Interestingly, while the linear regression algorithm only assigned ~7% of the photons within the IRL to melanin, it did not assign the photons in the ORL to melanin and accurately assigned them to A2E. Additionally, the linear regression model assigned ~1 3% of the melanin photons to the sclera, in regions surrounding the collagen bundles as well as within them. It is unlikely for melanin to be present within the collagen bundles, only SHG photons should be detected here, additionally, due to the fact that these tissues are not freshly excised, and FFPE, we suggest that the photons surrounding the collagen bundles belong to A2E and not melanin. Furthermore, we conclude that the LSqF algorithm is best suited for solving the inverse pro blem of the NMMMIs from mouse retinas. In addition to having the lowest error out of all three models tested, 105 it also accurately assigned photons to the histological locations where their presence had been previously documented. With additional source sign als provided, we suggest that using LSqF algorithm will become valuable technique to accurately assign and spectrally un -mix all endogenous photons on the pixel -level from unstained tissues . Moreover, this technique can be used as a baseline for endogenous signals in healthy tissues with the potential being used to diagnose abnormalities on a molecular level. 106 Chapter VI A supervised machine learning approach to diagnose human and canine oral squamous cell carcinoma from a very small dataset of nonlinear multiphoton multimodal microscopy images of unstained biopsies Abstract In this chapter, we explore the use of transfer learning to achieve the overall goal of classifying the NMMMIs of unstained oral cancer biopsies into three categories Ñhealthy, inflammatory, and neoplastic. We do this by initially training a neural network model to detect basic histological components of human tissues such as the stroma and mucosa, from a l arge (6,000 images) Kaggle dataset containing images of stained human colorectal cancer biopsies. We then perform experiments to optimize the modelÕs architecture, i.e. hidden layers, as well as the optimizers, hyperparameters, and API callback functions u sed before retraining the model on a new and much smaller dataset of 215 NMMMMIs. In addition to having different class labels, these images were obtained from an entirely different detector and thus have different features than the Kaggle dataset. We expl ore using tiling methods to subsample and expand our dataset as well as Òpiggy -backÓ off of the full -sized image labels and apply th ose same labels to all of the corresponding subsampled tiled images . This research shows that a classifier can be trained us ing transfer learning , and if the optimal ratio of frozen to retrainable layers is determined and used, transfer learning can improve classification accuracy by 10% over training on the smaller dataset alone. 6.1 Introduction Many medical diagnoses are centered around examination of histological slides of stained biopsied tissues by trained pathologists. Following initial examinations, if a physician finds an area of tissue that raises suspicion, surgical resection of the area in question is the next st ep. During the procedure, the physician will remove tissue from the area in question, thin slices at a time. If a rapid diagnosis of the tissue cannot be made in the operating room, these biopsies are sent to a pathologist where they are fixed, embedded in paraffin, sliced, and stained. These stains provide additional contrast and highlight certain histological features that the pathologist uses to make an expert decision of if there are any additional 107 diseased cells within that tissue section. This process will be continued until there are no longer any diseased cells in the biopsy and the margins of the tumor have been fully resected. While this is the gold standard, human involvement in the diagnostic process can lead to error in the diagnosis, where even an overlooked small cluster of cells can lead to a disease relapse. Over the past decade, integration of artificial intelligence (AI) in the fields of healthcare have greatly increased. Initially being more on the administrative side of healthcare, such as predicting the likelihood of a patient to miss their appointment [239] , more recently, AI has begun to show its merit in healthcare diagnostics [240,241] . From identifying gene clus ters from unrelated types of cancer to generate predictive models for breast cancer survival outcomes [242,243] to the recently established (2018) Precision Medicine Platform by The American Heart Association, to provide a secure workspace for deep learning in the health sciences [244,245] , the involvement AI in healthcare now has the potential to make significant impacts towards personalized medicine. The subset of AI, machine l earning for image classification, is now becoming even more popular with biopsy diagnostics in research, such as the work done by Beck et. al. where they used an unsupervised machine learning approach to discover stromal features associated with breast can cer survival [246] . Where ima ges are fed into a neural network model and used as training. There are two methods that an image classification model can be trained, either supervised, where each image has been professionally labeled to belong to a specific feature class, or unsupervise d, where the model determines the features of each image and from this, assigns each image to a specific class determined by the model. Both types of training require large image datasets comprised of tens to hundreds of thousands of images, and depending on the complexity of the classifier, i.e. binary classification (two possible classes) or multiple feature classification, can take thousands of epochs (one complete sweep through all of the data) to effectively train. A well accepted rule of thumb is the rule of 10 which says that in order to successfully train a model there needs to be 10 times the data for training than the number of trainable parameters. Though a timely endeavor, employing this type of AI in the clinic can improve patient quality of car e. With further exploration of this area, large companies, such as Google, as well as research groups have developed incredibly well -trained models that have drastically decreased 108 prediction error since 2010 [83,247] , and now outperform humans in their predictions from ImageNet datasets [247 Ð249]. The main goal of developing and training an effective image classification model is to achieve generalized learning. Generalized learning is similar to how a human would effectively study for an exam, instead of memorizing the examples, you want to have a solid understanding of the general knowledge and reasoning behind why the answer is just that. In an image classifier, if the model begins to ÒmemorizeÓ the training images (also called overfitt ing), the model will not extract the general features that are distinct to each class. This becomes an obvious issue when the testing image set is fed into the model and instead of using generalized features to classify each image, the model performs poorl y because these images are not the ones that were memorized. Models that memorize the training image set overfit the data, meaning that the accuracy and loss of the training set is higher and lower than that of the testing set, respectively or the case whe re the accuracy and loss of the testing set diverges significantly when compared to that of the training set. This is an issue frequently seen in image sets that are too small to train on and thus are not generalized enough. One approach to negate a lack of data is called transfer learning (TrL), where an image classifier model that is well -trained on large dataset is then either validated without retraining or partially retrained on a new problem with limited data. The idea is that features learned in the bigger dataset will transfer and be useful on a different classification problem. There are many methods of medical imaging, however, this research is employing an image classifier on multiphoton multimodal microscopy images (NMMMIs) of unstained human oral cancer biopsies excited with a 1070 nm ultrafast Yb -fiber laser. The primary adv antage of using an unstained method is the potential to significantly reduce the cost of sample preparation with an increase of information allowed by the frequency range of the newer technique. However, the disadvantage of this research is that this is a novel technology and we have a preliminary dataset of only 215 images. This limited dataset makes traditional machine learning methods difficult to validate. 109 Specifically, we aim use supervised learning to train a model on an incredibly small labeled dat aset with less than 215 512x512 images and three classes. Typically, the data used for an image classifier is divided into three sets; training, testing, and validation. Where the portion of the training set is largest (typically around 70 -80% of the image s), testing sets are comprised of 10 -15% of the data, and the remaining portion of the images that have not been allocated will comprise the validation dataset. In most cases, the validation set is used to evaluate the modelÕs performance and as a method o f fine -tuning the hyperparameters of the model. However, due to the smaller size of our original dataset Ñless than 215 -512x512-pixel images, we split our data into two categories, training and testing, and set aside a portion of the testing images to be us ed for validating our model. We employ different tiling sizes to increase the size of our data set and to determine the minimum image dimensions where useful features can still be extracted. We first utilize a previously labeled dataset from Kaggle. The c olorectal histology MNIST dataset contains 5000 64x64pixel images of labeled H&E -stained digital images with 8 different classes [250 Ð252]. We use this dataset to determine the optimal training model architecture and layer hierarchy and hyperparameters for our model. In addition to determining the optimal image dimensions of the NMMMIs, we evaluate the application of augmentation to our training set using the ImageDataGenerator object from Keras, and further determine the optimal parameters to be used for the augmentation [17] . This augmentation includes transformations such as rotation, scaling and mirroring and provides additional variation to the training set that will likely exist in actual data without t he cost of preparing more samples. An example of the transformations made during augmentation can be seen in Figure 51 , where we have applied the same transformations used in this work to an image of two cats so that the overall effect on the image is more apparent. 110 Figure 51 . Input image (left) and the resulting augmented images (3x3 panel on right), from using the Keras ImageDataGenerator. The operations performed on each image are identical to those used in this work and are shown in Table 7 [17,18] . Training on the NMMMI dataset will be accomplished using the same model architecture and hyperparameters as on the Kaggle dataset, following this, TrL was performed on the NMMMI dataset using the previously trained Kaggle model . Additionally, we explored the ideal combination of frozen layers (FL, ie. not retrainable) and retrainable layers ( TL) to reach maximum testing (validation) accuracy values. The potential of improving patient outcomes by integrating AI in healthcare diag nostics has already been shown to be a valuable resource, however, current model training methods rely strongly on large image datasets. This dependency creates a natural bias for which illnesses image classification can be used towards, with the most well -documented and examined diseases and cancers being those that prevail. We explore the use and effectiveness of employing augmented images obtained by different sampling methods in expanding considerably small (<215) image datasets with the goal of achievi ng an image classification model that avoids overfitting and can be termed generalized. If successful, our method can be used as a basis to expand image classification diagnostics to other under - represented and documented forms of cancer and other disease s, in addition to potentially reducing the costs of the current diagnostic methods by providing an all -in-one identification method that can replace multiple different staining modalities. 111 6.2 Materials and Methods 6.2.1 Excitation source , Sample preparation, and i mage acquisition The laser source and microscopy setup for these experiments is identical to that used in Chapter 4. The excitation source is a Yb -fiber laser oscillator generating pulses with sub -40 fs pulse durations (full -width half -maximum) (1.07 µm, 42 MHz) [123] (Chapter 4, Figure 31 ). The tissue slides were placed , coverslip -side down, on the stage of a Nikon TE2000 multiphoton inverted microscope for imaging (Chapter 4, Figure 32 ). A 40x water immersion (Carl Zeiss !, Immersol ! W, n e = 1.334 (23 ¡C), v e = 72) objective was employed with a working distance of 0.5 mm (Zeiss LD -C APOCHROMAT 1.1NA, Jena, Germany) to focus the beam on the tissue to a beam waist (beam diameter at the focus) of ~0.5 µm, favoring the generation of peak intensities high enough to induce multiphoton processes with pulse durations of 36 ± 1 fs. The peak intensity wa s maximized through the use of a pulse -shaper (MIIPS HD, BioPhotonic Solutions Inc., East Lansing, MI, USA) to compensate for the high -order dispersion along the beam path [194 Ð196]. Position and dwell time of the laser beam on the tissue was achieved using galvanometer mirrors. In addition to the excitation and microscope used to image this data, the samples and sample preparation are those demonstrated in Chapter 4. Images were obtained from a total of n= 6 biopsies and n = 36 imaging regions across the 6 FFPE prepared tissue slides. Further information regarding the sample preparation techniques are discussed in Chapter 4. All images used in this chapter are the single PMT images obtained alongside the single PMT and TCSPC data in Chapter 4. Signal detection was accomplished in the epi -direction using a single PMT detector . An optical filter was placed along the beam path, prior to the PMT to restrict the wavelength of photons detected to ~ 300 nm Ð 775 nm in conjunction with the dichroic mirror at the base of the objective. Each 512x512 -pixel 2D image is an averaged stack of 4 images, acquired for one second each with an average peak power of 3.2mW -6.2mW . All wide view images contain 4 -10 51 2x512-pixel images stitched together using the BigStitcher ImageJ package [198] . 112 6.2. 2 Image pre -processing and class distributions Figure 52 . Schematic showing the process of tiling the large -scale ( x,y -dimensions > 1000 µm) NMMMIs into multiple 64x64 -pixel (~40x40 µm) images to increase the size of the NMMMI dataset. White scale bar on large image is 200 µm. White scale bar on tiled image is 32 µm. Each stitched mosaic is combination of multiple 512 x512 -pixel images. To expand our dataset, we tile the 512x512 images into 64x64. 128x128, and 256x256 pixel images and use these tiled datasets to train our image classifier model. This allows us to determine the ideal size of the NMMMI s needed to effectively train our classifier while minimizing overfitting . While tiling our images to larger sizes than 64x64 pixels will reduce our total dataset size, the larger images may in turn have more information and features for the classifier to ac hieve generalization. While this tiling method will increase our database size, we realize that by extrapolating the same class labels of the mosaic NMMMI s to their tiled contributions, we are adding additional bias to our dataset. The ideal method to avo id this would be to individually classify each tiled image independently of the mosaic image classification, however, due to the fact that our mosaic images are obtained from bulk regions of tumor, inflammation, or healthy tissues, we are basing our decisi ons on the assumption that all cells within the mosaic region belong to a single class. Additionally, tiling the mosaic NMMMI s will indefinitely introduce ÒemptyÓ regions, where there is no tissue, such as in the top right corner of the image from Figure 5 2, which may also introduce bias to the 113 class assigned to that region. Individually labeling those regions with an ÒemptyÓ class could prevent the addition of such bias and will be done in future work. Two independent sets of the scaled image tiles were u sed in this work. The first being tiles obtained from the 512x512 -pixel NMMMI s. These tiles were obtained by dividing the original NMMMI s by an equal number of tiles with the desired dimensions. ABCCCD "3?E (13) Where l and w correspond to the length and width (in pixels) of the input NMMMI . The second set of images were rescaled NMMMI s to match the resolution of the Kaggle dataset. F",GHIIIJ GKLMMNO (14) PBCCCD "3?E?F, (15) Where the length (l) and width (w) of the original NMMMI are multiplied by a rescaling factor, f. The rescaling factor was determined by the resolution ( µm/pixel) of the NMMMI s and the Kaggle images and was determined to be 0.79 . This was done to asses s if the detectable features in a 64x64 -pixel Kaggle image are equivalent to those seen in a 64x64 -pixel NMMMI . The rescaled NMMMI s (R NMMMI ) are then divided into square tiles with 64x64, 128x128, and 256x256 -pixel dimensions. The class size distribution breakdown by original and final NMMMI dimensions can be seen in Figures 5 3a and 5 3b. The dataset sizes for the original (512x512 -pixels) NMMMIs belonging to the three classes Ñcancerous, inflammatory, healthy (normal), in Figure 53a, are organized into four groups, e.g. 64x64-pixels, 85x85 -pixels, 128x128 -pixels, and 256x256 -pixels. Figure 53 b, the class distributions for the rescaled NMMMIs provides similar information, however, the classes are organized into 3 groups for the tiled NMMMI dimensions. Due to the constraints of the new input dimensions of the rescaled NMMMIs Ñ384x384 pixels, we are unable to tile the NMMMIs into 85x85 -pixels images. 114 Figure 53 . Class distributions for the original (a) and rescaled (b) NMMMIs. The color of each bar plot represents each of the three classes, where red is for the Cancer class, orange is for Inflammation, and the yellow bars are for the Healthy class. Each plot has the class size distributions for each of the image dimension sizes used throughout this body of work , organized within groups based on the image dimensions along the x -axis . Annotated black brackets, surrounding each of the bar plots within the image dimension groups, shows the sum, or total number of tiled images within each group. In Figure 53 , it is clear that we have imbalanced classes, i.e. number of samples differs from one class to another. While we are aware that this may introduce bias into our classifier, by training our model to generalize cancerous tissues be st, we anticipate that having a multiclass dataset, instead of a binary class dataset, will help to counteract this bias. Furthermore, d ue to the fact that it is unlikely for physicians to intentionally biopsy healthy tissues, obtaining equivalent amounts of h ealthy samples to unhealthy, e.g. cancerous or inflammatory tissues, is difficult to achieve. Hence, the majority of the NMMMIs obtained from healthy regions belonged to the peripheral areas of the biopsies, where no abnormalities were detected, and the NM MMIs obtained from inflammatory regions were detected at the locations encompassing the margins separating the neoplastic and healthy cells. 6.2.3 Computational time All experiments were conducted on the Intel14 (Intel(R) Xeon(R) CPU E5 -2670 v2 @ 2.50GHz) and Intel16 (Intel(R) Xeon(R) CPU E5 -2680 v4 @ 2.40GHz) development node s at the High Performance Computing Center (HPCC) at Michigan State University. Figure 54 shows typical run times for training the 115 models us ing the Kaggle, original resolution, and new resolution NMMMIs, for different image dimensions (different data set sizes). Figure 54 . Typical run times for training the models on the Kaggle (left), original resolution NMMMIs (middle), and new resolutio n NMMMIs (right), for the different dataset sizes. Training times are represented in seconds form. Each color represents specific image dimensions. Diagonal pattern ed bar plots show for which image dimensions training was not performed . Note the decrease i n training time necessary for the NMMMIs that are rescaled to match the resolution of the Kaggle images. The bar plots in Figure 54 are grouped into three main data set categories ÑKaggle, Original Resolution NMMMIs, and New Resolution NMMMIs. Within each main category, there are four sub categories corresponding to the image dimensions, i.e. 64x64 -pixels, 85x85 -pixels, 128x128 -pixels, and 256x256-pixels . Due to the fact that in some cases, e.g. with the new resolution NMMMIs, it was not possible to subsam ple the full -sized NMMMIs (384x384 -pixels) into 85x85 -pixel tiles, therefore there was no training done for images of that size. In Figure 54 this is represented by d iagonally patterned bar plots . The outline colors of these plots correspond to the image d imensions where training was not performed for each type of data set . We note that the required training time (in seconds) decreases for the new resolution NMMMIs when compared to their original resolution counterparts. Interestingly, the size of the data set for the rescaled (new resolution) 64x64 -pixel NMMMIs is equal to the data set size of the original resolution 85x85-pixel NMMMIs ( Figure 53 ), however, the training time for the original resolution 85x85 -pixel NMMMIs is nearly 1.7x longer than the latt er. 116 Figure 55 . Typical training times for performing TL with a specified number of FL for the original and new resolution NMMMIs. The training times for the Keras recommended number of FL for the original and new resolution NMMMIs are shows by the solid red bar plot and the white bar plot with a red border, res pectively. These times were obtained by training on 64x64 -pixel images. Times are represented in seconds form. We note the relationship between training time and the number of FL, as the number of FL increases, training time decreases. In Figure 55 , we s how the typical training times (in seconds) for performing TL on the original and rescaled (new resolution) NMMMIs. The training times for training the Kaggle model on the original and rescaled NMMMIs while holding 9 layers frozen, as recommended by Keras, are shown as the solid red and the red -outlined bar plots, respectively. While we note the difference in training times between the two, the data set size of the original 64x64 -pixel NMMMIs is 5760 images, whereas the data set size of the rescaled 64x64 -pixel NMMMIs is 3240, therefore, we suggest that the decrease in training time required for the rescaled NMMMIs is most -likely related to the volume of the training data and not to the change in training methodology . In general, we note that training times decrease as the number of FL increases, this is to be expected as with more FL, less of the NMMMI images are used for retraining the model. 6.3 Results 6.3.1 Convolutional network training model Due to the large amount of data (>10,000 images) necessary for training a machine learning image classifier, performing these algorithms on small - (<1,000 images) and mid -sized (<10,000 images) individual and research -based datasets is uncommon. Unfortunately, in cases where obtaining more images 117 is impo ssible, problems that would benefit from an image classification algorithm, are disregarded. In order to avoid letting valuable data go to waste, we evaluate the performance of a convolutional network image classifier model on its ability to classify image s of unstained normal (healthy), inflammatory, and neoplastic oral cancer biopsied tissues, obtained using multiphoton multimodal microscopy. Initially, we determined the ideal model architecture that would give us the maximum testing (validation ) accuracy and minimal loss. To establish this baseline, we used an image dataset from Kaggle that we consider to be our synthetic data. Th e Kaggle dataset was a multi -class dataset (8 classes), containing 5000 labeled bright -field images of H&E stained col orectal (MNIST) tissues, each having dimensions of 64x64 -pixels . The motivation behind choosing this dataset was due to the histological basis of the labels, all of which identified some standard histological component or layer , such as the stroma and the mucosa , respectively . These components are natively present in all of the oral cancer NMMMI s. Therefore, we hypothesized that if we structured the ideal model for the Kaggle images , which we refer to as the Kaggle model, shown in Figure 56 , this same architecture can be applied to the oral cancer NMMMI s, as this architecture would be able to extract the low -level features of what constitutes tissue from the oral cavity . Figure 56 . Model parameters and architecture with 14 trainable layer s and the appropriate softmax layer to reflect the accurate number of classes for the images. 118 The Kaggle model is a convolutional network containing 14 layers, 447,273 trainable parameters (TrPs) , and 0 non -trainable parameters (NTrPs) . The only alteratio ns to the model architecture that were made when the NMMMI image datasets were used was dependent on the type of experiment being run , for the baseline NMMMI classification, only the softmax layer (dense_2) was changed to match the number of output classes . In the case of the experiments for the tiling dimensions study , which will be discussed further on, only the input image size was adjusted to match each scaled training image , and the softmax layer to match the number of classes . Additionally, for the experiments focused on TrL, any alterations to the layers of the Kaggle model architecture are covered in section 6.3.7. 6.3.2 Evaluation of a modelÕs performance and c ustomization of training model parameters and hyperparameters on the Kaggle dataset To evaluate a modelÕs performance, we plot the training and validation Ñusing a portion of the testing set not seen by the algorithm, referred to as Òtesting (validation)Ó throughout this work, accuracy and loss curves. Learning curves can provide valuable in formation about a modelÕs performance, such as if the model is overfitting or underfitting as well as if there is high bias in the model. Additionally, some information pertaining to the dataset, such as if there is high variation between the training and testing datasets as well as if the training set is unrepresentative itself can be extracted by a learning curve. Examples of these characteristics of learning curves are shown in Figure 57 using artificial data. 119 Figure 57 . Characteristics of learning curves that provide valuable information regarding a modelÕs performance and about the datasets. From left to right and top to bottom: (top) learning curves showing a good fit, underfitting, underfitting, (bottom) overfitting, unrepresentative training dat aset, high variance in training set, and high bias. Learning curves depicting a good fit ( Figure 57 a), have training and validation losses that decrease to a stable (steady state) value with a minimal gap between the two [234,253,254] . In the case of underfitting (Figures 5 7b and 5 7c), validation and training loss values may not change throughout the training cycle and appear as a flat line ( Figure 57 b), showing that the model was unable to learn anything from the training set, or in t he case of Figure 57 c, the training and validation loss values continually decrease, however, they do not reach a stable value during the training cycle and more training may be of good use [234,253,254] . Over time, the validation loss may increase and diverge from the training loss, this is an example of overfitting, where the model did not achieve good generalization from the training dataset ( Figure 57 d). Training data that is unrepresentative of the validation data, meaning that the training data does not provide enough information for the model to have a solid understanding of the concept being learned, are generally indicated by noisy loss curves, that while they improve with each epoch, there still remains a gap between the two [234,253,254] . Furthermore, a model trained on a dataset with high variance may have a large gap in between the loss and accuracy curves ( Figure 57 e), whereas a model with high bias may have a midpoint accuracy value between the accuracy a nd loss curves ( Figure 57 f, red dashed line). 120 Once the overall model architecture was optimized, we then experimented with the learning rate values and optimization functions available using the Keras library. We evaluated five different optimization func tions ÑAdadelta, Adagrad, Adamax, Adam, and Nadam. Each of these optimizers are adaptive gradient descent algorithms that assist in minimizing error functions used in gradient descent in order to minimize the loss and updating the calculation of the weights and biases following each epoch [255 Ð260]. These error functions are dependent on the internal learnable parameters of the model, therefore, there is not just a universal optimizer to use, the ideal optimizer for may differ from one model to the next. As a general comparison, e ach of the optimizers performs a type of gradient -based optimization, where they differ is with resp ect to the changes in their learning rates over an amount of time as well as with how the weights are updated. The mathematical proofs for these differences has been shown in previous work and will not be discussed here [255,256,258 Ð260]. Just as choosing the proper optimizer can significantly impact a modelÕs performance, choosing the ideal learning rate (LR) is a challenge and also significant, if a LR is too large, the global minima may be missed, however, choosing a LR that is too small can cause incredibly long convergence times [259] . A useful technique and argument available in the Keras library will make use of a LR scheduler, this will decrease the LR incrementally based on a certain threshold value [261] . The use of schedulers will be discussed further on. Here, we evaluated each of the five optimizers using four different learning rates Ñ0.01, 0.002, 0.001, and 0.0001. The combination of these learning rates was chosen based on their use as the default parameters within the Keras documentation [257] . In Figure 58 , we plot the accuracy values for each optimizer using each of the four LR values. When looking at these curves, the main take -away that we are trying to obtain is the ideal optimizer with its ideal LR value, we are looking for testing (validation) accuracy and testing (validation) loss values that are higher and lower than the corresponding training values, respectively. In each plot, the training accuracy curves are green, and the loss curves are blue, the t esting (validation) accuracy and loss curves are red and orange, respectively. When the learning rate is high, i.e. 0.01, we see divergence, in the case of the Adadelta and Adagrad and lack of learning anything, such as in the case of Adam and Nadam. Conve rsely, the highest 121 LR (0.01) was ideal for the Adamax optimizer and divergence occurred with lower LRs. Interestingly, contrarily to the Adam and Nadam optimizers, for lower LRs, i.e. 0.002, 0.001, and 0.0001, the accuracy values fail to increase to higher than the loss values Adadelta and Adagrad. While the Nadam optimizer performed similarly to Adam with a LR of 0.002 and 0.001, the steady -state (~last 25 epochs) training and testing (validation) accuracy (acc T, acc V) and loss (loss T,loss V) values for the Adam optimizer were acc T = 0.82, acc V = 0.82, loss T = 0.62, loss V = 0.62 at LR 0.002, and acc T = 0.88, acc V = 0.88, loss T = 0.49, and loss V = 0.50 for LR = 0.001, respectively. For the Nadam optimizer, these values were, acc T = 0.83, acc V = 0.83, loss T = 0.61, loss V = 0.61 at LR 0.002, and acc T = 0.86, acc V = 0.85, loss T = 0.49, and loss V = 0.57 for LR = 0.001, respectively. From Figure 58 , we see that the optimal combination of optimizer and LR for this specific model is the Adam optimizer with a LR of 0. 001. 122 Figure 58 . Training and testing ( validation ) accuracy plots for determining the ideal combination of optimizer algorithm and learning rate for the Kaggle dataset. The model algorithm that the accuracy results plotted in Figure 58 were obtained from were run for 1000 epochs, we show that in the case of the Adam and Nadam optimizers at either LR = 0.002 or LR = 0.001, there is very little change in the testing (validation) or training accuracy and loss achieved after the 123 500th epoch . From this, we hypothesize that 500 epochs should be sufficient for training our model and is the duration for how long the following experiments in the work will be trained for. 6.3.3 Baseline results of Kaggle model architecture without with augmentati on When the size of a dataset is considered to be small, overfitting, i.e. memorization, is a common problem with a ML classifier. One of the methods by which this is challenged is by use of augmentation. Augmentation refers to the process of making slig ht changes to the data so that the detectable features are not altered but merely presented in a different way. Some examples of i mage augmentation are rotating , resizing, skewing, or translating (vertically or horizontally) an input training image about i ts axis [262 Ð264]. While it is important to understand that augmentation will not increase your dataset size, it can help to prevent overfitting. Similarly, to how a student may study a specific mathematical concept by attempting to solve different practice problems that cover that same concept . Figure 59 . Sample of 25 non -augmented, labeled , and normalized input images from the Kaggle dataset which were fed into the ML model. Each image is 64x64 -pixels or ~32x32 µm in size, with a resolution of 0.49 µm/pixel. White scale bars are 10 µm. We trained our model, using both augmented and non -augmented input images from the Kaggle dataset to test the impact that augmentation would have. Figure 59 shows twenty -five of t he non-augmented labeled Kaggle images that were fed into our model. Each image is 64x64 -pixels, or ~32x32 µm with a single -color channel, normalized from 0 to 1. The ImageDataGenerator object from Keras was used to 124 perform the augmentation. The parameters and values used are presented in Table 7 . The values used for the parameters are the default values recommended by the Keras documentation [17] . Table 7 . ImageDataGenerator augmentation parameters and values applied to the training images. . Figure 60 . Comparison between non -augmenting (left) and augmenting (right) the Kaggle images prior to being fed into the model. Training accuracy and loss curves are green and blue, respectively. Testing (validation ) accuracy and loss are red and orange, respectively. Steady -state training and testing (validation ) accuracy and loss values are shown above all curves. Inset plots of sample images from each experiment are presented within each of the two plots. White scale bars are 10 µm. The models for both the augmented and non -augmented Kaggle datasets were train ed for 500 epochs, the training and testing ( validation ) accuracy and loss values are plotted in Figure 60 . Additionally, a sample image as well as the steady -state accuracy values are shown as insets to the main plot for each dataset used. The left plot c orresponds to the non -augmented images used by the model whereas the right plot shows the data resulting from augmented images being fed into the model for training. While in both cases, the training accuracy and loss are acceptable values, the testing ( validation ) accuracy of the model 125 trained on the non -augmented dataset plateaus well below the training accuracy value (~12%) and the loss increases exponentially when compared to the training loss. Conversely, while the testing ( validation ) accuracy is slig htly less than the training accuracy of the model supplied with the augmented images, 0.8390 versus 0.8462, respectively, there is no divergence or significant decrease in the accuracy values between the training and testing ( validation ), showing the impac t that augmentation had on the model trained on the Kaggle dataset. Hence, the augmented model has better performance than the non -augmented model. 6.3. 4 Baseline results of original resolution 64x64 -pixel NMMMI using Kaggle model architecture without and with augmentation Unlike the Kaggle dataset, the NMMMIs are obtained from unstained tissues. While imaging unstained tissues is valuable, images of stained tissues will have higher contrast, and a greater dynamic range of intensities. Furthermore, e ndogenous fluorescence has a lower quantum yield that the fluorescence emission of histological dyes, d ue to this, the NMMMIs are noisier than the stained Kaggle images. Nevertheless, we predict that though the accuracy values may be lower than those from the Kaggle model , augmentation will be useful in preventing overfitting on our model trained on the NMMMIs. Figure 61 . Twenty -five sample images of the NMMMIs that are used in the image classifier model. Each image is 64x64 -pixels, or ~40x40 µm in size, with 0.624 µm/pixel resolution , and normalized on a scale of 0 to 1 . The class labels for each image are along the bottom of each image, where the one -hot encoded labels are along the y -axis of each image. White scale bars are 10 µm. Prior to training the model, the 512x512 -pixel NMMMIs are tiled into multiple 64x64 -pixel images in order to match the size of the Kaggle images for input into the first layer of the classifier model . These 126 tiled NMMMIs are showed in Figure 61 . The method by which tiling was achieved can be seen in section 6.2.2. The normalized NMMMIs (0 to 1), are then either augmented or not augmented prior to training. Similarly, to Figure 60 , Figure 62 shows the accuracy curves from training the model on the non -augmented (left) and augmented (right) NMMMIs. As we expected, the accuracy and loss values from the model trained on the augmented NMMMIs are lower and higher than those from the model trained on the augmented Kaggle images, however, like in the case of the Kaggle model, we see a significant improvement on accuracy and loss values of the augmented NMMMI model than its non -augmented counterpart. In fact, we see the testing ( validation ) loss improve (decrease) by approximately 55%. Additionally, the augmented NMMMIs prevented overfitting, we did not see any divergence between the training and testing ( validation ) accuracy curves unlike in the case of the non -augmented NMMMIs . Furthermore , we note an improvement of 5% between the testing ( validation ) accuracy of the two cases. Figure 62 . Comparison between non -augmenting (left) and augmenting (right) the NMMMIs prior to being fed into the un -trained model. Training accuracy and l oss curves are green and blue, respectively. Testing (v alidation ) accuracy and loss are red and orange, respectively. Steady -state training and testing (validation ) accuracy and loss values are shown above all curves. Inset plots of sample images from each experiment are presented within each of the two plots. Key observation in this figure is that the loss curve is growing quickly in the non -augmented data while the loss is held under control with the augmented data. White scale bars are 10 µm. 127 6.3. 5 Updating the Adam optimizer parameters to better suit the NMMMI s and implementing the reduce learning rate callback from Keras The Adam optimizer performs gradient updates in mini batches, meaning that the number of iterations that the optimizer takes throughout a single epoch can be specified. In our case, we have a batch size of 16, therefore, B% (16) The Adam optimizer will update gradients every time the above equation is satisfied, where N corresponds to the number to training elements, and % corresponds to the batch size. Performing gradient updates in this manner not only decreases the computationa l cost of each epoch, but it also allows for more fine -tuning of the learning rate as the loss approaches the global minima [256,257] . In addition to the significance that determining the optimum LR can have on a modelÕs performance, the extent to how much this rate is altered during the training process is also important. As the we ap proach the global minima on a loss function , the slope, or rate of change at each update iteration, changes . In order not to miss the global minima, the LR needs to adapt to match the difference in slope of the gradient. The decay parameter (Table 8) for the Adam optimization function controls the extent to which the LR is changed while approaching the global minima. The decay value is used as a method of decreasing or increasing the LR. Depending whether on the initial LR is higher, i.e. LR > 0.01 or lowe r, i.e. LR < 0.002, altering the LR can improve stability and reduce loss for the former, or it can slow the convergence times for the global minima and have little effect on reducing loss for the latter [259] . We obser ved in Figure 61 that though our testing ( validation ) loss had decreased by applying augmentation to our NMMMIs, the signal -to-noise ratio (SNR) was still prevalent when compared to the training loss curves. Due to our already low initial LR of 0.001, we decided to prevent the Adam algorithm from decreasing the LR by setting the decay parameter to a value of 0.0 as well as implementing root mean square propagation (RMSProp) . Table 7 shows the original (default) Adam optimizer parameter values as well as the newer values we implemented. The default values of the optimizer were those recommended by Kingma, D. et.al. [256] . 128 Table 8 . Adam optimizer default and new parameter values used for the NMMMI model . Additionally, we implemented a Keras Callback function, ReduceLROnPlateau, which will reduce the learning rate by a factor specified by the user, once learning stagnates (plateaus) [261] . The callback will monitor a quantity, specified by the ÒmonitorÓ argument in Table 7 , for a set number of epochs (patience) until the minimum LR (min_lr) is reached. Figure 63 shows the ac curacy and loss curves for the training and testing ( validation ) set of NMMMIs using the adjusted Adam optimizer parameters and the ReduceLROnPlateau callback function. Conversely to the decay parameter for the Adam optimizer, the ReduceLROnPlateau paramet er performs on an epoch -level, whereas the decay parameter will reduce the LR after each mini -batch [257,261] . Additionally, because the ReduceLROnPlateau callback only reduces the LR, there is no concern about the updated LR becoming larger than the initial LR, which would increase variance. These features of ReduceLROnPlateau, will still allow us to reduce the LR to better approximate the global minima than a constant LR, and ensure our new LR only decreases. 129 Figure 63 . Accuracy results from the model trained on the NMMMI data with adjusted Adam optimizer parameter settings and implementing the Keras ReduceL ROnPlateau callback. Testing (v alidation ) accuracy and loss are red and orange, respectively. Steady -state training and testing ( validation ) accuracy and loss values are shown above all curves. Inset plots of sample images from each experiment are presented within each of the two plots. White scale bars are 10 µm. From looking at the NMMMI accuracy and loss curves (Figure 63 ), we can see that there are two major components against us, the gap between our loss and accuracy curves is high, and our curves are centered around 0.75 accuracy, i.e. our loss values never drop below 0.75, and our accuracy never increases beyond 0.75. Unfortunately, our curves being centered around such a high value is due to a significant amount of bias in our classifier, meaning that no matter how much additional data we feed the model, it is unlikely that our model will achieve good generalization. This was something that we had anticipated by extrapolating our labels from our mosaic (large view) images to their tiled counterparts. However, the large gap between the loss and accuracy curves is representative of high variance. There are a couple of ways that this can be alleviated, the first being to improve our data and the second being to simplify the model, with fewer or less complex features. 130 6.3. 6 Dimension s caling study for original and resized NMMMI s for Kaggle image resolution equivalency Table 9 . NMMMI image sizes and corresponding training and testing dataset sizes for the dimensions scaling study . Figure 64 . (left) 85x85 -pixel (~53x53 µm), (middle) 128x128 -pixel (~80x80 µm), and (right) 256x256-pixel (~160x160 µm) tiled normalized and labeled NMMMI s. Image class label is shown on the x -axis of each image whereas the one -hot encoded label corresponding to the class label is on the y -axis of each image. White scale bars are 10 µm. We first tested the impact the input image dimension s had on the model accuracy. The stained Kaggle images at 64x64 -pixels, have high contrast from the stains, this may impact how well the model is able to extract valuable features from the images. Due to the fact that the NMMMIs are unstained, we hypothesi zed that this may mean that by matching the image dimensions of our tiled NMMMIs (64x64 -pixels) to the Kaggle images, we have less extractable features compared to the Kaggle images. In other words, the features extracted from a 64x64 -pixel Kaggle image ar e not equivalent and far greater than t hose which can be extracted from a 64x64 -pixel unstained NMMMI . What may be equivalent to a 64x64 -pixel Kaggle image may be, in fact, a larger NMMMI. In addition to the 64x64 -pixel NMMMIs used for training and testing (validation ) (Figure 61 ), we also used 85x85 -pixel (~53x53 µm), 128x128 -pixel (~80x80 µm), and 256x256 -pixel (~160x160 µm) NMMMI tiles to train and validate our model. The data set size breakdown for each of the dimension sizes are shown in Figure 53 and Table 9 . Figure 64 shows nine of 131 the normalized and labeled NMMMIs for each of the new dimensions. The images corresponding to the 64x64-pixel NMMMIs can be seen in Figure 61 . Figure 65 . Accuracy curves for dimension study of NMMMI s. The models trained and tested ( validat ed) on 64x64 -pixel, 128x128 -pixel, and 256x256 -pixel training NMMMIs are shown from left to right, respectively. Testing (v alidation ) accuracy and loss are red and orange, respectively. Steady -state training and test ing ( validation ) accuracy and loss values are shown above all curves. Inset plots of sample images from each experiment corresponding to each dimension used are presented within each of the two plots. White scale bars are 10 µm. With larger dimensions, the histological and tissue -level organization is more apparent when looking at the NMMMIs in Figure 64 , however, this linear improvement on image quality with image size did not hold necessarily hold true in terms of training and testing ( validation ) loss and accuracy. Figure 65 shows the accuracy plots corresponding to the 85x85 -pixel, 128x128 -pixel, and 256x256 -pixel NMMMIs, respectively. We see the peak performance, in terms of testing ( validation ) accuracy and loss, on the 85x85 -pixel NMMMIs, with acc v = 0.6698 and loss v = 0.8060. These values improved by approximately 2% and 17% than those from the 64x64 -pixel NMMMIs ( Figure 63 ). On the larger NMMMIs, i.e. 128x128 -pixel and 256x256 -pixel, we see the testing ( validation ) loss and accuracy values worsen by approximately 4% and 4 -8%, respectively. Additionally, on these larger NMMMIs we see that at certain time points, prior to 200 epochs for the 128x128 -pixel images, and for all 500 epochs of the 256x256 -pixel images, there is no learning occurring and we are underfitting our data. This is most likely due to the fact that while our images are larger and have more information in them, the input data size for the 128x128 -pixel NMMMIs is only 25% and the data set size for the 256x256 -pixel NMMMIs is only 6.25% the amount of the original total data size for the 64x64 -pixel NMMMIs ( Table 9 ). 132 Table 10 . New dimensions of the input and output NMMMIs, the total NMMMIs and the corresponding training and testing allocations for the new dimens ions are also shown. Figure 66 . Rescaled NMMMI s (left) 64x64 -pixel (~ 40x40 µm), (middle) 128x128 -pixel (~80x80 µm), and (right) 192x192-pixel (~1 20x120 µm) tiled normalized and labeled NMMMIs. Image class label is shown on the x -axis of each image whereas the one -hot encoded label corresponding to the class label is on the y -axis of each image. White scale bars are 10 µm. The second method by which we attempted to improve our data was by rescaling our 512x512 -pixel NMMMIs to a size that would correlate to the resolution of the Kaggle images. The Kaggle images have a resolution of 0.49 µm/pixel, whereas the NMMMIs have a res olution of 0.624 µm/pixel, this corresponds to the Kaggle images having ~79% greater resolution than the NMMMIs, which could inherently be one of the reasons as to why the model architecture customized for the Kaggle dataset is not performing as well on th e NMMMI dataset. We predict that if we rescale the NMMMIs to a size that would permit the tiled NMMMIs to have the same level of features as the 64x64 -pixel Kaggle images, the Kaggle -designed model will perform better on the NMMMIs. Prior to tiling, we do wn-sized our 512x512 -pixel NMMMIs to 384x384 -pixels. This was 20 pixels smaller on each x - and y -axis than would equal the resolution ratio of the Kaggl e images to NMMMIs, however, these were the ideal initial dimensions in order to have output tiling dime nsions of 64x64 -pixels. Additionally, with the new input image dimensions of 384x384 -pixels, the tiling dimensions for the largest -scale NMMMIs were slightly smaller than the Kaggle image counterparts. The input dimensions, output dimensions, as well as th e total number NMMMIs and the number NMMMIs used for training and testing 133 are presented in Table 10 . The tiling dimensions for the rescaled NMMMIs were 64x64 -pixels, 128x128 -pixels, and 192x192 -pixels, and are shown as three -by-three image arrays in Figure 66, respectively. Figure 67 . Accuracy curves for scaling study of rescaled NMMMI s. The models trained and tested (validated) on 64x64 -pixels, 128x128 -pixels, and 192x192 -pixels training NMMMIs are shown from left to right, respectively. Testing (validation ) accuracy and loss are red and orange, respectively. Steady -state training and testing ( validation ) accuracy and loss values are shown above all curves. Inset plots of sample images from each experiment corresponding to each dimension used are presented within each of the two plots. White scale bars are 10 µm. We trained and validated the established model (section 6.3.5) on each of the dimension sizes of the rescaled NMMMIs. Figure 67 shows the accuracy plots for the 64x64 -pixel, 128x128 -pix el, and 192x192-pixel NMMMIs, respectively. Compared to the original -scale NMMMIs ( Figures 6 1 and 62), the model trained on the 64x64 -pixel rescaled NMMMIs had the highest validation accuracy (acc v = 0.6744) and lowest testing ( validation ) loss (loss v = 0.8065) compared to the models trained on the larger dimension NMMMIs. The model trained on the 128x128 -pixel NMMMIs had lower testing ( validation ) accuracy (acc v = 0.6049) and higher testing ( validation ) loss (loss v = 0.9720) than the largest dimension NMMMI tiles (192x192 -pixels), which had acc v = 0.6528 and loss v = 0.8374, confirming the nonlinear dependence on input dimensions of testing ( validation ) accuracy and loss values that was shown in Figure 65 . By training the model on rescaled NMMMIs (64x64 -pixels), the testing ( validation ) accuracy improved by ~2.5% (acc v(rescaled) = 0.6744) than the case of the original scaled 64x64 -pixel NMMMIs (acc v(original) = 0.6502, Figure 63 ) and nearly 17% for testing ( validation ) loss (loss v(rescaled) = 0.8065, los sv(original) = 0.9728). While the model trained on the original scale, 85x85 -pixels NMMMIs performed better than the one trained on original scale 64x64 -pixels NMMMIs, the model trained on the rescaled 64x64 -pixels had slightly 134 improved testing ( validation ) accuracy (~0.5%) and nearly the same testing ( validation ) loss ( $v_acc = 0.0005). Due to the fact that the total NMMMIs for both the original scale 85x85 -pixel and rescaled 64x64 -pixel NMMMIs is the same (3,240 NMMMIs, Tables 9 and 10), we predict that t he increase in testing (validation ) accuracy is due to the rescaling of the NMMMIs. We believe that the improvement on accuracy of the model trained on the rescaled 64x64 -pixel NMMMIs is related to the underlying architecture of the model. In all of our ex periments, we are using a convolutional network architecture rather than a neural network comprised of many fully connected layers. Unlike the input layer of the latter, where the input image is essentially transformed into a linear array based on the inpu t image dimensions Ñthe input layer dimensions corresponding to a 64x64 -pixel input image would be 1x 4096 nodes , the nodes of the convolutional input layer can be thought of an array equal to the input image dimensions, i.e. rows and columns of 64 x64 nodes [265] . The convolutional architecture has been shown to be ideal for image classification due to the fact that the special relationship of neighboring pixels is preserved [80,82,83] . Instead of the pixel intensity being the weight applied to the corresponding node in the hidden layer of a fully connected network , the weights passed to the following hidden layer in a convolutional neural network are calculated from the pixel intensities within a local array , with dimensions specified by a filter (3x3 -pixels in this case) [265] . Due to the increased resolution of the Kaggle images of 79% when compared to the NMMM Is, it is likely that the weights and biases which was calculated from the 3x3 -pixel kernel array and the resultant feature maps, i.e. the resultant total of all individual features extracted by the kernel at all local arrays within the input layer Ñ32 in t his case, was not as representative in the case of the original scaled NMMMIs as those from the rescaled NMMMIs and thus the feature extracted did not correlate to a robust generalization of the local array . Nevertheless , the gap between the testing ( validation ) loss and accuracy for the model fed the rescaled 64x64 -pixels NMMMIs improved from ~32% (loss v(original) - accv(original) = 0.3226, Figure 63 ) to ~13% (loss v(rescaled) - accv(rescaled) = 0.1321, Figure 67 ), when compared to the model trained on the original scale 64x64 -pixel NMMMIs, meaning that by rescaling the NMMMIs prior to training we reduced the variance in the model, which was one of the goals th e rescaling experiments were set to achieve. 135 6.3. 7 Improving testing ( validation ) accuracy o f NMMMI IC without increasing input data using transfer learning on the pre -trained Kaggle model Due to the fact that we are unable to obtain any additional NMMMIs, we perform T rL with our rescaled 64x64 -pixels NMMMIs on the saved model that was previously trained on the Kaggle images. We are interested in determining if T rL will aid in lowering the accuracy value of ~0.75 that both the NMMMI training and testing ( validation ) accuracy and loss curves are centered about to create a better generalized model. Figure 68 . Workflow schematic of transfer learning experiments with the pretrained Kaggle model on the rescaled 64x64 -pixel NMMMI dataset . The key in the top -right corner of the figure shows the meaning for the abbreviations used within the figure. The workflow is split into three parts, 1. The initial training of the Kaggle and NMMMI models, independently, 2. The FC study to determine the optimal ratio of FL to retrainable la yers, and 3. Appending the optimized number of retrainable layers determined in phase II to the last layer of the full Kaggle model. Ideally, the TrL experiments would be conducted in two stages, the first would determine the ideal number of FLs and retrainable layers, the number of retrainable layers is we will refer to as the finesse chunk width (FC). The second stage of the T rL experiments would append the optimal FC to the last layer of the full previously tr ained Kaggle model. However, we only conduct the experiments for phase I in this 136 work and the experiments for phase II will be completed in future work. Figure 68 shows the general workflow for the T rL experiments for both the work performed in phase I (th is research) as well as those for phase II in future work . In phase I, indicated by a Ò2Ó in Figure 68 , depicts the combination of the FL from the previously trained Kaggle model and the TL, where the rescaled 64x64 -pixel NMMMIs will be used for training. The initial placement of the FC, i.e. the layer number where retraining on the NMMMIs begins, was dictated by the recommendation presented in an example in the Keras documentation for TrL [266,267] . 6.3. 8 Following Keras documentation for the number of frozen and retrainable layers The model architecture, number of parameters per layer, total parameters (ToPrs) , total trainable parameters (ToTrPrs), and non -trainable parameters (NTrPrs) for the T rL study that utilizes the recommended ratio of FL to retrainable la yers, is shown in Figure 69 . Here, the first 9 layers of the previously trained Kaggle model are frozen, the remaining 5 layers are retrainable, and the last layer (of the 5 retrainable layers) Ñthe softmax layer, is altered to match the number of classes f or the NMMMIs. The new softmax layer (Dense) is named Òoutput_denseÓ whereas in the original model ( Figure 56 ), it was termed Òdense_2Ó. 137 Figure 69 . Transfer learning model parameters and architecture with 9 FL (non -retrainable) Kaggle layers, 4 TL, and an updated softmax layer to reflect the accurate number of classes for the NMMMI s. We performed T rL on both the original -scaled 64x64-pixel NMMMIs as well as the rescaled (~1:1 NMMMI to Kaggle ratio) 64x64 -pixel NMMMIs using the previously trained Kaggle image classifier model from section 6.3.3 with the optimized Adam parameters determined in section 6.3.5. Figure 70 shows the mean steady -state (final 25 epochs) accuracy values from retraining 5 of the 14 Kaggle model layers. Figure 70 . Steady -state accuracy values for recommended number of frozen layers for the models partially trained and then validated Ñusing a portion of the testing dataset, on both the (left) original resolution 64x64 pixel NMMMI s and the (right) Kaggle -matched reso lution 64x64 pixel NMMMI s. Matching the resolution of the NMMMIs to the Kaggle images shows improves accuracy from 64.1% to 65.8%. 138 Of the 5 retrainable layers, there are 418,564 parameters that are able to be trained on the NMMMI data, whereas there are only 28,064 parameters that are non -trainable as those are parameters within the 9 frozen layers of the model (Table 10 ). From looking at the testing ( validation ) accuracy for the rescaled NMMMIs in Figure 70 (left), while we have increased the value b y nearly 3% (2.73%) than the model trained on the original scale NMMMIs, the testing ( validation ) accuracy decreased by ~2% (1.63%) when compared to the model trained entirely on the rescaled, 64x64 -pixel, NMMMIs ( Figure 67 ). Using these metrics as evaluat ion parameters shows that while freezing the first 9 layers of the Kaggle model nearly follows the Keras recommendation for the percent of frozen to retrainable layers of 62.5% (freezing 9 of the 14 layers is 64.3%), the performance of our T rL model was le ss accurate than our optimized model with all 14 layers trained on the rescaled 64x64 -pixel NMMMIs. We suggest that the features extracted, and weights calculated by the 9 frozen layers from the Kaggle model are not representative of the NMMMIs . Neverthele ss, we believe that T rL is a valid technique to increase our accuracy for this specific set of images, and with the optimal combination of frozen and retrainable layers in the Kaggle model, we can improve our accuracy over the model used on the rescaled, 6 4x64 -pixel NMMMIs in section 6.3.6. 6.3. 9 Determining the optimal Finesse chunk width for transfer learning on the NMMMI data set Table 11 shows the parameter allocations for each of the T rL experiments with respect to the number of frozen layers. Layers 2, 3, 5, 8, and 12 are the only layers that contribute parameters to the total model, therefore, when the surrounding layers are frozen, there is no change in the number of TrPs or NTrPs. Nevertheless, by freezing the layers, whether or not they con tribute to the number of parameters Ñeither TrPs or NTrPs, we prevent the previously calculated weights from changing, which may still have an effect on the accuracy of the model. 139 Table 11 . Total, trainable, and non -trainable parameters with respect to number of frozen layers for transfer learning the Kaggle model on the scaled 64x64 -pixel NMMMIs . The accuracy for the training and validation (on a portion of the testing dataset) from the T rL experiments is shown as boxplots in Figure 71 , where the left plot corresponds to the training accuracy and the right plot corresponds to the testing (validation) accuracy . Each boxplot was calculated from the mean of the final 25 epochs out of 10 trails for each of the 12 TrL experiments. The mean accuracy valu es for the training and testing (validation) data is shown by the red lines of each plot. The mean accuracy values for each T rL experiments were calculated by excluding any outliers. Any value that fell either above or below the values determined by adding and subtracting 1.5 x interquartile range (IQR) from the third and first quartile (Q3, Q1), were determined to be outliers. Figure 71 . Steady -state accuracy values for the finesse chunk width study on the Kaggle -matched resolution 64x64 pixel NMMMI s. Testing (Validation) accuracy improves with fewer frozen layers than what was recommended by Keras ( Figure 70 ). 140 In both the training and testing (validation) boxplots, we see a general improvement on accuracy as the number of retrainable layers increas es and the number of frozen layers decreases . In t he training accuracy plot ( Figure 71 , left), the spread of the accuracy values is within a ~4% window and the distribution of values within each of the frozen layer assignments ranges from ~0.5 -1.0%, wherea s in the spread of the values for the testing (validation) accuracy of Figure 71 , is approximately 10%, with the distributions of each frozen layer being within 0.5%. The improvement of performance and the decrease in distribution for each frozen layer exp eriment is most -likely due to the use of drop -out layers in our model [268] . Nevertheless, w hen following the Keras recommendation and utilizing only the last 5 layers as retrainable layers ( Figure 70 ), the maximum validation (from the testing data set) we obtained was 65.8%, whereas the highest testing ( validation ) accuracy, shown in the right plot of Figure 71 , that our model achieved on the rescaled 64x64 -pixel NMMMIs was 70.1%. This is over a 6% (6.13%) increase in accuracy by holding only the first 4 layers of the Kaggle model frozen and retraining the remaining 10 layers on the rescaled NMMMIs. Figure 72 . Selected accuracy curves for transfer learning of the Kaggle model on the NMMMIs with 2 (a) , 4(b) , 9(c) , and 13 (d) frozen layers. Steady -state (last 25 epochs) metric values for each of the experiments are in the textbox located in the top -right corner of each plot. Observe that as the number of frozen layers increase the va riability in the results stabilizes and a general improvement on training and testing of the validation accuracy. Figure 72 shows the accuracy curves for selected frozen layer transfer learning experiments. From Figures 7 0 (right) and 71 (right), we show ed the general improvement on training and testing (validation) accuracy from holding 9 layers frozen, as was recommended in the Keras documentations, to holding 4 layers frozen for the NMMMIs . In addition to these results, in Figures 7 2a and 72d, we compare the results a. b. c. d. 141 from holding 2 and 13 layers frozen, respectively. One of the goals aimed to achieve from these FL experiments was to reduce the gap between the loss and accuracy values, though holding 4 layers frozen achieved the highest testing (validation) accuracy, holding 2 layers frozen reduced this gap from ~19% (18.9%) in the case of 9 FL, to ~14% in the case of 4 FL down to ~11%. Additionally, holding 2 layers frozen resulted in the lowest testing (validation) loss of 80.1 % than compared t o holding 4 layers frozen (loss v = 84.8%) or 9 layers frozen (loss v=84.6%). Additionally, there was less divergence of testing (validation) loss ( overfitting ) from the training loss seen in the T rL model holding 2 layers frozen (Figure 72a) than the model with 4 FL ( Figure 72 b). Furthermore, the testing (validation) accuracy for the 2FL T rL model was higher than the model trained solely on rescaled 64x64 -pixel NMMMIs in Figure 70 (left) by 1%. The combination of the lowest testing (validation) loss, least amount of overfitting, and higher testing (validation) accuracy values shown in Figure 72 a, for the T rL model with 2 FL seems to be ideal for use of NMMMIs. 6.4 Conclusion The long -term goal this work was to evaluate the combination of our NMMMIs and deep learning as a basis to develop an all -in-one identification method that can replace multiple different staining modalities, in order to improve patient outcomes by integrat ing AI in healthcare diagnostics. In particular, the work presented here demonstrates a first step to applying deep learning image classification to NMMMI histological images from datasets that are not large enough for conventional deep learning image clas sification methods. The initial model architecture optimization experiments showed that a 14 -layer, convolutional neural network, integrated with dropout layers and ReLU activation functions were ideal for working with the colorectal -histology MNIST Kaggle dataset and the NMMMIs. The Adam optimizer, with a LR = 0.001, outperformed the Nadam, Adagrad, Adadelta, and the Adamax optimizer functions with any of the four LRs that were evaluated, e.g. 0.01, 0.002, 0.001, and 0.0001. We determined that rescaling the NMMIs in order to artificially match the resolution of the Kaggle images was crucial. The combination of Keras callback functions ( ReduceLROnPlateau ) and implementation of RMSprop with the optimal architecture, optimizer function, and LR, resulted in a validation accuracy Ña subset of images from a 142 portion of the testing set, unseen to the classifier, of 67.4% and loss of 80.1% which was an improvement of 2% and 17% when compared to the model fed the original resolution NMMMIs. Furthermore, to improve these values in hopes of creating a robust generalized model, we employed TrL of the model previously trained on the Kaggle dataset with the recalled NMMMIs. We determined the optimal number of frozen and retrainable layers of the saved Kaggle model was, fou r FL and 10 retrainable layers, which performed better than the recommended ratio of FL to retrainable layers found in the Keras documentation. Using four FL in the transfer learning model resulted in a testing (validation) accuracy of 70.1%. We conclude t hat these results show promise, and with future work, such as labeling the tiled NMMMIs on an individual basis, re -optimizing the model architecture with respect to the NMMMIs and not an MNIST dataset, such as the Kaggle dataset used here, as well as perf orming TrL on a another model, previously trained on a larger dataset, will result in the metrics necessary to have a robust and generalized image classification model that can use NMMMI to diagnose cancer without use of expensive staining equipment and pa inful biopsies. 143 Chapter VII Summary and Outlook Although lasers are regularly used in other applications, their use in the clinic has yet to be approved. With characterized and optimized laser pulses, MP can be achieved at lower average peak powers, suc h as the work presented in this dissertation, where the range of average power used for imaging is between 3.2 mW -6.7mW. The near -IR wavelength (1070 nm), as well as variety of multiphoton excitation modalities, and the fact that the 3P processes are abov e the DNA -damaging UV range, of the Yb -fiber laser used in this work, makes its use desirable in biomedical applications. My research successfully shows the potential of using NMMM in tandem with computational methods to augment current diagnostic protoco ls used by the health care system with potential to improve patient outcomes as well as decrease pathology departmental costs. These results should facilitate the continued study and development of NMMM so that in the future, NMMM can be used for clinical applications. 7.1 Multiphoton excited hemoglobin fluorescence and third harmonic generation for non -invasive microscopy of stored blood We have investigated 2PEF and THG for label -free non -invasive RBC imaging. Unlike conventional laser microscopy systems (>100fs), the laser systems employed here produce very short pulses (15fs for the Ti:Sapphire and <45fs for the Yb -fiber lasers). Theref ore, these short -pulse sources deposit less thermal energy and reduce photo -thermal damage to the RBCs. 2PEF signal increases as the inverse of pulse duration, while THG signals increase as the inverse of the pulse duration squared [113] . Following successful 2PEF imaging of RBCs, we explored the source of the fluorescence and concluded it originated from two -photon excitation of the Soret band in hemoglobin based on fluorescence spectra, fluorescence lifetimes, as well as both linear and transient absorption data. The images are sufficiently detailed to assess morphological anomalies of RBCs non -destructiv ely without breaching sterility using commercially available compact femtosecond laser oscillators. 144 Multi -photon microscopy modalities such as THG and 2PEF can be used for non -invasive imaging of blood cells through the storage bag. Moreover, it was shown here that THG imaging provided the best resolution and image sensitivity for noninvasive imaging of stored RBCs without photodamage. We conclude that using compact and reliable ultrafast laser oscillators may lead to improvements in non -invasive blood anal ysis, including point -of-care assessment of RBC morphology. 7.2 Multimodal nonlinear optical imaging of unstained retinas in the epi -direction with a sub -40 fs Yb-fiber laser Following the validation of the non -invasive applications of NMMM, in Chapter I II we employed NMMM to establish the endogenous MP signals present in healthy excised and unstained mouse and Cynomolgus monkey retinas, using 2PEF, 3PEF, second harmonic generation (SHG), and THG. In this chapter, we presented the first epi -direction dete cted cross -section and depth -resolved images of unstained isolated retinas obtained using multiphoton microscopy with an ultrafast fiber laser centered at 1070 nm and a ~38 fs pulse duration. Moreover, we analyzed the spectral and temporal signatures of th e autofluorescence signals and showed two distinct regions; the first one from the nerve fiber layer to the inner receptor layer, and the second being the retinal pigmented epithelium and choroid. In summary, we presented the first epi -direction multimodal imaging of unstained isolated mouse and Cynomolgus monkey retinas with an ultrafast fiber laser centered at 1.07 µm. Measurements of the fluorescence spectra and lifetime from a thin cross -section of a mouse retina showed that emission from the ORL to the NFL have similar spectra, including a relatively long lifetime. The RPE and choroid have similar spectra, including a relatively short lifetime. We attribute a majority of the short lifetime signal to A2E, and a majority of the long lifetime signal to lip ofuscin or other lipofuscin degradation products. Interestingly, we show that FAD and NADH do not significantly contribute to the fluorescence emission from a 1.07 µm laser. This is different from most multiphoton microscopy studies where FAD and NADH are usually the strongest autofluorescent signals. In addition, depth resolved imaging of an unstained Cynomolgus monkey retina is also presented using the same laser and experimental setup. The depth resolved images from the 145 Cynomolgus monkey show that it is feasible to use our collection system to image the retina of live -animal subjects, and in the future of humans. 7.3 Tetra -modal multiphoton microscopy: A non -invasive technique for augmented histopathological analysis of oral squamous cell carcinoma biops ies Oral squamous cell carcinoma (OSCC), a form of head and neck cancer, is responsible for roughly 2-4% of cancer cases and can show little to no symptoms. If left untreated, OSCC can be fatal. Current methods of diagnosis involve time and cost consuming process that not only involves an uncomfortable biopsy procedure but also costs histology departments resources to prepare, stain, and mount these biopsied tissues on slides for a pathologist to manually inspect and interpret. By using the human eye as a diagnostic technique, the risk of an improper diagnosis is high. Here, we have shown how the use of NMMMI can be a useful tool to augment the current gold -standard of neoplasia diagnosis, H&E staining. In addition to matching H&E -like contrast in unstaine d excised tissues, we show our ability to distinguish elastin fibrils from collagen fibrils in the stroma and note changes in their structure and distribution in neoplastic tissues when compared to healthy tissues. Using these MM signals, our imaging techn ique can serve as a more direct way to evaluate the status of the basement membrane. Furthermore, we excite and detect additional 2PEF and 3PEF signals from cells in mild and severe inflammatory regions we attribute to plasma cells and lymphocytes, respec tively. Our inability to excite the nuclei of the squamous cell layer warrants our ability to distinguish the boundaries between nuclei and the surrounding cytoplasm. The combination of direct and indirect detection makes it plausible to employ existing statistical measurements used in diagnostics, on images acquired with our new technique. Furthermore, our 1070 nm excitation source permits the ability to image FFPE HOSCC biopsies directly from the paraffin block and maintain the same the multi -spectral con trast in depth -resolved images as in 2D NMMMIs obtained from slide -mounted unstained tissue slices. Our NMMMIs of unstained tissues can be used for qualitative and quantitative measurements, including the orientation angle measurements of collagen fibrils in 2D and depth -resolved images of healthy, MI, SI, and neoplastic regions. We show a pattern in the change in collagen angle orientation distinct to the state of the tissue. We note that these angles trend towards negative values as the region of interest is in closer 146 proximity to neoplastic regions. We conclude that with continued studies and improved instrumentation, the combination of endogenous 2PEF and 3PEF signal classification in inflammatory regions, evaluation of elastin fibril changes, and collag en orientation angle measurements show promise as a method of augmenting current gold standard HOSCC diagnostic protocols as well as increasing the likelihood of early detection. 7.4 Towards spectral unmixing of multiphoton multimodal images of unstained retinas using user -defined signal sources and inverse problem solving algorithms Due to the overlapping fluorescence and harmonic emission and wavelength spectra, separating the pixel -level contribution of each photon to its source can be difficult. Unfo rtunately, commercial algorithms that are able to spectrally un -mix signals rely on the using pure fluorescence spectra from fluorescently -labeled tissues as one of the input parameters. These commercial un -mixing software programs are not customizable, an d therefore, users cannot load pure emission spectra for fluorophores and harmonophores endogenous to unstained tissues, and apply those to the spectral un -mixing algorithm. In this chapter, we begin by evaluating the most commonly used method of solving the inverse problem of the NMMMIs of unstained mouse retinas, least squares fit. The initial performance of the algorithm was evaluated on virtual phantoms that mimic the fluorescent profiles and heterogenous histological patterns observed in the NMMMIs wh ere it was determined that the algorithm was less effective on distinguishing signals when there was significant overlap in the emission spectra of the source compounds, particularly in the case of FAD and NADH. Following the establishment of the baseli ne performance of the LSqF algorithm, we applied the algorithm to the NMMMIs of mouse retinas, where it was shown that the LSqF algorithm successfully resolved the photon sources for collagen in the sclera as SHG, melanin disks in the IRL as melanin, and visual cycle catabolites in the ORL as both concentrations of A2E, and very few photons assigned to the metabolites, FAD and NADH. We evaluated two other inverse problem solution algorithms, lasso and linear regression. Lasso was more effective at separatin g signals with significant overlapping spectra, e.g. NADH and FAD, however it was not accurate in assigning photons in the IRL and ORL, where nearly 13% 147 (~2,000/15,360) of the photons were assigned to FAD and less than 7% (~1000/15,360) of photons within these regions were assigned to melanin and A2E Ñthe main molecular components of those layers. Unlike the lasso algorithm, the linear regression algorithm was less effective at resolving the FAD and NADH photons within the retina regions, and assigned nearly all of the photons within the metabolically -active area of the RL to NADH and FAD to the photons within the choroid. Interestingly, while the linear regression algorithm only assigned ~7% of the photons within the IRL to melanin, it did not assign the photons in the ORL to melanin and accurately assigned them to A2E. Additionally, the linear regression model assigned ~13% of the melanin photons to the sclera, in regions surrounding the collagen bundles as well as within them. It is unlikely for melanin to be present within the collagen bundles, only SHG photons should be detected here, additionally, due to the fact that these tissues are not freshly excised, and FFPE, we suggest that the photons surrounding the collagen bundles belong to A2E and not melanin . Furthermore, we conclude that the LSqF algorithm is best suited for solving the inverse problem of the NMMMIs from mouse retinas. In addition to having the lowest error out of all three models tested, it also accurately assigned photons to the histolog ical locations where their presence had been previously documented. With additional source signals provided, we suggest that using LSqF algorithm will become valuable technique to accurately assign and spectrally un -mix all endogenous photons on the pixel -level from unstained tissues. Moreover, this technique can be used as a baseline for endogenous signals in healthy tissues with the potential being used to diagnose abnormalities on a molecular level. 7.5 A supervised machine learning approach to diagnose human and canine oral squamous cell carcinoma from a very small dataset of nonlinear multiphoton multimodal microscopy images of unstained biopsies The long -term goal this work was to evaluate the combination of our NMMMIs and deep learning as a basis to develop an all -in-one identification method that can replace multiple different staining modalities, in order to improve patient outcomes by integrating AI in healthcare diagnostics. In particular, the work presented here demonstrates a first step to appl ying deep learning image classification to NMMMI histological images from datasets that are not large enough for conventional deep learning image 148 classification methods. The initial model architecture optimization experiments showed that a 14 -layer, convol utional neural network, integrated with dropout layers and ReLU activation functions were ideal for working with the colorectal -histology MNIST Kaggle dataset and the NMMMIs. The Adam optimizer, with a LR = 0.001, outperformed the Nadam, Adagrad, Adadelta , and the Adamax optimizer functions with any of the four LRs that were evaluated, e.g. 0.01, 0.002, 0.001, and 0.0001. We determined that rescaling the NMMIs in order to artificially match the resolution of the Kaggle images was crucial. The combination of Keras callback functions (ReduceLROnPlateau) and implementation of RMSprop with the optimal architecture, optimizer function, and LR, resulted in a validation accuracy Ña subset of images from a portion of the testing set, unseen to the classifier, of 67 .4% and loss of 80.1% which was an improvement of 2% and 17% when compared to the model fed the original resolution NMMMIs. Furthermore, to improve these values in hopes of creating a robust generalized model, we employed TrL of the model previously traine d on the Kaggle dataset with the recalled NMMMIs. We determined the optimal number of frozen and retrainable layers of the saved Kaggle model was, four FL and 10 retrainable layers, which performed better than the recommended ratio of FL to retrainable lay ers found in the Keras documentation. Using four FL in the transfer learning model resulted in a testing (validation) accuracy of 70.1%. We conclude that these results show promise, and with future work, such as labeling the tiled NMMMIs on an individual basis, re -optimizing the model architecture with respect to the NMMMIs and not an MNIST dataset, such as the Kaggle dataset used here, as well as performing TrL on a another model, previously trained on a larger dataset, will result in the metrics necessar y to have a robust and generalized image classification model that can use NMMMI to diagnose cancer without use of expensive staining equipment and painful biopsies. 149 APPENDICES 150 APPENDIX -I Extra information on the fits from Chapter III Figure 73 (Left). Lifetime decay fits and the corresponding residual plots for the RPE and choroid for the wavelength range of 556 -594 nm. This figure shows the RPE and choroid fit poorly to a mono -exponential function. (Right). Lifetime decay fits and the corresponding residual plots for the ORL through NFL for the wavelength range of 610 -648 nm. This figure demonstrates that the ORL -NFL layers fit poorly to a tri -exponential function Figure 73 left and right show selected representative ca ses of how the criteria for excluding a certain fit type were. For the ORL through the NFL the tri -exponential fits were discarded in Table 4 as the values for t 2 and t 3 were identical ( Figure 73 ). In Table 5, where the wavelengths were held fixed, the val ue for a 1 in the fits became negative. For the choroid and RPE, the mono -exponential fits were discarded as the R 2 value was always below 0.91 ( Figure 73 right), which is well below the lowest accepted R 2 value in either of the tables (R 2=0.98) in this wor k. 151 APPENDIX -II A2E solution measurements from Chapter III Figure 74 . The spectra (Left) and lifetime (Right) from a 1 mM solution of A2E. The spectra peaks near 625 nm, which is the same peak seen in the choroid, RPE, and receptor layers. The lifetime of ~173 ps agrees with the literature values [29]. The emission spectrum of A2E is highly dependent on the excitation wavelength [142] . Therefore, we measured an isolated 1 mM solut ion to more accurately compare with the retina data. Our measurements show that the A2E solution has a peak wavelength ~625 nm, which is the same peak that can be seen in the choroid, RPE, and receptor layers. It is worth noting that the dichroic that was used to separate the fundamental from the fluorescence in the retina data starts to decrease in transmission at wavelengths greater than 650 nm. By the time the A2E solution was measured this optic had been replaced with one that has an edge at 750 nm. The lifetime was measured from the range ~600 -670 nm. The value of ~173 ps agrees well with the literature value [44] . 152 APPENDIX -III Publication List 1.!G.A. Murashova , Van Den Berg, N.S., Capes, E., Agnew, D., Rosenthal, E., Dantus, M., Qiu, Z., Spence, D., ÒTetra -modal multiphoton microscopy: A non -invasive technique for augmented histopathological analysis of oral squamous cell carcinoma biopsies,Ó Biomed. Opt. Express. (2019) (In progress). 2.!G.A. Murashova , Colbry, D., ÒA supervised machine learning approa ch to diagnose human and canine oral squamous cell carcinoma from a very small NMMM unstained images dataset.Ó (2019) (In Progress) 3.!G.A. Murashova , C.A. Mancuso, J.L. Canfield, S. Sakami, K. Palczewski, G. Palczewska, and M. Dantus, ÒMultimodal nonlinear optical imaging of unstained retinas in the epi -direction with a sub -40 fs Yb -fiber laser,Ó Biomed. Opt. Express 11, 5228 (2017). (Published) 4.!G.A. Murashova , C.A. Mancuso, S. Sakami, K. Palczewski, G. Palczewska, and M. Dantus, ÒEpidirection detected multimodal imaging of an unstained mouse retina with a Yb -fiber laser,Ó Proc. SPIE 10069, 100692K (2017). (Published) 5.!Saytashev, R. Glenn, G.A. Murashova , S. Osseiran, D. Spence, C.L. Evans, and M. Dantus, ÒMultiphoton excited hemoglobin fluorescence and third harmonic generation for noninvasive microscopy of stored blood,Ó Biomed. Opt. Express 7, 3449 -3460 (2016). (Published) 153 BIBLIOGRAPHY 154 BIBLIOGRAPHY 1. M. Gıppert -Mayer, "Elementary processes with two quantum transitions," Ann. der Phys. 18, 466Ð479 (2009). 2. I. V. Hertel and C. -P. Schulz, "Coherence and Photons," in (Springer, Berlin, Heidelberg, 2015), pp. 71 Ð134. 3. I. Saytashev, R. Glenn, G. A. Murashova, S. Osseiran, D. Spence, C. L. Evans, and M. Dantus, "Multiphoton excited hemoglobin fluorescence and third harmonic generation for non -invasive microscopy of stored blood," Biomed. Opt. Express 7, 3449 (2016). 4. W. Zheng, D. Li, Y. Zeng, Y. Luo, and J. Y. Qu, "Two -photon excited hemoglobin fluorescence," Biomed. Opt. Express 2, 71 Ð79 (2011). 5. E. Chaigneau, M. Oheim, E. Audinat, and S. Charpak, "Two -photon imaging of capillary blood flow in olfactory bulb glom eruli," Proc. Natl. Acad. Sci. U.S.A. 100, 13081 Ð13086 (2003). 6. R. Jimenez and F. E. Romesberg, "Excited state dynamics and heterogeneity of folded and unfolded states of cytochrome C," Phys. Chem 106, 9172 Ð9180 (2002). 7. W. Min, S. Lu, S. Chong, R. Roy, G. R. Holtom, and X. S. Xie, "Imaging chromophores with undetectable fluorescence by stimulated emission microscopy," Nature 461, 1105 Ð1109 (2009). 8. G. A. Murashova, C. A. Mancuso, J. L. Canfield, S. Sakami, K. Palczewski, G. Palczewska, and M. D antus, "Multimodal nonlinear optical imaging of unstained retinas in the epi -direction with a sub -40 fs Yb -fiber laser," Biomed. Opt. Express 8, 5228 (2017). 9. D. Fornell, "Recent Advances in MRI Technology," https://www.itnonline.com/article/recent -adv ances -mri -technology. 10. W. G. Bradley, "History of medical imaging.," Proc. Am. Philos. Soc. 152 , 349 Ð361 (2008). 11. A. M. Legendre, Nouvelles M”thodes Pour La D”termination Des Orbites Des Com‘tes - Adrien Marie Legendre - Google Books (1805). 12. M. Turk and A. Pentland, "Eigenfaces for recognition," J. Cogn. Neurosci. 3, 71 Ð86 (1991). 13. K. Fukushima, "Neocognitron: A self -organizing neural network model for a mechanism of pattern recognition unaffected by shift in position," Biol. Cybern. 36, 193 Ð202 (1980). 14. G. Schultz and G. Schultz, "Image Reconstruction in MRI," in Magnetic Resonance Imaging with Nonlinear Gradient Fields (Springer Fachmedien Wiesbaden, 2013), pp. 39 Ð101. 15. D. Brinks, R. Hildner, E. M. H. P. Van Dijk, F. D. Stefa ni, J. B. Nieder, J. Hernando, and N. F. Van Hulst, Ultrafast Dynamics of Single Molecules (Royal Society of Chemistry, 2014), Vol. 43. 16. V. V Lozovoy, B. Xu, Y. Coello, and M. Dantus, "Direct measurement of spectral phase for ultrashort laser pulses," Opt. Express 16, 592 (2008). 17. "Image Preprocessing - Keras Documentation," https://keras.io/preprocessing/image/. 155 18. "Ziff -deep -learning/CIFAR10 -DataGenerator.ipynb at master ' sukilau/Ziff -deep -learning," https://github.com/sukilau/Ziff -deep -learning/blob/master/2 -CIFAR10 -data -generator/CIFAR10 -DataGenerator.ipynb. 19. Maiman, "Optical and Microwave Experiments in Ruby," Phys. Rev. Lett. 4, 564 Ð567 (1960). 20. S. C. Shim, "Elements of Organic Photochemi stry," Photochem. Photobiol. 31, 187 Ð187 (1980). 21. A. G. Wedum, E. Hanel, and G. B. Phillips, "Ultraviolet Sterilization in Microbiological Laboratories," Source Public Heal. Reports 71, 11 (1956). 22. M. Born and E. Wolf, Principles of Optics Electr omagnetic Theory of Propagation, Interference and Diffraction of Light , 7th ed. (Pergamon Press, 2003). 23. H. A. Alturkistani, F. M. Tashkandi, and Z. M. Mohammedsaleh, "Histological Stains: A Literature Review and Case Study.," Glob. J. Health Sci. 8, 72Ð9 (2015). 24. A. H. Fischer, K. A. Jacobson, J. Rose, and R. Zeller, "Hematoxylin and eosin staining of tissue and cell sections.," CSH Protoc. 2008, pdb.prot4986 (2008). 25. M. A. Piccinin and J. Schwartz, Histology, Verhoeff Stain (StatPearls Publ ishing, 2019). 26. W. Denk, J. H. Strickler, and W. W. Webb, "Two -photon laser scanning fluorescence microscopy.," Science 248, 73 Ð6 (1990). 27. D. Yelin and Y. Silberberg, "Laser scanning third -harmonic -generation microscopy in biology.," Opt. Express 5, 169 Ð175 (1999). 28. G. L. Monroy, R. L. Shelton, R. M. Nolan, C. T. Nguyen, M. A. Novak, M. C. Hill, D. T. McCormick, and S. A. Boppart, "Noninvasive depth -resolved optical measurements of the tympanic membrane and middle ear for differentiating otit is media," Laryngoscope 125, E276 ÐE282 (2015). 29. N. G. Horton, K. Wang, D. Kobat, C. G. Clark, F. W. Wise, C. B. Schaffer, and C. Xu, "In vivo three -photon microscopy of subcortical structures within an intact mouse brain," Nat. Photonics 7, 205Ð209 (2 013). 30. K. Konig, "Multiphoton microscopy in life sciences," J. Microsc. 200, 83 Ð104 (2000). 31. A. Negrean and H. D. Mansvelder, "Optimal lens design and use in laser -scanning microscopy," Biomed. Opt. Express 5, 1588 (2014). 32. P. A. Franken, A. E. Hill, C. W. Peters, and G. Weinreich, "Generation of Optical Harmonics," Phys. Rev. Lett. 7, 118 Ð119 (1961). 33. R. W. Boyd, Nonlinear Optics (Academic Press, 2008). 34. B. Valeur and M. N. Berberan -Santos, Molecular Fluorescence ! : Principles and A pplications (Wiley -VCH, 2013). 35. National Eye Institute, "Facts About Age -Related Macular Degeneration | National Eye Institute," https://nei.nih.gov/health/maculardegen/armd_facts. 156 36. National Eye Institute, "Age -Related Macular Degeneration (AMD) | National Eye Institute," https://nei.nih.gov/eyedata/amd. 37. L. O. Lund University, "Einstein coefficients," in Springer Series in Surface Sciences (2016), Vol. 44, pp. 229 Ð254. 38. W. Becker, The Bh TCSPC Handbook , sixth (2010). 39. T. R. Kuphald t, "Lessons In Electric Circuits: DC," Open B. Proj. III, 1Ð526 (2009). 40. R. Glenn and M. Dantus, "Molecular level crossing and the geometric phase effect from the optical Hanle perspective," Phys. Rev. A 93, (2016). 41. D. Brinks, R. Hildner, E. M. H. P. van Dijk, F. D. Stefani, J. B. Nieder, J. Hernando, and N. F. van Hulst, "Ultrafast dynamics of single molecules," Chem. Soc. Rev. 43, 2476 (2014). 42. L. W. Peng, M. Dantus, A. H. Zewail, K. Kemnitz, J. M. Hicks, and K. B. Eisenthal, "Stepwise Solvation of the Intramolecular -Charge -Transfer Molecule p -( Dlmethy1amino)benzonitrlle," J . Phys. Chem 91, 6162 Ð6167 (1987). 43. A. Konar, V. V Lozovoy, and M. Dantus, "Solvation stokes -shift dynamics studied by chirped femtosecond laser pulses," J. Phys. Chem. Lett. 3, 2458 Ð2464 (2012). 44. D. Schweitzer, S. Schenke, M. Hammer, F. Schweitzer, S. Jentsch, E. Birckner, W. Becker, and A. Bergmann, "Towards metabolic mapping of the human retina," Microsc. Res. Tech. 70, 410 Ð419 (2007). 45. M. C. Skala, K. M. Riching, A. Gendron -Fitzpatrick, J. Eickhoff, K. W. Eliceiri, J. G. White, and N. Ramanujam, "In vivo multiphoton microscopy of NADH and FAD redox states, fluorescence lifetimes, and cellular morphology in precancerous epithelia.," Proc. Natl. Acad. Sci. 104, 19494 Ð19499 (2007). 46. M. Na irat, A. Konar, M. Kaniecki, V. V Lozovoy, and M. Dantus, "Investigating the role of human serum albumin protein pocket on the excited state dynamics of indocyanine green using shaped femtosecond laser pulses," Phys. Chem. Chem. Phys. Phys. Chem. Chem. Phy s 5872, 5872Ð5877 (2015). 47. L. J. Butler, "CHEMICAL REACTION DYNAMICS BEYOND THE BORN -OPPENHEIMER APPROXIMATION," Annu. Rev. Phys. Chem. 49, 125 Ð171 (1998). 48. J. R. Lakowicz, H. K. Szmacinski, K. Nowaczyk, and M. L. Johnson, "Fluorescence lifetime imaging of free and protein -bound NADH.," Proc. Natl. Acad. Sci. 89, 1271 Ð1275 (1992). 49. J. Liu, Y. Itagaki, S. Ben -Shabat, K. Nakanishi, and J. R. Sparrow, "The biosynthesis of A2E, a fluorophore of aging retina, involves the formation of the precurso r, A2 -PE, in the photoreceptor outer segment membrane," J. Biol. Chem. 275, 29354 Ð29360 (2000). 50. L. R. Weisel, P. Xi, Y. Andegeko, V. V. Lovozoy, and M. Dantus, "Greater signal and contrast in two -photon microscopy with ultrashort pulses," in Progress in Biomedical Optics and Imaging - Proceedings of SPIE (2008), Vol. 6860, p. 5. 157 51. P. Xi, Y. Andegeko, D. Pestov, V. V Lovozoy, and M. Dantus, "Two -photon imaging using adaptive phase compensated ultrashort laser pulses," J. Biomed. Opt. 14, 014002 (20 09). 52. W. R. Zipfel, R. M. Williams, and W. W. Webb, "Nonlinear magic: multiphoton microscopy in the biosciences.," Nat. Biotechnol. 21, 1369 Ð77 (2003). 53. G. T. Nogueira, B. Xu, Y. Coello, M. Dantus, and F. C. Cruz, "Broadband 2.12 GHz Ti:sapphire laser compressed to 5.9 femtoseconds using MIIPS.," Opt. Express 16, 10033 Ð10038 (2008). 54. D. Pestov, V. V Lozovoy, and M. Dantus, "Single -beam shaper -based pulse characterization and compression using MIIPS sonogram," Opt. Lett. 35, 1422 Ð4 (2010). 55. P. Xi, Y. Andegeko, D. Pestov, V. V Lovozoy, and M. Dantus, "Two -photon imaging using adaptive phase compensated ultrashort laser pulses," J. Biomed. Opt. 14, 014002 (2009). 56. M. Dantus, "SC352 Pulse Shaping Notes," 42 (2016). 57. Y. Coello, V. V. Lozovoy, T. C. Gunaratne, B. Xu, I. Borukhovich, C. Tseng, T. Weinacht, and M. Dantus, "Interference without an interferometer: a different approach to measuring, compressing, and shaping ultrashort laser pulses," J. Opt. Soc. Am. B 25, A140 (2008). 58. B. M. Dantus, V. Lozovoy, and I. Pastirk, "Analogous to a femtosecond Wheatstone bridge, the MIIPS system can perform pulse characterization and compensation rapidly and efficiently," 15 Ð17 (2003). 59. L. Bachmann, D. M. Zezell, A. da C. Ribeiro, L. Gom es, and A. S. Ito, "Fluorescence Spectroscopy of Biological Tissues ÑA Review," Appl. Spectrosc. Rev. 41, 575 Ð590 (2006). 60. A. R. Young, "Chromophores in human skin," Phys. Med. Biol. 42, 789 Ð802 (1997). 61. N. Ramanujam, "Fluorescence Spectroscopy In Vivo," in Encyclopedia of Analytical Chemistry , R.A. Meyers, ed. (John Wiley & Sons Ltd, 2009), pp. 20 Ð56. 62. N. Kollias and G. N. Stamatas, "Optical Non -Invasive Approaches to Diagnosis of Skin Diseases," J. Investig. Dermatology Symp. Proc. 7, 64 Ð75 (2002). 63. B. I. Tarnowski, F. G. Spinale, and J. H. Nicholson, "DAPI as a useful stain for nuclear quantitation.," Biotech. Histochem. 66, 297 Ð302 (1991). 64. "Protocols DAPI Staining Solution (ab228549) | Abcam," https://www.abcam.com/dapi -staining -solution -ab228549 -protocols.html#top -521. 65. S. M. Stigler, The History of Statistics ! : The Measurement of Uncertainty before 1900 (Belknap Press of Harvard University Press, 1986). 66. S. M. Stigler, "Mathematical Statistics in the Early States," Ann . Stat. 6, 239 Ð265 (1978). 67. F. R. Spellman, Handbook of Environmental Engineering (n.d.). 68. S. J. Miller, The Method of Least Squares (n.d.). 69. "4.4.3.1. Least Squares," https://www.itl.nist.gov/div898/handbook/pmd/section4/pmd431.htm. 158 70. P. C. Hansen, Discrete Inverse Problems ! : Insight and Algorithms (Society for Industrial and Applied Mathematics, 2010). 71. M. Bertero, "Introduction to Inverse Problems in Imaging," (1998). 72. E. Herrholz, D. Lorenz, G. Teschke, and D. Trede, Sparsity and Compressed Sensing in Inverse Problems (n.d.). 73. T. M. Mitchell, Machine Learning (McGraw Hill, 2017). 74. Yaser S. Abu -Mostafa, Malik Magdon -Ismail, and Hsuan -Tien Lin, Learning From Data (2012). 75. "Neural Network Architecture," https://www .dspguide.com/ch26/2.htm. 76. D.-A. Clevert, T. Unterthiner, and S. Hochreiter, "Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)," (2015). 77. J. Friedman, T. Hastie, and R. Tibshirani, The Elements of Statistical Learning: D ata Mining, Inference, and Prediction , Second (Springer -Verlag, 2009). 78. Sebastian Raschka and Vahid Mirjalili, Python Machine Learning: Machine Learning and Deep Learning with Python, Scikit -Learn, and TensorFlow, 2nd Edition (2017). 79. D. Soekhoe, P. van der Putten, and A. Plaat, "On the impact of data set size in transfer learning using deep neural networks," in Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Spr inger Verlag, 2016), Vol. 9897 LNCS, pp. 50 Ð60. 80. H. C. Shin, H. R. Roth, M. Gao, L. Lu, Z. Xu, I. Nogues, J. Yao, D. Mollura, and R. M. Summers, "Deep Convolutional Neural Networks for Computer -Aided Detection: CNN Architectures, Dataset Characteristi cs and Transfer Learning," IEEE Trans. Med. Imaging 35, 1285 Ð1298 (2016). 81. J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, "How transferable are features in deep neural networks?," in Advances in Neural Information Processing Systems (Neural informat ion processing systems foundation, 2014), Vol. 4, pp. 3320 Ð3328. 82. E. Shelhamer, J. Long, and T. Darrell, "Fully Convolutional Networks for Semantic Segmentation.," IEEE Trans. Pattern Anal. Mach. Intell. 39, 640 Ð651 (2017). 83. A. Krizhevsky, I. Sutskever, and G. E. Hinton, ImageNet Classification with Deep Convolutional Neural Networks (n.d.). 84. C. Xu, W. Zipfel, J. B. Shear, R. M. Williams, and W. W. Webb, "Multiphoton fluorescence excitation: new spectral windows for biol ogical nonlinear microscopy," Proc. Natl. Acad. Sci. U.S.A. 93, 10763 Ð10768 (1996). 85. I. Moon, F. Yi, Y. H. Lee, B. Javidi, D. Boss, and P. Marquet, "Automated quantitative analysis of 3D morphology and mean corpuscular hemoglobin in human red blood ce lls stored in different periods," Opt. Express 21, 30947 Ð30957 (2013). 86. S. M. Frank, B. Abazyan, M. Ono, C. W. Hogue, D. B. Cohen, D. E. Berkowitz, P. M. Ness, and V. M. Barodka, "Decreased erythrocyte deformability after transfusion and the effects o f 159 erythrocyte storage duration," Anesth. Analg. 116, 975 Ð981 (2013). 87. O. N. Salaria, V. M. Barodka, C. W. Hogue, D. E. Berkowitz, P. M. Ness, J. O. Wasey, and S. M. Frank, "Impaired red blood cell deformability after transfusion of stored allogeneic b lood but not autologous salvaged blood in cardiac surgery patients, Anesth," Anesth. Analg. 118, 1179 Ð1187 (2014). 88. A. B. Zimrin and J. R. Hess, "Current issues relating to the transfusion of stored red blood cells," Vox Sang 96, 93 Ð103 (2009). 89. G. Edgren, M. Kamper -Jorgensen, S. Eloranta, K. Rostgaard, B. Custer, H. Ullum, E. L. Murphy, M. P. Busch, M. Reilly, M. Melbye, H. Hjalgrim, and O. Nyr n, "Duration of red blood cell storage and survival of transfused patients (CME)," Transfusion 50, 118 5Ð1195 (2010). 90. S. D. Robinson, C. Janssen, E. B. Fretz, B. Berry, A. J. Chase, A. D. Siega, R. G. Carere, A. Fung, G. Simkus, W. P. Klinke, and J. D. Hilton, "Red blood cell storage duration and mortality in patients undergoing percutaneous coronary intervention," Am. Hear. J. 159, 876 Ð881 (2010). 91. A. H. M. van Straten, M. A. S. Hamad, A. A. van Zundert, E. J. Martens, J. F. ter Woorst, A. M. de Wolf, and V. Scharnhorst, "Effect of duration of red blood cell storage on early and late mortality af ter coronary artery bypass grafting," Thorac. Cardiovasc. Surg. 141, 231 Ð237 (2011). 92. S. D. Shukla, R. Coleman, J. B. Finean, and R. H. Michell, "The use of phospholipase c to detect structural changes in the membranes of human erythrocytes aged by st orage," Biophys. Acta 512, 341Ð349 (1978). 93. B. Blasi, A. DÕAlessandro, N. Ramundo, and L. Zolla, "Red blood cell storage and cell morphology," Med 22, 90 Ð96 (2012). 94. D. Fu, T. Ye, T. E. Matthews, B. J. Chen, G. Yurtserver, and W. S. Warren, "High -resolution in vivo imaging of blood vessels without labeling," Lett. 32, 2641 Ð2643 (2007). 95. W. Zheng, D. Li, Y. Zeng, Y. Luo, and J. Y. Qu, "Two -photon excited hemoglobin fluorescence.," Biomed. Opt. Express 2, 71 Ð9 (2010). 96. D. Li, W. Zheng, Y. Zeng, Y. Luo, and J. Y. Qu, "Two -photon excited hemoglobin fluorescence provides contrast mechanism for label -free imaging of microvasculature in vivo," Opt. Lett. 36, 834Ð836 (2011). 97. R. D. Schaller, J. C. Johnson, and R. J. Saykally, "Nonlinear chem ical imaging microscopy: near -field third harmonic generation imaging of human red blood cells," Anal. Chem. 72, 5361 Ð5364 (2000). 98. G. O. Clay, A. C. Millard, C. B. Schaffer, J. Aus -der -Au, P. S. Tsai, J. A. Squier, and D. Kleinfeld, "Spectroscopy of third -harmonic generation: evidence for resonances in model compounds and ligated hemoglobin," J. Opt. Soc. Am. B 23, 932 Ð950 (2006). 99. B. Nie, I. Saytashev, A. Chong, H. Liu, S. N. Arkhipov, F. W. Wise, and M. Dantus, "Multimodal microscopy with sub -30 fs Yb fiber laser oscillator," Biomed. Opt. Express 3, 1750 Ð1756 (2012). 100. B. Nie, D. Pestov, F. W. Wise, and M. Dantus, "An ultr afast fiber laser with self -similar evolution 160 in the gain segment," Opt. Photonics News 22, 47 (2011). 101. S. Lu, W. Min, S. Chong, G. R. Holtom, and X. S. Xie, "Label -free imaging of heme proteins with two -photon excited photothermal lens microscopy," Appl. Phys. Lett. 96, (2010). 102. J. A. Meyer, J. M. Froelich, G. E. Reid, W. K. Karunarathne, and D. M. S pence, "Metal -activated C-peptide Facilitates Glucose Clearance and the Release of a Nitric Oxide Stimulus via the GLUT1 Transporter," Diabetologia 51, 175 Ð182 (2007). 103. Y. Wang, A. Giebink, and D. M. Spence, "Microfluidic evaluation of red cells coll ected and stored in modified processing solutions used in blood banking," Integr. Biol. 6, 65 Ð75 (2014). 104. J. T. Dodge, C. Mitchell, and D. J. Hanahan, "The preparation and chemical characteristics of hemoglobin -free ghosts of human erythrocytes," Arc h. Biochem. Biophys. 100, 119 Ð130 (1963). 105. R. Dasgupta, S. Ahlawat, R. S. Verma, A. Uppal, and P. K. Gupta, "Hemoglobin degradation in human erythrocytes with long -duration near -infrared laser exposure in Raman optical tweezers," J. Biomed. Opt 15, ( 2010). 106. J. P. Greer, D. A. Arber, B. Glader, A. F. List, R. T. Means, F. Paraskevas, and G. M. Rodgers, WintrobeÕs Clinical Hematology (Lippincott Williams & Wilkins p. 3094, 2013). 107. K. Jaferzadeh and I. Moon, "Quantitative investigation of red blood cell three -dimensional geometric and chemical changes in the storage lesion using digital holographic microscopy," J. Biomed. Opt. 20, (2015). 108. G. Brecher and M. Bessis, "Present status of spiculed red cells and their relationship to the disco cyte -echinocyte transformation: a critical review," Blood 40, 333 Ð344 (1972). 109. B. F. Rodak, Diagnostic Hematology (1995). 110. B. J. Bain, Blood Cells: A Practical Guide , 5th ed. (NJ, 2015). 111. Y. Barad, H. Eisenberg, M. Horowitz, and Y. Silber berg, "Nonlinear scanning laser microscopy by third harmonic generation," Appl. Phys. Lett. 70, 922 Ð924 (1997). 112. D. Debarre, N. Olivier, and E. Beaurepaire, "Signal epidetection in third -harmonic generation microscopy of turbid media," Opt. Exress 15, 8913 Ð8924 (2007). 113. Y. Zeng, B. Yan, Q. Sun, S. K. Teh, W. Zhang, Z. Wen, and J. Y. Qu, "Label -free in vivo imaging of human leukocytes using two -photon excited endogenous fluorescence," J. Biomed. Opt. 18, (2013). 114. G. O. Clay, C. B. Schaffer, and D. Kleinfeld, "Large two -photon absorptivity of hemoglobin in the infrared range of 780 -880 nm," J. Chem. Phys. 126, (2007). 115. A. A. Rehms and P. R. Callis, "Two -photon fluorescence excitation spectra of aromatic amino acids," Chem. Phys. Lett. 208, 276 Ð282 (1993). 116. B. Kierdaszuk, I. Gryczynski, M. -W. A, A. Bzowska, D. Shugar, and J. R. Lakowicz, "Fluorescence of tyrosine and tryptophan in proteins using one - and two -photon excitation," Photochem. Photobiol. 61, 319 Ð 324 (1995). 161 117. S. Yo shida, T. Iizuka, T. Nozawa, and M. Hatano, "Studies on the charge transfer band in high spin state of ferric myoglobin and hemoglobin by low temperature optical and magnetic circular dichroism spectroscopy," Biochim. Biophys. Acta 405, 122 Ð135 (1975). 118. L. M. Snyder, F. Garver, S. C. Liu, L. Leb, J. Trainor, and N. L. Fortier, "Demonstration of haemoglobin associated with isolated, purified spectrin from senescent human red cells," Br. J. Haematol. 61, 415 Ð419 (1985). 119. P. P. Ho and R. R. Alfano, "Optical Kerr Effect in Liquids," Phys. Rev. A. 20, 2170 Ð2187 (1979). 120. M. Mler, J. Squier, K. R. Wilson, and G. J. Brakenhoff, "3D microscopy of transparent objects using third -harmonic generation," J. Microsc. 191, 266 Ð274 (1998). 121. R. M. Williams, W. R. Zipfel, and W. W. Webb, "Interpreting second -harmonic generation images of collagen I fibrils.," Biophys. J. 88, 1377 Ð86 (2005). 122. P. Xi, Y. Andegeko, L. R. Weisel, V. V Lozovoy, and M. Dantus, "Greater signal, increased depth, and less photobleaching in two -photon microscopy with 10 fs pulses," Opt. Commun. 281, 1841Ð1849 (2008). 123. B. Nie, D. Pestov, F. W. Wise, and M. Dantus, "Generation of 42 -fs and 10 -nJ pulses from a fiber laser with self -similar evolution in the gain s egment," Opt. Express 19, 12074 (2011). 124. D. Huang, E. A. Swanson, C. P. Lin, J. S. Schuman, W. G. Stinson, W. Chang, M. R. Hee, T. Flotte, K. Gregory, C. A. Puliafito, and J. G. Fujimoto, "Optical coherence tomography.," Science 254, 1178 Ð81 (1991). 125. M. R. Hee, J. A. Izatt, E. A. Swanson, D. Huang, J. S. Schuman, C. P. Lin, C. A. Puliafito, and J. G. Fujimoto, "Optical Coherence Tomography of the Human Retina," Arch. Ophthalmol. 113, 325Ð332 (1995). 126. M. Dantus, Femtosecond Laser Shaping: F rom Laboratory to Industry (CRC Press, 2017). 127. J. C. Sutherland and K. P. Griffin, "Absorption Spectrum of DNA for Wavelengths Greater than 300 nm," Radiat. Res. 86, 399 (1981). 128. Y. Mattley, "Measuring DNA Absorbance with the STS -UV Microspectr ometer," 1 Ð3 (n.d.). 129. S. L. Jacques, "Physics in Medicine & Biology Optical properties of biological tissues: a review Optical properties of human tissues A N Bashkatov, E A Genina, V I Kochubey et al. - Optical absorption and scattering properti es of bulk porcine muscle phantoms from int," Phys. Med. Biol 58, (2013). 130. F. Bevilacqua, D. Piguet, P. Marquet, J. D. Gross, B. J. Tromberg, and C. Depeursinge, "In vivo local determination of tissue optical properties: applications to human brain," Appl. Opt. 38, 4939 (1999). 131. W. Denk, J. H. Strickler, and W. W. Webb, "Two -photon laser scanning fluorescence microscopy," Science (80 -. ). 248, 73 Ð76 (1990). 132. E. A. Gibson, O. Masihzadeh, T. C. Lei, D. A. Ammar, and M. Y. Kahook, "Multiphoto n 162 Microscopy for Ophthalmic Imaging," J. Ophthalmol. 2011 , 1Ð11 (2011). 133. S. Huang, A. A. Heikal, and W. W. Webb, "Two -photon fluorescence spectroscopy and microscopy of NAD(P)H and flavoprotein.," Biophys. J. 82, 2811 Ð25 (2002). 134. S. He, C. Ye, Q. Sun, C. K. S. Leung, and J. Y. Qu, "Label -free nonlinear optical imaging of mouse retina," Biomed. Opt. Express 6, (2015). 135. J. M. Bueno, E. J. Gualda, and P. Artal, "Adaptive optics multiphoton microscopy to study ex vivo ocular tissues," J. Biome d. Opt. 15, 066004 (2010). 136. F. Aptel, N. Olivier, A. Deniset -Besseau, J. -M. Legeais, K. Plamann, M. -C. Schanne -Klein, and E. Beaurepaire, "Multimodal Nonlinear Imaging of the Human Cornea," Investig. Opthalmology Vis. Sci. 51, 2459 (2010). 137. D. D”barre, W. Supatto, A. -M. M. Pena, A. Fabre, T. Tordjmann, L. Combettes, M. -C. C. Schanne -Klein, and E. Beaurepaire, "Imaging lipid bodies in cells and tissues using third -harmonic generation microscopy," Nat. Methods 3, 47 Ð53 (2006). 138. O. Masihzadeh , T. C. Lei, S. R. Domingue, M. Y. Kahook, R. A. Bartels, and D. A. Ammar, "Third harmonic generation microscopy of a mouse retina.," Mol. Vis. 21, 538 Ð47 (2015). 139. G. Palczewska, A. Maeda, M. Golczak, E. Arai, Z. Dong, L. Perusek, B. Kevany, and K. Palczewski, "Receptor MER tyrosine kinase proto -oncogene (MERTK) is not required for transfer of bis -retinoids to the retinal pigmented epithelium," J. Biol. Chem. 291, 26937 Ð26949 (2016). 140. R. W. Young and D. Bok, "Participation of the retinal pigment epithelium in the rod outer segment renewal process.," J. Cell Biol. 42, 392 Ð403 (1969). 141. M. E. Boulton, "Studying melanin and lipofuscin in RPE cell culture mode ls.," Exp. Eye Res. 126, 61 Ð7 (2014). 142. J. Sparrow and T. Duncker, "Fundus Autofluorescence and RPE Lipofuscin in Age -Related Macular Degeneration," J. Clin. Med. 3, 1302 Ð1321 (2014). 143. M. L. Katz, W. L. Stone, and E. A. Dratz, "Fluorescent pigme nt accumulation in retinal pigment epithelium of antioxidant -deficient rats," Investig. Ophthalmol. Vis. Sci. 17, 1049 Ð1058 (1978). 144. J. J. Weiter, F. C. Delori, G. L. Wing, and K. A. Fitch, "Retinal pigment epithelial lipofuscin and melanin and choro idal melanin in human eyes," Investig. Ophthalmol. Vis. Sci. 27, 145 Ð152 (1986). 145. F. C. Delori, C. K. Dorey, G. Staurenghi, O. Arend, D. G. Goger, and J. J. Writer, "In Vivo Fluorescence of the Ocular Fundus Exhibits Retinal Pigment Epithelium Lipofu scin Characteristics," 36, 718 Ð729 (1995). 146. C. J. Kennedy, P. E. Rakoczy, and I. J. Constable, "Lipofuscin of the retinal pigment epithelium: a review.," Eye (Lond). 9, 763 Ð771 (1995). 147. F. Holz, R. Spaide, A. C. Bird, and S. Schmitz -Valckenberg , "Lipofuscin of the Retinal Pigment Epithelium," in Atlas of Fundus Autofluorscence Imaging , F. Holz, R. Spaide, A. C. Bird, and S. 163 Schmitz -Valckenberg, eds. (Springer Berlin Heidelberg, 2007), pp. 3 Ð16. 148. S. Warburton, W. E. Davis, K. Southwick, H. Xin, A. T. Woolley, G. F. Burton, and C. D. Thulin, "Proteomic and phototoxic characterization of melanolipofuscin: correlation to disease and model for its origin.," Mol. Vis. 13, 318 Ð29 (2007). 149. C. A. Curcio and M. Johnson, "Structure, Function, an d Pathology of BruchÕs Membrane," in Retina Fifth Edition (2012), Vol. 1, pp. 465 Ð481. 150. J. R. Sparrow, E. Gregory -Roberts, K. Yamamoto, A. Blonska, S. K. Ghosh, K. Ueda, and J. Zhou, "The bisretinoids of retinal pigment epithelium.," Prog. Retin. Eye Res. 31, 121 Ð35 (2012). 151. Y. Coello, B. Xu, T. L. Miller, V. V Lozovoy, and M. Dantus, "Group -velocity dispersion measurements of water, seawater, and ocular components using multiphoton intrapulse interference phase scan," (n.d.). 152. H. Gross, F. Blechinger, and B. Achtner, "Handbook of optical systems, Volume 4: Survey of optical instruments," Handb. Opt. Syst. 4, 1092 (2008). 153. L. I. of America and N/A, ANSI Z136.1 - Safe Use of Lasers (2014). 154. V. V. Lozovoy, I. Pastirk, and M. Dantus, "Multiphoton intrapulse interference. IV. Ultrashort laser pulse spectral phase characterization and compensation," Opt. Lett. 29, 775 Ð777 (2004). 155. T. E. Oliphant, "Python for scientific computing," Comput. Sci. Eng. 9, 10 Ð20 (2007). 156. N. L. Mata, J. Weng, and G. H. Travis, "Biosynthesis of a major lipofuscin fluorophore in mice and humans with ABCR -mediated retinal and macular degeneration," Proc. Natl. Acad. Sci. 97, 7154Ð7159 (2000). 157. L. K. Barthel and P. A. Raymond, "Impr oved method for obtaining 3 -microns cryosections for immunocytochemistry.," J. Histochem. Cytochem. 38, 1383 Ð8 (1990). 158. M. W. Conklin, P. P. Provenzano, K. W. Eliceiri, R. Sullivan, and P. J. Keely, "Fluorescence lifetime imaging of endogenous fluorophores in histopathology sections reveals differences between normal and tumor epithelium in carcinoma in situ of the breast, " Cell Biochem. Biophys. 53, 145 Ð157 (2009). 159. P. V Sarthy and D. M. Lam, "Isolated cells from a mammalian retina," Brain Res 176, 208 Ð212 (1979). 160. J. R. Sparrow, C. a Parish, M. Hashimoto, and K. Nakanishi, "Pigmented Epithelial Cells in Cultur e," Invest. Ophthalmol. Vis. Sci. 40, 2988 Ð2995 (1999). 161. M. Balu, I. Saytashev, J. Hou, M. Dantus, and B. J. Tromberg, "Sub Q40 fs, 1060 Qnm Yb Qfiber laser enhances penetration depth in nonlinear optical microscopy of human skin," J. Biomed. Opt. 20, 120501 (2015). 162. M. Boulton, F. Docchio, P. Dayhaw -Barker, R. Ramponi, and R. Cubeddu, "Age -related changes in the morphology, absorption and fluorescence of melanosomes and lipofuscin granules of the retinal pigment epithelium," Vision Res. 30, 1291 Ð1303 (1990). 164 163. S. Watanabe, G. Matthews, M. L. Woodruff, I. V. Peshenko, A. B. Savchenko, C. L. Makino, Y. -S. Ho, G. L. Fain, and A. M. Dizhoor, "Regional distribution of cGMP -activated ion channels in the plasma membrane of the rod photoreceptor.," J. Neurosci. 8, 2334 Ð7 (1988). 164. B. Chang, N. L. Hawes, R. E. Hurd, M. T. Davisson, S. Nusinowitz, and J. R. Heckenlively, "Retinal degeneration mutants in the mouse," Vision Res. 42, 517 Ð525 (2002). 165. M. Hoon, H. Okawa, L. Della Santina, and R. O. L. Wong, "Functional architecture of the retina: development and disease.," Prog. Retin. Eye Res. 42, 44 Ð84 (2014). 166. I. Rocha -Mendoza, D. R. Yankelevich, M. Wang, K. M. Reiser, C. W. Frank, and A. Knoesen, "Sum Frequency Vibrational Spectroscopy: Th e Molecular Origins of the Optical Second -Order Nonlinearity of Collagen," Biophys. J. 93, 4433 Ð4444 (2007). 167. A. D. Marmorstein, L. Y. Marmorstein, H. Sakaguchi, and J. G. Hollyfield, "Spectral Profiling of Autofluorescence Associated with Lipofuscin , Bruch Õ s Membrane , and Sub -RPE Deposits in Normal and AMD Eyes," 43, 2435 Ð2441 (2017). 168. W. A. N. (William A. N. Dorland, DorlandÕs Illustrated Medical Dictionary. (Saunders/Elsevier, 2012). 169. M. L. Katz, C. -L. Gao, and L. M. Rice, "Formatio n of lipofuscin -like fluorophores by reaction of retinal with photoreceptor outer segments and liposomes," Mech. Ageing Dev. 92, 159 Ð174 (1996). 170. W. G. Robison and M. L. Katz, "Vitamin A and Lipofuscin," in The Microenvironment and Vision , J. B. Shef field and S. R. Hilfer, eds. (Springer New York, 1987), pp. 95 Ð122. 171. J. R. Sparrow, Y. Wu, T. Nagasaki, K. D. Yoon, K. Yamamoto, and J. Zhou, "Fundus autofluorescence and the bisretinoids of retina.," Photochem. Photobiol. Sci. 9, 1480 Ð9 (2010). 172. A. R. Kreimer, G. M. Clifford, P. Boyle, and S. Franceschi, "Human Papillomavirus Types in Head and Neck Squamous Cell Carcinomas Worldwide: A Systematic Review," Cancer Epidemiol Biomarkers Prev 14, 467 Ð75 (2005). 173. B. Neville and T. A. Day, "Oral cancer and precancerous lesions," CA. Cancer J. Clin. 52, 195 Ð215 (2002). 174. J. O. Boyle, J. Hakim, W. Koch, P. Van Der Riet, R. H. Hruban, R. A. Roa, R. Correo, Y. J. Eby, J. M. Ruppert, and D. Sidransky, "The Incidence of p53 Mutations Increases wit h Progression of Head and Neck Cancer," CANCER Res. 53, 447 Ð4480 (1993). 175. R. Baral, S. Patnaik, and B. R. Das, "Co -overexpression of p53 and c -myc proteins linked with advanced stages of betel - and tobacco -related oral squamous cell carcinomas from e astern India," Eur. J. Oral Sci. 106, 907 Ð913 (1998). 176. J. Caamano, S. Y. Zhang, E. A. Rosvold, B. Bauer, and A. J. P. Klein -Szanto, "p53 Alterations in Human Squamous Cell Carcinomas and Carcinoma Cell Lines," Am. J. Pathol. 142, (1993). 177. L. Yan, F. Chen, F. Liu, Y. Qiu, J. Wang, J. Wu, X. Bao, Z. Hu, X. Peng, X. Lin, L. Cai, L. Lin, and B. He, "Differences in modifiable factors of oral squamous cell carcinoma in the upper and 165 lower of oral fissure.," Oncotarget 8, 75094 Ð75101 (2017). 178. A. K. Markopoulos, "Current Aspects on Oral Squamous Cell Carcinoma," Open Dent. J. 6, 126 Ð130 (2012). 179. D. Neena, S. Siddharth, P. Keyuri B, J. Munira, and N. Doshi, "HISTOLOGICAL GRADING OF ORAL CANCER: A COMPARISON OF DIFFERENT SYSTEMS AND THEIR RELATION TO LYMPH NODE METASTASIS," Natl. J. COMMUNITY Med. 2, (2011). 180. World Health Organization, Global Cancer (2008). 181. X. Li, H. Li, X. He, T. Chen, X. Xia, C. Yang, and W. Zheng, "Spectrum - and time -resolved endogenous multiphoton signals reveal quantitative differentiation of premalignant and malignant gastric mucosa.," Biomed. Opt. Express 9, 453 Ð471 (2018). 182. A. K. Glaser, N. P. Reder, Y. Chen, E. F. McCarty, C. Yin, L. Wei, Y. Wang, L. D. True, and J. T. C. Liu, "Light -sheet microscopy for slide -free non -destructive pathology of large clinical specimens," Nat. Biomed. Eng. 1, 0084 (2017). 183. S. You, Y. Sun, E. J. Chaney, Y. Zhao, J. Chen, S. A. Boppart, and H. Tu, "Slide -free virtual histochemistry (Part II): detection of field cancerization," Biomed. Opt. Express 9, 5253 (2018). 184. S. You, Y. Sun, E. J. Chaney, Y. Zhao, J. Chen, S. A . Boppart, and H. Tu, "Slide -free virtual histochemistry (Part I): development via nonlinear optics," Biomed. Opt. Express 9, 5240 (2018). 185. S. You, H. Tu, E. J. Chaney, Y. Sun, Y. Zhao, A. J. Bower, Y. -Z. Liu, M. Marjanovic, S. Sinha, Y. Pu, and S. A . Boppart, "Intravital imaging by simultaneous label -free autofluorescence -multiharmonic microscopy," Nat. Commun. 9, 2125 (2018). 186. Y. Sun, S. You, H. Tu, D. R. Spillman, E. J. Chaney, M. Marjanovic, J. Li, R. Barkalifa, J. Wang, A. M. Higham, N. N. Luckey, K. A. Cradock, Z. George Liu, and S. A. Boppart, "Intraoperative visualization of the tumor microenvironment and quantification of extracellular vesicles by label -free nonlinear imaging," Sci. Adv. 4, eaau5603 (2018). 187. L. Nolte, G. C. Antonop oulos, L. R−misch, A. Heisterkamp, T. Ripken, and H. Meyer, "Enabling second harmonic generation as a contrast mechanism for optical projection tomography (OPT) and scanning laser optical tomography (SLOT)," Biomed. Opt. Express 9, 2627 (2018). 188. B. Z hang, M. Sun, Y. Yang, L. Chen, X. Zou, T. Yang, Y. Hua, and M. Ji, "Rapid, large -scale stimulated Raman histology with strip mosaicing and dual -phase detection," Biomed. Opt. Express 9, 2604 (2018). 189. E. Yew, C. Rowlands, and P. T. C. So, "Applicatio n of Multiphoton Microscopy in Dermatological Studies: a Mini -Review NIH Public Access," J Innov Opt Heal. Sci 7, 1330010 (2014). 190. J. Xu, D. Kang, Y. Zeng, S. Zhuo, X. Zhu, L. Jiang, J. Chen, and J. Lin, "Multiphoton microscopy for label -free identif ication of intramural metastasis in human esophageal squamous cell carcinoma.," Biomed. Opt. Express 8, 3360 Ð3368 (2017). 191. T. Yoshitake, M. G. Giacomelli, L. M. Quintana, H. Vardeh, L. C. Cahill, B. E. Faulkner -Jones, J. 166 L. Connolly, D. Do, and J. G. Fujimoto, "Rapid histopathological imaging of skin and breast cancer surgical specimens using immersion microscopy with ultraviolet surface excitation," Sci. Rep. 8, 4476 (2018). 192. D. Hanahan and R. A. Weinberg, "The Hallmarks of Cancer," Cell 100 , 5 7Ð70 (2000). 193. D. Hanahan and R. A. Weinberg, "Hallmarks of cancer: The next generation," Cell 144, 646 Ð674 (2011). 194. "Pillow Ñ Pillow (PIL Fork) 5.1.1 documentation," https://pillow.readthedocs.io/en/5.1.x/. 195. J. Schindelin, I. Arganda -Carreras, E. Frise, V. Kaynig, M. Longair, T. Pietzsch, S. Preibisch, C. Rueden, S. Saalfeld, B. Schmid, J. -Y. Tinevez, D. J. White, V. Hartenstein, K. Eliceiri, P. Tomancak, and A. Cardona, "Fiji: an open -source platform for bio logical -image analysis," Nat. Methods 9, 676 Ð682 (2012). 196. C. A. Schneider, W. S. Rasband, and K. W. Eliceiri, "NIH Image to ImageJ: 25 years of image analysis.," Nat. Methods 9, 671 Ð5 (2012). 197. TOFRA, "TOFRA, Inc. - Low Cost Components for Micro scope Automation," http://www.tofrainc.net/index.html. 198. S. Preibisch and D. Hırl, "BigStitcher - ImageJ," https://imagej.net/BigStitcher. 199. Daniel Sage, "Quantitative orientation analysis -OrientationJ," http://bigwww.epfl.ch/demo/orientationj/th eoretical -background.pdf. 200. R. Rezakhaniha, A. Agianniotis, J. T. C. Schrauwen, A. Griffa, D. Sage, C. V. C. Bouten, F. N. van de Vosse, M. Unser, N. Stergiopulos, ' A Agianniotis, ' J T C Schrauwen, ' A Griffa, ' D Sage, ' C V C Bouten, ' F N Van De Vosse, ' M Unser, ' N Stergiopulos, J. T. C. Schrauwen, A. Griffa, D. Sage, C. V. C. Bouten, and A. Agianniotis, "Experimental investigation of collagen waviness and orientation in the arterial adventitia using confocal laser scanning microscopy," Biomech. Model. Mechanobiol. 11, 461 Ð473 (2012). 201. D. Zink, A. H. Fischer, and J. A. Nickerson, "Nuclear structure in cancer cells," Nat. Rev. Cancer 4, 677 Ð687 (2004). 202. D. Kass, M. Rabinovitch, R. Visse, and H. Nagase, "Matrix Metalloproteinases and Ti ssue Inhibitors of Metalloproteinases," (n.d.). 203. M. Egeblad and Z. Werb, "New functions for the matrix metalloproteinases in cancer progression," Nat. Rev. Cancer 2, 161 Ð174 (2002). 204. F. Antonicelli, G. Bellon, L. Debelle, and W. Hornebeck, "Ela stin QElastases and Inflamm QAging," Curr. Top. Dev. Biol. 79, 99 Ð155 (2007). 205. V. Bhakhar and N. Bhavsar, "Matrix Metalloproteinases (MMPs) in Oral Squamous Cell Carcinoma," Sch. J. Appl. Med. Sci. (SJAMS 5, 3191 Ð3197 (2017). 206. T. Klein and R. Bischoff, "Physiology and pathophysiology of matrix metalloproteases.," Amino Acids 41, 271 Ð90 (2011). 167 207. N. Choudhry, S. Sarmad, N. U. A. Waheed, and A. J. Gondal, "Estimation of serum matrix metalloproteinases among patients of oral squamous cell carcinoma.," Pakistan J. Med. Sci. 35, 252Ð256 (2019). 208. K. Y. P. Liu, X. J. D. Lu, Y. S. Zhu, N. Le, H. Kim, and C. F. Poh, "Plasma -Derived Inflammatory Proteins Predict Oral Squamous Cell Carcinoma," Front. Oncol. 8, 1 Ð11 (2018). 209. A. Hirshberg, M. Lib, A. Kozlovsky, and I. Kaplan, "The influence of inflammation on the polarization colors of collagen fibers in the wall of odontogenic keratocyst," Oral Oncol. 43, 278 Ð282 (2007). 210. C. Goertzen, H. Mahdi, C. Laliberte, T. Meirson , D. Eymael, H. Gil -Henn, and M. Magalhaes, "Oral inflammation promotes oral squamous cell carcinoma invasion," Oncotarget 9, 29047 Ð29063 (2018). 211. L. V. Galv‰o -Moreira and M. C. F. N. da Cruz, "Oral microbiome, periodontitis and risk of head and neck cancer," Oral Oncol. 53, 17 Ð19 (2016). 212. L. M. Coussens and Z. Werb, "Inflammation and cancer," Nature 420, 860 Ð867 (2002). 213. E. J. Wood, Cellular and Molecular Immunology (5th Ed.): Abbas A. K., and Lichtman, A. H. (John Wiley & Sons, Ltd, 2004 ), Vol. 32. 214. M. Sudhakara, V. Reshma, N. Khan, and S. R. Amulya, "Uncommon features in conventional oral squamous cell carcinoma.," J. Oral Maxillofac. Pathol. 20, 316 Ð9 (2016). 215. A. L. Mescher, "Histology & Its Methods of Study," in JunqueiraÕs Basic Histology, 14e (McGraw -Hill Education, 2016). 216. B. R. Masters, "The Development of Fluorescence Microscopy," (n.d.). 217. P. Theer and W. Denk, On the Fundamental Imaging -Depth Limit in Two -Photon Microscopy (2006). 218. J. Squier and M. Mler, "High resolution nonlinear microscopy: A review of sources and methods for achieving optimal imaging," Rev. Sci. Instrum. 72, 2855 (2001). 219. M. D. Young, J. J. Field, K. E. Sheetz, R. A. Bartels, and J. Squier, "A pragmatic g uide to multiphoton microscope design.," Adv. Opt. photonics 7, 276 Ð378 (2015). 220. A. Bindewald -Wittich, M. Han, S. Schmitz -Valckenberg, S. R. Snyder, G. G. Giese, J. F. Bille, and F. G. Holz, "Two -photon -excited fluorescence imaging of human RPE cells with a femtosecond Ti:sapphire laser," Investig. Ophthalmol. Vis. Sci. 47, 4553 Ð4557 (2006). 221. G. Palczewska, Z. Dong, M. Golczak, J. J. Hunter, D. R. Williams, N. S. Alexander, and K. Palczewski, "Noninvasive two -photon microscopy imaging of mouse r etina and retinal pigment epithelium through the pupil of the eye," Nat. Med. 20, 785 Ð789 (2014). 222. R. Sharma, D. R. Williams, G. Palczewska, K. Palczewski, and J. J. Hunter, "Two -photon autofluorescence imaging reveals cellular structures throughout the retina of the living primate eye," Investig. Ophthalmol. Vis. Sci. 57, 632 Ð646 (2016). 168 223. "4.1.4.1. Linear Least Squares Regression," https://www.itl.nist.gov/div898/handbook/pmd/section1/pmd141.htm. 224. B. Nie, D. Pestov, F. W. Wise, and M. Dant us, "Generation of 42 -fs and 10 -nJ pulses from a fiber laser with self -similar evolution in the gain segment," Opt. Express 19, 12074 (2011). 225. Y. Coello, V. V. Lozovoy, T. C. Gunaratne, B. Xu, I. Borukhovich, C. Tseng, T. Weinacht, and M. Dantus, "In terference without an interferometer: a different approach to measuring, compressing, and shaping ultrashort laser pulses," J. Opt. Soc. Am. B (2008). 226. V. V. Lozovoy, I. Pastirk, and M. Dantus, "Multiphoton intrapulse interference IV Ultrashort laser pulse spectral phase characterization and compensation," Opt. Lett. 29, 775 (2004). 227. P. Xi, Y. Andegeko, D. Pestov, V. V Lovozoy, and M. Dantus, "Two -photon imaging using adaptive phase compensated ultrashort laser pulses," J. Biomed. Opt. 14, 01400 2 (2009). 228. W. Becker, The Bh TCSPC Handbook , sixth (2010). 229. "numpy.ravel Ñ NumPy v1.17 Manual," https://docs.scipy.org/doc/numpy/reference/generated/numpy.ravel.html. 230. "numpy.reshape Ñ NumPy v1.17 Manual," https://docs.scipy.org/doc/numpy/r eference/generated/numpy.reshape.html. 231. F. Santosa and W. W. Symes, "Linear Inversion of Band -Limited Reflection Seismograms," SIAM J. Sci. Stat. Comput. 7, 1307 Ð1330 (1986). 232. R. Tibshirani, "Ridge shrinkage and selection via the lasso," 5, 267 Ð288 (1996). 233. T. Hastie, R. T. Martin, W. Hastie, ¥ Tibshirani, and ¥ Wainwright, Statistical Learning with Sparsity The Lasso and Generalizations Statistical Learning with Sparsity (n.d.). 234. G. (Gareth M. James, An Introduction to Statistical Learning ! : With Applications in R (n.d.). 235. P. D. Kiser, M. Golczak, and K. Palczewski, "Chemistry of the retinoid (visual) cycle," Chem. Rev. 114 , 194 Ð232 (2014). 236. A. Maeda and K. Palczewski, "Retinal degeneration in animal models with a defect ive visual cycle," Drug Discov. Today Dis. Model. 10, e163 Ðe172 (2013). 237. A. MuŒiz, W. A. Greene, M. L. Plamper, J. H. Choi, A. J. Johnson, A. T. Tsin, and H. -C. Wang, "Retinoid Uptake, Processing, and Secretion in Human iPS -RPE Support the Visual Cycle," Investig. Opthalmology Vis. Sci. 55, 198 (2014). 238. R. W. Boyd, Nonli near Optics (Academic Press, 2008). 239. S. Ostling, J. Wyckoff, S. L. Ciarkowski, C. -W. Pai, H. M. Choe, V. Bahl, and R. Gianchandani, "The relationship between diabetes mellitus and 30 -day readmission rates," Clin. Diabetes Endocrinol. 3, 3 (2017). 240. D. R. C, "Machine Learning in Medicine HHS Public Access," Circulation 132, 1920 Ð1930 (2015). 169 241. P. Mobadersany, S. Yousefi, M. Amgad, D. A. Gutman, J. S. Barnholtz -Sloan, J. E. Vel⁄zquez Vega, D. J. Brat, and L. A. D. Cooper, "Predicting cancer ou tcomes from histology and genomics using convolutional networks.," Proc. Natl. Acad. Sci. U. S. A. 115, E2970 ÐE2979 (2018). 242. W. Y. Cheng, T. H. O. Yang, and D. Anastassiou, "Development of a prognostic model for breast cancer survival in an open chal lenge environment," Sci. Transl. Med. 5, (2013). 243. W.-Y. Cheng, T. -H. O. Yang, and D. Anastassiou, "Biomolecular Events in Cancer Revealed by Attractor Metagenes," PLoS Comput. Biol. 9, e1002920 (2013). 244. T. A. Kass -Hout, L. M. Stevens, and J. L. Hall, "American Heart Association precision medicine platform," Circulation 137, 647 Ð649 (2018). 245. "AHA - Precision Medicine Platform," https://precision.heart.org/. 246. A. H. Beck, A. R. Sangoi, S. Leung, R. J. Marinelli, T. O. Nielsen, M. J. van de Vijver, R. B. West, M. van de Rijn, and D. Koller, "Systematic Analysis of Breast Cancer Morphology Uncovers Stromal Features Associated with Survival," Sci. Transl. Med. 3, (2011). 247. "Real -Time, In Situ Intelligent Video Analytics: Harnessing The P ower of GPUs for Deep Learning Applications | DSIAC," https://www.dsiac.org/resources/journals/dsiac/winter -2017-volume -4-number -1/real -time-situ -intelligent -video -analytics. 248. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A . Karpathy, A. Khosla, M. Bernstein, A. C. Berg, L. Fei -Fei, O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei -Fei, ImageNet Large Scale Visual Recognition Challenge (n.d.) . 249. "ILSVRC2017," http://image -net.org/challenges/LSVRC/2017/index#det. 250. J. N. Kather, C. A. Weis, F. Bianconi, S. M. Melchers, L. R. Schad, T. Gaiser, A. Marx, and F. G. Zıllner, "Multi -class texture analysis in colorectal cancer histology," Sci . Rep. 6, (2016). 251. "Colorectal Histology MNIST | Kaggle," https://www.kaggle.com/kmader/colorectal -histology -mnist. 252. J. N. Kather, F. G. Zıllner, F. Bianconi, S. M. Melchers, L. R. Schad, T. Gaiser, A. Marx, and C. -A. Weis, "Collection of textur es in colorectal cancer histology," (2016). 253. M. J. Anzanello and F. S. Fogliatto, "Learning curve models and applications: Literature review and research directions," Int. J. Ind. Ergon. 41, 573 Ð583 (2011). 254. and A. C. Ian Goodfellow, Yoshua Be ngio, Deep Learning (Springer US, 2017), Vol. 19. 255. M. D. Zeiler, "ADADELTA: An Adaptive Learning Rate Method," (2012). 256. D. P. Kingma and J. Ba, "Adam: A Method for Stochastic Optimization," (2014). 257. "Optimizers - Keras Documentation," http s://keras.io/optimizers/. 258. T. Dozat, "Incorporating Nesterov Momentum into Adam | OpenReview," (n.d.). 170 259. S. Ruder, "An overview of gradient descent optimization algorithms," (2016). 260. J. Duchi JDUCHI and Y. Singer, Adaptive Subgradient Methods for Online Learning and Stochastic Optimization * Elad Hazan (2011), Vol. 12. 261. "Callbacks - Keras Documentation," https://keras.io/callbacks/. 262. Y. Xu, R. Jia, L. Mou, G. Li, Y. Chen, Y. Lu, and Z. Jin, "Improved Relation Classification b y Deep Recurrent Neural Networks with Data Augmentation," (2016). 263. L. Perez and J. Wang, "The Effectiveness of Data Augmentation in Image Classification using Deep Learning," (2017). 264. C. Shorten and T. M. Khoshgoftaar, "A survey on Image Data A ugmentation for Deep Learning," J. Big Data 6, (2019). 265. Neural Networks and Deep Learning (n.d.). 266. "Building powerful image classification models using very little data," https://blog.keras.io/building -powerful -image -classification -models -using -very -little-data.html. 267. "Applications - Keras Documentation," https://keras.io/applications/. 268. N. Srivastava, G. Hinton, A. Krizhevsky, and R. Salakhutdinov, Dropout: A Simple Way to Prevent Neural Networks from Overfitting (2014), Vol. 15.