PUBLICATION
Parallelized computational 3D video microscopy of freely moving organisms at multiple gigapixels per second
- Authors
- Zhou, K.C., Harfouche, M., Cooke, C.L., Park, J., Konda, P.C., Kreiss, L., Kim, K., Jönsson, J., Doman, T., Reamey, P., Saliu, V., Cook, C.B., Zheng, M., Bechtel, J.P., Bègue, A., McCarroll, M., Bagwell, J., Horstmeyer, G., Bagnat, M., Horstmeyer, R.
- ID
- ZDB-PUB-231009-58
- Date
- 2023
- Source
- Nature photonics 17: 442450442-450 (Journal)
- Registered Authors
- Bagnat, Michel, McCarroll, Matthew N.
- Keywords
- 3D imaging, behavioral imaging, camera array, computational microscopy, parallelized microscopy, self-supervised learning
- MeSH Terms
- none
- PubMed
- 37808252 Full text @ Nat Photonics
Citation
Zhou, K.C., Harfouche, M., Cooke, C.L., Park, J., Konda, P.C., Kreiss, L., Kim, K., Jönsson, J., Doman, T., Reamey, P., Saliu, V., Cook, C.B., Zheng, M., Bechtel, J.P., Bègue, A., McCarroll, M., Bagwell, J., Horstmeyer, G., Bagnat, M., Horstmeyer, R. (2023) Parallelized computational 3D video microscopy of freely moving organisms at multiple gigapixels per second. Nature photonics. 17:442450442-450.
Abstract
Wide field of view microscopy that can resolve 3D information at high speed and spatial resolution is highly desirable for studying the behaviour of freely moving model organisms. However, it is challenging to design an optical instrument that optimises all these properties simultaneously. Existing techniques typically require the acquisition of sequential image snapshots to observe large areas or measure 3D information, thus compromising on speed and throughput. Here, we present 3D-RAPID, a computational microscope based on a synchronized array of 54 cameras that can capture high-speed 3D topographic videos over an area of 135 cm2, achieving up to 230 frames per second at spatiotemporal throughputs exceeding 5 gigapixels per second. 3D-RAPID employs a 3D reconstruction algorithm that, for each synchronized snapshot, fuses all 54 images into a composite that includes a co-registered 3D height map. The self-supervised 3D reconstruction algorithm trains a neural network to map raw photometric images to 3D topography using stereo overlap redundancy and ray-propagation physics as the only supervision mechanism. The resulting reconstruction process is thus robust to generalization errors and scales to arbitrarily long videos from arbitrarily sized camera arrays. We demonstrate the broad applicability of 3D-RAPID with collections of several freely behaving organisms, including ants, fruit flies, and zebrafish larvae.
Genes / Markers
Expression
Phenotype
Mutations / Transgenics
Human Disease / Model
Sequence Targeting Reagents
Fish
Orthology
Engineered Foreign Genes
Mapping