|
Fig. 1 The general framework of ZeChat behavioral analysis
(A) To analyze a ZeChat recording, a separate video clip is first generated for each fish by cropping out the ZeChat arena in which it is located. Each cropped video clip is orientated so that the transparent window is always aligned to the top edge of the clip. Each frame is then preprocessed to preserve positional, postural, and motion-related information. The preprocessed images are fed into an autoencoder for feature extraction. The main principal components of the extracted feature vector are each converted to a spectrogram by time-frequency analysis. The resulting spectral feature vectors are embedded into a 2-dimensional map and classified to distinct behavioral categories by nonlinear embedding and classification.
(B) The 3-dimensional design of the 40-unit ZeChat testing array.
(C) A screenshot of ZeChat recording, which is also zoomed in to show an independent testing unit.
(D) Intermediate and resulting images of the preprocessing procedure. The fish is first tracked to remove background (tracked). Consecutive tracked frames are subtracted (silhouette). In parallel, the tracked fish is colored by dense optical flow (dense optical flow). Finally, the dense optical flow image is masked by the silhouette to generate a merged image (merge).
(E) Training the convolutional autoencoder. Preprocessed images (left, input images) are fed into the 7-layer convolutional autoencoder (middle) to be reconstructed (right, reconstructed images). The encoder layers are responsible for compressing the input image into a latent representation space in the form of a latent vector, which is then used to reconstruct the input image by the decoder layers.
(F) Training dataset embedded into a 2-dimensional ZeChat map. A reference map containing 3,000 datapoints (red) was first embedded using t-SNE. Kernel t-SNE was then used to embed an additional 60,000 datapoints (blue).
(G) Probability density function (PDF) of the ZeChat map containing 10,000 randomly selected datapoints. Generated by convolving the ZeChat map with a Gaussian.
(H) PDF of the ZeChat map was segmented into 80 distinct behavioral categories by performing a watershed transform.