Overview of the task and methods. (a) We aim to segment 3D instances in a completely unlabeled target domain IY by leveraging the images IX and masks SX in the source domain (i.e., unsupervised domain adaptation). Instead of (b) conducting image translation (e.g., via CycleGAN [9]) and instance segmentation as two separate steps, we propose (c) Cyclic Segmentation GAN (CySGAN) to unify the two functionalities using weight sharing, which is optimized with both image translation as well as supervised and semi-supervised segmentation losses.

Architecture details of CySGAN. Given an image sampled from IY, the generator G predicts both the transferred image in IX and the BCD segmentation representations SY. Then the generator F takes only the translated image as input and predicts both the reconstructed image and segmentation representations. Specifically, BCD stands for “binary foreground mask, “contour map,” and “distance transform map.” We visualize the predicted BCD representations in the dashed yellow boxes. The two generators have exactly the same architecture, but the weights are not shared as they are optimized to translate images in different domains. Only the generator G is needed to segment IY images at inference time (the output channel for translation can also be removed).

Different segmentation losses for two domains. (a) For an annotated image in X, we compute the supervised losses of predicted segmentation representations against the label. (b) For an unlabeled image in Y, we enforce structural consistency between predicted representations (as the underlying structures should be shared) and also segmentation-based adversarial losses to improve the quality of predictions in the absence of paired labels.

Restore augmented regions with an adapted cycle-consistency strategy. We show four consecutive slices of (a) augmented real IY input, (b) synthesized IX volume, (c) reconstructed IY volume and (d) real IY volume w/o augmentations. By forcing the cycle consistency of (c) to (d), the model learns to restore corrupted regions with 3D context.

Visualization of the NucExM dataset. We sample a sub-volume of size (1024, 1024, 100) from the V1 volume of NucExM. (Left) The expansion microscopy (ExM) image volume visualized using Napari. (Right) The corresponding 3D segmentation masks visualized using Neuroglancer.

Statistics of the source (EM) and target (ExM) datasets. We show the distribution of (a) instance size (in terms of voxels) and (b) nearest-neighbor distance between nuclei centers. The density plots are normalized by the total number of instances in each volume. We also show (c) the voxel intensity distribution in object (foreground) and non-object (background) regions for both volumes. The domain gap is characterized by different intensity distributions and contrast.

Visual comparisons of segmentation results. (a) ExM image, (b) ground-truth instances, (c) Cellpose [3], (d) StarDist [4] and (e) CySGAN results. The red arrows highlight false negatives in Cellpose predictions and overlapping masks from StarDist. We also show (f-h) the predicted segmentation representations of U3D-BCD used in CySGAN. Note that all the nuclei instances are 3D as shown in Fig. 5. We present representative 2D slices in this visualization to demonstrate the model performance.