By Semere Gerezgiher Tesfay

Autonomous driving relies heavily on robust 3D perception systems, which require large-scale labeled datasets for training. However, annotating 3D data (LiDAR, camera) is expensive and time-consuming. Self-supervised learning (SSL) offers a solution by leveraging unlabeled data, but existing methods often borrow ideas from 2D vision without addressing 3D-specific challenges like sparsity and irregularity in point clouds.

Enter UniPAD, a novel SSL framework that introduces 3D volumetric differentiable rendering for pre-training. Unlike contrastive or masked autoencoding (MAE) methods, UniPAD reconstructs continuous 3D shapes and their 2D projections through neural rendering, enabling better feature learning for both LiDAR and camera-based models.

1. First 3D Differentiable Rendering for SSL in Autonomous Driving

   • Uses neural rendering to implicitly encode 3D geometry and appearance.

2. Unified Pre-training for 2D & 3D Modalities

   • Works with LiDAR, cameras, or fused inputs.

3. Memory-Efficient Ray Sampling

   • Reduces computational costs while maintaining accuracy.

4. State-of-the-Art Performance

   • Achieves 73.2 NDS (detection) and 79.4 mIoU (segmentation) on nuScenes.

Existing SSL methods for 3D perception suffer from:

• Contrastive Learning: Sensitive to positive/negative sample selection.

• MAE-based Methods: Struggle with sparse, irregular point clouds.

• Lack of Unified 2D-3D Pre-training: Most approaches specialize in one modality.

UniPAD addresses these by:
Reconstructing masked 3D scenes via rendering (no contrastive pairs needed).
Handling both LiDAR and camera data in a single framework.

UniPAD consists of:

1. Modal-Specific Encoder (processes LiDAR/camera inputs).

2. Unified 3D Volumetric Representation (combines modalities).

3. Neural Rendering Decoder (reconstructs masked regions).

   

• Input: Masked LiDAR point clouds ($P$) or multi-view images ($I$ ).

• Masking: Block-wise masking (size and ratio: 8 and 0.8 for LiDAR, 32 and 0.3 for images).

• Encoders:

  • LiDAR: VoxelNet extracts hierarchical features ($F_p$).

  • Camera: ConvNeXt extracts 2D features ($F_c$), lifted to 3D $X_p$ via

Both modalities are transformed into a shared voxel grid $V \in \mathbb{R}^{(X \times Y \times Z\times 3)}$:

For images, 3D voxel coordinates $X_p \in \mathbb{R}^{(X \times Y \times Z \times 3)}$ are projected to 2D:
$X'_p = T_{c2i} T_{l2c} X_p$

• $T_{l2c}$ = LiDAR-to-camera transform.

• $T_{c2i}$ = Camera-to-image transform.

Voxel features $V$ are computed via bilinear ($B$) and trilinear ($T$) interpolation:
$V = B(X'ₚ, F_c) \cdot T(X'ₚ, \phi(F_c))$

where $\phi$ is a convolutional layer with softmax activation. hile $B$ and $T$ retrieve the 2D features and scaling factor, respectively.
A projection layer (3D convs) further refines $V$.

Reconstructs masked regions using differentiable volume rendering:

1. Ray Sampling: For each pixel, cast a ray $r_i = o +t_id_i$ (camera origin $o$, direction $d_i$).

Memory-Friendly Ray Sampling:

• Dilation: Skips rays at fixed intervals ($I=16$).

• Random: Selects $K=512$ random rays.

• Depth-Aware: Prioritizes rays near objects (LiDAR-guided).

  

Loss Function:
Combines L1 losses for RGB and depth:
$L = \frac{\lambda_{\text{RGB}}}{K} \sum_{i=1}^K |Ŷᵢᴿᴳᴮ - Yᵢᴿᴳᴮ| + \frac{\lambda_{\text{depth}}}{K^+} \sum_{i=1}^{K^+} |\hat{Y}_i^{depth} - Y_i^{depth}|$

used MMDetection3D and train on 4× NVIDIA A100 GPUs.

• Image resolution: 1600 × 900

• Voxel size: [0.075, 0.075, 0.2]

• Data augmentation: Random scaling, rotation, and masking (image mask: size 32, ratio 0.3; point mask: size 8, ratio 0.8).

• Encoders:

  • Image: ConvNeXt-small

  • Point cloud: VoxelNet

• Voxel representation: $180 \times 180 \times 5$

• Feature projection: $3 \times 3$ conv, reducing to 32-Dim.

• Decoders:

  • SDF: 6-layer MLP

  • RGB: 4-layer MLP

• Rendering: 512 rays/image, 96 points/ray.

• Loss weights: $\lambda_{RGB}$ = 10, $\lambda_{depth}$ = 10.

• Optimizer: AdamW (LR: 2e−5 for points, 1e−4 for images).

• Training schedule: 12 epochs (pre-training), 12–20 epochs (fine-tuning on 20–50% data (point-image)).

• No CBGS or cut-and-paste augmentation

• LiDAR Detection: +9.1 NDS over baselines.

• Camera Detection: +7.7 NDS.

• Multi-Modal Fusion: 73.2 NDS, matching SOTA.

• Segmentation: 79.4 mIoU (nuScenes val set).

1. Continuous 3D Supervision: Rendering enforces geometric consistency.

2. Modality-Agnostic: Shared voxel space bridges 2D/3D gaps.

3. Efficiency: Depth-aware ray sampling cuts compute costs.

UniPAD rethinks SSL for autonomous driving by merging neural rendering with multi-modal pre-training. Its flexibility and performance make it a promising backbone for future perception systems.

Future Work: Extending to dynamic scenes and higher-resolution voxels.

• Fig 1: UniPAD architecture (encoder → voxelization → rendering).

• Fig 2: Comparison of ray sampling strategies.

• Fig 3: UniPAD experiment result sample visualizations.

Would you like a deeper dive into the rendering math or ablation studies? Let me know in the comments! 💨

• UniPAD https://arxiv.org/pdf/2310.08370

By Semere Gerezgiher Tesfay

Autonomous driving relies heavily on robust 3D perception systems, which require large-scale labeled datasets for training. However, annotating 3D data (LiDAR, camera) is expensive and time-consuming. Self-supervised learning (SSL) offers a solution by leveraging unlabeled data, but existing methods often borrow ideas from 2D vision without addressing 3D-specific challenges like sparsity and irregularity in point clouds.

Enter UniPAD, a novel SSL framework that introduces 3D volumetric differentiable rendering for pre-training. Unlike contrastive or masked autoencoding (MAE) methods, UniPAD reconstructs continuous 3D shapes and their 2D projections through neural rendering, enabling better feature learning for both LiDAR and camera-based models.

1. First 3D Differentiable Rendering for SSL in Autonomous Driving

   • Uses neural rendering to implicitly encode 3D geometry and appearance.

2. Unified Pre-training for 2D & 3D Modalities

   • Works with LiDAR, cameras, or fused inputs.

3. Memory-Efficient Ray Sampling

   • Reduces computational costs while maintaining accuracy.

4. State-of-the-Art Performance

   • Achieves 73.2 NDS (detection) and 79.4 mIoU (segmentation) on nuScenes.

Existing SSL methods for 3D perception suffer from:

• Contrastive Learning: Sensitive to positive/negative sample selection.

• MAE-based Methods: Struggle with sparse, irregular point clouds.

• Lack of Unified 2D-3D Pre-training: Most approaches specialize in one modality.

UniPAD addresses these by:
Reconstructing masked 3D scenes via rendering (no contrastive pairs needed).
Handling both LiDAR and camera data in a single framework.

UniPAD consists of:

1. Modal-Specific Encoder (processes LiDAR/camera inputs).

2. Unified 3D Volumetric Representation (combines modalities).

3. Neural Rendering Decoder (reconstructs masked regions).

   

• Input: Masked LiDAR point clouds ($P$) or multi-view images ($I$ ).

• Masking: Block-wise masking (size and ratio: 8 and 0.8 for LiDAR, 32 and 0.3 for images).

• Encoders:

  • LiDAR: VoxelNet extracts hierarchical features ($F_p$).

  • Camera: ConvNeXt extracts 2D features ($F_c$), lifted to 3D $X_p$ via

Both modalities are transformed into a shared voxel grid $V \in \mathbb{R}^{(X \times Y \times Z\times 3)}$:

For images, 3D voxel coordinates $X_p \in \mathbb{R}^{(X \times Y \times Z \times 3)}$ are projected to 2D:
$X'_p = T_{c2i} T_{l2c} X_p$

• $T_{l2c}$ = LiDAR-to-camera transform.

• $T_{c2i}$ = Camera-to-image transform.

Voxel features $V$ are computed via bilinear ($B$) and trilinear ($T$) interpolation:
$V = B(X'ₚ, F_c) \cdot T(X'ₚ, \phi(F_c))$

where $\phi$ is a convolutional layer with softmax activation. hile $B$ and $T$ retrieve the 2D features and scaling factor, respectively.
A projection layer (3D convs) further refines $V$.

Reconstructs masked regions using differentiable volume rendering:

1. Ray Sampling: For each pixel, cast a ray $r_i = o +t_id_i$ (camera origin $o$, direction $d_i$).

Memory-Friendly Ray Sampling:

• Dilation: Skips rays at fixed intervals ($I=16$).

• Random: Selects $K=512$ random rays.

• Depth-Aware: Prioritizes rays near objects (LiDAR-guided).

  

Loss Function:
Combines L1 losses for RGB and depth:
$L = \frac{\lambda_{\text{RGB}}}{K} \sum_{i=1}^K |Ŷᵢᴿᴳᴮ - Yᵢᴿᴳᴮ| + \frac{\lambda_{\text{depth}}}{K^+} \sum_{i=1}^{K^+} |\hat{Y}_i^{depth} - Y_i^{depth}|$

used MMDetection3D and train on 4× NVIDIA A100 GPUs.

• Image resolution: 1600 × 900

• Voxel size: [0.075, 0.075, 0.2]

• Data augmentation: Random scaling, rotation, and masking (image mask: size 32, ratio 0.3; point mask: size 8, ratio 0.8).

• Encoders:

  • Image: ConvNeXt-small

  • Point cloud: VoxelNet

• Voxel representation: $180 \times 180 \times 5$

• Feature projection: $3 \times 3$ conv, reducing to 32-Dim.

• Decoders:

  • SDF: 6-layer MLP

  • RGB: 4-layer MLP

• Rendering: 512 rays/image, 96 points/ray.

• Loss weights: $\lambda_{RGB}$ = 10, $\lambda_{depth}$ = 10.

• Optimizer: AdamW (LR: 2e−5 for points, 1e−4 for images).

• Training schedule: 12 epochs (pre-training), 12–20 epochs (fine-tuning on 20–50% data (point-image)).

• No CBGS or cut-and-paste augmentation

• LiDAR Detection: +9.1 NDS over baselines.

• Camera Detection: +7.7 NDS.

• Multi-Modal Fusion: 73.2 NDS, matching SOTA.

• Segmentation: 79.4 mIoU (nuScenes val set).

1. Continuous 3D Supervision: Rendering enforces geometric consistency.

2. Modality-Agnostic: Shared voxel space bridges 2D/3D gaps.

3. Efficiency: Depth-aware ray sampling cuts compute costs.

UniPAD rethinks SSL for autonomous driving by merging neural rendering with multi-modal pre-training. Its flexibility and performance make it a promising backbone for future perception systems.

Future Work: Extending to dynamic scenes and higher-resolution voxels.

• Fig 1: UniPAD architecture (encoder → voxelization → rendering).

• Fig 2: Comparison of ray sampling strategies.

• Fig 3: UniPAD experiment result sample visualizations.

Would you like a deeper dive into the rendering math or ablation studies? Let me know in the comments! 💨

• UniPAD https://arxiv.org/pdf/2310.08370