We introduce SimDINO and SimDINOv2 by simplifying widely-used SSL algorithms (i.e. DINO and DINOv2) via coding rate regularization. These simplified algorithms lead to several key benefits:

• Simplicity: Removing many empirically-selected components from the original DINO pipeline.
• Robustness: More robust to variations in architecture, datasets and other configuations.
• Performance: Leading to better learned represetntation and performance on downstream tasks.
• Interpretability: Providing a theoretically motivated way to prevent representation collapse.

Overall, making design choices explicit and principled lead to simplicity and improved performance in vision SSL.

Our experiments demonstrate that SimDINO and SimDINOv2 achieve superior performance compared to their original counterparts:

Quantitative Results

Feature Visualizations

Our models exhibit strong emergent properties similar to the original DINO families. Below we show different types of visualizations demonstrating the models' capabilities:

Attention Maps

Average self-attention maps showing object-centric attention patterns

PCA Visualization

Top three principal components of key features in RGB format

Saliency Maps

Regions attended by the [CLS] token

• Attention Maps
• PCA Analysis
• Saliency Maps

Optimization Dynamics

We visualize the training dynamics of training ViT-B/16 under SimDINO and DINO. X-axis denotes the training epochs while Y-axis indicates the k-NN performance on ImageNet-1K.

• Left: Both models are trained on ImageNet-1K. We omit the earlier epochs for better visualization.
• Right: Both models are trained on COCO train2017 (roughly 1/10th of ImageNet-1K). As a verification experiment, this shows that SimDINO requires substantially less tuning and much easier to optimize.

The core insight of our work is that representation collapse can be prevented directly through coding rate regularization, eliminating the need for many empirically-designed components. Take DINO as an example:

It turns out we can remove many empirically-designed components and replace them with a simple coding rate term in the loss function.

Removing Empirically-Designed Components:

• A weight-normalized linear layer that projects features to high-dim (~65,536) outputs for clustering.
• Balancing operations to prevent representation collapse (e.g., centering, sharpening).
• Miscellaneous hyperparameters (e.g., temperature schedules, centering momentum).

Opting for Simple, Explicit Solutions:

• Feature alignment loss $\mathcal{L}_{\text{align}}$ via simple Euclidean distance: $d_{\ell^{2}}(z_1,z_2) = \frac{1}{2}\|z_1 - z_2\|^2$.
• Anti-collapse loss $\mathcal{L}_{\text{rate}}$ based on explicit coding rate regularization: $$R_{\epsilon}(\Gamma) := \frac{1}{2}\text{logdet}({I + \frac{d}{\epsilon^{2}}\Gamma}),$$ where $\Gamma = \text{Cov}(Z)$ is the covariance of global-view features computed on a batch of images, $d$ is feature dimension and $\epsilon$ denotes the distortion error (Refer to the paper for more specifics).
• Combining these two losses leads to the complete objective $\mathcal{L} = \mathcal{L}_{\text{align}} + \lambda \mathcal{L}_{\text{rate}}$, where the balancing factor $\lambda$ can be explicitly derived based on gradient norm (See Appendix C).

This leads to a much simpler pipeline we term SimDINO:

This streamlined design makes the models more robust to architectural choices and hyperparameter settings while achieving better downstream performance. The coding rate term provides a theoretically motivated way to prevent collapse, replacing many empirical design choices in the original DINO pipeline. Similar simplifications can be made to DINOv2, leading to SimDINOv2. Refer to the paper for more details.