Lifelong Representations: A Survey on Continual Self-Supervised Learning for Vision Models

Overview & Why SSL Resists Forgetting

Traditionally, continual learning (CL) assumes access to curated, labeled data—an assumption that fails in real-world deployments like lifelong robotics or continual stream ingestion. While self-supervised learning (SSL) provides a scalable alternative by deriving supervisory signals directly from data structure, models still suffer from catastrophic forgetting over extended training horizons.

Our analysis shows that SSL objectives appear inherently more robust to forgetting than supervised ones due to two main properties:

• Task-Agnostic Representations: Unlike supervised training, which collapses feature spaces into task-specific discriminative subspaces (causing the tunnel effect), SSL preserves broader, structurally rich features that generalize better across sequential distributions.
• Flatter Minima in the Loss Landscape: Self-supervised optimization tends to converge to wider, flatter minima, allowing model parameters to adapt to new tasks with less disruption of previously learned representations.

Taxonomy of CSSL & Key Insights by Family

We organize existing CSSL approaches into a unified taxonomy of six families based on their primary forgetting-mitigation strategies:

1. Distillation Methods

• Approach: Knowledge distillation regularizes the model by encouraging consistency between the representations or similarity matrices of the current network and a past/teacher model (e.g., MedCoSS, CCL).
• Key Insight: While standard feature distillation preserves past structure, overly rigid alignment can restrict the model's plasticity. New methods such as projected feature distillation show that allowing representations to change in a controlled manner preserves plasticity.

2. Weight Regularization

• Approach: Penalizes changes to parameters deemed important for previous tasks through quadratic penalties or parameter freezing based on estimated importance metrics.
• Key Insight: Prioritizes retention over plasticity. While parameter-selective regularization reduces catastrophic forgetting on unimodal benchmarks, it often over-constrains updates.

3. Replay Methods

• Approach: Maintains a memory buffer of past samples to interleave during the training of new tasks, either by direct batch concatenation, distillation/contrastive objectives (CCL), data interpolation (LUMP, Cro-Mo Mixup), or informative sample selection (ASR, CUCL, EDSR).
• Key Insight: Replay remains one of the most effective strategies for preventing forgetting. However, performance hinges heavily on buffer diversity, sample selection criteria, and the ratio of replayed data to new task data.

4. Architectural Methods

• Approach: Mitigates forgetting at the parameter level by isolating parameters, such as dynamically adding task-specific modules (Branch-Tuning), expanding capacity online (KIERA), or selectively freezing correlated layers (PTLF). In vision-language setups, parameter-efficient tuning like LoRA (C-CLIP, FoMo-in-Flux) is frequently employed.
• Key Insight: Freezing parameters yields good retention, while expanding architecture improves plasticity. However, both require careful structural design to balance flexibility against computational overhead and scalability, and many rely on explicit task boundaries and may underperform in online settings.

5. Model Merging

• Approach: Consolidates knowledge across sequentially trained or expert models by directly aggregating their parameters in weight space (e.g., recursive merging in TiC-CLIP).
• Key Insight: Serves as a computationally lightweight alternative to continuous replay or distillation. However, sequential merging requires navigating weight-space drift over time to prevent feature degradation across successive updates.

6. Objective-Level Adaptation

• Approach: Modifies the self-supervised objective directly to enhance inherent stability on non-stationary data streams without relying on external buffers or teacher models (e.g., Continual Multi-Patch Learning (CMP)).
• Key Insight: Objective-level tweaks can naturally enforce consistency across spatial or temporal views, offering a memory-free direction for CSSL.

Open Challenges & Future Directions

Despite rapid progress, CSSL remains at an early stage of development. Moving beyond adapting supervised CL techniques requires addressing four core open challenges:

BibTeX

@misc{masip2026lifelongrepresentationssurveycontinual,
      title={Lifelong Representations: A Survey on Continual Self-Supervised Learning for Vision Models}, 
      author={Sergi Masip and Alicja Dobrzeniecka and Jonathan Swinnen and Joachim Collin and Bartłomiej Twardowski and Szymon Łukasik and Tinne Tuytelaars},
      year={2026},
      eprint={2607.09785},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.09785}, 
}