Citation :

Neeraj, Poonam Nandal, "An Empirical Evaluation of Regularization, Replay, and Architectural Methods Employed in Continual Learning using Split MNIST, CIFAR10, CIFAR100," International Journal of Engineering Trends and Technology (IJETT), vol. 74, no. 9, pp. 349-363, 2026. Crossref, https://doi.org/10.14445/22315381/IJETT-V74I9P124

Abstract

Human beings can learn throughout their lives; they learn new concepts, refine their existing skills, employ their experiences in various situations, and build memories for prolonged retention. The ability to learn throughout one's lifetime is known as continual learning, and it is challenging to replicate this capacity in an artificial neural network. Deep models tend to overwrite previously learned information when exposed to sequential tasks, a phenomenon known as catastrophic forgetting. To address catastrophic forgetting, several methods have been proposed. In this study, we focus on a comprehensive empirical evaluation of three major categories of continual learning methods: regularization-based methods, replay-based methods, and architecture-based methods. The experiment uses multiple benchmark settings. For a simple image classification task, use Split-MNIST. For representing more complex tasks, use the Split CIFAR10 and Split CIFAR100 datasets (with and without pretraining). All datasets are used under class-incremental and Task-Incremental Learning scenarios. Each scenario faces distinct challenges. To ensure transparency, each method is evaluated using a shared backbone architecture. Performance of each method in both scenarios is reported using various standard continual learning metrics, such as Average Accuracy (ACC), Average Forgetting and Backward Transfer (BWT), along with measurements of total training time and CPU/GPU memory usage to show the resource efficacy. This paper presents a remarkable difference in terms of levels of difficulty and the relative efficacy of various tactics in both scenarios, task-incremental and Class-Incremental Learning. The performance of Task-Incremental Learning is better than Class-Incremental Learning for all continual learning methods.

Keywords

Continual Learning, Catastrophic Forgetting, Class-Incremental Learning, Task-Incremental Learning.

References

[1] James Kirkpatrick et al., “Overcoming Catastrophic Forgetting in Neural Networks,” Proceedings of the National Academy of Sciences, vol. 114, no. 13, pp. 3521-3526, 2017.
[CrossRef] [Google Scholar] [Publisher Link]

[2] Cuong V. Nguyen et al., “Variational Continual Learning,” arxiv, pp. 1-18, 2018.
[CrossRef] [Google Scholar] [Publisher Link]

[3] Michael McCloskey, and Neal J. Cohen, “Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem,” Psychology of Learning and Motivation, vol. 24, pp. 109-165, 1989.
[CrossRef] [Google Scholar] [Publisher Link]

[4] Ian J. Goodfellow et al., “An Empirical Investigation of Catastrophic Forgetting in Gradient-based Neural Networks,” arXiv, pp. 1-9, 2013.
[CrossRef] [Google Scholar] [Publisher Link]

[5] Delphine Oudiette, and Ken A. Paller, “Upgrading the Sleeping Brain with Targeted Memory Reactivation,” Trends in Cognitive Sciences, vol. 17, no. 3, pp. 142-149, 2013.
[CrossRef] [Google Scholar] [Publisher Link]

[6] Gido M. van de Ven et al., “Hippocampal Offline Reactivation Consolidates Recently Formed Cell Assembly Patterns during Sharp Wave-Ripples,” Neuron, vol. 92, no. 5, pp. 968-974, 2016.
[CrossRef] [Google Scholar] [Publisher Link]

[7] Arslan Chaudhry et al., “On Tiny Episodic Memories in Continual Learning,” arXiv, pp. 1-15, 2019.Top of Form
[CrossRef] [Google Scholar] [Publisher Link]

[8] Hanul Shin, et al., “Continual Learning with Deep Generative Replay,” Advances in Neural Information Processing Systems, vol. 30, 2017.
[Google Scholar] [Publisher Link]

[9] Pietro Buzzega et al., “Dark Experience for General Continual Learning: A Strong, Simple Baseline,” Advances in Neural Information Processing Systems, vol. 33, pp. 15920-15930, 2020.
[Google Scholar] [Publisher Link]

[10] Friedemann Zenke, Ben Poole, and Surya Ganguli, “Continual Learning through Synaptic Intelligence,” Proceedings of the 34^th International Conference on Machine Learning (PMLR), vol. 70, pp. 3987-3995, 2017.
[Google Scholar] [Publisher Link]

[11] Rahaf Aljundi et al., “Memory Aware Synapses: Learning What (not) to Forget,” Computer Vision (ECCV): 15^th European Conference, Munich, Germany, vol. 11207, pp. 1-11, 2018.
[CrossRef] [Google Scholar] [Publisher Link]

[12] Zhizhong Li, and Derek Hoiem, “Learning without Forgetting,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 12, pp. 2935-2947, 2017.
[CrossRef] [Google Scholar] [Publisher Link]

[13] Arun Mallya, and Svetlana Lazebnik, “PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, pp. 7765-7773, 2018.
[CrossRef] [Google Scholar] [Publisher Link] 

[14] Andrei A. Rusu et al., “Progressive Neural Networks,” arXiv, pp. 1-16, 2016.
[CrossRef] [Google Scholar] [Publisher Link] 

[15] Joan Serra et al., “Overcoming Catastrophic Forgetting with Hard Attention to the Task,” Proceedings of the 35^th International Conference on Machine Learning, vol. 80, pp. 4548-4557, 2018.
[Google Scholar] [Publisher Link] 

[16] David Lopez-Paz, and Marc'Aurelio Ranzato, “Gradient Episodic Memory for Continual Learning,” Advances in Neural Information Processing Systems (NeurIPS), vol. 30, 2017.
[Google Scholar] [Publisher Link]

[17] Gido M. van de Ven, Tinne Tuytelaars, and Andreas S. Tolias, “Three Types of Incremental Learning,” Nature Machine Intelligence, vol. 4, no. 12, pp. 1185-1197, 2022.
[CrossRef] [Google Scholar] [Publisher Link]  

[18] Sylvestre-Alvise Rebuffi et al., “iCaRL: Incremental Classifier and Representation Learning,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, pp. 5533-5542, 2017.
[CrossRef] [Google Scholar] [Publisher Link]  

[19] Arslan Chaudhry et al., “Efficient Lifelong Learning with A-GEM,” arxiv, pp. 1-20, 2019.
[CrossRef] [Google Scholar] [Publisher Link]  

[20] Nicolas Y. Masse, Gregory D. Grant, and David J. Freedman, “Alleviating Catastrophic Forgetting using Context-Dependent Gating and Synaptic Stabilization,” Proceedings of the National Academy of Sciences, vol. 115, no. 4, pp. E10467-E10475, 2018.
[CrossRef] [Google Scholar] [Publisher Link]

[21] Gido M. van de Ven, Hava T. Siegelmann, and Andreas S. Tolias, “Brain-Inspired Replay for Continual Learning with Artificial Neural Networks,” Nature Communications, vol. 11, no.  1, pp. 1-14, 2020.
[CrossRef] [Google Scholar] [Publisher Link] 

[22] Jonathan Schwarz et al., “Progress and Compress: A Scalable Framework for Continual Learning,” Proceedings of the 35^th International Conference on Machine Learning, pp. 4528-4537, 2018.
[Google Scholar] [Publisher Link] 

[23] David Rolnick et al., “Experience Replay for Continual Learning,” Advances in Neural Information Processing Systems, vol. 32, 2019.
[Google Scholar] [Publisher Link] 

[24] Arslan Chaudhry et al., “Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence,” European Conference on Computer Vision, Springer, Cham, vol. 11215, pp. 556-572, 2018.
[CrossRef] [Google Scholar] [Publisher Link] 

[25] Rahaf Aljundi, et al., “Online Continual Learning with Maximal Interfered Retrieval,” Advances in Neural Information Processing Systems, vol. 32, 2019.
[Google Scholar] [Publisher Link] 

[26] Rahaf Aljundi et al., “Gradient-based Sample Selection for Online Continual Learning,” Advances in Neural Information Processing Systems, vol. 32, 2019.
[Google Scholar] [Publisher Link] 

[27] Jihwan Bang et al., “Rainbow Memory: Continual Learning with a Memory of Diverse Samples,” 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, pp. 8214-8223, 2021.
[CrossRef] [Google Scholar] [Publisher Link] 

[28] Chenshen Wu et al., “Memory Replay GANs: Learning to Generate Images from New Categories without Forgetting,” Advances in Neural Information Processing Systems, vol. 31, 2018.
[Google Scholar] [Publisher Link] 

[29] Ye Xiang et al., “Incremental Learning using Conditional Adversarial Networks,” 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), pp. 6618-6627, 2019.
[CrossRef] [Google Scholar] [Publisher Link] 

[30] Quentin Jodelet et al., “Class-Incremental Learning using Diffusion Model for Distillation and Replay,” 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), Paris, France, pp. 3417-3425, 2023.
[CrossRef] [Google Scholar] [Publisher Link] 

[31] James Smith et al., “Always be Dreaming: A New Approach for Data-Free Class Incremental Learning,” 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, pp. 9354-9364, 2021.
[CrossRef] [Google Scholar] [Publisher Link] 

[32] Gido M. van de Ven, and Andreas S. Tolias, “Three Scenarios for Continual Learning,” arXiv, pp. 1-18, 2019.
[CrossRef] [Google Scholar] [Publisher Link] 

[33] Pingbo Pan et al., “Continual Deep Learning by Functional Regularisation of Memorable Past,” Advances in Neural Information Processing Systems, vol. 33, pp. 4453-4464, 2020.
[Google Scholar] [Publisher Link]