Jan 28, 2023 · The basic idea is to adopt a restricted orthogonal constraint allowing parameters optimized in the direction oblique to the whole frozen space ...

Jan 28, 2023 · To address this challenge, we propose our Restricted Orthog- onal Gradient prOjection (ROGO) framework to facilitate forward knowledge transfer.

Continual learning aims to avoid catastrophic forgetting and effectively leverage learned experiences to master new knowledge. Existing gradient projection ...

Aug 16, 2023 · The basic idea is to adopt a restricted orthogonal constraint allowing parameters optimized in the direction oblique to the whole frozen space ...

We present the Orthogonal Gradient Descent (OGD) method, which accomplishes this goal by projecting the gradients from new tasks onto a subspace in which the ...

This work presents a geometrical analysis of the Large Step Gradient Descent (LGD) dictionary learning algorithm. LGD updates the atoms of the dictionary using ...

All of them constrain the gradient to be fully orthogonal to the feature space. Right: We propose a space decoupling (SD) algorithm to decouple the feature ...

However, such restrictive orthogonal gra- dient updates hamper the learning capability of the new tasks resulting in sub-optimal performance. To improve new ...

In this paper, we present a system where the gradients produced on a training minibatch can avoid interfering with gradi- ents produced on previous tasks. The ...

We show how gradient descent orthogonal to these spaces enable us to learn continually without forgetting. Learning Task 1: We learn the first task (τ = 1) ...