The Selective $G$-Bispectrum and its Inversion: Applications to $G$-Invariant Networks

The $G$-Bispectrum is the Fourier transform of the $G$-Triple Correlation ($G$-TC). Historically, higher-order spectra like found initial applications in the context of classical signal processing as generalizations of the two-point autocorrelation [31, 3, 24]. The work of Kakarala [14] illuminated the relevance of the $G$-Bispectrum for invariant theory, as it is the lowest-degree spectral invariant that is complete [30]. Since then, it has appeared in diverse settings such as vision science [34], machine learning [18, 19], and 3D modeling [17].

The computational complexity of the $G$-Bispectrum has severely limited the reach of its applications. The most salient example of this limitation is in machine learning and deep learning. Convolutional Neural Network (CNN) [21, 22] reflect and exploit the translational symmetry of the visual world. Group-Equivariant CNNs ($G$-CNNs) [7, 20] do just this, with more general group-equivariant convolutions to exploit symmetries like rotational symmetries. In both cases, one typically wants to preserve transformations throughout the layers of a network (i.e., to be group-equivariant), and remove them only at the end when “canonicalizing” an image for classification (i.e., to be group-invariant). While the theory of equivariant layers has been thoroughly developed [8, 33], less attention has been paid to the theory of invariant layers [12]. This is where the $G$-Bispectrum enters the picture, and where its computational cost has strongly limited its integration into deep learning.

Commonly, invariance in $G$-CNNs is achieved by simply taking an average or maximum over the transformation group (Average or Max $G$-Pooling, respectively). However, as noted by Sanborn & Miolane [27], this is a highly lossy operation removing information about the structure of the signal. While the max operation is indeed invariant (the max of an image is the same as the max of an image rotated by $90$ degrees), it is excessively invariant: one could permute all of the pixels in the image and without changing the maximum, but with none of the same structure (see Figure 7). To address this, Sanborn & Miolane [27] used the $G$-TC as a $G$-invariant layer that is complete—that is, it removes group transformations with no loss of signal structure. This approach achieves demonstrable gains in accuracy and robustness [27], but it is computationally expensive.

Indeed, the space complexity of the $G$-TC, i.e, its number of coefficients, scales as $\mathcal{O}(|G|^{2})$, where $|G|$ is the size of the group. As each coefficient demands for $\mathcal{O}(|G|)$ operations, the computational cost or the time complexity of the $G$-TC is $\mathcal{O}(|G|^{3})$. An alternative would be to use the $G$-Bispectrum as the pooling layer. However, both its space and time complexities are $\mathcal{O}(|G|^{2})$. By comparison, the Max $G$-pooling layer features a $\mathcal{O}(|G|)$ computational cost and returns a scalar output. This raises the question of whether one can achieve complete invariance and adversarial robustness without sacrificing too much in terms of computational efficiency.

Contributions. In this work, we prove for the first time that we can significantly reduce the computational complexity of the $G$-Bispectrum. This result has important implications for signal processing and deep learning on groups, for which the $G$-Bispectrum is a foundational computational primitive. Our contributions are:

• •

  We provide a general algorithm that reduces the computational complexity of the $G$-Bispectrum from $\mathcal{O}(|G|^{2})$ to $\mathcal{O}(|G|)$ in space complexity and from $\mathcal{O}(|G|^{2})$ to $\mathcal{O}(|G|\log|G|)$ in time complexity if an FFT is available on $G$. We term it the selective $G$-Bispectrum. The algorithm can be applied to any finite group.

• •

  We prove that the selective $G$-Bispectrum is complete for the most important finite groups used in practice, i.e., all discrete commutative groups, the dihedral groups of any order, the octahedral and full octahedral group. This significantly extends the work of [10, 14, 26], who first showed this for some finite, commutative groups, where it was demonstrated that the $G$-Bispectrum can be computed with only $|G|$ space complexity.

• •

  We use the selective $G$-Bispectrum to propose a new $G$-invariant layer that strikes a balance between robustness and efficiency. In particular, it is more expensive than the Max $G$-pooling, but cheaper than the $G$-TC pooling.It is also cheaper than the full $G$-bispectral pooling of $\mathcal{O}(|G|^{2})$ time and $\mathcal{O}(|G|^{2})$ space complexity. The selective $G$-Bispectrum is more robust than the max $G$-pooling, and almost as robust as the $G$-TC.

• •

  We run extensive experiments on the MNIST [23] and EMNIST [6] datasets to evaluate how each invariance layer (Max $G$-pooling, $G$-TC, selective $G$-Bispectrum) impacts accuracy and speed on classification tasks. We achieve the expected results: Our layer is faster than the $G$-TC and full $G$-Bispectrum and more accurate than Max $G$-pooling.

• •

  We present several findings important to the design of invariant layers to guide further advances in the field of geometric deep learning. In particular, we show that the accuracy and speed advantages of the selective $G$-Bispectrum is most striking for $G$-CNNs with low number of convolutional filters. Conversely, increasing the number of filters in the $G$-Convolutions allows the Max $G$-Pooling to catch up on the accuracy.This demonstrates that the $G$-bispectral pooling will be particularly interesting for neural networks operating under a smaller parameter budget.

We hope that the proposed reduction of the $G$-Bispectrum complexity will further open areas of research in signal processing on groups, that were previously prohibited due to the high complexity of the operation.

Given a real signal defined on a finite group $\Theta:G\mapsto\mathbb{R}$, the $G$-Triple Correlation ($G$-TC) [14] is the lowest order polynomial that is complete, i.e., that conserves all of the information of the signal $\Theta$, up to group action by $G$.

The original triple-correlation was introduced for the classical framework of translations of a one-dimensional signal, i.e., where $X=\mathbb{Z}$ and $(G,\ \cdot)=(\mathbb{Z},+)$. The $G$-triple correlation from Definition 2.1 extends the original definition to any finite group $(G,\ \cdot)$. In our setting, the signal $\Theta:G\mapsto\mathbb{R}$ will be obtained after the $G$-convolution of a function $f:X\mapsto\mathbb{R}^{c}$, representing a continuous image with $c$ channels, with a filter $\phi:X\mapsto\mathbb{R}^{c}$, and the $G$-TC will be applied channel-by-channel. Importantly, the $G$-TC layer has computational complexity $\mathcal{O}(|G|^{3})$ and outputs $\mathcal{O}(|G|^{2})$ coefficients.

The $G$-TC operation has a Fourier equivalent: the $G$-Bispectrum. Indeed, the definition of the Discrete Fourier Transform (DFT) can be extended to any finite group (see, e.g., [9]), as recalled below.

The $G$-Bispectrum $\beta(\Theta)$ is defined as $\mathcal{F}(T(\Theta))$, with $\mathcal{F}$ evaluated over the group $G\times G$. Kakarala [14] proposed a closed-form expression for the $G$-Bispectrum $\beta(\Theta)$ directly in terms of $\mathcal{F}(\Theta)$. We recall it in Theorem 2.3.

For commutative groups, Theorem 2.3 simplifies to a more compact expression, recalled in Theorem A.12. However, both the space and time complexity of the $G$-Bispectrum remain $O(|G|^{2})$.

To the authors’ best knowledge, there is no generic analytical formula for computing $C_{\rho_{1},\rho_{2}}$ for an arbitrary group $G$. However, there exist formulas for specific classes of groups (see Appendix E.2). Additionally, there exist packages for computing these for many groups using packages such as escnn [4].

The $G$-TC and the $G$-Bispectrum are desirable computational primitives for signal processing and deep learning because they are complete $G$-invariants (for generic data $\Theta,\widetilde{\Theta}$). Indeed, this completeness property make them very interesting for building invariance layers in $G$-CNNs, as they are selectively invariant. We define complete $G$-invariance next.

The $G$-CNN architecture, first proposed in [7], is illustrated on Figure 1. The input signal $f:X\mapsto\mathbb{R}$, typically an image, is processed through a $G$-Convolution layer using filters $\{\phi_{k}\}_{k=1}^{K}$. The output is feature maps $\{\Theta_{k}\}_{k=1}^{K}$ that form a set of $K$ real-valued signals with domain $G$. This $G$-Convolution layer is traditionally followed by a $G$-invariance layer. The most common is the Max $G$-Pooling layer. More recent works have proposed two alternatives based on the $G$-TC and the full $G$-Bispectrum: the $G$-TC Pooling [27] and the (full) $G$-Bispectrum [28] respectively, where the latter requires the computations of the Fourier transforms of the feature maps, preferably computed using a Fast Fourier Transform (FFT) algorithm on $G$ [9]. When testing the impact of the choice of $G$-invariance layer, the output of the invariance layer is typically fed to a Secondary Neural Network (NN) to perform the desired task, e.g., image classification. The Secondary NN often takes the form of a Multi-Layer Perceptron (MLP).

Experimental results have demonstrated the superior accuracy and adversarial robustness of the $G$-CNN equipped with a $G$-TC and $G$-Bispectrum invariance layer [28, 27]. However, both methods inherit the high space and time complexity of their respective operations. This raises the question of whether we can reduce this computational complexity.

We introduce a novel tool for signal processing on groups: the selective $G$-Bispectrum. The selective $G$-Bispectrum $\beta_{sel}$ is a subset of all coefficients of the $G$-Bispectrum $\beta$ (Definition 2.3), only conserving well-chosen pairs of irreps $(\rho_{1},\rho_{2})$. Which pairs of irreps to select depends on the group of interest. This is possible due to redundancies and symmetries in the full object. Below, we provide an algorithmic procedure to compute the selective $G$-Bispectrum for any finite group $G$ that features at most $|Irreps|(\leq|G|)$ coefficients. The procedure is summarized in Algorithm 1. We have the following proposition.

In Algorithm 1, we note that the choice of $\rho_{1}$ is important, since some $\rho_{1}$ will not allow the user to recover all of the irreps and therefore not ensure the completeness of the selective. We illustrate the computation of the selective $G$-Bispectrum in Figure 2, where we choose $\tilde{\rho}_{1}=\rho_{6}$.

The inversion of the selective $G$-Bispectrum $\beta_{sel}(\Theta)$ is reconstructing a signal $\mathcal{F}(\widetilde{\Theta})$ from the $G$-Fourier coefficients in the list $L_{\rho}$ such that $\widetilde{\Theta}=\alpha(g,\Theta)$ for some $g\in G$ (The $G$-Bispectrum is $G$-invariant, hence, $\Theta$ can only be recovered at best up to group action). Once $\mathcal{F}(\widetilde{\Theta})$ is known, $\widetilde{\Theta}$ can be obtained using the Inverse Fourier Transform. If the selective $G$-Bispectrum can be inverted, then, by definition, it is complete in the sense of Theorem 2.4.

Previous authors had looked into the $G$-Bispectrum inversion problem. It is well known that $|G|=n$ coefficients are enough for the cyclic group $(C_{n},\cdot)=(\mathbb{Z}/\mathbb{Z}_{n},+\mod n)$.

Similarly for a product of two such groups, we have the following theorem.

From now on, we assume that the Fourier transform $\mathcal{F}(\Theta)$ only features non-zero elements, or invertible matrices in the case of non-scalar Fourier coefficients. This assumption is supported by the zero probability of encountering this corner case (an arbitrarily small perturbation of any signal makes this assumption true).

We first extend the above results to all commutative groups. The proof relies on the fact that every finite commutative group is the direct sum of finitely many cyclic groups.

See Appendix D for the proof and derivation of the inversion for the specific case of commutative groups. We note that our approach to inversion is symbolic, in that a solution can be expressed explicitly as a formula in terms of the input. Other approaches are also possible to determine an inverse, such as using least squares [11] or more recent spectral methods [5].

We now extend the result to dihedral groups. Dihedral groups are ubiquitous in signal processing and deep learning because they represent the group of rotations and reflections.

The proof is provided in Appendix E. We now extend the result to octahedral and full octahedral groups, that are related to the symmetries of the octahedron. These groups are very important in signal processing and deep learning of 3D images.

A sketch of proof is provided in Appendix F given the redundancy of the procedure. We see that the selective $G$-Bispectrum uses only $4$ coefficients, compared to $25$ coefficients needed for the full $G$-Bispectrum for the octahedral group. For the full octahedral group, it requires only $6$ coefficients compared to the $100$ coefficients of the full $G$-Bispectrum. In Figure 3, we compare the full and selective $G$-Bispectra of the dihedral group $D_{4}$ (symmetries of the square) and the octahedral group.

Our implementation of the selective $G$-bispectrum layer is based on the gtc-invariance repository, implementing the $G$-CNN with $G$-convolution and $G$-TC layer [27] and relying itself on the escnn library [4, 32].

We propose an experimental assessment of the newly proposed selective $G$-Bispectrum layer by comparing it with the Avg $G$-pooling, the Max $G$-pooling, the $G$-TC as invariance operations after the $G$-convolution of a $G$-CNN on the classification problems of the MNIST dataset of handwritten digits [23], the EMNIST dataset of handwritten letters [6] with standard train-test division. These datasets count 10 and 26 classes, respectively. We obtain transformed versions of the datasets – $G$-MNIST/EMNIST – by applying a random action $g\in G$ on each image in the original dataset.

The objective of our experiments is to isolate the speed-up of the $G$-Bispectrum layer. Hence, we consider architectures that only differ by the invariance layer in the classification task, following the experimental set up by [27]. The neural network architecture is composed of a $G$-convolution, a $G$-invariance layer, and finally a Multi-Layer-Perceptron (MLP), itself composed of three fully connected layers with ReLU nonlinearity. Finally, a fully connected linear layer is added to perform classification. The MLP’s widths are tuned to match the number of parameters across each neural network model. The details are given in Appendix G. We highlight here that the pursued objective is to compare the differences in performances of the $G$-invariance layers, not to provide the state-of-the-art accuracy on the datasets involved. Henceforth, we do not optimize the architectures to reach the highest possible accuracy. We set simple architectures providing interpretable results for analysis. The experiments a performed using $8$ cores of a NVIDIA A30 GPU.

Table 1 recalls the theoretical complexities of the different layers. The computational cost of computing the selective $G$-Bispectrum is $\mathcal{O}(|G|\log|G|)$ if an FFT algorithm is available on $G$ [9], and $\mathcal{O}(|G|^{2})$ with classical DFT. On Figure 4, we report the average training times on $\mathrm{SO}(2)/\mathrm{O}(2)$-MNIST for 10 runs as the discretization $C_{n}/D_{n}$ of $\mathrm{SO}(2)/\mathrm{O}(2)$ varies. In the first case, we use the FFT and observe that the Max $G$-pooling and $G$-Bispectrum training time scale linearly whereas it scales quadratically for the $G$-TC. For $\mathrm{O}(2)$, we perform a classic DFT on $D_{n}$ so that the $G$-Bispectrum scales worth. However, an FFT could be implemented to speed-up the process.

We compare the performances of the $G$-Bispectrum layer with respect to the $G$-TC, the Max $G$-pooling and the Avg $G$-pooling models, trained on the $\mathrm{SO}(2)$/$\mathrm{O}(2)$-MNIST/EMNIST datasets and we assess the accuracy by averaging the validation accuracy over 10 runs. The classification accuracy is provided in Table 2. For the experiments in Table 2, the following pattern holds: at equivalent number of parameters, the more computationally expensive the pooling layer, the better the accuracy. However, the use of the $G$-TC becomes prohibitive when $|G|$ increases. In the next section, we discuss the settings where each invariance layer should be preferred, and highlight each invariance layer’s strengths and weaknesses.

The first observation from Table 2 is though the selective $G$-Bispectrum is complete, the model obtains slightly lower accuracy than $G$-TC. This observation might be surprising at first, since we prove mathematically in Section 4 that the selective $G$-Bispectrum is complete just as the full version. An explanation to this lies in the paradoxes of the Universal Approximation Theorem [13]. Just because an arbitrarily large MLP can theoretically fit any function, this does not imply that it will happen for a practical, limited MLP. In practice, we hypothesize that the redundancy of the $G$-TC allows the MLP to distinguish inputs more easily. If the size of the model allows it, the $G$-TC or the full $G$-Bispectrum will provide better accuracy. However, when the size of the group is big, their use is often out of reach while the selective $G$-Bispectrum is scalable. In Table 2, we also notice that the Max $G$-pooling performs well compared to the others even though it is not complete. This is because we have many filters that allow for refined classification. Indeed, assume $f,\phi_{k}$ are black-and-white images with $N$ pixels. In consequence, $\max_{g}\Theta_{k}(g)\in\{0,1,...,N\}$ for $k=1,2,...,K$. The Max $G$-pooling allows a maximum separation of $(N+1)^{K}$ classes. In practice, this value is not reached, but it explains why Max $G$-pooling performs well. Figure 5 highlights this dependency of the Max $G$-pooling on the number of filters since the accuracy drops to less than $60\%$ with 2 filters. In comparison, the $G$-TC and the selective $G$-Bispectrum, which are complete, keep an accuracy above $85\%$ with 2 filters.

To conclude our numerical experiments, we study the robustness of the selective $G$-Bispectrum to adversarial attacks, following the analysis in Sanborn & Miolane [27, Figure 2]. Given an image $\widehat{f}:X\mapsto\mathbb{R}^{c}$ and a filter $\phi:X\mapsto\mathbb{R}^{c}$, they numerically verified the robustness (=completeness) of the $G$-TC by showing that

Indeed, Sanborn & Miolane [27, Figure 2] shows that only images that are identical up to rotation/reflection can yield the same $C_{n}/D_{n}$-TC. That is, the $G$-CNN with $G$-TC can not be “fooled” since only input in the same orbit yield the same output. It is well-known that the $G$-convolution $\phi*f$ is $G$-equivariant. Hence, a strictly equivalent experiment is to show that

On Figure 6, we show that the selective $G$-Bispectrum $\beta_{sel}$ is robust to adversarial attacks by solving

The signals are indeed recovered up to a translation, i.e., a group action of $C_{30}$. Moreover, despite (3) only optimizes using the selective $G$-Bispectrum, the full $G$-Bispectrum is correctly recovered.