119

Holographic neural networks∗

Robert Manger†

Abstract. Holographic neural networks are a new and promising

type of artificial neural networks. This article gives an overview of
the holographic neural technology and its possibilities. The theoretical
principles of holographic networks are first reviewed. Then, some other
papers are presented, where holographic networks have been applied or
experimentally evaluated. A case study dealing with currency exchange
rate prediction is described in more detail.

Key words: neurocomputing, artificial neural networks, holographic

neural technology, prediction problems

Sažetak. Holografske neuronske mreže. Holografske neuronske

mreže predstavljaju jednu novu i obećavajuću vrstu umjetnih neuronskih
mreža. Članak daje prikaz holografske neuronske tehnologije i njenih
mogućnosti. Prvo se objašnjavaju teoretski principi na kojima se za-
snivaju holografske mreže. Zatim se prikazuju neki drugi radovi koji se
bave primjenom ili eksperimentalnom evaluacijom holografskih mreža.
Detaljnije je opisan studijski primjer prognoziranja valutnih tečajeva.

Ključne riječi: neuronsko računanje, umjetne neuronske mreže,

holografska neuronska tehnologija, problemi prognoziranja

1. Introduction
Neurocomputing [4] is a technological discipline concerned with information process-
ing systems (for example neural networks) that autonomously develop operational
capabilities in adaptive response to an information environment. Neurocomput-
ing is a fundamentally new and different approach to information processing. It
is a first alternative to programmed computing, which has dominated information
processing for the last 50 years.
An artificial neural network is a data processing structure (real or simulated)
that bears some resemblance to a natural neural tissue. More precisely, it is a set
of interconnected basic processing elements called neurons. For an input (called
stimulus) this set automatically produces an output (response). Furthermore, the
∗ The lecture presented at the Mathematical Colloquium in Osijek organized by Croatian

Mathematical Society - Division Osijek, December ???, 1998.

† Department of Mathematics, University of Zagreb, Bijenička cesta 30, HR-10 000 Zagreb,

Croatia, e-mail: [email protected]

120 R. Manger

network can be trained to faithfully reproduce a predefined collection of stimulus-

response associations. After successful training (learning), the used associations are
also slightly generalized, i.e. the network produces a plausible response to a new
stimulus that is similar to some of the learned stimuli.
Typical applications of neural networks include pattern recognition, classifica-
tion, prediction (forecasting), control of complex systems, signal processing. Some
more concrete examples are: optical character recognition, voice recognition, credit
scoring, bankruptcy prediction, stock forecasting, robot control, acoustic noise fil-
tering.
A problem suitable to be solved by neural networks has the following general
characteristics. It reduces to a correspondence between some kind of stimuli and
responses. However, there exists no simple mathematical model of that correspon-
dence. Instead, concrete examples of stimuli and required responses are given.
These examples are comprehensive enough to express all important aspects of the
problem.
A solution obtained by neural networks has the following characteristics. It is
obtained by designing a suitable network and by network training. Thus it avoids
the classical algorithmic approach and programming. Unfortunately, it is a black-
box solution, which does not give any explanation to its responses, decisions, etc.
There are many types of neural networks, which differ in various details including
neuron structure, network topology and training algorithms. Holographic networks
are yet another class of neural networks, which have recently been proposed [11].
Although conforming to the general paradigm, these networks are unusual in many
aspects, and they provide an alternative to conventional network types [4].
The aim of this presentation is to inform the reader about holographic neu-
ral networks, its peculiarities, and its possible applications. We believe that this
novel type of networks deserves more attention, and that it should play a more
prominent role in the future. Apart from this introduction, the text consists of
two major sections and a conclusion. Section 2 reviews the theory of holographic
networks: through this review the holographic neural process is analyzed, and its
main properties are explained in more detail. Section 3 refers to some other papers,
which consider possible applications of holographic networks, or where those net-
works have been evaluated on experimental data; special emphasis is put on a case
study dealing with currency exchange rate prediction. R

2. Theoretical issues
The main difference between holographic and conventional neural networks is that
a holographic neuron is more powerful than a conventional one, so that it is func-
tionally equivalent to a whole conventional network. Consequently, a holographic
network usually requires a very simple topology consisting of only few neurons. An-
other characteristic of the holographic technology is that it represents information
by complex numbers operating within two degrees of freedom (value and confi-
dence). Also an important property is that holographic training is accomplished
by direct (almost non-iterative) algorithms, while conventional training is based on
relatively slow “back-propagation” (gradient) algorithms.
A holographic neuron is sketched in Figure 1. As we can see, it is equipped
 Holographic neural networks 121

with only one input channel and one output channel. However, both channels carry
whole vectors of complex numbers. An input vector S is called a stimulus and it
has the form
S = [λ1 eiθ1 , λ2 eiθ2 , . . . , λn eiθn ].
An output vector R is called a response and its form is

R = [γ1 eiφ1 , γ2 eiφ2 , . . . , γm eiφm ].

All complex numbers above are written in polar notation, so that moduli (magni-
tudes) are interpreted as confidence levels of data, and arguments (phase angles)
serve as actual values of data. The neuron internally holds a complex n × m matrix
X = [xjk ], which serves as a memory for recording associations.

S - X -R

Figure 1. A holographic neuron

Now we will explain the basic learning process. Learning one association between
a stimulus S and a desired response R requires that the correlation between the
j-th stimulus element and the k-th response element is accumulated in the (j, k)-th
entry of the memory matrix. More precisely:

xjk += λj γk ei(φk −θj ) .

The same formula can be written in the matrix-vector form:

X+= S̄ τ R. (1)

Here S̄ τ denotes the conjugated transpose of the vector S.

To accomplish training with multiple stimulus-response associations, the basic
learning step (1) must be repeated in turn for each association. Note that all asso-
ciations must be enfolded onto the same memory matrix. Each learned association
will slightly disturb the other learned associations.
The computed response R∗ to a new stimulus
∗ ∗ ∗
S ∗ = [λ∗1 eiθ1 , λ∗2 eiθ2 , . . . , λ∗n eiθn ]

is obtained as a matrix-vector product

1 ∗
R∗ = S X. (2)
c
The normalization coefficient is usually taken as
n
X
c= λ∗k .
k=1
122 R. Manger

Now there follows an analysis of the computed response. Suppose that the
associations
(S (t) , R(t) ), t = 1, 2, . . . , p,
have previously been learned. Let us consider the k-th response element, 1 ≤ k ≤ m.
According to (1) and (2) we have:
n p p n
∗ 1 X ∗ iθj∗ X (t) (t) i(φ(t) (t) 1 X (t) iφ(t) X ∗ (t) i(θj∗ −θj(t) )
γk∗ eiφk = λj e λj γk e k −θj ) = γ e k λj λj e .
c j=1 t=1
c t=1 k j=1

The above formula can be rearranged in the following way:

p
X
∗ (t)
γk∗ eiφk = Λ(t) eiΨ ,
t=1

where
 1/2
(t) n n
γk  X ∗ (t) (t)
X (t) (t)
Λ(t) = [ λj λj cos(θj∗ − θj )]2 + [ λ∗j λj sin(θj∗ − θj )]2  ,
c j=1 j=1
" Pn #
∗ (t) ∗ (t) (t)
(t) −1 j=1 λj λj sin(θj − θj + φk )
Ψ = tan Pn .
∗ (t) ∗ (t) (t)
j=1 λj λj cos(θj − θj + φk )

Thus the chosen response element is a sum of many components. Each component
corresponds to one of the learned associations.
Now let us consider the case where the new stimulus S ∗ is approximately equal
to one of the previously learned stimuli. Suppose that for some l, 1 ≤ l ≤ p,

S ∗ ≈ S (l) .

Suppose also that for all j and t,

(t) (t)
λ∗j ≈ 1, λj ≈ 1, γk ≈ 1.

Then the above expressions for Λ(t) and Ψ(t) indicate that the l-th component of
the response has a relatively big confidence level and an explicit direction:
(l)
Λ(l) ≈ 1, Ψ(l) ≈ φk .

The other components usually have smaller confidence levels and different direc-
tions. It means, for t 6= l:

Λ(t) << 1, Ψ(t) = · · · ? · · · .

It is expected that these other components will neutralize in a manner analogous

to random walk. Consequently, the generated response will be approximately equal
to the desired learned response, as shown in Figure 2.
 Holographic neural networks 123

H
@
PP ¡
A
@
¤¤º ££±
¤£
error ¤ £
¤£
¤£
¤£
¤£
d£¤

Figure 2. Response to a learned stimulus

An important aspect of the holographic neural technology is data pre- or post-

processing. For instance, data conversion is needed to switch between external
(real or integer) and internal (complex) representation, i.e. original application data
should be transformed into arguments (angles) and vice versa. Some commonly
used types of data conversion are: linear conversion (real values from a known fi-
nite range), sigmoid or arctan conversion (real values from an unknown or infinite
range), step-function conversion (a finite set of values). As we will see in the next
section, additional stimulus preprocessing procedures can be used to control the
holographic process in some way.
In addition to the already presented formula (1), the holographic neural tech-
nology provides also some other more advanced learning modes. The following en-
hanced learning formula takes into account the prior knowledge accumulated within
the neuron, and tries to minimize distortion of the previously learned associations.
Let S again be a given stimulus, and R the desired response. The idea is to compute
first the response that would already have been generated:
1
R0 = SX.
c
Then compute the difference between the generated and the desired response:

Rdif = R − R0 .

Finally, learn the association between the stimulus and the above difference, by
using the old formula (1):
X+= S̄ τ Rdif .
The resulting formula, which can replace (1), is
µ ¶
τ 1
X+= S̄ R − SX . (3)
c

In fact, this formula is used as default since it assures better performance than (1).
As before, to accomplish training on a set of stimulus-response associations,
the enhanced learning step (3) has to be repeated for each association in the set.
Note that the order of steps now becomes important, namely the first association
is more distorted by subsequent encodings than the last one. Therefore, the whole
learning cycle should be repeated several times in order to stabilize. So we end up
124 R. Manger

with a form of iterative training. Still, the number of needed iterations (so called
epochs) is considerably smaller than in traditional “back propagation” algorithms,
i.e. according to [2] it is never greater than 20.
Holographic networks also allow a special regime called training with a reduced
memory profile. When this regime is applied, the previously learned stimulus-
response associations are gradually forgotten as training progresses. Consequently,
more recently learned associations expose stronger influence on a response than
older ones. The memory profile is expressed in percentages (100% - permanent
memory, < 100% - reduced memory), and it is controlled through periodical re-
scaling (reduction) of the entries in the memory matrix X.
Finally, let us note that holographic networks allow incremental training. It
means that an already trained neuron can subsequently learn an aditional stimulus-
response association. The latter is not true for traditional networks, where adding
a new training example usually means starting the whole training procedure from
scratch. Incremental training of a holographic neuron is possible for both learning
formulas (1) and (3), and for any memory profile. Again, the additional learning
step will slightly distort the prior knowledge. However, this distortion is not visible
if a reduced memory profile is used.

3. Applications and experiments

Holographic neural networks can be applied to the same problems as the other
network types. However, the process of designing a holographic application is quite
specific. Namely, the network topology for most problems turns out to be trivial or
very simple. Rather than with topology, the designer is much more concerned with
data preprocessing. Adequate preprocessing procedures can, for instance, be used
to improve accuracy in reproducing the learned associations, or to expand learning
capacity, or even to control generalization properties.
Generally speaking, holographic networks are very suitable for those problems
where stimuli are long vectors with symmetrically (uniformly) distributed argu-
ments. A longer stimulus vector assures a greater learning capacity, i.e. a greater
number of stimulus-response associations that can be learned. Symmetry in argu-
ments assures accuracy in reproducing the learned stimulus-response associations.
Long stimuli are usually obtained by discretization of continuous functions or digi-
talization of images. Symmetry of long vectors can be improved by adequate pre-
processing, so called stimulus symmetrization based on the sigmoid function [12].
There are few demo examples in [2] that illustrate the applicability of holo-
graphic networks to problems dealing with continuous signals and images. The
examples include: satellite control, image filtering, recognition of a detail in a large
picture.
Holographic networks can also be applied to classification problems. However,
some difficulties may arise if the number of attributes used for classification is small.
Then it is necessary to artificially enlarge the stimulus vector, by a preprocessing
procedure called stimulus expansion. The standard expansion procedure [12], based
on higher order product terms, may prove inadequate for an extremely short stim-
ulus. A more suitable expansion method, based on sines and cosines, has been
proposed in [6]. The same paper also introduces a new symmetrization method,
 Holographic neural networks 125

and evaluates both methods on the iris flowers benchmark classification problem.
Other examples of classification with holographic networks have been described in
[10, 5]; the concrete problems considered there comprise credit scoring and neuro-
logical diagnosis.
It has been suggested in [1] that holographic networks can be applied to data
compression. Namely, a holographic neuron can be used to memorize the set of
values from a file (stimulus: the value identifier, response: the value itself). After
training, the neuron should be able to approximately reproduce any value. If the
memory matrix inside the neuron happens to be smaller than the original file, we can
speak about (lossy) data compression. This idea has been explored experimentally
in [8]. The obtained results indicate that the considered holographic compression
method works well only for very regular (smooth, redundant) files.
We believe that holographic networks are very suitable for prediction (forecast-
ing) problems, specially if the considered system is dominated by short-term trends.
Namely, a natural training regime for such problems is incremental training with a
low memory profile (knowledge should be constantly revised). The recommended
regime can easily be realized with holographic networks, but much harder with tra-
ditional network types. To illustrate this, we now present original results dealing
with currency exchange rate prediction.
In our experiments we used authentic data from “Zagrebačka banka” (Bank of
Zagreb) comprising the exchange rates of seven currencies (ATS, CHF, DEM, FRF,
GBP, ITL, USD) for each working day between 1st October 1992 and 1st October
1993. To eliminate unwanted effects of the domestic inflation, we chose ATS as the
reference currency and expressed the other six currencies in terms of ATS. Some
basic statistical parameters are shown in Table 1.

currency mean standard minimum maximum

deviation
ATS/CHF 0.1278 0.0024 0.1236 0.1322
ATS/DEM 0.1421 0.0000 0.1415 0.1422
ATS/FRF 0.4843 0.0063 0.4774 0.5040
ATS/GBP 0.0577 0.0013 0.0550 0.0612
ATS/ITL 131.4096 5.0543 119.9376 142.1060
ATS/USD 0.0879 0.0036 0.0815 0.1016

Table 1. Statistical parameters for the exchange rates data

The described data were interpreted as a set of stimulus-response pairs. In each

pair, the stimulus consisted of exchange rates for five consecutive days, and the
response comprised the next-day rates. A suitably sized holographic neuron was
chosen. The stimulus-response examples were presented to the neuron in chrono-
logical order. Each example was first used for testing (i.e. prediction), and then for
additional training. The mean absolute prediction errors were recorded. The whole
procedure was repeated with different levels of memory profile. The obtained results
are summarized in Table 2. The same table also contains comparable results which
would have been obtained by the traditional moving averages prediction method
[9].
126 R. Manger

prediction memory mean absolute prediction error

method profile ATS/ ATS/ ATS/ ATS/ ATS/ ATS/
CHF DEM FRF GBP ITL USD
holographic 100% 0.0004 0.0000 0.0021 0.0004 1.6089 0.0007
80% 0.0004 0.0000 0.0021 0.0004 1.6089 0.0007
20% 0.0004 0.0000 0.0021 0.0004 1.5986 0.0006
10% 0.0004 0.0000 0.0023 0.0004 1.3994 0.0006
5% 0.0003 0.0000 0.0012 0.0003 0.6818 0.0006
3% 0.0003 0.0000 0.0008 0.0003 0.6766 0.0006
2% 0.0003 0.0000 0.0008 0.0003 0.6748 0.0006
1% 0.0003 0.0000 0.0008 0.0003 0.6738 0.0006
moving
averages n/a 0.0005 0.0000 0.0010 0.0004 0.9930 0.0008

Table 2. Prediction of exchange rates

As we can see from Table 2, the performance of our holographic prediction

method depends on the memory profile used. With a high memory profile (persis-
tent memory) the results are in some aspects worse than those obtained by moving
averages. With a low memory profile (short-term memory) the holographic method
clearly outperforms moving averages. Some other results based on the same data
set are available in [7].
Finally, let us note that a software package called the HNeT system is now
available, which enables easy experimenting with holographic networks and rapid
application development. The first version of the HNeT system [1] was designed for a
PC with MS-DOS and a transputer expansion card. The present version runs under
MS Windows and consists of two separate products: HNeT Discovery [2] and HNeT
Professional [3]. HNeT Discovery is an emulator that serves for experimenting,
prototyping and evaluation. HNeT Professional is a library of subroutines that can
be used together with MS Visual Basic or Visual C++ for application development.

4. Conclusion
Holographic neural networks are in some aspects superior to traditional network
types. For instance, they are more suitable for prediction problems, thanks to
technical feasibility of incremental training with a reduced memory profile. Also,
holographic networks assure quicker convergence during training, and are easier to
use.
The most important phase in designing a holographic application is choosing
adequate data preprocessing. Therefore, it is important that a diversity of prepro-
cessing procedures are available, so that conflicting requirements of various applica-
tions can always be accommodated. Many procedures have already been proposed
and experimentally tested. At this moment, a more reliable mathematical analysis
of the existing methods is needed.
Holographic neural networks are still an obscure technology. There are not
many papers or books that treat or even mention this type of networks. One of the
reasons may be the reluctance of the traditional “connectionist” neurocomputing
 Holographic neural networks 127

community. However, the situation could change very soon, thanks to the software
support that is now available for holographic networks.

References
[1] AND Corporation, HNeT Neural Development System, Version 1.0 , Hamil-
ton, Ontario, 1990.

[2] AND America Ltd., HNeT Discovery Package - Version 1.3 for Windows,
Oakville, Ontario, 1993.
[3] AND America Ltd., Using the HNeT Professional Development System,
Oakville, Ontario, 1993.
[4] R. Hecht-Nielsen, Neurocomputing, Addison-Wesley, Reading, Massachus-
etts, 1990.
[5] R. Ho, J. G. Sutherland, I. Bruha, Neurological fuzzy diagnoses: holo-
graphic versus statistical versus neural methods, in: Frontier Decision Support
Concepts (V. L. Plantamura, B. Souček and G. Visaggio, Eds.), John Wiley and
Sons, New York, 1994, 155–169.

[6] R. Manger, B. Souček, New preprocessing methods for holographic neural

networks, in: Proceedings of the International Conference on Artificial Neural
Nets and Genetic Algorithms - ANNGA’93 (R. F. Albrecht, C. R. Reeves and
N. C. Steele, Eds.), Springer-Verlag, Vienna, 1993, 190–197.

[7] R. Manger, M. Mauher, Using holographic neural networks for currency ex-
change rates prediction, in: Proceedings of the 16th International Conference
on Information Technology Interfaces - ITI’94 (V. Čerić and V. Hljuz-Dobrić,
Eds.), University Computing Centre, Zagreb, 1994, 143–150.
[8] R. Manger, Holographic neural networks and data compression, Informatica,
21(1997), 667–676.
[9] A. F. Siegel, Statistics and Data Analysis, John Wiley, New York, 1988.
[10] B. Souček, J. G. Sutherland, G. Visaggio, Holographic decision support
system: credit scoring based on quality metrics, in: Frontier Decision Support
Concepts (V. L. Plantamura, B. Souček and G. Visaggio, Eds.), John Wiley and
Sons, New York, 1994, 171–182.
[11] J. G. Sutherland, Holographic model of memory, learning and expression,
International Journal of Neural Systems 1(1990) 256–267.
[12] J. G. Sutherland, The holographic neural method, in: Fuzzy, Holographic
and Parallel Intelligence (B. Souček, Ed.), John Wiley and Sons, New York,
1992, 30–63.