Hopfield's Neural Network Model

I've tried my best to explain this fascinating topic in fairly simple language.

📅 31 Jul 2022

Disclaimer: This post reflects my own understanding of the network, and is not reviewed.

+ Why did you publish it then?
- Shut up!

Hebbian Theory

Before we dive into the HNNM (Hopfield Neural Network Model) called Hebbian Theory or as some people may call it, Hebb Rule. (It’s called Hebb Rule because a gentleman named Donald Hebb has discovered this rule.) Examples always work, don’t they? Take a look at this picture and try to figure out what’s going on in there, once you’ve had a [vague] idea of it, read the following.

Hebbian Plasticity This is a picture I’ve taken from this YouTube video from Daniel Kochli.

Imagine someone puffing air into your eyes. You’d close your eyes, unconsciously, won’t you? That’s because the synapse (connection) between your somatosensory neurons (the neurons who “sense” the puffing) and your motor neurons (those who command your eye muscles to blink) is strong a mighty.

Now imagine a \(1000Hz\) tone (Approximately the highest note a female voice can produce.) being played for you. In an initial and normal state of consciousness, you won’t blink. Why? Because the synapse between the auditory neuron is not that strong to activate/amuse the motor cell that much.

We’ll go further: play a \(1000Hz\) tone for you and puff off air into your eyes, at the same time. What happens is that the weaker synapse actually get stronger. Who keeps company with the wolf will learn to howl. Surprisingly, after doing that for plenty of times, you’ll blink when you hear a \(1000Hz\) tone. That’s extraordinary, isn’t it? Believe me, it’s going to be even more interesting.

- Wait! That’s just a neural explanation for Classical Conditioning, isn’t it?
+ Hell Yeah!

So if \(V_{i} = V_{j}\) (\(V\) is the value of our neuron, which takes either \(1\) (on) or \(-1\) (off).) the weight of connection between these two will increase (by \(1\)): \(W_{ij} + 1\). And otherwise, if \(V_i \neq V_j\), \(W_{ij} - 1\). You can easily write the code for this simple function.

Introduction to the Hopfield Neural Network Model

You can read this word: H_pfi_ld Net_ork, though it’s not a “complete” one. Your brain is reconstructing this phrase, by using something we call content-addressable memory/auto-associative memory. The process kind of remained a mystery until someone handsome named John Hopfield suggested a mathematical model in 1982, which is 40 years ago. I won’t get into the mathematics a lot, because we don’t quite need much of it to have an intuition of the subject.

How does it work, anyway? Well, that’s fairly simple: in the model, we have some “nodes” and these nodes/neurons are linked to each other, and these links have some “weights”. It may seem absurd and very abstract, so let me show you a simple visualized model:

A Simple Hoppfield Network

Hint: Patterns are learned through changing the weights (\(W\)) of links.

Tip: Don’t feel bad if you can’t relate the topics discussed. Once you’ve finished reading this article you can think through it again.

The Process

Let’s say each node (neuron) has only two possible states. It can be either on (\(+1\)) or off (\(-1\)). (Some may use 0 instead of \(-1\), but I’m not going to use the binary system here.) The initial state of all weights is zero. (They have no weights, so no pattern is learned to the time being.) Let’s teach this pattern to our network of 10 neurons: \([1,-1,1,-1]\).

As I previously said, these neurons are interconnected, AND now, we can update the weights of these links according to the Hebb Rule, so that we can consider the pattern as a learned pattern. I’m going to call this phase the learning phase. This is the weights’ matrix after one update:

\[\begin{bmatrix}0 & -1 & 1 & -1 \\ -1 & 0 & -1 & 1 \\ 1 & -1 & 0 & -1 \\ -1 & 1 & -1 & 0 \end{bmatrix}\]

Fun Fact I: Why is the diameter of matrix all zero? Well, because each neuron is not connected to itself, so it has no self-weight. In other words, \(W_{ii} = 0\).

Fun Fact II: In the Hopfield Model, all weights are symmetric, which means that the weight between node \(i\) and node \(j\) is equal to the weight between node \(j\) and node \(i\). (\(W_{ij} = W_{ji}\))

The desired pattern is learned (stored in weights) now. What we need to do is to give another pattern, preferably not so distinct, to the network and change it, so that the system can relate it to the learned pattern. I’ll call this the testing phase.

Rules of the Game

Now we’ll get into the real Hopfield Model Rules. neuron \(i\)’s state is determined by the sum of other neuron’s values times their connection weight. If the sum is non-negative, then the value of \(i\) is going to be \(1\), and otherwise if it’s negative, the value is going to be \(-1\):

\[V_i = +1 \space \space if \space \space \displaystyle\sum_{i \neq j} W_{ij} V_j \geq 0\] \[V_i = -1 \space \space if \space \space \displaystyle\sum_{i \neq j} W_{ij} V_j < 0\]

Now, I’ll give the pattern \([1,-1,1,-1]\) to our neural Hopfield Network. It’s patently not so different than the learned pattern, in fact I just changed one value. If we run the testing phase, we’ll see that the pattern is rebuilt into the learned pattern after several (or even one) session(s) of testing using our rules and the generated weights’ matrix. That’s awesome, right? Hail Hopfield!

Physical Interpretation

This model has a physical derivative, too. In physics, we know that objects tend to lower their energy level as much as possible. Energy is to objects what responsibilities are to human beings. They want to opt out of it. That’s what happens with the Hopfield Model: the learned pattern(s) have the lowest energy level among other patterns, so other patterns tend to fall down the hill of energy. This is a simple visualization:

Energy Lanscape

Image taken from Hopfield Networks: Neural Memory Machines by Ethan Course.

This is going to be updated, for sure, and I’m probably going to publish a very basic implementation of the Hopfield in Python, soon. By the way, I would greatly appreciate any feedback on this post.