đ 31 Jul 2022
Disclaimer: This post reflects my own understanding of the network, and is not reviewed.
+ Why did you publish it then?
- Shut up!
Hebbian Theory
Before we dive into the HNNM (Hopfield Neural Network Model) called Hebbian Theory or as some people may call it, Hebb Rule. (Itâs called Hebb Rule because a gentleman named Donald Hebb has discovered this rule.) Examples always work, donât they? Take a look at this picture and try to figure out whatâs going on in there, once youâve had a [vague] idea of it, read the following.
This is a picture Iâve taken from this YouTube video from Daniel Kochli.
Imagine someone puffing air into your eyes. Youâd close your eyes, unconsciously, wonât you? Thatâs because the synapse (connection) between your somatosensory neurons (the neurons who âsenseâ the puffing) and your motor neurons (those who command your eye muscles to blink) is strong a mighty.
Now imagine a \(1000Hz\) tone (Approximately the highest note a female voice can produce.) being played for you. In an initial and normal state of consciousness, you wonât blink. Why? Because the synapse between the auditory neuron is not that strong to activate/amuse the motor cell that much.
Weâll go further: play a \(1000Hz\) tone for you and puff off air into your eyes, at the same time. What happens is that the weaker synapse actually get stronger. Who keeps company with the wolf will learn to howl. Surprisingly, after doing that for plenty of times, youâll blink when you hear a \(1000Hz\) tone. Thatâs extraordinary, isnât it? Believe me, itâs going to be even more interesting.
- Wait! Thatâs just a neural explanation for Classical Conditioning, isnât it?
+ Hell Yeah!
So if \(V_{i} = V_{j}\) (\(V\) is the value of our neuron, which takes either \(1\) (on) or \(-1\) (off).) the weight of connection between these two will increase (by \(1\)): \(W_{ij} + 1\). And otherwise, if \(V_i \neq V_j\), \(W_{ij} - 1\). You can easily write the code for this simple function.
Introduction to the Hopfield Neural Network Model
You can read this word: H_pfi_ld Net_ork, though itâs not a âcompleteâ one. Your brain is reconstructing this phrase, by using something we call content-addressable memory/auto-associative memory. The process kind of remained a mystery until someone handsome named John Hopfield suggested a mathematical model in 1982, which is 40 years ago. I wonât get into the mathematics a lot, because we donât quite need much of it to have an intuition of the subject.
How does it work, anyway? Well, thatâs fairly simple: in the model, we have some ânodesâ and these nodes/neurons are linked to each other, and these links have some âweightsâ. It may seem absurd and very abstract, so let me show you a simple visualized model:
Hint: Patterns are learned through changing the weights (\(W\)) of links.
Tip: Donât feel bad if you canât relate the topics discussed. Once youâve finished reading this article you can think through it again.
The Process
Letâs say each node (neuron) has only two possible states. It can be either on (\(+1\)) or off (\(-1\)). (Some may use 0 instead of \(-1\), but Iâm not going to use the binary system here.) The initial state of all weights is zero. (They have no weights, so no pattern is learned to the time being.) Letâs teach this pattern to our network of 10 neurons: \([1,-1,1,-1]\).
As I previously said, these neurons are interconnected, AND now, we can update the weights of these links according to the Hebb Rule, so that we can consider the pattern as a learned pattern. Iâm going to call this phase the learning phase. This is the weightsâ matrix after one update:
\[\begin{bmatrix}0 & -1 & 1 & -1 \\ -1 & 0 & -1 & 1 \\ 1 & -1 & 0 & -1 \\ -1 & 1 & -1 & 0 \end{bmatrix}\]Fun Fact I: Why is the diameter of matrix all zero? Well, because each neuron is not connected to itself, so it has no self-weight. In other words, \(W_{ii} = 0\).
Fun Fact II: In the Hopfield Model, all weights are symmetric, which means that the weight between node \(i\) and node \(j\) is equal to the weight between node \(j\) and node \(i\). (\(W_{ij} = W_{ji}\))
The desired pattern is learned (stored in weights) now. What we need to do is to give another pattern, preferably not so distinct, to the network and change it, so that the system can relate it to the learned pattern. Iâll call this the testing phase.
Rules of the Game
Now weâll get into the real Hopfield Model Rules. neuron \(i\)âs state is determined by the sum of other neuronâs values times their connection weight. If the sum is non-negative, then the value of \(i\) is going to be \(1\), and otherwise if itâs negative, the value is going to be \(-1\):
\[V_i = +1 \space \space if \space \space \displaystyle\sum_{i \neq j} W_{ij} V_j \geq 0\] \[V_i = -1 \space \space if \space \space \displaystyle\sum_{i \neq j} W_{ij} V_j < 0\]Now, Iâll give the pattern \([1,-1,1,-1]\) to our neural Hopfield Network. Itâs patently not so different than the learned pattern, in fact I just changed one value. If we run the testing phase, weâll see that the pattern is rebuilt into the learned pattern after several (or even one) session(s) of testing using our rules and the generated weightsâ matrix. Thatâs awesome, right? Hail Hopfield!
Physical Interpretation
This model has a physical derivative, too. In physics, we know that objects tend to lower their energy level as much as possible. Energy is to objects what responsibilities are to human beings. They want to opt out of it. Thatâs what happens with the Hopfield Model: the learned pattern(s) have the lowest energy level among other patterns, so other patterns tend to fall down the hill of energy. This is a simple visualization:
Image taken from Hopfield Networks: Neural Memory Machines by Ethan Course.
This is going to be updated, for sure, and Iâm probably going to publish a very basic implementation of the Hopfield in Python, soon. By the way, I would greatly appreciate any feedback on this post.