Tag: Hebbian AI- No Backpropagation

The firing rate of biological neurons is limited by the refractory period, a brief period after activation during which a second activation is not possible.

I managed to sync neurons firing within the same pattern… But that does not seem to be enough… Neighboring neurons, not part of the current pattern, remain in an unknown state. That is problematic because the current pattern can activate also close enough neurons that happened to be in a close to activation state… Inhibitory neurons cannot inhibit neurons that activate at at the same time.. So I figured I need a refractory period… after a firing event, synaptic strength should be decreased to a defined level. This in turn would create chaos within the next layer, because some cycles would bring zero signal while the previous layers is in a refractory cycle.. Maybe next layer should work on a different frequency … should update slower… Well, I’m sure there are other problems that will show up and I don’t like having to deal with the firing rate, that would add additional complexity making things even harder to understand.

Am I sure that this is the desired behavior ? Not at all, but unless I have this working perfect, I cannot confirm or discard the hypothesis.. Right now, I’m not getting stable patterns (good or bad). The response depends on the previous pattern to some extent. If previous pattern was far, I get a (stable) response, if it was close, I get a different response.

Since my previous post, I focused only on understanding the source of asymmetries from my system. I disabled all secondary dendrites, because they are very hard to predict, but no luck. Then I thought maybe the inhibitory neurons are to blame, because I’m not so sure what should happen with individual synapses when their dendrite is inhibited. Should they still be under the LTP and LTD mechanisms ? Maybe, maybe not.

So after some adjustments in the inhibitory mechanism I decided to remove inhibition entirely. Yet, the asymmetries were still there. What I discovered from plotting potentials from individual neurons, was that they have a wave like behavior and they can get out of phase….. So two neurons would increase potential while the third would be in a decreasing phase…

Why this happens ? Because each neuron can be part of multiple patterns … So pattern A would activate neuron 1 but not neurons 2 and 3. When I switch to pattern B involving all 3 neurons, neuron 1 is in a different phase from neuron 2 and 3 , resulting in a non activation event… I’m assuming this is where small basket cells or chandelier cell may come into play, to regulate this out of phase situation… I’m unsure of what to do next… I can squash the waves easily enough when starting a new pattern, but this would only help in short term, wont work for a moving pattern… But most likely I will take the easy way out and see what happens later on..

With that said, brain waves might be real after all 🙂

When I started this project, I thought the invariance comes from the properties of complex cells, to respond to multiple similar signals (for example to respond to all vertical lines from their respective receptive fields)… I tried to simulate how this would work but my final conclusion was not that final… It seemed that it would not work, but I could never do a good enough simulation to say for sure.

Another way of obtaining invariance might be the movement of the eye, focus, so the invariance would not be created in the brain but in the eye itself.. This still may be the case, I have not explored this option at all.

But from my latest blunders it seems the mechanism of invariance might be something very peculiar… It may come from inhibition.. While small patterns behave more or less predictable, bigger patterns have become totally unpredictable due to multiple overlapping inhibitions.. The number of neurons activating is not proportional to the input signal.. One might see more firing for smaller patterns than from bigger patterns.. Very strange indeed..

I don’t have a coherent theory about training.. Sure I could get a single letter to work, but the second one is not certain, unless I understand what I’m looking for..

I tried already hundreds of patterns and I’m no closer to understanding what would be the best output. I tried small patterns, big patterns… nothing really helped..

After many trials and errors I’m starting to see the light… First signs of invariance, and how that might work, are here..

It’s a very small matrix with only 2 layers (in fact there are 8 layers if I count input/output and inhibitory layers), 4×4. If pattern is moved on the right side, then another set of inhibitory neurons control the output, and the response is very different, like having an entirely different pattern. I was theorizing that this is how it should behave, but I was never very sure. Also I observed clustering that I have never predicted, but in hindsight I should have. They were messing my beautiful patterns.. so I inhibited the secondary dendrites as well… all possible dendrites are now controlled by inhibitory neurons , still I see asymmetries that are troublesome.

Can this be coincidental ? Yes and no, yes because the patterns are not perfectly as predicted, no, because is close to what I was predicting, and prediction show that it should work.

Anyway some things are clear… This invariance, can only work on limited fields. Theoretically the size of the field is directly proportional to the number of hidden layers (the bigger the field, the more hidden layers). And this is where I’m going next… But there still work to be done here on this tiny set-up.

First of all, that’s my current setup, nothing fancy.

Second, I have made some progress but it’s uncertain. I don’t really have clear criteria for determining good/bad, am I going in a good direction or is it all bad.

About inhibitory neurons… My model failed in many ways and I concluded that maybe the biological model was not so bad after all. My inhibitory synapse kept being removed from all neurons after a while because with it in place nothing would work.. So neurons do help other neuron escape inhibition 😀 ….. to my astonishment..

I figured that a solution to my problem with neuronal reach could be (possibly should be) the orientation columns seen in biology. That’s where I’m going now. The feed-back issue is on ice for now, still have to solve other problems first.

What I call a computational cell is made up of 4 normal neurons governed by an inhibitory neuron as shown in my previous post. Now I increased the number of computational cells to 4 and everything became a big mess… Some neurons are now part of up to 4 cells… or 2.. or 3.. As synapses are further away from the neuron body, their contribution to the overall potential (activation potential), decreases. So the more synapses on a single neuron the more complexity resulting in a poor understanding of what’s going on and adjusting/optimizing the algorithms 🙁

Still some cell behave more or less predictable, predictable enough to realized that I have an additional problem, which I predicted, but I was hoping it will get somehow solved by the added complexity, something that I could not predict, but I could observe when actually running the full network. It did not happen. Far away patterns cannot connect to each other because I don’t have a fully connected network, neurons bind only in a limited area around their position (in the matrix) .Moreover things that worked on a single computational cell, such as a single neuron responding to all vertical lines, do not work among multiple computational cells. Last problem that I encountered is that now, the network, has become visible slower and I only have 128 neurons running wild… That’s because I’m forming now, tones of dendrites to respond to various patterns or perhaps there is something wrong within my latest modifications some wild loops working for no reasons… doesn’t bode well for the future though…

Overall I’d say I’m stuck with no clear way going forward.

Why am I writing this blog ? Because nobody is reading it 🙂 Well I write because by doing so I’m usually clarifying some thing in my head, sometimes laying out the problem clearly is a big step forward . For the same reasons I also speak to my friends about AI. I’m such a bore, I can tell you that much 🙂

I was hoping for more progress, but I did not put in the work required. Over the holidays I’ve been mostly lazy. But anyway, I did manage to have the inhibitory neurons working. The project is very small and may seem even smaller to the untrained eye 🙂 I worked on a 2 by 2 matrix, with only 2 layers, one layer to receive the pattern the other layer to show the output. Second layer is what I call a compression layer since it’s doing a data reduction more or less like a complex cell would, a single cell responds to similar pattern such as a vertical lines .

Why is this important at all ? Because otherwise a pattern eventually will spread all over the network in obscure ways. Say you have a pattern for letter A… some layers down the line you may find that pattern stored as an AAAAAAA chain without an inhibitory effect.

How important is this in the bigger picture ? Not nearly enough, but now I can move onto the next big issue, make the network learn based on feed-back… What does that mean ? it means that now I should have a 3rd layer which should associate 2 unrelated patterns as being the same, based on my feed-back.. take the two pattern above, a vertical line and a horizontal line. A third layer should have a single neuron firing whether the vertical line is showing or the horizontal one is… because I say so, you see 😀

When ? God knows 🙂

From my many simulations I concluded that inhibitory neurons (why in the world are they called Interneurons?) have to be there to stop information replicating endlessly on all neurons. So each inhibitory neuron will prevent the same information to be replicated in the cluster of neurons governed by a particular inhibitory neuron.

They also determine the level of discrimination, the smallest information that can be classified will depend on the number of inhibitory neurons. The more inhibitory neurons the smaller the information that can be classified, more discrimination. If one would want to have a pixel wise discrimination the number of inhibitory neurons should equal the number of pixels.

There is an additional role, to regulate frequency, but I cannot test this one since my frequency is fixed and because of that I don’t need this role added to my code.

On the update front… well, I learn every day something new, mostly by discarding ideas that don’t fit the general algorithms. I’ve reached a point where I need to code all the new ideas and see actual results. I cannot go further till I get a better understanding of the impact of the new ideas.

I’ve settled some concept that were somehow in limbo, what is the relation between the information contained in two different dendrites of the same neuron, or how many layers should my network contain, to accomplish a certain task.

But some older problems came back into focus… what is the reach of a single neuron ? Answering this question is not really a must for the layers doing data classification but it gets important for what I call decisions layers … Decision layers can be viewed as output layers for simple networks, but in my model they can be used to sandwich classification layers , that means I can have more than one in a network. For the next 2 weeks I’ll be working only on programing the new concepts.. see what I get. What I should get is identifying letter A at different positions and sizes as being letter A, and not a different pattern, on a 252 x 252 matrix…

I’ve spent already too much time on this topic with no real insights. Most of papers I read said something along the lines: they link together to overcame inhibition. But perhaps is not meant to be taken at face value. They can overcame inhibition only by being active, but if they are already active, what do they overcome then ? Moreover, we say they are inhibited, but in fact they are active ?

Assume such a dendritic structure: D(S1, S2), where S1 and S2 are 2 synapses. Assume S1 and S2 are not able to activate dendrite D by themselves and they need and additional S3 for the dendrite to become active and activate generic Neuron N1. That means the neuron providing the synapse S3, lets call it N2 has to fire first. This way may be possible to establish directionality as long as N1 does not link back to N2. But what is “direction”… Hubel and Wiesel experiments did have direction but in real life light is always there.. Could direction be the constant movement of the eye ? Even so, why would direction be important ?

Can’t find a believable reason for having a true inter-layer link. When I say “true inter-layer”, I’m trying to discard the possibility where neurons actually link in a feed-forward like network, but where a biological layer is actually made of many “neuronal” layers..