More failures, but a new learning theory

While I was focusing on fixing the synchronization issue, I lost sight of another serious issue. Once I introduced the semantics of dendrites, I lost the learning mechanism.. Not only that but I also lost the inhibition mechanism.. Inhibition could have been fixed somehow, but I realized that without the current mechanism of inhibition is not possible to synchronize neuronal activity. Maybe I need 2 inhibitions to get back to the previous state.

Anyway when synchronization was somehow fixed, I realized that there was no learning anymore. Of course I did not think through the changes I introduced with the semantics of dendrites… I read more neuroscience articles watched some online lectures.. got disappointed on the lack of clarity but eventually I came up with a new theory for neuronal learning inspired by what I learned. I went and ran multiple simulation scenarios on an Excel sheets and it seems to work.

The new theory is unfortunately much more complicated, meaning many things could go wrong but it has some clear advantages and is much more in line with what’s known (or assumed) in biology:

  • learning now integrates firing rates (which I despise because it makes understanding more difficult)
  • multiple synapses on multiple dendrites can activate now to generate an activation potential.
  • there is a new mechanism for “dendritic growth”, which is to say that I now have a rule, based on activity, for when a dendrite can accept connections. The model does not tell me when to seek new connections though..

The drawback ? Firing rates… I still use the concept as defined in my previous post so I’m not using time but cycles to calculate a firing rate. I’m still hoping that I won’t have to use actual time for firing rates. Also there are still many unknowns, LTD is not so clear anymore, so I may still end up with yet another failure. In terms of coding, it should not be difficult to implement, but it will take some time to understand if something is “right” or “wrong”.

Firing Rates and Neuronal Synchronization

I’m defining firing rate as:

Cycles needed by a single presyanptic neuron to activate a postsynaptic neuron,

Assume a single presynaptic neuron could activate a single postsynaptic neuron. If the presynaptic neuron has to activate 3 times before it can activate the postysnaptic neuron then the firing rate is 1/3. The firing rate is then a function of how much potential can a single synapse bring per activation. So close (proximal) synapses to the neuron body would generate higher firing rates than distal synapses. So the firing rate is not entirely depended on the firing rate of the presynapstic neuron.

Another definition is for “semantics of dendrites” .

The semantics describes the relationship between two dendrites. If we define a Dendrite a set of synapses (with values ON/OFF) , then the semantic between dendrite A and B will be the percentage of synapses from dendrite B being ON, while dendrite A is generating an activation potential.. Still working on this definition.

I hit a dead end with the current development branch. I calculated the theoretical dendrites and then tried to actually obtain all of them within my simulation by doing a precise training. It did not take long to realize that most of the dendrites could not actually form because they were being blocked on way or another. So I decided to switch to a more advance model in which I included the semantics of dendrites. Till now dendrites were independent. The model is actually what I started with but was too complex to handle and it had a theoretical drawback, the AI would go “blind”. I implemented most of the changes and indeed the blindness problem is there and I don’t have yet a solution. What do I mean by blindness ? Is some sort of over-training which leads to synapses being definitively removed from neurons.

But more concerning than blindness is still neuronal synchronization. I also implemented a form of “firing rates”. But that immediately added a more chaotic behavior. Running same pattern would end up with different responses on different cycles, first would show up most probable response then the neurons will go through a cooling cycle and the next most probable response (if any possible), would show up…

Firing rates bring more chaos to an already chaotic model

Neuronal refractory period

The firing rate of biological neurons is limited by the refractory period, a brief period after activation during which a second activation is not possible.

I managed to sync neurons firing within the same pattern… But that does not seem to be enough… Neighboring neurons, not part of the current pattern, remain in an unknown state. That is problematic because the current pattern can activate also close enough neurons that happened to be in a close to activation state… Inhibitory neurons cannot inhibit neurons that activate at at the same time.. So I figured I need a refractory period… after a firing event, synaptic strength should be decreased to a defined level. This in turn would create chaos within the next layer, because some cycles would bring zero signal while the previous layers is in a refractory cycle.. Maybe next layer should work on a different frequency … should update slower… Well, I’m sure there are other problems that will show up and I don’t like having to deal with the firing rate, that would add additional complexity making things even harder to understand.

Am I sure that this is the desired behavior ? Not at all, but unless I have this working perfect, I cannot confirm or discard the hypothesis.. Right now, I’m not getting stable patterns (good or bad). The response depends on the previous pattern to some extent. If previous pattern was far, I get a (stable) response, if it was close, I get a different response.

Brain waves

Since my previous post, I focused only on understanding the source of asymmetries from my system. I disabled all secondary dendrites, because they are very hard to predict, but no luck. Then I thought maybe the inhibitory neurons are to blame, because I’m not so sure what should happen with individual synapses when their dendrite is inhibited. Should they still be under the LTP and LTD mechanisms ? Maybe, maybe not.

So after some adjustments in the inhibitory mechanism I decided to remove inhibition entirely. Yet, the asymmetries were still there. What I discovered from plotting potentials from individual neurons, was that they have a wave like behavior and they can get out of phase….. So two neurons would increase potential while the third would be in a decreasing phase…

Two neurons out of phase

Why this happens ? Because each neuron can be part of multiple patterns … So pattern A would activate neuron 1 but not neurons 2 and 3. When I switch to pattern B involving all 3 neurons, neuron 1 is in a different phase from neuron 2 and 3 , resulting in a non activation event… I’m assuming this is where small basket cells or chandelier cell may come into play, to regulate this out of phase situation… I’m unsure of what to do next… I can squash the waves easily enough when starting a new pattern, but this would only help in short term, wont work for a moving pattern… But most likely I will take the easy way out and see what happens later on..

With that said, brain waves might be real after all 🙂

What is the basis for “invariance” ?

When I started this project, I thought the invariance comes from the properties of complex cells, to respond to multiple similar signals (for example to respond to all vertical lines from their respective receptive fields)… I tried to simulate how this would work but my final conclusion was not that final… It seemed that it would not work, but I could never do a good enough simulation to say for sure.

Another way of obtaining invariance might be the movement of the eye, focus, so the invariance would not be created in the brain but in the eye itself.. This still may be the case, I have not explored this option at all.

But from my latest blunders it seems the mechanism of invariance might be something very peculiar… It may come from inhibition.. While small patterns behave more or less predictable, bigger patterns have become totally unpredictable due to multiple overlapping inhibitions.. The number of neurons activating is not proportional to the input signal.. One might see more firing for smaller patterns than from bigger patterns.. Very strange indeed..

Stuck in training…

I don’t have a coherent theory about training.. Sure I could get a single letter to work, but the second one is not certain, unless I understand what I’m looking for..

I tried already hundreds of patterns and I’m no closer to understanding what would be the best output. I tried small patterns, big patterns… nothing really helped..

First steps in invariance for size and position

After many trials and errors I’m starting to see the light… First signs of invariance, and how that might work, are here..

It’s a very small matrix with only 2 layers (in fact there are 8 layers if I count input/output and inhibitory layers), 4×4. If pattern is moved on the right side, then another set of inhibitory neurons control the output, and the response is very different, like having an entirely different pattern. I was theorizing that this is how it should behave, but I was never very sure. Also I observed clustering that I have never predicted, but in hindsight I should have. They were messing my beautiful patterns.. so I inhibited the secondary dendrites as well… all possible dendrites are now controlled by inhibitory neurons , still I see asymmetries that are troublesome.

Can this be coincidental ? Yes and no, yes because the patterns are not perfectly as predicted, no, because is close to what I was predicting, and prediction show that it should work.

Anyway some things are clear… This invariance, can only work on limited fields. Theoretically the size of the field is directly proportional to the number of hidden layers (the bigger the field, the more hidden layers). And this is where I’m going next… But there still work to be done here on this tiny set-up.

Inhibitory Neurons part N

First of all, that’s my current setup, nothing fancy.

Second, I have made some progress but it’s uncertain. I don’t really have clear criteria for determining good/bad, am I going in a good direction or is it all bad.

About inhibitory neurons… My model failed in many ways and I concluded that maybe the biological model was not so bad after all. My inhibitory synapse kept being removed from all neurons after a while because with it in place nothing would work.. So neurons do help other neuron escape inhibition 😀 ….. to my astonishment..

I figured that a solution to my problem with neuronal reach could be (possibly should be) the orientation columns seen in biology. That’s where I’m going now. The feed-back issue is on ice for now, still have to solve other problems first.

Multiple computational cell acting together – AI

What I call a computational cell is made up of 4 normal neurons governed by an inhibitory neuron as shown in my previous post. Now I increased the number of computational cells to 4 and everything became a big mess… Some neurons are now part of up to 4 cells… or 2.. or 3.. As synapses are further away from the neuron body, their contribution to the overall potential (activation potential), decreases. So the more synapses on a single neuron the more complexity resulting in a poor understanding of what’s going on and adjusting/optimizing the algorithms 🙁

Still some cell behave more or less predictable, predictable enough to realized that I have an additional problem, which I predicted, but I was hoping it will get somehow solved by the added complexity, something that I could not predict, but I could observe when actually running the full network. It did not happen. Far away patterns cannot connect to each other because I don’t have a fully connected network, neurons bind only in a limited area around their position (in the matrix) .Moreover things that worked on a single computational cell, such as a single neuron responding to all vertical lines, do not work among multiple computational cells. Last problem that I encountered is that now, the network, has become visible slower and I only have 128 neurons running wild… That’s because I’m forming now, tones of dendrites to respond to various patterns or perhaps there is something wrong within my latest modifications some wild loops working for no reasons… doesn’t bode well for the future though…

Overall I’d say I’m stuck with no clear way going forward.

Why am I writing this blog ? Because nobody is reading it 🙂 Well I write because by doing so I’m usually clarifying some thing in my head, sometimes laying out the problem clearly is a big step forward . For the same reasons I also speak to my friends about AI. I’m such a bore, I can tell you that much 🙂

Inhibitory Neurons Working – update

I was hoping for more progress, but I did not put in the work required. Over the holidays I’ve been mostly lazy. But anyway, I did manage to have the inhibitory neurons working. The project is very small and may seem even smaller to the untrained eye 🙂 I worked on a 2 by 2 matrix, with only 2 layers, one layer to receive the pattern the other layer to show the output. Second layer is what I call a compression layer since it’s doing a data reduction more or less like a complex cell would, a single cell responds to similar pattern such as a vertical lines .

Left showing the patterns, right showing the compression layers. Showing only 2 pattern but 3 are possible, the cross one is missing from this picture.

Why is this important at all ? Because otherwise a pattern eventually will spread all over the network in obscure ways. Say you have a pattern for letter A… some layers down the line you may find that pattern stored as an AAAAAAA chain without an inhibitory effect.

How important is this in the bigger picture ? Not nearly enough, but now I can move onto the next big issue, make the network learn based on feed-back… What does that mean ? it means that now I should have a 3rd layer which should associate 2 unrelated patterns as being the same, based on my feed-back.. take the two pattern above, a vertical line and a horizontal line. A third layer should have a single neuron firing whether the vertical line is showing or the horizontal one is… because I say so, you see 😀

When ? God knows 🙂