TECHNOLOGIES
I don’t have a coherent theory about training.. Sure I could get a single letter to work, but the second one is not certain, unless I understand what I’m looking for..
I tried already hundreds of patterns and I’m no closer to understanding what would be the best output. I tried small patterns, big patterns… nothing really helped..
After many trials and errors I’m starting to see the light… First signs of invariance, and how that might work, are here..
It’s a very small matrix with only 2 layers (in fact there are 8 layers if I count input/output and inhibitory layers), 4×4. If pattern is moved on the right side, then another set of inhibitory neurons control the output, and the response is very different, like having an entirely different pattern. I was theorizing that this is how it should behave, but I was never very sure. Also I observed clustering that I have never predicted, but in hindsight I should have. They were messing my beautiful patterns.. so I inhibited the secondary dendrites as well… all possible dendrites are now controlled by inhibitory neurons , still I see asymmetries that are troublesome.
Can this be coincidental ? Yes and no, yes because the patterns are not perfectly as predicted, no, because is close to what I was predicting, and prediction show that it should work.
Anyway some things are clear… This invariance, can only work on limited fields. Theoretically the size of the field is directly proportional to the number of hidden layers (the bigger the field, the more hidden layers). And this is where I’m going next… But there still work to be done here on this tiny set-up.
First of all, that’s my current setup, nothing fancy.
Second, I have made some progress but it’s uncertain. I don’t really have clear criteria for determining good/bad, am I going in a good direction or is it all bad.
About inhibitory neurons… My model failed in many ways and I concluded that maybe the biological model was not so bad after all. My inhibitory synapse kept being removed from all neurons after a while because with it in place nothing would work.. So neurons do help other neuron escape inhibition 😀 ….. to my astonishment..
I figured that a solution to my problem with neuronal reach could be (possibly should be) the orientation columns seen in biology. That’s where I’m going now. The feed-back issue is on ice for now, still have to solve other problems first.
What I call a computational cell is made up of 4 normal neurons governed by an inhibitory neuron as shown in my previous post. Now I increased the number of computational cells to 4 and everything became a big mess… Some neurons are now part of up to 4 cells… or 2.. or 3.. As synapses are further away from the neuron body, their contribution to the overall potential (activation potential), decreases. So the more synapses on a single neuron the more complexity resulting in a poor understanding of what’s going on and adjusting/optimizing the algorithms 🙁
Still some cell behave more or less predictable, predictable enough to realized that I have an additional problem, which I predicted, but I was hoping it will get somehow solved by the added complexity, something that I could not predict, but I could observe when actually running the full network. It did not happen. Far away patterns cannot connect to each other because I don’t have a fully connected network, neurons bind only in a limited area around their position (in the matrix) .Moreover things that worked on a single computational cell, such as a single neuron responding to all vertical lines, do not work among multiple computational cells. Last problem that I encountered is that now, the network, has become visible slower and I only have 128 neurons running wild… That’s because I’m forming now, tones of dendrites to respond to various patterns or perhaps there is something wrong within my latest modifications some wild loops working for no reasons… doesn’t bode well for the future though…
Overall I’d say I’m stuck with no clear way going forward.
Why am I writing this blog ? Because nobody is reading it 🙂 Well I write because by doing so I’m usually clarifying some thing in my head, sometimes laying out the problem clearly is a big step forward . For the same reasons I also speak to my friends about AI. I’m such a bore, I can tell you that much 🙂
I was hoping for more progress, but I did not put in the work required. Over the holidays I’ve been mostly lazy. But anyway, I did manage to have the inhibitory neurons working. The project is very small and may seem even smaller to the untrained eye 🙂 I worked on a 2 by 2 matrix, with only 2 layers, one layer to receive the pattern the other layer to show the output. Second layer is what I call a compression layer since it’s doing a data reduction more or less like a complex cell would, a single cell responds to similar pattern such as a vertical lines .
Why is this important at all ? Because otherwise a pattern eventually will spread all over the network in obscure ways. Say you have a pattern for letter A… some layers down the line you may find that pattern stored as an AAAAAAA chain without an inhibitory effect.
How important is this in the bigger picture ? Not nearly enough, but now I can move onto the next big issue, make the network learn based on feed-back… What does that mean ? it means that now I should have a 3rd layer which should associate 2 unrelated patterns as being the same, based on my feed-back.. take the two pattern above, a vertical line and a horizontal line. A third layer should have a single neuron firing whether the vertical line is showing or the horizontal one is… because I say so, you see 😀
When ? God knows 🙂
From my many simulations I concluded that inhibitory neurons (why in the world are they called Interneurons?) have to be there to stop information replicating endlessly on all neurons. So each inhibitory neuron will prevent the same information to be replicated in the cluster of neurons governed by a particular inhibitory neuron.
They also determine the level of discrimination, the smallest information that can be classified will depend on the number of inhibitory neurons. The more inhibitory neurons the smaller the information that can be classified, more discrimination. If one would want to have a pixel wise discrimination the number of inhibitory neurons should equal the number of pixels.
There is an additional role, to regulate frequency, but I cannot test this one since my frequency is fixed and because of that I don’t need this role added to my code.
On the update front… well, I learn every day something new, mostly by discarding ideas that don’t fit the general algorithms. I’ve reached a point where I need to code all the new ideas and see actual results. I cannot go further till I get a better understanding of the impact of the new ideas.
I’ve settled some concept that were somehow in limbo, what is the relation between the information contained in two different dendrites of the same neuron, or how many layers should my network contain, to accomplish a certain task.
But some older problems came back into focus… what is the reach of a single neuron ? Answering this question is not really a must for the layers doing data classification but it gets important for what I call decisions layers … Decision layers can be viewed as output layers for simple networks, but in my model they can be used to sandwich classification layers , that means I can have more than one in a network. For the next 2 weeks I’ll be working only on programing the new concepts.. see what I get. What I should get is identifying letter A at different positions and sizes as being letter A, and not a different pattern, on a 252 x 252 matrix…
I’ve spent already too much time on this topic with no real insights. Most of papers I read said something along the lines: they link together to overcame inhibition. But perhaps is not meant to be taken at face value. They can overcame inhibition only by being active, but if they are already active, what do they overcome then ? Moreover, we say they are inhibited, but in fact they are active ?
Assume such a dendritic structure: D(S1, S2), where S1 and S2 are 2 synapses. Assume S1 and S2 are not able to activate dendrite D by themselves and they need and additional S3 for the dendrite to become active and activate generic Neuron N1. That means the neuron providing the synapse S3, lets call it N2 has to fire first. This way may be possible to establish directionality as long as N1 does not link back to N2. But what is “direction”… Hubel and Wiesel experiments did have direction but in real life light is always there.. Could direction be the constant movement of the eye ? Even so, why would direction be important ?
Can’t find a believable reason for having a true inter-layer link. When I say “true inter-layer”, I’m trying to discard the possibility where neurons actually link in a feed-forward like network, but where a biological layer is actually made of many “neuronal” layers..
I’ve spent most of my time reading about inhibitory neurons. I can’t say I found the holy grail. Too much data to sift through, so still searching. However, I tried also to do some theoretical simulations, went through about 20 models but nothing seemed to make sense. I also created a smaller version of my AI coding with just 3 neurons, two of them excitatory and one inhibitory. They obviously (only in hindsight obvious) can’t do inhibition in all cycles, they do inhibition only in 50% of the cycles, moreover, if I allow excitatory neurons to bind among themselves in a circular fashion (A binds to B and B binds to A) then in some cases they will will keep exciting each other in the absence of an outside input. A is active and will activate B and then B is active and will activate A. In hindsight, also obvious :-)..
But what are the roles an inhibitory neuron can play ?
In summary I have no working hypothesis on what the inhibitory neurons do.
A bit of a good news is that I found this course online from MIT, through their MIT OpenCourseWare: Sensory Systems, held by professor Peter H. Schiller. That’s an excellent course, lots of information, well structured and well presented. Many thanks to MIT for allowing free access to such great resources.
I added in the inter-layer neurons, hoping to solve some problems related to network transmission. Basically, because I don’t use full connections and because the neurons have to find their partners by themselves, I ended up with some non-trivial problems, if neurons are too far away on a 2D surface, they can’t find each other, so I thought I’d use other “inter-layer” neurons, to link them..
Adding inter-layer neurons did not help with long range connections but it did do an interesting thing, now it can reconstruct learned pattern using only parts of the patterns, some sort of a localized Hopfield network. Running a pattern many times eventually excites other patterns from the surrounding neighborhood, and, at the extreme, it excites all the network… all layers.. a total confusion… That’s the reason I need inhibitory neurons, to stop that uncontrolled excitation.
Anyway, this new observation, doesn’t help me at all. Over the weekend I watched some youtubes on neuroscience looking for inspiration .. very disappointing, there is an inflation of scientists trying to feed their families. So I decided to restrict my feed to well established universities, other talks promise much and deliver nothing. Now I’m watching the 12 lectures from Allen Institute: Coding and Vision 101.
So where I am at ? I can detect lines of some orientations, not all, but then I can’t link lines of the same orientations to a single neuron in the next layer because they are far apart and I build the network in the assumption the neurons link to neurons close by, don’t travel the world to find their pair. For long range links I was envisioning the use of inter-layer neurons … Even if everything would work perfectly, I still don’t see the next step, in the brain, simple cells link with multiple complex cells responding to the same type of signal… to what purpose ? How does that help in understanding the shape of an object ? Are they there just for invariance some sort of pooling to account for different positions on a surface, and basically have nothing to do with shape recognition ?
I have made some serious advances with my AI. Now I have definite pattern learning + pattern discrimination = pattern recognition. No doubts anymore, that this is working, but I also know that this is only the very first 0.1% of the development needed to truly claim a different type of AI – the Hebbian AI..
Here is the most relevant data:
I’ve also created a some sort of graphical interface and made a video with the whole process:
I have tested the code with 19 different patterns and 1 of the 19 patterns was misunderstood (letter M is sometimes confused with letter H), you can see it on the video. That was to be expected, at some point, some patterns, are going to be misunderstood. How many patterns can it learn ? All possible patterns within that matrix (25 by 25 pixels), but as more patterns are added more misunderstandings should occur. Why ? because I don’t really have what in current AI is called “feature extraction”… I only have a single feature extracted, the whole pattern. This works with basically one (1) hidden layer and the extraction is made by the output layer…
This is also observed in the fact that as you go with more and more pattern learning, the learning process, becomes more difficult, it take more cycles to be learned. This should not happen with more “feature extraction” along the way… Learning should become faster, as more an more small pattern become common among the bigger patterns.
Any way this is a big achievement and a proof of concept, even if in infancy.
What’s next ? Of course feature extraction… My first target is to have a big line learned and then a smaller line be recognized as the same “line”. Right now, the smaller line pattern is considered an incomplete pattern and blocked from reaching the output layer. How am I planning to go ahead ? Well, I have wondered for a long time, why our brain, has neurons that bind among themselves within the same “layer”. I cannot find information showing differences between neuron that bind to neurons from the same layer and neurons that bind to neurons from a higher layer… They appear to be the same… I don’t know the ratio.. how many bind intra-layer, how many bind to the next layer. Do they have the same learning rate ? Do they form as many/as few dendrites/synapses ? Do they have a shorter axon from the beginning ? are they attracted the same by the Nitric Oxide (retrograde signalling) ? Is mirroring happening only for inter-layer binding neurons or it also happens to inter-layer binding neurons ?
( Mirror neurons are neurons that fire both when an individual performs an action and when the individual sees[14] or hears[15] another perform a similar action from wiki).
So there are many many unknowns, not sure about the time frame I’ll be able to put them all together… I’m hoping to have something working by the end of the year… but who knows, I may decide to take a more brute force approach and use what I already have, multiply the single feature extraction, but that should prove slow, inefficient and inflexible… Anyway, small line/big line – here I come !!!!