Another update on my lack of progress :)

I’ve spent most of my time reading about inhibitory neurons. I can’t say I found the holy grail. Too much data to sift through, so still searching. However, I tried also to do some theoretical simulations, went through about 20 models but nothing seemed to make sense. I also created a smaller version of my AI coding with just 3 neurons, two of them excitatory and one inhibitory. They obviously (only in hindsight obvious) can’t do inhibition in all cycles, they do inhibition only in 50% of the cycles, moreover, if I allow excitatory neurons to bind among themselves in a circular fashion (A binds to B and B binds to A) then in some cases they will will keep exciting each other in the absence of an outside input. A is active and will activate B and then B is active and will activate A. In hindsight, also obvious :-)..

But what are the roles an inhibitory neuron can play ?

  • they should adjust the frequency of firing. Here I have to say that I don’t understand why people are still considering firing rates as being meaningful., Firing rates dependent of the intensity of the stimulus, if I shout louder there is an increase in firing rate but I can’t understand things faster. So information at some point has to be slowed down to integrate with the rest of the information. For example if I say a word in which some of the letter are louder then I would lose the meaning of the word because some sounds will arrive and be process faster than the rest of the word. But maybe I’m missing something.
  • they should provide feedback from other area, but feedback can come also from regular neurons.
  • in my model I need them to extinguish excitation propagating through all the network, stopping the circular activation from A to B but I can’t find a reasonable way of doing this.
  • They should also act as a secondary mechanism in data analysis, the first mechanism being the normal neurons. To be more precise, they should be able to prevent the same pattern being learned by more than one neuron, at least this is one of my working hypothesis. How to do this has remained elusive so far… Another way to do this, through topological means, is still in the cards, but I have made not progress on this front either.
  • Another function could be, to force dendrites to make more synapses, at least in some cases, where the inhibitory neuron binds to dendrites and not to the neuron body.

In summary I have no working hypothesis on what the inhibitory neurons do.

A bit of a good news is that I found this course online from MIT, through their MIT OpenCourseWare: Sensory Systems, held by professor Peter H. Schiller. That’s an excellent course, lots of information, well structured and well presented. Many thanks to MIT for allowing free access to such great resources.

AI – No more progress :(

I added in the inter-layer neurons, hoping to solve some problems related to network transmission. Basically, because I don’t use full connections and because the neurons have to find their partners by themselves, I ended up with some non-trivial problems, if neurons are too far away on a 2D surface, they can’t find each other, so I thought I’d use other “inter-layer” neurons, to link them..

Adding inter-layer neurons did not help with long range connections but it did do an interesting thing, now it can reconstruct learned pattern using only parts of the patterns, some sort of a localized Hopfield network. Running a pattern many times eventually excites other patterns from the surrounding neighborhood, and, at the extreme, it excites all the network… all layers.. a total confusion… That’s the reason I need inhibitory neurons, to stop that uncontrolled excitation.

Anyway, this new observation, doesn’t help me at all. Over the weekend I watched some youtubes on neuroscience looking for inspiration .. very disappointing, there is an inflation of scientists trying to feed their families. So I decided to restrict my feed to well established universities, other talks promise much and deliver nothing. Now I’m watching the 12 lectures from Allen Institute: Coding and Vision 101.

So where I am at ? I can detect lines of some orientations, not all, but then I can’t link lines of the same orientations to a single neuron in the next layer because they are far apart and I build the network in the assumption the neurons link to neurons close by, don’t travel the world to find their pair. For long range links I was envisioning the use of inter-layer neurons … Even if everything would work perfectly, I still don’t see the next step, in the brain, simple cells link with multiple complex cells responding to the same type of signal… to what purpose ? How does that help in understanding the shape of an object ? Are they there just for invariance some sort of pooling to account for different positions on a surface, and basically have nothing to do with shape recognition ?

AI- finally, pattern recognition

I have made some serious advances with my AI. Now I have definite pattern learning + pattern discrimination = pattern recognition. No doubts anymore, that this is working, but I also know that this is only the very first 0.1% of the development needed to truly claim a different type of AI – the Hebbian AI..

Here is the most relevant data:

First we have the learning step, before the black line. each pattern will activate a different neuron from the output layer, I only use 4 output neurons here. Then after the black line I “read” the patterns again, expecting that, the same pattern to light up the same output neuron.

I’ve also created a some sort of graphical interface and made a video with the whole process:

I have tested the code with 19 different patterns and 1 of the 19 patterns was misunderstood (letter M is sometimes confused with letter H), you can see it on the video. That was to be expected, at some point, some patterns, are going to be misunderstood. How many patterns can it learn ? All possible patterns within that matrix (25 by 25 pixels), but as more patterns are added more misunderstandings should occur. Why ? because I don’t really have what in current AI is called “feature extraction”… I only have a single feature extracted, the whole pattern. This works with basically one (1) hidden layer and the extraction is made by the output layer…

This is also observed in the fact that as you go with more and more pattern learning, the learning process, becomes more difficult, it take more cycles to be learned. This should not happen with more “feature extraction” along the way… Learning should become faster, as more an more small pattern become common among the bigger patterns.

Any way this is a big achievement and a proof of concept, even if in infancy.

What’s next ? Of course feature extraction… My first target is to have a big line learned and then a smaller line be recognized as the same “line”. Right now, the smaller line pattern is considered an incomplete pattern and blocked from reaching the output layer. How am I planning to go ahead ? Well, I have wondered for a long time, why our brain, has neurons that bind among themselves within the same “layer”. I cannot find information showing differences between neuron that bind to neurons from the same layer and neurons that bind to neurons from a higher layer… They appear to be the same… I don’t know the ratio.. how many bind intra-layer, how many bind to the next layer. Do they have the same learning rate ? Do they form as many/as few dendrites/synapses ? Do they have a shorter axon from the beginning ? are they attracted the same by the Nitric Oxide (retrograde signalling) ? Is mirroring happening only for inter-layer binding neurons or it also happens to inter-layer binding neurons ?

( Mirror neurons are neurons that fire both when an individual performs an action and when the individual sees[14] or hears[15] another perform a similar action from wiki).

So there are many many unknowns, not sure about the time frame I’ll be able to put them all together… I’m hoping to have something working by the end of the year… but who knows, I may decide to take a more brute force approach and use what I already have, multiply the single feature extraction, but that should prove slow, inefficient and inflexible… Anyway, small line/big line – here I come !!!!

What is a Hebbian AI ?

“Cells that fire together wire together.” – see it on wiki

A Hebbian AI is the term I used to describe my AI project, yet, what a Hebbian AI might be is not clearly defined. The mathematical description is extremely limited and is totally different from what I’m using for this AI.

SO what do I mean by Hebbian AI ?

  • it has a learning algorithm based on: ” Cells that fire together wire together “, so it is not using regression for learning, so no back-propagation function. It does not do ANY kind of regressions with all the downsides it brings.
  • The network is made WITHOUT connections. Connections between neurons are made dynamically. There is both creation and destruction of connections.
  • The neurons are using only 0 and 1 as responses, so every type of signal can be correlated and understood, for example an IR sensor can be integrated with an audio signal … They are both treated the same.

Possible advantages over current AIs:

  • It could be way faster for some tasks. That remains to be determined with some sort of benchmarks. So far the code is in Python with zero optimization… no multi thread no GPU use.. nothing. Even so, for what it does at this moment, should still prove faster than whatever sophisticated AI out there.
  • It can immediately integrate all kind of data, visual, audio, pressure, IR, 2-3 video inputs .. everything.
  • It has causality, as far as causality can be determined from observable data, built in.
  • It has temporal awareness, so it matters what is learned first
  • Understanding is probabilistic, sort of a Bayesian learning

Possible dis-advantages over current AIs:

  • It does not do regressions, so it should not be able to do some task, that are possible today with current AI.
  • It is somehow more imprecise (no values between 0 and 1).
  • It can’t be made totally parallel when computing neuronal responses… many processes need to remain serial (so postsynaptic neuron cannot fire before the pre-synaptic neuron ).

I’ll update this article once I understand more :).

AI – Houston we have touchdown

The bad news first. Seem my previous post was some sort of a random luck error. The code did not behave as expected, but I was too exited anyway, and rushed to post it. (not that anyone read this blog..lol). So I went back to the drawing board working only with enough data to have some meaning but otherwise the simplest data possible: [1,0,0,1]. Then after I was convinced everything worked well (enough..lol) I moved onto a bit more complex data.. a 5×5 array with two patterns:

  • a right line /
  • a left line \

Then after even more work, mostly figuring out how my important variables (number of connections, filter size, learning factor and so on) affect the results. I am no where near understanding all the subtle influences are having, but I have some important rules to guide me. Last layer has to have all neurons firing to decide when something was learned, and read data from the discrimination layer (second to last). Then the magic happened.. I was able to repeatedly see the same difference between the two patterns: \, /.

Also, as predicted, I needed to see a learning curve (which was not happening in my previous post, there was no apparent learning, the signal would just drop)

Here’s the data:

Learning two patterns: / and \. The patterns are 2 png files of 5 by 5 pixels each. Left graph showing the discrimination layer, second one (number of cycles vs active neurons in the learning layer (last layer in this configuration)

How confident am I that this is not yet another fluke ? Well, is not 100%, that’s for sure, I need to run more (simple) patterns and then more complex patterns to get to 95% confidence :). I’ve changed my mind too many times and now I’m more skeptical than ever.. even when things do go as predicted.

My set up so far: Input Layer (5 by 5 matrix), 2 Hidden Layers (5 by 5) and an Output Layer (4 by 4), a total of 91 neurons. Output Layer = Learning Layer, Last Hidden layer acts as a discrimination layer.

AI-Update – learning hand written numbers

Progress has been slow for a while, but after a certain development threshold was reached, things have started moving faster. I now have a good release candidate for my neuron code. I also spent a lot of time reading and doing theoretical calculations. While the Neuron can be well described in mathematical terms, the network is too complex to describe in mathematical terms. The code can handle now 2D images, with at least thousands of cycles and at least thousands of neuron into the net. Perhaps more, I have not tried. Speed is not limiting me at the moment, 1000 cycles with 535 neurons took 20 seconds to finish, so I’m delaying any speed improvements for now, I was planing moving parts of code to C but other things are more important for now.

So what is new ? LEARNING hand written numbers from images.

learning hand written numer zero and 3
Learning hand written number zero ( 0 ) and 3

The output layer is a 4 by 4 matrix, but neurons are presented on a single row. All output neurons have a 0 / 1 response. After 150 cycles learning number 0, all neurons have “learned” the number and I was expecting all neurons to light up. They do, on certain configurations, but here one is off. After the initial process, I fed the network more images (different images) of number 0 (zero). I expect common features to be kept and at some point to obtain only a single output neuron in an active state, rest of them to be off. In the image above I’m showing some preliminary results where I get results only for different images of number zero, and I get no result for images of number 3. For training I’m using a very small MNIST data base with hand written numbers. What about CNN ( Convolutional Neural Networks ), pooling and such ? I’m using none of those. The image if fed directly into the hidden layer neurons. I worked for a bit to implement a receptive field algorithm. While I believe that to be a very useful and smart way of reducing complexity, I don’t believe is needed for simple images like this.

What’s Next?

Learning all numbers and letters, at the same time. I have hoped that the current set up will allow for at least 2 numbers learning at the same time, but I had no success so far. Keep in mind that the network has many parameters that need to be optimized, such as number of synapses (connections per neuron), number of hidden layers, neurons per layer, learning factor and so on. Right now I’m using this configuration: Input layer a 25 by 25 matrix with 3 hidden layers (25 by 25) and a 4 by 4 output layer.