XLANDC

I have made some serious advances with my AI. Now I have definite pattern learning + pattern discrimination = pattern recognition. No doubts anymore, that this is working, but I also know that this is only the very first 0.1% of the development needed to truly claim a different type of AI – the Hebbian AI..

Here is the most relevant data:

I’ve also created a some sort of graphical interface and made a video with the whole process:

I have tested the code with 19 different patterns and 1 of the 19 patterns was misunderstood (letter M is sometimes confused with letter H), you can see it on the video. That was to be expected, at some point, some patterns, are going to be misunderstood. How many patterns can it learn ? All possible patterns within that matrix (25 by 25 pixels), but as more patterns are added more misunderstandings should occur. Why ? because I don’t really have what in current AI is called “feature extraction”… I only have a single feature extracted, the whole pattern. This works with basically one (1) hidden layer and the extraction is made by the output layer…

This is also observed in the fact that as you go with more and more pattern learning, the learning process, becomes more difficult, it take more cycles to be learned. This should not happen with more “feature extraction” along the way… Learning should become faster, as more an more small pattern become common among the bigger patterns.

Any way this is a big achievement and a proof of concept, even if in infancy.

What’s next ? Of course feature extraction… My first target is to have a big line learned and then a smaller line be recognized as the same “line”. Right now, the smaller line pattern is considered an incomplete pattern and blocked from reaching the output layer. How am I planning to go ahead ? Well, I have wondered for a long time, why our brain, has neurons that bind among themselves within the same “layer”. I cannot find information showing differences between neuron that bind to neurons from the same layer and neurons that bind to neurons from a higher layer… They appear to be the same… I don’t know the ratio.. how many bind intra-layer, how many bind to the next layer. Do they have the same learning rate ? Do they form as many/as few dendrites/synapses ? Do they have a shorter axon from the beginning ? are they attracted the same by the Nitric Oxide (retrograde signalling) ? Is mirroring happening only for inter-layer binding neurons or it also happens to inter-layer binding neurons ?

( Mirror neurons are neurons that fire both when an individual performs an action and when the individual sees^[14] or hears^[15] another perform a similar action from wiki).

So there are many many unknowns, not sure about the time frame I’ll be able to put them all together… I’m hoping to have something working by the end of the year… but who knows, I may decide to take a more brute force approach and use what I already have, multiply the single feature extraction, but that should prove slow, inefficient and inflexible… Anyway, small line/big line – here I come !!!!

“Cells that fire together wire together.” – see it on wiki

A Hebbian AI is the term I used to describe my AI project, yet, what a Hebbian AI might be is not clearly defined. The mathematical description is extremely limited and is totally different from what I’m using for this AI.

SO what do I mean by Hebbian AI ?

• it has a learning algorithm based on: ” Cells that fire together wire together “, so it is not using regression for learning, so no back-propagation function. It does not do ANY kind of regressions with all the downsides it brings.
• The network is made WITHOUT connections. Connections between neurons are made dynamically. There is both creation and destruction of connections.
• The neurons are using only 0 and 1 as responses, so every type of signal can be correlated and understood, for example an IR sensor can be integrated with an audio signal … They are both treated the same.

Possible advantages over current AIs:

• It could be way faster for some tasks. That remains to be determined with some sort of benchmarks. So far the code is in Python with zero optimization… no multi thread no GPU use.. nothing. Even so, for what it does at this moment, should still prove faster than whatever sophisticated AI out there.
• It can immediately integrate all kind of data, visual, audio, pressure, IR, 2-3 video inputs .. everything.
• It has causality, as far as causality can be determined from observable data, built in.
• It has temporal awareness, so it matters what is learned first
• Understanding is probabilistic, sort of a Bayesian learning

Possible dis-advantages over current AIs:

• It does not do regressions, so it should not be able to do some task, that are possible today with current AI.
• It is somehow more imprecise (no values between 0 and 1).
• It can’t be made totally parallel when computing neuronal responses… many processes need to remain serial (so postsynaptic neuron cannot fire before the pre-synaptic neuron ).

I’ll update this article once I understand more :).

The bad news first. Seem my previous post was some sort of a random luck error. The code did not behave as expected, but I was too exited anyway, and rushed to post it. (not that anyone read this blog..lol). So I went back to the drawing board working only with enough data to have some meaning but otherwise the simplest data possible: [1,0,0,1]. Then after I was convinced everything worked well (enough..lol) I moved onto a bit more complex data.. a 5×5 array with two patterns:

• a right line /
• a left line \

Then after even more work, mostly figuring out how my important variables (number of connections, filter size, learning factor and so on) affect the results. I am no where near understanding all the subtle influences are having, but I have some important rules to guide me. Last layer has to have all neurons firing to decide when something was learned, and read data from the discrimination layer (second to last). Then the magic happened.. I was able to repeatedly see the same difference between the two patterns: \, /.

Also, as predicted, I needed to see a learning curve (which was not happening in my previous post, there was no apparent learning, the signal would just drop)

Here’s the data:

How confident am I that this is not yet another fluke ? Well, is not 100%, that’s for sure, I need to run more (simple) patterns and then more complex patterns to get to 95% confidence :). I’ve changed my mind too many times and now I’m more skeptical than ever.. even when things do go as predicted.

My set up so far: Input Layer (5 by 5 matrix), 2 Hidden Layers (5 by 5) and an Output Layer (4 by 4), a total of 91 neurons. Output Layer = Learning Layer, Last Hidden layer acts as a discrimination layer.

Progress has been slow for a while, but after a certain development threshold was reached, things have started moving faster. I now have a good release candidate for my neuron code. I also spent a lot of time reading and doing theoretical calculations. While the Neuron can be well described in mathematical terms, the network is too complex to describe in mathematical terms. The code can handle now 2D images, with at least thousands of cycles and at least thousands of neuron into the net. Perhaps more, I have not tried. Speed is not limiting me at the moment, 1000 cycles with 535 neurons took 20 seconds to finish, so I’m delaying any speed improvements for now, I was planing moving parts of code to C but other things are more important for now.

So what is new ? LEARNING hand written numbers from images.

The output layer is a 4 by 4 matrix, but neurons are presented on a single row. All output neurons have a 0 / 1 response. After 150 cycles learning number 0, all neurons have “learned” the number and I was expecting all neurons to light up. They do, on certain configurations, but here one is off. After the initial process, I fed the network more images (different images) of number 0 (zero). I expect common features to be kept and at some point to obtain only a single output neuron in an active state, rest of them to be off. In the image above I’m showing some preliminary results where I get results only for different images of number zero, and I get no result for images of number 3. For training I’m using a very small MNIST data base with hand written numbers. What about CNN ( Convolutional Neural Networks ), pooling and such ? I’m using none of those. The image if fed directly into the hidden layer neurons. I worked for a bit to implement a receptive field algorithm. While I believe that to be a very useful and smart way of reducing complexity, I don’t believe is needed for simple images like this.

What’s Next?

Learning all numbers and letters, at the same time. I have hoped that the current set up will allow for at least 2 numbers learning at the same time, but I had no success so far. Keep in mind that the network has many parameters that need to be optimized, such as number of synapses (connections per neuron), number of hidden layers, neurons per layer, learning factor and so on. Right now I’m using this configuration: Input layer a 25 by 25 matrix with 3 hidden layers (25 by 25) and a 4 by 4 output layer.