Why do neurons link inter-layer ?

I’ve spent already too much time on this topic with no real insights. Most of papers I read said something along the lines: they link together to overcame inhibition. But perhaps is not meant to be taken at face value. They can overcame inhibition only by being active, but if they are already active, what do they overcome then ? Moreover, we say they are inhibited, but in fact they are active ?

Assume such a dendritic structure: D(S1, S2), where S1 and S2 are 2 synapses. Assume S1 and S2 are not able to activate dendrite D by themselves and they need and additional S3 for the dendrite to become active and activate generic Neuron N1. That means the neuron providing the synapse S3, lets call it N2 has to fire first. This way may be possible to establish directionality as long as N1 does not link back to N2. But what is “direction”… Hubel and Wiesel experiments did have direction but in real life light is always there.. Could direction be the constant movement of the eye ? Even so, why would direction be important ?

Can’t find a believable reason for having a true inter-layer link. When I say “true inter-layer”, I’m trying to discard the possibility where neurons actually link in a feed-forward like network, but where a biological layer is actually made of many “neuronal” layers..

Another update on my lack of progress :)

I’ve spent most of my time reading about inhibitory neurons. I can’t say I found the holy grail. Too much data to sift through, so still searching. However, I tried also to do some theoretical simulations, went through about 20 models but nothing seemed to make sense. I also created a smaller version of my AI coding with just 3 neurons, two of them excitatory and one inhibitory. They obviously (only in hindsight obvious) can’t do inhibition in all cycles, they do inhibition only in 50% of the cycles, moreover, if I allow excitatory neurons to bind among themselves in a circular fashion (A binds to B and B binds to A) then in some cases they will will keep exciting each other in the absence of an outside input. A is active and will activate B and then B is active and will activate A. In hindsight, also obvious :-)..

But what are the roles an inhibitory neuron can play ?

  • they should adjust the frequency of firing. Here I have to say that I don’t understand why people are still considering firing rates as being meaningful., Firing rates dependent of the intensity of the stimulus, if I shout louder there is an increase in firing rate but I can’t understand things faster. So information at some point has to be slowed down to integrate with the rest of the information. For example if I say a word in which some of the letter are louder then I would lose the meaning of the word because some sounds will arrive and be process faster than the rest of the word. But maybe I’m missing something.
  • they should provide feedback from other area, but feedback can come also from regular neurons.
  • in my model I need them to extinguish excitation propagating through all the network, stopping the circular activation from A to B but I can’t find a reasonable way of doing this.
  • They should also act as a secondary mechanism in data analysis, the first mechanism being the normal neurons. To be more precise, they should be able to prevent the same pattern being learned by more than one neuron, at least this is one of my working hypothesis. How to do this has remained elusive so far… Another way to do this, through topological means, is still in the cards, but I have made not progress on this front either.
  • Another function could be, to force dendrites to make more synapses, at least in some cases, where the inhibitory neuron binds to dendrites and not to the neuron body.

In summary I have no working hypothesis on what the inhibitory neurons do.

A bit of a good news is that I found this course online from MIT, through their MIT OpenCourseWare: Sensory Systems, held by professor Peter H. Schiller. That’s an excellent course, lots of information, well structured and well presented. Many thanks to MIT for allowing free access to such great resources.

AI – No more progress :(

I added in the inter-layer neurons, hoping to solve some problems related to network transmission. Basically, because I don’t use full connections and because the neurons have to find their partners by themselves, I ended up with some non-trivial problems, if neurons are too far away on a 2D surface, they can’t find each other, so I thought I’d use other “inter-layer” neurons, to link them..

Adding inter-layer neurons did not help with long range connections but it did do an interesting thing, now it can reconstruct learned pattern using only parts of the patterns, some sort of a localized Hopfield network. Running a pattern many times eventually excites other patterns from the surrounding neighborhood, and, at the extreme, it excites all the network… all layers.. a total confusion… That’s the reason I need inhibitory neurons, to stop that uncontrolled excitation.

Anyway, this new observation, doesn’t help me at all. Over the weekend I watched some youtubes on neuroscience looking for inspiration .. very disappointing, there is an inflation of scientists trying to feed their families. So I decided to restrict my feed to well established universities, other talks promise much and deliver nothing. Now I’m watching the 12 lectures from Allen Institute: Coding and Vision 101.

So where I am at ? I can detect lines of some orientations, not all, but then I can’t link lines of the same orientations to a single neuron in the next layer because they are far apart and I build the network in the assumption the neurons link to neurons close by, don’t travel the world to find their pair. For long range links I was envisioning the use of inter-layer neurons … Even if everything would work perfectly, I still don’t see the next step, in the brain, simple cells link with multiple complex cells responding to the same type of signal… to what purpose ? How does that help in understanding the shape of an object ? Are they there just for invariance some sort of pooling to account for different positions on a surface, and basically have nothing to do with shape recognition ?

AI- finally, pattern recognition

I have made some serious advances with my AI. Now I have definite pattern learning + pattern discrimination = pattern recognition. No doubts anymore, that this is working, but I also know that this is only the very first 0.1% of the development needed to truly claim a different type of AI – the Hebbian AI..

Here is the most relevant data:

First we have the learning step, before the black line. each pattern will activate a different neuron from the output layer, I only use 4 output neurons here. Then after the black line I “read” the patterns again, expecting that, the same pattern to light up the same output neuron.

I’ve also created a some sort of graphical interface and made a video with the whole process:

I have tested the code with 19 different patterns and 1 of the 19 patterns was misunderstood (letter M is sometimes confused with letter H), you can see it on the video. That was to be expected, at some point, some patterns, are going to be misunderstood. How many patterns can it learn ? All possible patterns within that matrix (25 by 25 pixels), but as more patterns are added more misunderstandings should occur. Why ? because I don’t really have what in current AI is called “feature extraction”… I only have a single feature extracted, the whole pattern. This works with basically one (1) hidden layer and the extraction is made by the output layer…

This is also observed in the fact that as you go with more and more pattern learning, the learning process, becomes more difficult, it take more cycles to be learned. This should not happen with more “feature extraction” along the way… Learning should become faster, as more an more small pattern become common among the bigger patterns.

Any way this is a big achievement and a proof of concept, even if in infancy.

What’s next ? Of course feature extraction… My first target is to have a big line learned and then a smaller line be recognized as the same “line”. Right now, the smaller line pattern is considered an incomplete pattern and blocked from reaching the output layer. How am I planning to go ahead ? Well, I have wondered for a long time, why our brain, has neurons that bind among themselves within the same “layer”. I cannot find information showing differences between neuron that bind to neurons from the same layer and neurons that bind to neurons from a higher layer… They appear to be the same… I don’t know the ratio.. how many bind intra-layer, how many bind to the next layer. Do they have the same learning rate ? Do they form as many/as few dendrites/synapses ? Do they have a shorter axon from the beginning ? are they attracted the same by the Nitric Oxide (retrograde signalling) ? Is mirroring happening only for inter-layer binding neurons or it also happens to inter-layer binding neurons ?

( Mirror neurons are neurons that fire both when an individual performs an action and when the individual sees[14] or hears[15] another perform a similar action from wiki).

So there are many many unknowns, not sure about the time frame I’ll be able to put them all together… I’m hoping to have something working by the end of the year… but who knows, I may decide to take a more brute force approach and use what I already have, multiply the single feature extraction, but that should prove slow, inefficient and inflexible… Anyway, small line/big line – here I come !!!!

What is a Hebbian AI ?

“Cells that fire together wire together.” – see it on wiki

A Hebbian AI is the term I used to describe my AI project, yet, what a Hebbian AI might be is not clearly defined. The mathematical description is extremely limited and is totally different from what I’m using for this AI.

SO what do I mean by Hebbian AI ?

  • it has a learning algorithm based on: ” Cells that fire together wire together “, so it is not using regression for learning, so no back-propagation function. It does not do ANY kind of regressions with all the downsides it brings.
  • The network is made WITHOUT connections. Connections between neurons are made dynamically. There is both creation and destruction of connections.
  • The neurons are using only 0 and 1 as responses, so every type of signal can be correlated and understood, for example an IR sensor can be integrated with an audio signal … They are both treated the same.

Possible advantages over current AIs:

  • It could be way faster for some tasks. That remains to be determined with some sort of benchmarks. So far the code is in Python with zero optimization… no multi thread no GPU use.. nothing. Even so, for what it does at this moment, should still prove faster than whatever sophisticated AI out there.
  • It can immediately integrate all kind of data, visual, audio, pressure, IR, 2-3 video inputs .. everything.
  • It has causality, as far as causality can be determined from observable data, built in.
  • It has temporal awareness, so it matters what is learned first
  • Understanding is probabilistic, sort of a Bayesian learning

Possible dis-advantages over current AIs:

  • It does not do regressions, so it should not be able to do some task, that are possible today with current AI.
  • It is somehow more imprecise (no values between 0 and 1).
  • It can’t be made totally parallel when computing neuronal responses… many processes need to remain serial (so postsynaptic neuron cannot fire before the pre-synaptic neuron ).

I’ll update this article once I understand more :).

AI – Houston we have touchdown

The bad news first. Seem my previous post was some sort of a random luck error. The code did not behave as expected, but I was too exited anyway, and rushed to post it. (not that anyone read this blog..lol). So I went back to the drawing board working only with enough data to have some meaning but otherwise the simplest data possible: [1,0,0,1]. Then after I was convinced everything worked well (enough..lol) I moved onto a bit more complex data.. a 5×5 array with two patterns:

  • a right line /
  • a left line \

Then after even more work, mostly figuring out how my important variables (number of connections, filter size, learning factor and so on) affect the results. I am no where near understanding all the subtle influences are having, but I have some important rules to guide me. Last layer has to have all neurons firing to decide when something was learned, and read data from the discrimination layer (second to last). Then the magic happened.. I was able to repeatedly see the same difference between the two patterns: \, /.

Also, as predicted, I needed to see a learning curve (which was not happening in my previous post, there was no apparent learning, the signal would just drop)

Here’s the data:

Learning two patterns: / and \. The patterns are 2 png files of 5 by 5 pixels each. Left graph showing the discrimination layer, second one (number of cycles vs active neurons in the learning layer (last layer in this configuration)

How confident am I that this is not yet another fluke ? Well, is not 100%, that’s for sure, I need to run more (simple) patterns and then more complex patterns to get to 95% confidence :). I’ve changed my mind too many times and now I’m more skeptical than ever.. even when things do go as predicted.

My set up so far: Input Layer (5 by 5 matrix), 2 Hidden Layers (5 by 5) and an Output Layer (4 by 4), a total of 91 neurons. Output Layer = Learning Layer, Last Hidden layer acts as a discrimination layer.

AI-Update – learning hand written numbers

Progress has been slow for a while, but after a certain development threshold was reached, things have started moving faster. I now have a good release candidate for my neuron code. I also spent a lot of time reading and doing theoretical calculations. While the Neuron can be well described in mathematical terms, the network is too complex to describe in mathematical terms. The code can handle now 2D images, with at least thousands of cycles and at least thousands of neuron into the net. Perhaps more, I have not tried. Speed is not limiting me at the moment, 1000 cycles with 535 neurons took 20 seconds to finish, so I’m delaying any speed improvements for now, I was planing moving parts of code to C but other things are more important for now.

So what is new ? LEARNING hand written numbers from images.

learning hand written numer zero and 3
Learning hand written number zero ( 0 ) and 3

The output layer is a 4 by 4 matrix, but neurons are presented on a single row. All output neurons have a 0 / 1 response. After 150 cycles learning number 0, all neurons have “learned” the number and I was expecting all neurons to light up. They do, on certain configurations, but here one is off. After the initial process, I fed the network more images (different images) of number 0 (zero). I expect common features to be kept and at some point to obtain only a single output neuron in an active state, rest of them to be off. In the image above I’m showing some preliminary results where I get results only for different images of number zero, and I get no result for images of number 3. For training I’m using a very small MNIST data base with hand written numbers. What about CNN ( Convolutional Neural Networks ), pooling and such ? I’m using none of those. The image if fed directly into the hidden layer neurons. I worked for a bit to implement a receptive field algorithm. While I believe that to be a very useful and smart way of reducing complexity, I don’t believe is needed for simple images like this.

What’s Next?

Learning all numbers and letters, at the same time. I have hoped that the current set up will allow for at least 2 numbers learning at the same time, but I had no success so far. Keep in mind that the network has many parameters that need to be optimized, such as number of synapses (connections per neuron), number of hidden layers, neurons per layer, learning factor and so on. Right now I’m using this configuration: Input layer a 25 by 25 matrix with 3 hidden layers (25 by 25) and a 4 by 4 output layer.

Artificial Intelligence -Update- One Neuron in action !

After many trials and many grandiose ideas, I simplified everything to bare minimum. Only a single neuron. Fix that part first. I gave up on sophisticated connections, but those functions are still available, gave up on complex distributions. And here I am… a single neuron + 3 input Neurons. The input neurons do almost nothing, they are there to establish input connections with “real” neurons. And without further due, this is it:

I have a pattern of 3 digits but the pattern can be infinite. The learning part comes from repetition. See the same pattern twice and that is it… the third time the pattern is recognized. I taught the AI 2 patterns [1,1,0] and [0,0,1]. Then I asked it for other 3 patterns: [1,0,0], [1,1,1] and [0,1,0]. I was hoping for a “Pattern Unknown”, all over, but instead I got an unexpected: “Pattern recognized !” for pattern [1,1,1]. So if it’s composed of two know patterns, is still considered known.


I went for an actual neural net, meaning many neurons connected somehow. The full connection (each neuron with each neuron), proved useless, all neurons became symmetric with same output. So next step is to go for an asymmetrical connection between neurons. I’ll first try the asymmetrical_1D and then asymmetrical_2D, and eventually get back to random spherical distribution.

Artificial Intelligence – some ideas

While taking a brake from game development, I though to myself, why not ?? Everybody is doing it ! So I started working on an AI project. Did some literature research for a month or so and then decided what I wanna do. Some sort of a Hebbian type of AI without a correction function (backpropagation).. Well sort of, there is a self correction based on frequency of data and what data is associated with what. Working in Pyhton with the Matplotlib , using, for now, a random spherical distribution for network topology.

Blue -is the input layer, Green dots is the output layer. Trying to learn 3 letters.

Advertising with Google

We’ve started a small advertising campaign with Google Ads for our new game BackSO. For some reasons I always believed this should be something very costly and maybe complex. Well, it was not. In fact was very simple and the prices were actually very good. You can limit the showings to a certain budget/day, and the budget can actually be very small. Overall I’m very happy with how Google is doing its job. Of course the game is doing rather poorly but that is not because of Google :). For for those of you running small gigs don’t be afraid of Google. You can get very fast some users and if you have the Firebase statistics implemented you can get a very large amount of data for the price of a Starbucks coffee. You can the go on on improving your project of maybe decide to start anew.

I also tried a Fiverr small gig, my conclusion is that it is not worth it. The audience is not targeted and get your money wasted for no real benefit. Better to get with the giants, at least you get some real data that can be used to improve your game, before trying out other campaigns.

For Google Ads, start with a global campaign, all over the world, at the smallest price Google can offer per install. Adjust your game and perhaps do a targeted campaign in richer countries, that perhaps require higher prices for installs but may give you better return on your investment.