Neuromorphic chips and spiking neurons

"Computational neuroscientists think that spiking neural networks will bring them closer to understanding the brain than is possible with current deep neural networks."

Read our newsletter feature Spiking Neural Networks, from @anilananth.https://t.co/SQGfedesib pic.twitter.com/K4YO4kViEN

— Simons Institute for the Theory of Computing (@SimonsInstitute) December 18, 2021

From the linked article, "Spiking Neural Networks":

Artificial intelligence researchers, on the other hand, would like to build deep neural networks that have both the brain’s remarkable abilities and its extraordinary energy efficiency. The brain consumes only about 20 watts of power. If the brain achieves its ends partly because of spiking neurons, some think that energy-efficient deep artificial neural networks (ANNs) would also need to follow suit.

But spiking neural networks have been hamstrung. The very thing that made them attractive — communicating via spikes — also made them extremely difficult to train. The algorithms that ran on IBM’s chip, for instance, had to be trained offline at considerable computational cost.

That’s set to change. Researchers have developed new algorithms to train spiking neural networks that overcome some of the earlier limitations. And at least for networks of tens of thousands of neurons, these SNNs perform as well as regular ANNs. Such networks would likely be better at processing data that has a temporal dimension (such as speech and videos) compared with standard ANNs. Also, when implemented on neuromorphic chips, SNNs promise to open up a new era of low-energy, always-on devices that can operate without access to services in the cloud.

One of the key differences between a standard ANN and a spiking neural network is the model of the neuron itself. Any artificial neuron is a simplified computational model of biological neurons. A biological neuron receives inputs into the cell body via its dendrites; and based on some internal computation, the neuron may generate an output in the form of a spike on its axon, which then serves as an input to other neurons. Standard ANNs use a model of the neuron in which the information is encoded in the firing rate of the neuron. So the function that transforms the inputs into an output is often a continuous valued function that represents the spiking rate. This is achieved by first taking a weighted sum of all the inputs and then passing the sum through an activation function. For example, a sigmoid turns the weighted sum into a real value between 0 and 1.

In a spiking artificial neuron, on the other hand, the information is encoded in both the timing of the output spike and the spiking rate. The most commonly used model of such a spiking neuron in artificial neural networks is called the leaky integrate-and-fire (LIF) neuron. Input spikes cause the neuron’s membrane potential — the electrical charge across the neuron’s cell wall — to build up. There are also processes that cause this charge to leak; in the absence of input spikes, any built-up membrane potential starts to decay. But if enough input spikes come within a certain time window, then the membrane potential crosses a threshold, at which point the neuron fires an output spike. The membrane potential resets to its base value. Variations on this theme of an LIF neuron form the basic computational units of spiking neural networks.

In 1997, Wolfgang Maass of the Institute of Theoretical Computer Science, Technische Universität, Graz, Austria, showed that such SNNs are computationally more powerful, in terms of the number of neurons needed for some task, than ANNs with rate-coding neurons that use a sigmoid activation function. He also showed that SNNs and ANNs are equivalent in their ability to compute some function (an important equivalence, since an ANN’s claim to fame is that it is a universal function approximator: given some input, an ANN can be trained to approximate any function to transform the input into a desired output).

There's much more at the link, much of it having to do with how to modify backpropagation so it works and can scale with SNNs.

Conclusion:

All this bodes well for the day when spiking neural networks can be implemented on the numerous neuromorphic chips that are in development. The hope is that such networks can be both trained and deployed using dedicated hardware that sips rather than sucks energy.