Entropic Neuronal Summation

Context

These notes are part of the Assothink project.

Natural intelligences, like the human brain, and artificial intelligences, like Assothink, involve numerous nodes exchanging signals.

Focusing on one node, it is assumed that this node receives a finite (discrete) number of input signals (inflows). The input signals may be considered and described as positive real values.

These signals determine the local excitation level of the node.

The question is purely mathematical: how do the various inflow units combine into a global inflow value ?

[ Note: les notations mathématiques utilisées ici sont construites à partir de cette aide ] 

Principle

The first answer to the question above is 'summation'. Neuroscience document do not discuss this point (as far as we know), but it is generally assumed that summation effects occur. It would be hard to prove that the simple summation effect is the correct model for the combination of inflows, but hard also to prove that it is not the correct model. So we can just make assumptions.

In this document, another assumption is built: the ENS (Entropic Neuronal Summation).

Obviously the model is quite similar to a simple summation model, but there is one point attracting our attention. The inflow coming from two different sources operates more than the inflow coming from one single source, event when the simple summation of the values is identical. There is no demonstration for this. It is based on introspective considerations.

But anyway, why would it be worse than the simple summation model ?

The critical point mentioned above, the difference between the summation model and the ENS model is that we want for the ENS a function $\Psi ()$ , such that, when $a>0$

$\Psi (a/2,a/2)>\Psi (a)$

In other words, the $\Psi ()$ function has (besides some properties common with the simple summation model) some specific properties.

Properties

The $\psi ()$ function shares these properties with the simple summation model:

$\psi (...)\geq 0$ $\psi ()=0$ $\psi (0)=0$ $\psi (...,0)=\psi (...)$ $\psi (...,a,b,...)=\psi (...,b,a,...)$ $\psi (...,z)>\psi (...)\ \ \ \ (z>0)$

We also want some additional properties:

$\Psi (a/2,a/2)>\Psi (a)$ $\Psi (a,a)>\Psi (x,2a-x)\ \ \;(0<x<a)$

ENS model

Let us first define various writings. The arguments of the functions (all $a_{i}$ positive) are written

$\Psi (a_{i})\ \ or\ \ \Psi (a_{0},a_{1},a_{2},...)$

The sum of the arguments (the simple summation value) is

$\Sigma =\Sigma (a_{0},a_{1},a_{2},...)=\Sigma _{i}(a_{i})$

The relative weight of the arguments are

$p_{i}=a_{i}/\Sigma$

And we also define

$q_{i}=\Sigma /a_{i}=p_{i}^{-1}$

In this document all logarithms use base 2

$log(x)=log_{2}(x)$

As a convenient short writing, we also use

$flog(x)=x\ log(x)$

Let $\mathrm {E}$ be the entropic uncertainty measure (see explanations there) of the $p_{i}$ values

$\mathrm {E} (a_{i})=-\Sigma (p_{i}\ log(p_{i}))=-\Sigma (flog(p_{i}))$

Now we define our ENS function:

$\Psi (a_{i})=\Sigma (a_{i})\ (\kappa \ +\ (1-\kappa )\ \mathrm {E} (a_{i}))$

where the $\kappa$ value is a null or positive real parameter value.

Developments

Some simple mathematical steps allow to transform the $\Psi$ formulation. They lead to:

$\Psi =\kappa \Sigma \ +(1-\kappa )(flog(\Sigma )\ -\ \Sigma _{i}(flog(a_{i})))$

When the $\kappa$ parameters is 1.0 the ENS function falls back to the simple summation model, losing the specific properties specified above.

When the $\kappa$ parameter is 0.0, it turns into a strictly entropic summation, involving for instance $\Psi (x)=0$ (null when only 1 argument is provided!).

Realistic values used for the Assothink developments should reside around $\kappa$ = 0.5.

Computation

The ENS formula has a practical convenient property. It may be computed incrementally. In other words, it is not necessary to memorize all the input values before computing the ENS result. It is sufficient to accumulate the sum of the $a_{i}$ values and the sum of the $flog(a_{i})$ values.When all input is terminated, the accumulated values are used to produce the ENS result.

On the other hand, any flow occurring in the Assothink engine requires a logarithm computation (accurate or not), and this consumes a lot of CPU (given flow event more than once per nanosec on usual computers).

So the trade-off between the CPU cost and the improved targeting of the flows has to be decided according to practical tests.

Fast x.log(x) computation

It is possible, with a low accuracy, to quickly compute

$flog(x)=x\ log(x)$

by using the following approximation (Taylor development around 1.0, see wolfram)

$flog(x)\asymp$

Results example

Here are some instructive results computed with $\kappa$=0.5:

args=(2.0,2.0)                SUM=4.00 ENS=4.00
args=(3.0,3.0)                SUM=6.00 ENS=6.00
args=(2.0,2.0,2.0)            SUM=6.00 ENS=7.75
args=(4.0,4.0)                SUM=8.00 ENS=8.00
args=(4.0)                    SUM=4.00 ENS=2.00
args=(3.0,1.0)                SUM=4.00 ENS=3.62
args=(3.9,0.1)                SUM=4.00 ENS=2.34

All expected properties are respected (which was obvious from the formulae).

Conclusion

As part of the Assothink model, it is assumed (believed?) that the ENS function is a useful model for the computation of excitation flows between nodes of the active jelly.

Its is however not proved.

Anyway it is tested as part of the Assothink software implementation from April 2013.