[Comp-neuro] Review announcement
rinkus at comcast.net
Thu Jul 24 06:33:59 CEST 2008
Excellent post, Dr. Minai,
You may be interested in a poster that I just gave at AREADNE, which
describes a new theory of how noise is used, as you suggest, to inject
variety into the assignment of sparse distributed codes.
Suppose you have a sparse distributed coding layer where N out of M cells
are chosen for any representation.
Suppose you have a mechanism for computing the familiarity, G (normalized
between 0 and 1), of an input, X, before assigning a code to it.
Finally, you have a mechanism that adds an amount of noise, inversely
proportional to G (i.e., directly proportional to novelty) into the code
For completely familiar inputs (G=1), the mechanism adds no noise, allowing
the synaptic inputs to cells to be the dominant influence in choosing which
cells become active, resulting in code completion (recognition).
For completely novel inputs (G=0), the mechanism add so much noise that it
swamps out the effects of synaptic inputs, making all cells equally likely
to be chosen (i.e., N cells chosen from the uniform distribution). In this
case, the expected intersection between the newly chosen code and any
previously assigned code will be at the level of chance, resulting in code
separation (learning). More generally, this mechanism varies the amount of
noise added into the winner selection process from zero when G=1 to high
enough to swamp out the synaptic inputs when G=0.
Overall, this mechanism ensures that the more similar the inputs are, the
more similar (i.e., more overlapped) their codes are.
The concept can be seen in more detail in my poster available at top of the
publications page http://people.brandeis.edu/~grinkus/Publications
of my Web site http://people.brandeis.edu/~grinkus/
Gerard Rinkus, PhD
Visiting Scientist, Lisman Lab
Volen Center for Complex Systems
Brandeis University, Waltham, MA
From: comp-neuro-bounces at neuroinf.org
[mailto:comp-neuro-bounces at neuroinf.org] On Behalf Of Ali Minai
Sent: Tuesday, July 22, 2008 1:42 PM
To: comp-neuro at neuroinf.org; Etienne B. Roesch
Subject: Re: [Comp-neuro] Review announcement
Noise does this and much, much more. It can inject variety, break symmetry,
generate novelty, provide energy, facilitate search, carry signal, and do
many other things. Indeed, the only time noise is really a problem is when
one is trying to do achieve a pre-determined goal (e.g., following a
pre-computed trajectory). Since natural systems - notably the nervous system
- rarely (if ever) try to do this, they thrive on noise. Perhaps we should
give the phenomenon a less pejorative name. "Noise" signals such a linear
Ali A. Minai
Associate Head for Electrical Engineering
Department of Electrical & Computer Engineering
University of Cincinnati
Cincinnati, OH 45221-0030
Phone: (513) 556-4783
Fax: (513) 556-7326
Email: aminai at ececs.uc.edu
minai_ali at yahoo.com
--- On Tue, 7/22/08, Etienne B. Roesch <Etienne.Roesch at pse.unige.ch> wrote:
From: Etienne B. Roesch <Etienne.Roesch at pse.unige.ch>
Subject: Re: [Comp-neuro] Review announcement
To: comp-neuro at neuroinf.org
Cc: comp-neuro-bounces at neuroinf.org
Date: Tuesday, July 22, 2008, 11:28 AM
Yeah, I am loving the discussion! More, more!
As an early postdoc, I still have in my working memory the classes I went
through in grad school, and I remember this connectionist lecturer arguing
that noise was actually a good thing for classifier-like systems (and by
extension neural nets, and by extension plausible neural nets -- which are
not classifiers stricto senso I agree) in that it allows an easier
discrimination of the input in a probabilistic context. Given that
redundancy of information/signal plays a big part in how the brain does the
job, wouldn't noise be a clever mechanism to discriminate close-to-threshold
stimuli? What do you think?
Le 22 juil. 08 à 17:17, jim bower a écrit :
I am actually in a remote part of brazil at the moment, so limited to typing
on my blackberry.
Impressive typing skills, I have to admit. ;-)
However, yes I was curious if a discussion could be induced. That was
originally what this mailing list was set up for, I know, because I started
it. ;-). However things have become a bit complacent so I figured what the
Again limited in my ability to respond but a couple of things. I think as
computational neurobiologists or scientists in general, we need to be aware
of the extent which what we can measure (oscillations, synchronous spikes,
etc) limits the way we think about how things work. Many many years ago now
when cortical oscillations became more generally interesting to people once
found in visual cortex we suggested based on our realistic cortical models
that they were an epiphenomina more (loosly) reflecting and underlying
mechanism for coordinating communication and processing between regions than
carriers of any information themselves. I continue to believe or set my
primary assumption that until proven otherwise, every spike is significant
for something and worse yet so is the lack of a spike.
(Certainly in digital coding 0s are as important as 1s.
Yes "serious scientists" prefer more constrained and defined discussions
than this. - but we can easily get lost "drinking our own whisky". As a
famous computational math-bio guy is fond of saying.
Truth is all these issues really remain wide open.
But and the big but, no evidence that nature is sloppy or unsophisticated.
One last point, the assumption that in fact nature is very sophisticated and
that the structure of the brain deeply reflects a complex, sophisticated
function pushes in the direction of first building models reflecting that
structure, even if you are still clueless about function.
I am in brazil teaching at the latin american school for computational
neuroscience, where realistic modeling lives on. ;-)
Best to all
Sent via BlackBerry by AT&T
Department of Computing
London SW7 2AZ
Comp-neuro mailing list
Comp-neuro at neuroinf.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Comp-neuro