[Comp-neuro] Re: Attractors, variability and noise

Todd Troyer todd.troyer at utsa.edu
Wed Aug 13 15:59:54 CEST 2008


Much of the early part of this discussion focused on definitions.  The word
³attractor² gets thrown around a lot, but most formal definitions apply only
to autonomous systems.  When you start changing the inputs or start talking
about ³dynamic attractors² then you start blurring the separation of time
scales used to treat activation dynamics as distinct from the timescales  of
changing input and changing network structure.  In this case, a more
detailed definition is in order.

Todd


On 8/12/08 11:55 AM, "Ali Minai" <minai_ali at yahoo.com> wrote:

> When we go looking for the "meaning" of attractors, or - even more
> problematically - identity of meaning across attractors, aren't we back to the
> homunculus or dualism? Shouldn't all "meaning" ultimately be grounded in
> physiological response (sensor activity, behavior, autonomic activity,
> biochemical activity, etc.) and all else regarded as post-hoc constructs that
> we use for descriptive convenience? Or do people think that "appleness" exists
> independently of "applehood" - if you know what I mean? :-).
> 
> Ali
> 
> 
> ---------------------------------------------------------------------
> Ali A. Minai
> Complex Adaptive Systems Lab
> Associate Professor
> Department of Electrical & Computer Engineering
> University of Cincinnati
> Cincinnati, OH 45221-0030
> 
> Phone: (513) 556-4783
> Fax: (513) 556-7326
> Email: aminai at ece.uc.edu
>           minai_ali at yahoo.com
> 
> WWW: http://www.ece.uc.edu/~aminai/
> 
> ----------------------------------------------------------------------
> 
> --- On Tue, 8/12/08, Mario Negrello <mnegrello at gmail.com> wrote:
>> From: Mario Negrello <mnegrello at gmail.com>
>> Subject: [Comp-neuro] Re: Attractors, variability and noise
>> To: r.gayler at gmail.com, comp-neuro at neuroinf.org
>> Date: Tuesday, August 12, 2008, 10:27 AM
>> 
>> Ross, List,
>> 
>>> >
>>> > I'll make some responses to your post, although they speak rather
>>> > indirectly
>>> > to issues of variability and noise.  Also, although I am working with
>>> > connectionist systems my perspective is one of artificial intelligence
>>> > engineering - I am trying to
>>  achieve functional performance without
>>> > any
>>> > specific concern for biological plausibility.  (Having said that, the
>>> > mechanisms I am investigating are not obviously incompatible with
>>> > neural
>>> > implementation.)
>> 
>> This degree of correspondance is my main interest right now. I'd like
>> to know to what extent analytic arguments about NNs of different sorts
>> are more than mere analogies. Functional performance is at the core of
>> the issue, as function has the troubles of being defined subjectively.
>>> >
>>>> >> What do you think of recurrent neural networks (RNNs), with their
>>>> >> wealth
>>> > of attractors, as a model for variability/noise?
>>> >
>>> > In general, I have no idea.  [...] Arathorn claims that this
>>> > parallels some aspects of cortical
>>> > architecture.  I wouldn't know about that.  The point is that this
>>> > allows a
>>> > level of variability that
>>  is greater than most people would assume
>>> > when
>>> > thinking about RNNs.  The point is that there are multiplicative
>>> > interactions and there are internal degrees of freedom (the
>>> > selection of the
>>> > transforms) - so that effectively the system has attractors that
>>> > evolve over
>>> > the same time scale as the settling of the RNN.
>> 
>> I am somewhat familiar with arathorn's work, but his claims of
>> plausibility (as you point out) may be more particular than general.
>> In any case, to understand networks with high level of variability is
>> a must for us.
>> 
>> You say attractors evolve as they are being visited. I wonder to what
>> extent that is the case, for instance, in mammalian motor systems,
>> where there are obvious advangates of having a somewhat stable system.
>> One can think of other examples, consider invertebrates with many
>> fewer neurons and networks that are somewhat
>>  stabler. What systems are
>> emblematic of the role of multiplicative interactions?
>> 
>>> > Let me return to an argument from cognitive function.  We can
>>> > recognise
>>> > novel situations almost as rapidly as familiar ones.  All the time
>>> > we are
>>> > exposed to and deal with novel situations (e.g. my hovercraft is
>>> > full of
>>> > eels).  If we assume (as I do) that RNNs are a natural basis for
>>> > neural
>>> > computation, then you think in terms of attractors.  Familiar
>>> > situations are
>>> > recognised rapidly by the process of settling into an attractor.
>>> > But even
>>> > novel situations are recognised so rapidly that it suggests they too
>>> > are
>>> > recognised by the same process.  However, it is implausible to
>>> > believe that
>>> > the brain comes pre-stocked with an attractor for every situation
>>> > which
>>> > might ever arise (this is the dynamic
>>  systems equivalent of the
>>> > "grandmother
>>> > cell" argument).  I believe the solution to this is to have dynamic
>>> > attractors, i.e. attractors that are created on the fly to meet the
>>> > need of
>>> > the moment.  Where there are multiplicative interactions one RNN can
>>> > modulate the dynamics of another RNN leading to new attractors.
>> 
>> Intersting point about dynamic attractors. You talk about the creation
>> of new attractors for recognition. But is it not the case that a fixed
>> network structure already has an implicit attractor structure?
>> Different initial conditions can take the network to attractors that
>> have not yet been visited, but putatively already existed. This
>> assumes fixed network structures, somewhat. The extent to which
>> plasticity will impact on the visited attractors, is the extent to
>> which we can talk about newly created attractors. Even so, plasticity
>> may happen
>>  parallely. The brain may be in constant change, but the
>> changes maybe modularized, hence, some attractor structures can
>> remained unchanged while others do change. Would you agree with this?
>> 
>> But the brain structure changes less and less with age. So, one can
>> think new thoughts, but these are compositional. Using the attractor
>> structure already available, but in new combinations.
>> 
>> Could you comment on the following question: To what extent
>> multiplicative interactions maintain attractor structures unaltered?
>> 
>>> >
>>> > The notion is to have the attractor corresponding to the entire
>>> > input arise
>>> > as a function of the attactors corresponding to components of the
>>> > input.
>>> > This relates to the notion of systematicity, which holds that a
>>> > system able
>>> > to represent some situations (e.g. John loves Mary) will
>>> > *necessarily* be
>>> > able to represent other
>>  related situations (e.g. Mary loves John).
>>> > This
>>> > arises as a consequence of representations being composites of
>>> > components
>>> > (which can be interchanged).
>> 
>> So, that's the idea with compositionality of attractors. It is amusing
>> that we connectionists are driven back to explain stuff that old AI
>> had to assume. Looking for mechanisms compositionality in neural
>> networks leads to interesting conclusions about the 'meaning' of
>> attractors.
>> 
>> In neural networks both orders, J loves M, and M loves J, may have
>> different implications. Say, they evoke different emotions from the
>> perspective of the thinker of the thoughts. E.g., J loves M implies
>> Jealousy and M loves J implies disappointment.
>> 
>> In a sense, one can think of RNNs lodging not only the attractors that
>> enable compositional thougths, but also having attractor structures
>> that induce some further implications. For
>>  that, in principle, one
>> wouldn't need to generate new attractors, in the sense that the net
>> struct is fixed in a certain time scale. Or one could think that some
>> changes in the network will have the same 'meaning'. That is, they
>> will produce similar dispositions, or evoke similar categories of
>> outputs. In that case, there must be a many to one mapping from
>> attractors to categories. This is to say that a lot of neural changes
>> may be irrelevant for categorization. It goes back to the idea that
>> variation which is not useful is noise. The querstion becomes: from
>> the perspective of the organism, when are two attractors the same? In
>> other words, when do two attractors have the same meaning?
>> 
>> The answer to that question seem to me to point out that it is not
>> only noise that has to be defined in terms of the observer, but also
>> 'stability'.  Two attractors (or transients) that have the same stable
>>  
>> 
>> meaning, will be equivalent. In order to verify stability, one will
>> have to make subjective assumptions about the meaning of the
>> attractors. If the attractors predispose the same responses, they are
>> essentially the same. Correct?
>> 
>>> > The only point I really want to make out of this is that when you are
>>> > dealing with RNNs in cognitive systems you should allow for the
>>> > likelihood
>>> > that the attractors are dynamic and novel unless you are probing with
>>> > exceptionally impoverished stimuli (which is probably the norm in
>>> > electrophysiology).
>> 
>> It's a good point. But even with empoverished stimuli, there can be a
>> lot of variability in the neurophysiology (unless one probes very
>> close to the lower level sensory sheets). Interestingly, and i think
>> this is a major point,
>> 
>> 
>> 
>> PS. Some might have noticed that there was a posting coming from my
>> address with
>>  splintered phrases. I sent it by mistake, apologies for
>> that!
>> 
>> 
>>> > -----Original Message-----
>>> > From: Mario Negrello [mailto:mnegrello at gmail.com]
>>> > Sent: Saturday, 2 August 2008 12:33 AM
>>> > To: r.gayler at mbox.com.au
>>> > Cc: comp-neuro at neuroinf.org
>>> > Subject: Attractors, variability and noise
>>> >
>>> > Dear Ross, Jim, David and list,
>>> >
>>> > First off, thank you for amazing discussion. I hope there is still
>>> > some
>>> > momentum in it.
>>> > I take the opportunity to summarize and combine a couple of points,
>>> > and pose
>>> > a question.
>>> >
>>> > David Tam, among others, remarked that
>>>> >> "So do neurons (or the brain) use noise in its computation?  If
>> the
>>>> >> neurons care about those signals, it is not noise.  If they don't
>>>> >> care, yes, then it is noise."
>>> >
>>> > This is a good working definition for noise, as it takes the
>>> >
>>  'receiver' into
>>> > consideration. I'll phrase it in terms of variability, if i may
>>> > (it's for a
>>> > purpose):
>>> > - When the receiving system is indifferent to incoming variability,
>>> > then
>>> > that variability is noise.
>>> >
>>> > Jim and others insisted that it is hard to tell noise and
>>> > variability apart,
>>> > when the systems are complex.
>>>>> >>> the more sophisticated a coding system, the harder it is likely to
>>  
>>>>> >>> be
>>>>> >>> to distinguish signal from "noise".
>>> >
>>> > 1. As Ross points out, concerning abstract connectionist models, the
>>> > response of a single neuron may look random, because we d.on't see the
>>> > multidimensional pattern of activity, by recording one neuron.
>>> > 2. And as david adds, neural computation is distinction, if there is
>>> > no
>>> > distinction, no computation.
>>> > 3. Moreover, Jim added that (3) neural
>>  computation is likely to be
>>> > sequential.
>>> >
>>> > The points above fit nicely with the idea that brain networks are
>>> > analogous
>>> > to recurrent neural networks, which have, instead of a lot of noise,
>>> > complex
>>> > transients.
>>> >
>>> > The question: What do you think of recurrent neural networks (RNNs),
>>> > with
>>> > their wealth of attractors, as a model for variability/noise?
>>> > Large networks produce much repeatable variability in
>> 'unpredictable'
>>> > oscillatory patterns. But with knowledge of the network structure,
>>> > much can
>>> > be known about the possible dynamical patterns that the system may
>>> > produce.
>>> > And with respect to functional levels,  one can distinguish between
>>> > meaningful and meaningless variability. (by functional levels, i
>>> > mean levels
>>> > of the network we may attrbitute particular functions, say perceptual
>>> >
>>  categorization or motor pattern formation).
>>> >
>>> > Regarding the items above:
>>> > To (1), It is self-evident when one takes a large network of, say,
>>> > sigmoidal
>>> > units, and looks at the activity of any one neuron. Though the path
>>> > of the
>>> > transients in phase space is highly structured, the activity of the
>>> > single
>>> > neuron is scarcely predictable without knowledge of the network
>>> > structure.
>>> > But if one has knowledge about the structure, she also has info about
>>> > possible activities patterns.
>>> >
>>> > Will these patterns resemble those of more complex biological
>>> > networks? My
>>> > guess is yes, to the extent of the level of abstraction introduced.
>>> >
>>> > To (2), there is something that can be said in terms of distinction
>>> > mechanisms, and transient activity. If one considers the transient
>>> > activity
>>> > of a module/area as an
>>  open path in phase space (an orbit), then the
>>> > activity of one neuron must be a projection of all the incoming
>>> > activity,
>>> > onto the hyperplane defined by the projections weight matrix (attached
>>> > diagram).
>>> >
>> 
>> 
>> 
>> |============]M[=============|
>> |www.firingrates.blogspot.com|
>> 
>> Every machine is the spiritualization of an organism.
>> 
>> Theo van Doesburg (1883-1931)
>> 
>> _______________________________________________
>> Comp-neuro mailing list
>> Comp-neuro at neuroinf.org
>> http://www.neuroinf.org/mailman/listinfo/comp-neuro
> 
> 
> _______________________________________________
> Comp-neuro mailing list
> Comp-neuro at neuroinf.org
> http://www.neuroinf.org/mailman/listinfo/comp-neuro

------------------------------------------------------
Todd Troyer
UT San Antonio, Biology Dept.
One UTSA Circle
San Antonio TX 78249
Todd.troyer at utsa.edu
W: 210-458-5487, FAX: 210-458-5658
www.utsa.edu/troyerlab
------------------------------------------------------


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.neuroinf.org/pipermail/comp-neuro/attachments/20080813/91971416/attachment-0001.html


More information about the Comp-neuro mailing list