[Comp-neuro] Re: Attractors, variability and noise

Mario Negrello mnegrello at gmail.com
Wed Aug 13 19:39:38 CEST 2008

To Ali and Brad, about meaning of attractors
> When we go looking for the "meaning" of attractors, or - even more  
> problematically - identity of meaning across attractors, aren't we  
> back to the homunculus or dualism?
> Shouldn't all "meaning" ultimately be grounded in physiological  
> response (sensor activity, behavior, autonomic activity, biochemical  
> activity, etc.) and all else regarded as post-hoc constructs that we  
> use for descriptive convenience? Or do people think that "appleness"  
> exists independently of "applehood"

When we measure physiological responses, we have a partial picture of  
the system. We have to assume a lot before we may even measure, since  
all the variables we assess empirically are selected by us, conforming  
our biases as scientific observers. Variables are by necessity the  
product of selection. They do not exhaust the phenomenon. Measured  
variables are only aspects of the phenomenon we study. Moreover, a  
controlled experiment requires the definition of what is meant by  
control, that is, the context and all the other things that are 'kept  

So, those things that you call 'post-hoc constructs' are not only for  
'descriptive convenience'. They are abstractions over the multiple  
observations of experiments. One can measure the patch clamp, but only  
the Hodkin Huxley model may explain the shape of an action potential.  
Theories are about the interrelations of the variables measured in  
experiment. You may say that that is obvious, and you'd be right. But  
what is not obvious is that there may be concepts that we 'invent' in  
theory, that make the results from experiment, make sense, in a more  
general sense. The physiology exists, but the differential equations  
of the models, that fit so well, do not.

Brad put it, inspiredly  "The Tyranny of ideas exists precisely  
because we think with ideas." The hermeneutic circle, as some called  
it. I find it reassuring to realize that many are so keenly aware of  
that. Attractors are short-hands, limited in scope, highly dependent  
on the theory they are a part of, worse than all, they are not real.  
But they are accessories to the conceptualization, and analogical  
understanding, of what may be going on down in biology. Indeed,  
despite all the contingencies that may be raised.

> - if you know what I mean? :-).

(i think i did, but you will be the judge)

To Todd:
> Much of the early part of this discussion focused on definitions.   
> The word “attractor” gets thrown around a lot, but most formal  
> definitions apply only to autonomous systems.  When you start  
> changing the inputs or start talking about “dynamic attractors” then  
> you start blurring the separation of time scales used to treat  
> activation dynamics as distinct from the timescales  of changing  
> input and changing network structure.  In this case, a more detailed  
> definition is in order.
Very true. I tried to be more specific in the paper. But inevitably, i  
had to introduce other concepts beyond those of pure dynamical  
system's theory, that were designed to increase the scope of the terms  
in the theory. If you're curious:
[1]	M. Negrello and F. Pasemann. Attractor landscapes in active  
tracking (special issue: the mind as a complex adaptive system).  
Journal of Adaptive Behavior, 2008.

To Brad and Jim, I am just happy to notice that  constructivistic  
ideas made a headway in brain sciences (pardon the pun!)

Kind regards.

On 12/ago/08, at 18:55, Ali Minai wrote:

> When we go looking for the "meaning" of attractors, or - even more  
> problematically - identity of meaning across attractors, aren't we  
> back to the homunculus or dualism? Shouldn't all "meaning"  
> ultimately be grounded in physiological response (sensor activity,  
> behavior, autonomic activity, biochemical activity, etc.) and all  
> else regarded as post-hoc constructs that we use for descriptive  
> convenience? Or do people think that "appleness" exists  
> independently of "applehood" - if you know what I mean? :-).
> Ali
> ---------------------------------------------------------------------
> Ali A. Minai
> Complex Adaptive Systems Lab
> Associate Professor
> Department of Electrical & Computer Engineering
> University of Cincinnati
> Cincinnati, OH 45221-0030
> Phone: (513) 556-4783
> Fax: (513) 556-7326
> Email: aminai at ece.uc.edu
>           minai_ali at yahoo.com
> WWW: http://www.ece.uc.edu/~aminai/
> ----------------------------------------------------------------------
> --- On Tue, 8/12/08, Mario Negrello <mnegrello at gmail.com> wrote:
> From: Mario Negrello <mnegrello at gmail.com>
> Subject: [Comp-neuro] Re: Attractors, variability and noise
> To: r.gayler at gmail.com, comp-neuro at neuroinf.org
> Date: Tuesday, August 12, 2008, 10:27 AM
> Ross, List,
> >
> > I'll make some responses to your post, although they speak rather
> > indirectly
> > to issues of variability and noise.  Also, although I am working  
> with
> > connectionist systems my perspective is one of artificial  
> intelligence
> > engineering - I am trying to
>  achieve functional performance without
> > any
> > specific concern for biological plausibility.  (Having said that,  
> the
> > mechanisms I am investigating are not obviously incompatible with
> > neural
> > implementation.)
> This degree of correspondance is my main interest right now. I'd like
> to know to what extent analytic arguments about NNs of different sorts
> are more than mere analogies. Functional performance is at the core of
> the issue, as function has the troubles of being defined subjectively.
> >
> >> What do you think of recurrent neural networks (RNNs), with their
> >> wealth
> > of attractors, as a model for variability/noise?
> >
> > In general, I have no idea.  [...] Arathorn claims that this
> > parallels some aspects of cortical
> > architecture.  I wouldn't know about that.  The point is that this
> > allows a
> > level of variability that
>  is greater than most people would assume
> > when
> > thinking about RNNs.  The point is that there are multiplicative
> > interactions and there are internal degrees of freedom (the
> > selection of the
> > transforms) - so that effectively the system has attractors that
> > evolve over
> > the same time scale as the settling of the RNN.
> I am somewhat familiar with arathorn's work, but his claims of
> plausibility (as you point out) may be more particular than general.
> In any case, to understand networks with high level of variability is
> a must for us.
> You say attractors evolve as they are being visited. I wonder to what
> extent that is the case, for instance, in mammalian motor systems,
> where there are obvious advangates of having a somewhat stable system.
> One can think of other examples, consider invertebrates with many
> fewer neurons and networks that are somewhat
>  stabler. What systems are
> emblematic of the role of multiplicative interactions?
> > Let me return to an argument from cognitive function.  We can
> > recognise
> > novel situations almost as rapidly as familiar ones.  All the time
> > we are
> > exposed to and deal with novel situations (e.g. my hovercraft is
> > full of
> > eels).  If we assume (as I do) that RNNs are a natural basis for
> > neural
> > computation, then you think in terms of attractors.  Familiar
> > situations are
> > recognised rapidly by the process of settling into an attractor.
> > But even
> > novel situations are recognised so rapidly that it suggests they too
> > are
> > recognised by the same process.  However, it is implausible to
> > believe that
> > the brain comes pre-stocked with an attractor for every situation
> > which
> > might ever arise (this is the dynamic
>  systems equivalent of the
> > "grandmother
> > cell" argument).  I believe the solution to this is to have dynamic
> > attractors, i.e. attractors that are created on the fly to meet the
> > need of
> > the moment.  Where there are multiplicative interactions one RNN can
> > modulate the dynamics of another RNN leading to new attractors.
> Intersting point about dynamic attractors. You talk about the creation
> of new attractors for recognition. But is it not the case that a fixed
> network structure already has an implicit attractor structure?
> Different initial conditions can take the network to attractors that
> have not yet been visited, but putatively already existed. This
> assumes fixed network structures, somewhat. The extent to which
> plasticity will impact on the visited attractors, is the extent to
> which we can talk about newly created attractors. Even so, plasticity
> may happen
>  parallely. The brain may be in constant change, but the
> changes maybe modularized, hence, some attractor structures can
> remained unchanged while others do change. Would you agree with this?
> But the brain structure changes less and less with age. So, one can
> think new thoughts, but these are compositional. Using the attractor
> structure already available, but in new combinations.
> Could you comment on the following question: To what extent
> multiplicative interactions maintain attractor structures unaltered?
> >
> > The notion is to have the attractor corresponding to the entire
> > input arise
> > as a function of the attactors corresponding to components of the
> > input.
> > This relates to the notion of systematicity, which holds that a
> > system able
> > to represent some situations (e.g. John loves Mary) will
> > *necessarily* be
> > able to represent other
>  related situations (e.g. Mary loves John).
> > This
> > arises as a consequence of representations being composites of
> > components
> > (which can be interchanged).
> So, that's the idea with compositionality of attractors. It is amusing
> that we connectionists are driven back to explain stuff that old AI
> had to assume. Looking for mechanisms compositionality in neural
> networks leads to interesting conclusions about the 'meaning' of
> attractors.
> In neural networks both orders, J loves M, and M loves J, may have
> different implications. Say, they evoke different emotions from the
> perspective of the thinker of the thoughts. E.g., J loves M implies
> Jealousy and M loves J implies disappointment.
> In a sense, one can think of RNNs lodging not only the attractors that
> enable compositional thougths, but also having attractor structures
> that induce some further implications. For
>  that, in principle, one
> wouldn't need to generate new attractors, in the sense that the net
> struct is fixed in a certain time scale. Or one could think that some
> changes in the network will have the same 'meaning'. That is, they
> will produce similar dispositions, or evoke similar categories of
> outputs. In that case, there must be a many to one mapping from
> attractors to categories. This is to say that a lot of neural changes
> may be irrelevant for categorization. It goes back to the idea that
> variation which is not useful is noise. The querstion becomes: from
> the perspective of the organism, when are two attractors the same? In
> other words, when do two attractors have the same meaning?
> The answer to that question seem to me to point out that it is not
> only noise that has to be defined in terms of the observer, but also
> 'stability'.  Two attractors (or transients) that have the same stable
> meaning, will be equivalent. In order to verify stability, one will
> have to make subjective assumptions about the meaning of the
> attractors. If the attractors predispose the same responses, they are
> essentially the same. Correct?
> > The only point I really want to make out of this is that when you  
> are
> > dealing with RNNs in cognitive systems you should allow for the
> > likelihood
> > that the attractors are dynamic and novel unless you are probing  
> with
> > exceptionally impoverished stimuli (which is probably the norm in
> > electrophysiology).
> It's a good point. But even with empoverished stimuli, there can be a
> lot of variability in the neurophysiology (unless one probes very
> close to the lower level sensory sheets). Interestingly, and i think
> this is a major point,
> PS. Some might have noticed that there was a posting coming from my
> address with
>  splintered phrases. I sent it by mistake, apologies for
> that!
> > -----Original Message-----
> > From: Mario Negrello [mailto:mnegrello at gmail.com]
> > Sent: Saturday, 2 August 2008 12:33 AM
> > To: r.gayler at mbox.com.au
> > Cc: comp-neuro at neuroinf.org
> > Subject: Attractors, variability and noise
> >
> > Dear Ross, Jim, David and list,
> >
> > First off, thank you for amazing discussion. I hope there is still
> > some
> > momentum in it.
> > I take the opportunity to summarize and combine a couple of points,
> > and pose
> > a question.
> >
> > David Tam, among others, remarked that
> >> "So do neurons (or the brain) use noise in its computation?  If
> the
> >> neurons care about those signals, it is not noise.  If they don't
> >> care, yes, then it is noise."
> >
> > This is a good working definition for noise, as it takes the
> >
>  'receiver' into
> > consideration. I'll phrase it in terms of variability, if i may
> > (it's for a
> > purpose):
> > - When the receiving system is indifferent to incoming variability,
> > then
> > that variability is noise.
> >
> > Jim and others insisted that it is hard to tell noise and
> > variability apart,
> > when the systems are complex.
> >>> the more sophisticated a coding system, the harder it is likely to
> >>> be
> >>> to distinguish signal from "noise".
> >
> > 1. As Ross points out, concerning abstract connectionist models, the
> > response of a single neuron may look random, because we d.on't see  
> the
> > multidimensional pattern of activity, by recording one neuron.
> > 2. And as david adds, neural computation is distinction, if there is
> > no
> > distinction, no computation.
> > 3. Moreover, Jim added that (3) neural
>  computation is likely to be
> > sequential.
> >
> > The points above fit nicely with the idea that brain networks are
> > analogous
> > to recurrent neural networks, which have, instead of a lot of noise,
> > complex
> > transients.
> >
> > The question: What do you think of recurrent neural networks (RNNs),
> > with
> > their wealth of attractors, as a model for variability/noise?
> > Large networks produce much repeatable variability in
> 'unpredictable'
> > oscillatory patterns. But with knowledge of the network structure,
> > much can
> > be known about the possible dynamical patterns that the system may
> > produce.
> > And with respect to functional levels,  one can distinguish between
> > meaningful and meaningless variability. (by functional levels, i
> > mean levels
> > of the network we may attrbitute particular functions, say  
> perceptual
> >
>  categorization or motor pattern formation).
> >
> > Regarding the items above:
> > To (1), It is self-evident when one takes a large network of, say,
> > sigmoidal
> > units, and looks at the activity of any one neuron. Though the path
> > of the
> > transients in phase space is highly structured, the activity of the
> > single
> > neuron is scarcely predictable without knowledge of the network
> > structure.
> > But if one has knowledge about the structure, she also has info  
> about
> > possible activities patterns.
> >
> > Will these patterns resemble those of more complex biological
> > networks? My
> > guess is yes, to the extent of the level of abstraction introduced.
> >
> > To (2), there is something that can be said in terms of distinction
> > mechanisms, and transient activity. If one considers the transient
> > activity
> > of a module/area as an
>  open path in phase space (an orbit), then the
> > activity of one neuron must be a projection of all the incoming
> > activity,
> > onto the hyperplane defined by the projections weight matrix  
> (attached
> > diagram).
> >
> |============]M[=============|
> |www.firingrates.blogspot.com|
> Every machine is the spiritualization of an organism.
> Theo van Doesburg (1883-1931)
> _______________________________________________
> Comp-neuro mailing list
> Comp-neuro at neuroinf.org
> http://www.neuroinf.org/mailman/listinfo/comp-neuro
> _______________________________________________
> Comp-neuro mailing list
> Comp-neuro at neuroinf.org
> http://www.neuroinf.org/mailman/listinfo/comp-neuro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.neuroinf.org/pipermail/comp-neuro/attachments/20080813/e189cb8e/attachment-0001.html

More information about the Comp-neuro mailing list