Laboratory of Neural Networks
REVIEW: OSCILLATORY NEURAL NETWORKS FOR ATTENTION MODELING
Oscillatory models of attention: A class of artificial neural networks which simulate psychological and neurobiological aspects of attention basing on synchronous dynamics of network elements.
1. Motivation. Biological bases. Tasks
2. Model building units and connection architectures.
3. Oscillatory models of attention with the central unit
3.1. Basic principles
3.2. Partial synchronization
3.3. Object selection and novelty detection
3.4. The tracking of moving objects
4. Work in progress: Object segmentation and attention
1. Motivation. Biological bases. Tasks
Attention is traditionally considered as a filter that extracts from the incoming information the part that is most important at a given moment and that should be processed in more detail. Electrode recordings and functional brain imaging in animals and humans revealed two types of attentional modulation of neural activity in the cortex. First, additional excitation of neurons representing attended stimuli is observed while neural activity evoked by unattended stimuli is reduced to a low level. Second, attention increases the synchronization of neural activity representing attending stimuli in various parts of the cortex.
The main psychological phenomena subject to attention modeling include consecutive selection and segmentation of objects in the visual scene, conjunction-feature search, and the tracking of moving objects. It is supposed that the models of attention should be able to simulate these facts by biologically plausible artificial neural networks. The oscillatory models of attention try to simulate the evidence on neural activity, temporal characteristics of attention focusing (e.g., response time) and error profiles basing on oscillatory and synchronous dynamics in neural networks.
The interest to oscillatory models of attention is conditioned by the fact that they can provide a unified approach to the problems of feature binding and attention. This approach is formulated as a temporal correlation hypothesis which states that synchrony of neural activity is a common mechanism of representation of stimuli in the cortex. It is assumed that the synchronized input results in high post-synaptic potentials effectively amplifying the activity of post-synaptic neurons. Therefore the role of attentional control might be to increase the synchronization in attended stimuli representation and to desynchronize the activity between attended and unattended representations. Recently the experimental evidence has been obtained showing that increased synchronization in the visual cortex is a result of synchronizing influence projected from the so-called central executive of the attention system that is located in the prefrontal cortex.
2. Model building units and connection architectures
Oscillatory models of attention preserve the same principles of neural network design that have been developed in the theory of oscillatory neural networks. The networks are built from oscillatory units or there is a mixture of oscillatory units and traditional excitatory and inhibitory artificial neurons. An oscillatory unit can represent a pacemaker neuron or be a neural oscillator that represents two interacting populations of excitatory and inhibitory neurons working in the oscillatory regime. The following oscillatory units are most common: Hodgkin-Huxley neurons, interacting populations of integrate-and-fire neurons, Wilson-Cowan neural oscillators, Van der Pol oscillators, Kuramoto-type pulse oscillators, Kuramoto’s phase oscillators.
The model architectures reproduce conventional architectures of multilayer networks usually with lateral connections in addition to between layer connections. There is also a special architecture that is rarely used in conventional neural network design but is helpful in oscillatory neural networks. It is called the architecture with a central element. This architecture is characterized by the existence of a unit that has global connections with all other units of the network and that controls the dynamics of the whole system. The central unit is used to select a particular object in the focus of attention. It can be done by activating the units that represent this object and inhibiting all other units or by synchronizing the activity of the central unit with the activity of a particular assembly of units.
3. Oscillatory models of attention with the central unit
3.1. Basic principles
The basic principle of functioning of oscillatory models of attention with a central unit is that the focus of attention is represented by those oscillators that are entrained or work in-phase with the central unit. It is assumed that the central unit (CU) performs the role of the central executive of the attention system while other units code the features of external stimuli. Below these units will be referred to as feature units (FU). The architecture of the network with a central unit is shown in Figure 1. CU contains one oscillatory element or a small number of such elements. FUs form a one layer network with or without lateral connections. There are feedforward and feedback connections between FUs and the CU.
Figure 1. Architecture of connections for the network with a central unit (CU). The brown arrow shows the assignment of natural frequencies of feature units (FU). Black arrows show excitatory/synchronizing connections. These connections are used (1) to bind the FUs that code an object into a synchronous assembly and (2) to synchronize an assembly of FUs with the CU. The hollow arrow shows desynchronizing connections that are used to avoid simultaneous synchronization of the CU with several assemblies of FUs.
3.2. Partial synchronization (Kazanovich & Borisyuk, 1999; Borisyuk & Kazanovich, 2003; Chik et al., 2009a, Chik et al., 2009b)
Depending on parameter values (natural frequencies of oscillators, connection strengths, time delays in connections) different types of synchronous dynamics are possible. For attention modeling, the most important regime is the regime of partial synchronization. This regime corresponds to the situation when several objects are simultaneously presented as an external stimulus, but only one object is selected in the focus of attention. During partial synchronization the CU is phase-locked by some assembly of FUs representing the features of an object that is currently included in the attention focus, while other FUs are not included into synchronization with the CU. The diagram of different types of synchronous dynamics in the case of Hodgkin-Huxley type pacemaker neurons is shown in Figure 2.
Figure 2. Dynamical profile of the system with two groups of FUs. The axes show the external currents I1 and I2 (in milliamperes) which determine the intrinsic frequencies of groups A and B of FUs, respectively. The dynamical behaviors are labeled as:
global synchronization – all FUs are synchronized by the CU;
transitional state – complex types of synchronous dynamics between the CU and FUs;
partial synchronization – synchronization of the CU with a group of FUs, spiking activity in the other group is inhibited;
quiescence – spiking activity in both groups in inhibited, only fluctuations of the membrane potential survive (Chik et al., 2009a).
For some parameter values complex chaotic behavior is possible in the system when partial synchronization automatically switches from one group of features to another (Figure 3). This regime can be used for modeling perceptual multi-stability.
Figure 3. Modeling perceptional bistability. Intermittent activity in two groups of FUs is shown. Traces of the membrane potential represent three FUs from group A and three from group B (Chik et al., 2009b).
3.3. Object selection and novelty detection (Borisyuk & Kazanovich, 2004)
Oscillatory models of attention with the central unit have been applied to modeling consecutive selection of objects in the visual scene (Figure 4). To prevent selection of the same object in the focus of attention, the activity of oscillators representing an object that has already been selected in the focus of attention is inhibited for some time to allow focusing attention on another object. The attention system can be combined with the system for novelty detection. In this case if the same object is present at different locations of the same image it is included in the focus of attention only once when it is recognized as a new object. The attempts to include the same object into the focus of attention are suppressed by the novelty detection system which recognizes the object as familiar.
Figure 4. Consecutive selection of objects (characters) from the image of the word HELLO. Vertical axes represent the amplitude of oscillations of the unit corresponding to a particular pixel. The order of selection is not from left to right but with the advantage of characters with the higher number of pixels. Time is given in seconds (Borisyuk & Kazanovich, 2004).
3.4. The tracking of moving objects (Kazanovich & Borisyuk, 2006)
Oscillatory models of attention are helpful for the tracking of moving objects since they mostly operate not in physical but in phase-frequency space where object representation is invariant to object location. The important property of tracking is that there is a possibility to always keep only a given number of target objects in the focus of attention. This is because assemblies of synchronously running FUs compete for the synchronization with the CU and only a given number of assemblies can win this competition at any moment of time. An example of the tracking of moving objects among several distracters is shown in Figure 5. It has been found in psychological experiments that subjects can track up to 5 identical targets among a set of distracters, but the probability of an error in attending the targets increases if the number of targets increases. This observation was reproduced in the model due to the fact that phase space is rather restricted and different oscillators disturb each other making the partial synchronization of the CU with a particular assembly of FUs unstable.
Figure 5. Multiple object tracking: Spontaneous attention switching. The frames show the evolution of the input image in time (the frames are ordered from left to right and from top to bottom). The input image to the attention system contains 10 identical moving objects (black squares). The system simultaneously tracks 5 objects (colors are used to mark the areas covered by attention) among 5 distractors (shown as black squares in the figure). Frames (2-4) show the moment when attention spontaneously switches from the target object to a distractor (green marker jumps from one object to another) which results in the error of tracking. Frames (5-8) show spontaneous attention switching from one distractor to another. The number of tracking errors does not increase (Kazanovich & Borisyuk, 2006).
4. Work in progress: Object segmentation and attention (Borisyuk et al., 2009)
The problem of focusing attention on an object is closely related to the problem of object segmentation because the selected object should be segmented from other objects in the image and from the background. Segmentation of objects in a visual scene is a classical problem in the theory of image processing. To extract the boundaries between objects, various types of filters have been widely used both in artificial vision and neural network models. Unfortunately, the resulting boundaries are contaminated with noise. Also some spurious objects may appear. A combination of contour extraction algorithms and a two layer network of phase oscillators with the central element can help to improve the results of attention focusing on an object in the real colored image. The first layer obtains a raw contour of an object as an input and uses this contour to form a coherent representation of the internal part of the object. The second layer spreads this representation to the pixels of the object boundary. The central unit desynchronizes the activity of oscillators representing the attended object and other objects of the image. Figure 6 presents an example of system performance in the case of an image taken from a visual stream of the robot camera.
The development of most dynamic neural models was aimed to reproduce separate cognitive functions. We think that essential progress can be achieved when a complex model that combines different cognitive functions in a single device will be developed. These cognitive functions should include image segmentation, attention, novelty detection, pattern recognition, short- and long-term memory. This will allow the finding of a solution to such problems as separation of overlapping objects, searching a given object in a complex scene and the tracking of moving objects on a complex background. Also, the possibility to reproduce neurobiological and psychological data related to these tasks will radically increase. The implementation of such a system based on the principles of oscillatory dynamics, synchronization, and resonance is an important task for the near future.
Chik D., Borisyuk R., Kazanovich Y. (2009a) Selective attention model with spiking elements. Neural Networks, 22, 890-900.
Chik D., Borisyuk R., Kazanovich Y. (2009b) Visual perception of ambiguous figures: Synchronization based neural models. Biological Cybernetics, 100, 491-504.
Borisyuk R.M., Kazanovich Y.B. (2003) Oscillatory neural network model of attention focus formation and control. BioSystems, 71, 29-38.
Borisyuk R., Kazanovich Y. (2004) Oscillatory model of attention-guided object selection and novelty detection. Neural Networks, 17, 899-915.
Borisyuk R., Kazanovich Y., Chik D., Tikhanoff V., Cangelosi A. (2009) A neural model of selective attention and object segmentation in the visual scene: An approach based on partial synchronization and star-like architecture of connections. Neural Networks, 22, 707-719,
Kazanovich Y.B., Borisyuk R.M. (1999) Dynamics of neural networks with a central element. Neural Networks, 12, 149-161.
Kazanovich Y.B., Borisyuk R.M. (2006) An oscillatory neural model of multiple object tracking. Neural Computation, 18, 1413-1440.