Computational neuroscience for learning from a small sample
Abstract
Deep neural networks have been remarkably useful for image classification and phoneme recognition. Combined with reinforcement learning algorithms, deep neural networks have outperformed human experts in simulated video games and the game “Go”. To achieve such successes, millions of images, hundreds of millions of phonemes, and tens of millions of games have been utilized as training data sets in the supervised learning or training trials in the reinforcement learning. Meanwhile, in the 2015 DARPA robotics challenge final competition (2015 DARPA Robotics Challenge Finals), many humanoid robots fell while walking on sand, going up stairs, turning bulbs, or getting out of a car. A small number of humanoids completed all the tasks, but they were extremely slower than humans. By age 5, human infants are able to execute all of the above tasks more quickly and reliably than humanoid robots developed by world premier researchers. What could be the reasons of this dramatic contrast between success and failure for simulated versus real-world tasks by artificial intelligence? In the simulated video games and “Go”, the degrees of freedom of the controlled system were relatively small, there were no hidden variables, and state transitions were deterministic without noise and perfectly described by simple rules. Thus, the computer simulations were exactly correct without errors. For the final reason, tens of millions of simulated games are generated by software players, and they can be used efficiently for DeepQ learning (a Q-learning algorithm of reinforcement learning combined with deep neural network learning). In contrast, a humanoid robot in the real world is a complicated nonlinear dynamical system with huge degrees of freedom. Indeed, hidden states can be situated far above measured sensory signals and far below issued motor commands. Many physical processes, including contact and friction, are difficult to model. Mainly for the final reason, quantitatively reliable simulations of humanoid robots in real-world environments are extremely difficult even if not impossible. Thus, reinforcement learning in humanoids designed to operate in the real world has been typically conducted using real experimental trials. However, when humanoids fall, they are often damaged such that no further trials can be accumulated before painful, expensive and laborious repairs are made. In artificial intelligence, or more precisely, in neural networks learning and machine learning, it is well established that when a learning system with a fixed degrees of freedom n is utilized, approximately 10n training samples are necessary. If it is possible to conduct tens of millions of learning trials, a large learning system, such as deep neural networks, can be utilized. However, if only 100 trials can be accumulated, only very simple learning systems with ten degrees of freedom should be utilized to avoid over-fitting problems in learning. I postulate that these differences in the number of training samples and consequently resulting allowed degrees of freedom of the control systems readily explain the dramatic contrast between the success of the simulated learning and the failure of the real-world learning mentioned above.
Animal brains are confronted with sensorimotor problems that are much more challenging than those faced by humanoid robots. Animal bodies are flexible and possess an enormous number of muscles, sensors, and motor neurons. Neurons are slow-computing devices with a significant degree of noise. Thus, physical modeling of animal movements is very difficult, as there are many degrees of freedom, hidden variables, a high noise level, and a risk of injury or death in the case of failure. The human brain contains 10 to the 11th neurons and 10 to the 14th synapses. As a learning control system it has enormous degrees of freedom. If we assume that the number of synapses correspond to the degree of freedom of the learning system, and that a single reinforcement learning trial can be obtained within 10 seconds, then it follows that an animal brain will need 10 to the 15th training trials, and thus 10 to the 16th seconds for learning time to avoid over-fitting. This period is much longer than an animal life. In contrast to this estimate, humans learn motor control very quickly. For example, humans can learn new dynamic environment within a few trials. Human infants learn to walk after only several thousands falls. Through computational neuroscience research of sensorimotor learning, I hope to understand a mystery to brake the common sense in artificial intelligence: 10 to the 11th degrees-of-freedom learning system can learn to control an extremely complicated nonlinear dynamical system only after 1,000 failures. Kawato and Samejima (2007) reviewed several computational schemes for enabling efficient reinforcement learning from a small training samples. They include internal models, sparse estimation algorithms, multiple- paired forward and inverse models, and a hierarchical reinforcement learning algorithms. Attention, consciousness, metacognition, and episodic memory are important research topics in cognitive neuroscience, and have recently attracted the interests of artificial intelligence researchers with the hope that they could provide computational mechanisms to decrease high dimensionality of data in learning. They may play essential roles in constructing abstract concepts, dimensions and attributes that are high-level representations necessary in the upper layers of hierarchical reinforcement learning. With respect to reducing the dimensionality of high-dimensional data, electrical synapses that transmit information via gap junctions are attractive elements in neuronal circuits because they tend to synchronize neurons and effectively reduce the degrees of freedom of the circuit.
The cerebellum is important for motor control and motor learning and plays very important roles in multi-joint movements such as walking. The inferior olivary (IO) nucleus sends climbing fiber inputs to Purkinje cell (PC), the only output of all motor coordination in the cerebellar cortex, and possesses the highest density of gap junctions in the mammalian brain. As a good candidate for a neuronal system that plays a central role in motor learning and that may be useful in investigating the above-mentioned disparity between the large degrees of freedom of learning systems and conditions where only a small number of training trials are available, I focus on the olivo-cerebellar system. Of special interest is the network of IO neurons, which may control the degrees of freedom by adjusting their synchronous/asynchronous firing activities to provide an adaptive framework for the learning machinery. In the cerebellar motor learning, it has been known that the IO neurons transmit error signals to the PC, inducing plasticity at the parallel fiber-PC synapses. Recent investigations have also revealed multiple plasticity mechanisms as well as evidence that parallel fiber-evoked simple spikes to PCs contribute to cerebellum-dependent learning to some extent. One dominant view over the last several decades suggests that complex spikes transmitted through the climbing fibers provide instructive signals to the PCs to drive learning. To examine the functions of the IO, computational modeling has been one of the promising driving forces. As the carrier of the teaching signals, the IO has been modeled to provide the climbing fiber inputs in the simulation studies of the cerebellar learning. To explore the IO dynamics in detail, a class of simplified conductance-based models has been developed to reproduce experimental observation of sub- threshold oscillations. Further details of the electrophysiological properties of the IO neurons have been described by multiple compartment models, which have been applied to elucidate experimental observation of the sub-threshold activities, to examine the capability of their information transmission, and to estimate conductance levels of the IO network from experimental data. Owing to the advanced experimental methods as well as the rapid growth in computer power, the computational models have been nowadays utilized for quantitative understanding of the experimentally measured IO dynamics and furthermore for testing hypotheses regarding IO functions. Here, I review recent advances in the computational modeling of the olivo- cerebellar system.
