Gene Expression in Neurons Solves a Brain Evolution Puzzle

The neocortex stands out as a stunning achievement of biological evolution. All mammals have this swath of tissue covering their brain, and the six layers of densely packed neurons within it handle the sophisticated computations and associations that produce cognitive prowess. Since no animals other than mammals have a neocortex, scientists have wondered how such a complex brain region evolved.

The brains of reptiles seemed to offer a clue. Not only are reptiles the closest living relatives of mammals, but their brains have a three-layered structure called a dorsal ventricular ridge, or DVR, with functional similarities to the neocortex. For more than 50 years, some evolutionary neuroscientists have argued that the neocortex and the DVR were both derived from a more primitive feature in an ancestor shared by mammals and reptiles.

Now, however, by analyzing molecular details invisible to the human eye, scientists have refuted that view. By looking at patterns of gene expression in individual brain cells, researchers at Columbia University showed that despite the anatomical similarities, the neocortex in mammals and the DVR in reptiles are unrelated. Instead, mammals seem to have evolved the neocortex as an entirely new brain region, one built without a trace of what came before it. The neocortex is composed of new types of neurons that seem to have no precedent in ancestral animals.

The paper describing this work, which was led by the evolutionary and developmental biologist Maria Antonietta Tosches, was published last September in Science.

This process of evolutionary innovation in the brain isn’t limited to the creation of new parts. Other work by Tosches and her colleagues in the same issue of Science showed that even seemingly ancient brain regions are continuing to evolve by getting rewired with new types of cells. The discovery that gene expression can reveal these kinds of important distinctions between neurons is also prompting researchers to rethink how they define some brain regions and to reassess whether some animals might have more complex brains than they thought.

Active Genes in Single Neurons

Back in the 1960s, the influential neuroscientist Paul MacLean proposed an idea about brain evolution that was wrong but still had a lasting impact on the field. He suggested that the basal ganglia, a grouping of structures near the base of the brain, were a holdover from a “lizard brain” that evolved in reptiles and was responsible for survival instincts and behaviors. When early mammals evolved, they added a limbic system for the regulation of emotions above the basal ganglia. And when humans and other advanced mammals arose, according to MacLean, they added a neocortex. Like a “thinking cap,” it sat at the top of the stack and imparted higher cognition.

This “triune brain” model captivated the public imagination after Carl Sagan wrote about it in his 1977 Pulitzer Prize-winning book The Dragons of Eden. Evolutionary neuroscientists were less impressed. Studies soon debunked the model by showing conclusively that brain regions do not evolve neatly one on top of another. Instead, the brain evolves as a whole, with older parts undergoing modifications to adapt to the addition of new parts, explained Paul Cisek, a cognitive neuroscientist at the University of Montreal. “It’s not like upgrading your iPhone, where you load up a new app,” he said.

The best-supported explanation for the origin of new brain regions was that they evolved mostly by duplicating and modifying preexisting structures and neural circuits. To many evolutionary biologists, such as Harvey Karten of the University of California, San Diego, the similarities between the mammalian neocortex and the reptilian DVR suggested that they are, in evolutionary terms, homologous—that they both evolved from a structure passed down from an ancestor shared by mammals and reptiles.

But other researchers, including Luis Puelles of the University of Murcia in Spain, disagreed. In the development of mammals and reptiles, they saw signs that the neocortex and the DVR took shape through completely different processes. This hinted that the neocortex and DVR evolved independently. If so, their similarities had nothing to do with homology: They were probably coincidences dictated by the functions and constraints on the structures.