Neuroscience is at a crossroads, and the path we choose could define the future of our understanding of the brain. Are we building models so complex that they become useless, like a map the size of the kingdom it represents? Over the past 25 years, artificial neural networks have grown from modest 60,000-parameter models like LeNet in 1998 to behemoths like Llama 3, boasting 70 billion parameters in 2024. While these large models have revolutionized tasks like image recognition and provided valuable tools for studying animal brains, their sheer scale raises a critical question: Are we sacrificing interpretability and practicality for the sake of size?
But here's where it gets controversial: What if the key to unlocking the brain's mysteries lies not in larger models, but in smaller, more manageable ones? So-called "toy models," often consisting of just two to three neurons, offer a highly interpretable and practical framework for studying neural circuits. Despite being dismissed by some as oversimplifications, these tiny models have a storied history in neuroscience. In 1943, Warren McCulloch and Walter Pitts used such models to demonstrate how neural circuits could implement logical functions, laying the groundwork for our understanding of neural computation. Decades later, Anthropic employed similarly small networks to explore the phenomenon of superposition, where neurons respond to multiple unrelated inputs. These examples highlight how toy models can serve as powerful tools for developing theories and benchmarking methods.
And this is the part most people miss: While large models struggle with interpretability, toy models excel in it, often while remaining amenable to the same manipulations and analyses. For instance, recent studies have shown that tasks traditionally used in psychophysics—like binary decision-making or responding to sensory stimuli—can be solved by tiny artificial neural networks with just one to four neurons. This raises a provocative question: If toy models can solve toy tasks, is it truly useful to study complex models performing the same simplified experiments?
Consider the case of Centaur, a large language model with over 70 billion parameters, used to model human behavior in simple psychophysics tasks. While Centaur is approximately 10 million times larger than a roundworm and 1,000 times larger than a fruit fly in terms of neuron-to-neuron connections, it’s debatable whether such scale is necessary for understanding behaviors that even these tiny organisms exhibit. Following Jorge Luis Borges’ parable, Centaur risks becoming a map larger than the kingdom itself—impressive in detail but impractical in use.
So, what can we learn from these large models? Proponents argue that their complexity makes them ideal for developing experimental methods and analysis tools that can later be applied to real brains. However, without a ground-truth understanding of how these models work, it’s difficult to determine if the insights gained are meaningful. Toy models, by contrast, offer clarity and precision, allowing researchers to develop and test methods in a controlled environment before scaling up. For example, a study using a tiny neural network with a single hidden neuron demonstrated that silencing individual neurons or connections fails to provide an accurate understanding of network function. The researchers proposed a multi-lesion method, which was later successfully applied to a 56-billion-parameter model, underscoring the value of starting small.
But the debate doesn’t end here. Are toy models truly all neuroscience needs? There are phenomena that emerge only at scale, such as the hierarchical feature representation in deep visual networks. And as neuroscientists tackle more naturalistic, complex tasks, toy models may fall short. Yet, even in these cases, the principle of Occam’s razor applies: our models and theories should be as simple as possible while remaining effective. If our goal is to understand the brain, not just create intricate maps, we must prioritize creativity and conciseness in our toy models.
So, I leave you with this thought-provoking question: In the quest to unravel the brain’s complexities, are we better served by starting small and scaling up, or by diving headfirst into the depths of massive models? Share your thoughts in the comments—let’s spark a conversation that could shape the future of neuroscience.
