Neural networks are remarkably successful, but still fail on tasks which are very easy for humans, like understanding simple regularities such as 10101010... or aaabbb..., and learning basic arithmetic operations like addition and multiplication.

To make networks generalize better, we replace standard objectives with a computable version of Kolmogorov Complexity, the Minimum Description Length principle (MDL), which balances the network's architecture size with its accuracy.

Using an evolutionary architecture search guided by MDL, we find small and perfect networks that can handle tasks which are notoriously hard for traditional networks, like basic addition, and recognition of formal languages such as Dyck-1, aⁿbⁿ, aⁿb²ⁿ, aⁿb^mc^n+m, and aⁿbⁿcⁿ. MDL networks are very small, and often contain only one or two hidden units, which makes it possible to prove that they are correct for all strings. No other neural network that we know of has been proven do to that.

Modern language models are trained on huge corpora that amount to years or even lifetimes of human linguistic experience. Can we use this fact to learn about the initial state of a human child acquiring language?

Building on work by Wilcox et al. (2022), we examine the knowledge of state-of-the-art language models, including GPT-j and GPT-3, regarding important syntactic constraints. We find that all models fail to acquire an adequate knowledge of these phenomena, delivering predictions that clash with the judgments of human speakers. Since these models are trained on data that go above and beyond the linguistic experience of children, our findings support the claim that children are equipped with innate linguistic biases that these models don't have.