Universal linguistic inductive biases via meta-learning

Tom McCoy, Erin Grant, Paul Smolensky, Tom Griffiths, Tal Linzen

This post accompanies the paper found here and the code found here.

Introduction

Language acquisition involves a complex interplay between the data and the learner. The importance of data is clear: we can only learn a language if we have experience with it. Less obviously, acquisition is also guided by properties of the learner called inductive biases, which determine how the learner will generalize beyond the utterances they have encountered. As an example, consider the following pattern:

• gbg → .ga.ba.ga.
• zz → .za.za.
• trv → ???

What output should replace the question marks? You probably answered .ta.ra.va. even though other answers are conceivable; e.g., the provided outputs are consistent with a rule that involves reversing the input, so the answer could instead be .va.ra.ta. If you answered .ta.ra.va. instead of .va.ra.ta., it suggests that you have an inductive bias for preserving the input order.

Which inductive biases enable humans to acquire language? The answer is hotly debated. To facilitate computational modeling aimed at answering this question, we introduce a framework for giving particular linguistic inductive biases to a model. Such a model can then be used to explore the effects of those inductive biases, and to see which biases yield the most human-like generalization behavior.

In our framework, the inductive biases are encoded in the initial state of a neural network. This initial state is found with meta-learning, a technique through which a model discovers how to acquire new languages more easily via exposure to many possible languages. By controlling the properties of the languages that are used during meta-learning, we can control the set of inductive biases that meta-learning imparts.

Try it out!

To demonstrate this framework, we use the linguistic domain of syllable structure. For a given language, the model must learn how to map a sequence of sounds (e.g., tarava) into a sequence of syllables (e.g., .ta.ra.va.). Each language has restrictions on what types of syllables are allowed, and sounds may need to be inserted or deleted to meet these restrictions. For example, if a language requires all syllables to end with a vowel, the input kep might map to .ke.pa. (if the language uses insertion) or .ke. (if the language uses deletion).

The demo below shows how a neural network initialized with meta-learning can learn syllable structure mappings much more quickly than a neural network with a standard random initialization. This demo trains the models in your browser using randomly-generated data, so none of the results that you see are cherry-picked.

The meta-initialized model can typically learn any of these syllable structure mappings from fewer than 100 examples—sometimes substantially fewer. In contrast, after the same number of examples, the outputs of the randomly initialized model are usually nowhere near the correct answers.

In the class of languages that we include in our experiment, certain inputs will map to the same output across all languages; such mappings are therefore absolute universals, or properties that all languages possess. For example, an input of the form CVCV will always map to .CV.CV. (where C stands for any consonant, V stands for any vowel, and periods indicate word and syllable boundaries). Even though some languages allow syllables of the form .CVC. and .V., the input CVCV will only ever map to .CV.CV., never to .CVC.V. The meta-initialized model typically gets the correct output for such inputs without any training in the language, indicating that the meta-learning process has imparted these absolute universals. You can test these absolute universals by clicking "Show advanced settings" in the demo.

How many examples does the randomly-initialized model need before it begins performing well? The table below should give some sense of this; it contains data from one language for each of the 8 abstract categories of languages that we use in our experiments. On these languages, the randomly-initialized model typically needs a few thousand examples, compared to a few dozen for the meta-initialized model.

Generalization

The main goal of our framework is to give a model a certain set of inductive biases. While the ability to learn from a small number of examples is one piece of evidence that a model has acquired some useful inductive biases, a more direct way to study a learner's inductive biases is through generalization: how does the learner handle novel types of examples?

The figure below shows how meta-learning facilitates three types of generalization: generalization to a novel length, generalization to novel symbols, and generalization to novel input structures (we refer to examples of the last category as implicational universals). In each case, we train our model on a training set which has certain types of examples withheld, then evaluate it on the withheld class of examples. Click the tabs above the figure to explore these three types of generalization.

Across all three cases, the meta-initialized model generalizes well while the randomly-initialized model generalizes poorly. These results indicate that meta-learning has successfully imparted a bias for the length-invariance of syllable patterns, a bias for assuming two universal classes of symbols (namely, consonants and vowels), and a bias for associating certain mappings with certain other mappings (a.k.a. implicational universals).

Linguistic universals are typically divided into two categories: absolute universals, which are properties that all languages possess, and implicational universals, which are cases where the presence of one property in a language implies the presence of another property. The previous section showed that meta-learning had instilled some absolute universals in the model, while this section shows that it has also instilled some implicational universals. Thus, meta-learning can impart both of the major categories of linguistic universals.

See the paper for more discussion of these experiments and for investigation of three other inductive biases using an alternate technique, namely ease of learning (that is, which types of languages are easiest for a model to learn?)

How does it work?

Our method can be summarized as follows:

1. Determine the set of linguistic inductive biases that you wish to impart to a model.
2. Define a space of languages encoding those inductive biases.
3. Have a model meta-learn from this space of languages; this process is intended to give the model an initial parameter setting from which it can easily learn any language in the space of languages you have defined.
4. Analyze the model to verify that meta-learning has given it the intended inductive biases.

The core component of our method is meta-learning; the particular type of meta-learning that we use is MAML. In MAML, a model is exposed to a variety of tasks, each of which comes with a limited amount of data. After exposure to each task, the model's weights are adjusted so that, if it were taught the same task again, it would perform better. As MAML proceeds, the model converges to an initial state from which it can readily learn any task in the distribution of tasks it has been shown. The demo below gives an example of MAML.

If meta-learning is successful, the model's initialization will encode some set of inductive biases that are useful for learning the space of tasks that the model is being shown. In the demo, the initialization that the model converges to encodes a bias for the slope to be close to -3 and for the intercept to be close to -2.

In this simple example, it would have been easy to set the model's parameters by hand in a way that encoded this bias. However, such hand-coding of biases is not practical with standard neural networks, which typically have far more parameters (often tens of millions of parameters, or even 175 billion parameters) which interact with each other in difficult-to-interpret ways. In such cases where manual parameter setting is impractical, meta-learning can automatically discover parameter settings that encode the desired biases.

In our application of meta-learning, each “task” is a language, and the goal is to find a set of initial parameter values that allow the model to quickly learn any language. We hypothesize that, if we carefully construct the space of languages used during meta-learning to encode a desired set of inductive biases, we can impart these inductive biases in order to study how they affect generalization behavior.

Conclusions and future directions

We have shown how meta-learning can impart inductive biases specified by the modeler. While the meta-learned biases are not as transparent as those encoded in probabilistic symbolic models, analysis of the model's learning behavior can be used to evaluate whether meta-learning has produced the desired biases, as we have shown. Our case study demonstrates that linguistic inductive biases that have previously been framed in symbolic terms can be reformulated in the context of neural networks, facilitating cognitive modeling that combines the power of neural networks with the controlled inductive biases of symbolic methods.

In future work, we plan to use this approach to study open questions in cognitive science. For example, by creating a model that has a particular bias which has been argued to be important for human cognition, we can empirically test whether the bias has the explanatory power that it has been argued to have. Alternately, while our case study involved giving a model certain pre-selected biases, this approach could instead be applied to naturally-occurring linguistic data to lend insight into the inductive biases that shaped this data. Finally, this approach might have applications in Artificial Intelligence: By giving targeted, cognitively-motivated learning biases to models, we may be able to decrease the gap between the learning capabilities of AI models and humans.

See the full paper for more details. If you have questions or comments, email tom.mccoy@jhu.edu.

*****

Some fine print about this website: The demos differ from the experiments in the paper in a few ways designed to decrease the computation load for the website. First, the models on the website are GRUs, while in the paper we used LSTMs. Second, training in the demos uses a batch size of 1 with stochastic gradient descent, while the paper uses a batch size of 10 with the Adam optimizer. Thus, the performance of the demo models is not guaranteed to be the same as the performance of the models used in the paper. Finally, on this page, we have excluded any examples containing lowercase L or capital i, as the similarity of these letters sometimes makes examples hard to read, but both letters were available in the experiments in the paper.