How to Review Linguistic Problems

The process of reviewing a problem submission has 4 main steps (not necessarily done in this order):

  1. Perform a linguist review, which establishes that all data used in the problem are accurate
  2. Write a solution path, which establishes that the problem is solvable and has exactly one solution
  3. If the problem has any issues that prevent the above two steps from being completed, revise the problem so that it is accurate and solvable
  4. (Optional) Revise the problem in ways that make it more fun or engaging

Thus, your job is not only to review the problem but also to revise it to fix any problems that you find when reviewing it. However, in some cases you might decide that a problem has unfixable flaws, or that its flaws would require too much work to fix; in such cases, you would not produce a revised version of the problem but instead write down the reasons for why the problem is not worth pursuing any further.

Below we provide guidance on how to carry out each of these steps. We know that this document is long, but please read it in its entirety; the information here is important for ensuring that you are reviewing problems in the most effective manner.

One point to reiterate: You do not need to carry out these steps in the order they are listed above. In practice, you are likely to end up doing all four of them roughly simultaneously. That's fine—the goal is just to end up with everything that we need (namely, a usable problem, a linguist review, and a solution path), and it doesn't matter what path you took to get there.

Performing the linguist review

NACLO problems usually include linguistic data, such as words or sentences in the language that the problem focuses on. It is important for the data to be accurate. If there are mistakes, that’s an issue for several reasons: it can be disrespectful to the speakers of the language, it can mislead people solving the problem, and it can hurt NACLO's image.

Therefore, one of your roles when vetting the problem is to check the accuracy of the data. There are several ways that we can satisfy ourselves that the data are accurate:

  1. Author expertise: If the author is a native speaker of the language, or a linguist who is an expert in the language, then we can ask them “Are you confident that all the examples in the problem are accurate?” If they say yes, that is sufficient verification.
  2. Verification with a reliable source: If every example in the problem appears in its entirety in a reputable source, then we can also be confident in the accuracy of the problem.
  3. Careful analysis of the rules of the language: Often, problem authors do not use the exact examples that appear in sources. Instead, they will generate new examples (e.g., they might read the grammar rules described in a scholarly source and then apply them to produce a new example). This is completely fine to do, but it can also be error-prone. Therefore, if the author has done this, we need to carefully check all of their examples by looking into the relevant sources and making sure that the examples are consistent with the source.
  4. Verification with a native speaker: If we are not certain of the sources, it can be very productive to consult with a native speaker of the language, since native speakers are the ultimate authority. However, when we do this, it is important to be courteous in our interactions with native speakers - so before reaching out to any native speakers, please consult with the rest of the PC first.

Whichever of these situations applies, you should write down your conclusions about the accuracy of the data in the "pc_comments" document associated with the problem. Depending on which scenario applies, this could be very brief or very lengthy. Specifically, if Situation 1 or Situation 4 applies, you can merely state something like "[INSERT NAME] is a linguist who is an expert in the language, and they have verified the accuracy of the data." However, if Situation 2 applies, you should state which source(s) you used and where each example appears; e.g., ""All of the examples are taken directly from BOOKTITLE. Specifically, they are example 17 on page 93, example 2 on page 111, example 45 on page 113, and examples 1 through 7 on page 248." And if Situation 3 applies, you should specify which source(s) you used, which rules/data you took from those sources, and where those rules/data appeared; e.g., "The problem is based on a grammatical phenomenon described on pages 93 to 95 of BOOKTITLE. Several of the examples are taken from that passage (specifically, examples 1 through 5 in the problem correspond to sentences 3 to 7 on page 94 of the book). To round out the problem, I constructed six additional examples by applying the rules on pages 95 and 113 to vocabulary items from TITLE OF DICTIONARY. I simplified the rules slightly by ignoring the intervocalic voicing rule described at the bottom of page 95."

Here are some additional clarifications:

Whatever you end up concluding from your linguist review, make sure to write it in the "pc_comments" document associated with the problem, so that no one else needs to duplicate the work you have done!

Writing a solution path

One requirement for each problem that we release is that it must have one and only one solution. The purpose of a solution path is to verify that this is the case.

The solution path is like a proof that starts with the provided data and uses it to establish the solution to the problem, step by step. As a simple example, consider this problem:

A solution path for this problem might look as follows:

There are a couple of points to note about the solution path:

Note that it can be acceptable for a problem to have multiple solutions in some cases; e.g., maybe the language has multiple ways to express something, meaning that there may legitimately be multiple ways to translate an English sentence into the language of focus. In that case, the problem should say something like "there may be multiple translations; you only need to provide one."

If it would be helpful to see more examples of solution paths, feel free to ask Tom! When you make a solution path, make sure to add it to the "pc_comments" document associated with the problem. (Or, if it gets very long, you can also create a new document for it in the Google Drive folder associated with the problem).

In addition to checking the accuracy of the data and the solvability of the problem, there are a few more factors to look out for. If the problem includes any of the following, it may need to be revised to avoid the issue, or it may need to be rejected if the issue cannot be avoided. A reason to be aware of these points is that some of them can require substantial revisions; for example, if you need to replace some of the vocabulary in the problem, the replacements might change the problem's logic, therefore requiring you to redo the solution path:

Revising the problem to overcome flaws

If the problem has any flaws, part of your job is to revise it so in order to overcome those flaws. Specifically, if there are any flaws that make the data inaccurate (as revealed by the linguist review) or that lead to multiple possible solutions (as revealed by the fact that you cannot create a solution path), then these issues need to be fixed.

There is no specific formula to follow for how to fix flaws in a problem; figuring out how to do it is often like solving a puzzle in its own right! Here are some strategies that can often be effective:

In some cases, it may be impossible to overcome the problem's flaws. For example, maybe we really need more examples, but the literature just doesn't contain any suitable examples. In those cases, we will have to reject the problem—which is too bad, but does happen sometimes.

Revising the problem to make it more fun or engaging

One of our goals is to make our problems engaging; we want solvers to come away from the contest feeling happy and excited about linguistics. To help with this goal, as you think about the problem, think about whether it is currently enjoyable, and about whether there is anything you can do to make it enjoyable (or more enjoyable, if it is already enjoyable!)

As in the previous section, there is no single formula for how to do this, but here are some ways that you can try to make a problem more engaging:

General guidelines

Beyond the procedures outlined above, here are a few more important points to keep in mind: