Testing and Debugging Ontologies

I'm a firm believer in using Agile methods for developing software. I'll never forget the first time I read Kent Beck's book Extreme Programming Explained. It was as if someone had finally put down in words the way I always tried to develop software. One of the principles of Agile is test-driven development.

Test-driven development applies to ontology design as well. When developing an ontology, one of the most common errors that I see new users commit is to spend all their initial work developing the classes, properties, and axioms while leaving the individuals (the test data for an ontology) until the end. This seems natural but it can make debugging the ontology much more difficult. The problem is that the reasoner will often only detect errors in your ontology when there are individuals. If you create a complex ontology with many axioms and only add individuals as the last step there may be many errors that are only detected by the reasoner once individuals are defined. When there are multiple inconsistencies in an ontology the feedback from the reasoner is often much less informative and rather than having the reasoner focus on the specific classes or properties causing the problem you get a very generic error message that is of little value. Here are some best practices I use for taking a test-driven approach to ontology design:

1. Run the reasoner frequently. I usually run it after every change that could impact reasoning.

2. Define individuals to test your ontology at the same time as you develop your ontology.

3. Make all your test individuals instances of a class called TestData or something similar. That way you keep them separate from real data and can easily delete them when you go to load real data.

4. Make sure that every leaf class (a class with no subclasses) has at least one instance.

5. Make sure that each property has at least one instance with values.

6. Use an ontology evaluation tool. These tools can find anti-patterns such as inconsistent naming schemes and many more. Two excellent tools are OOPS! and (for those who follow the OBO standards) ROBOT.

7. Postpone adding Disjoint With axioms until you have added individuals, run the reasoner, and inspected the results. A common problem is that an individual will be assigned to an instance of some class that the designer didn't intend as a result of some domain or range definition. It is easier to find these errors by inspecting your individuals to see if the reasoner has asserted them to be instances of classes you didn't expect than to have the reasoner make the entire ontology inconsistent due to violations of a disjoint axiom.

8. Don't add axioms for every possible condition in the ontology. For example, if the domain for hasFather is Person it typically isn't needed to add an axiom to the Person class such as hasFather some Person. Only add axioms if they satisfy some use case or story or provide some value added to differentiate individuals or automatically recognize an individual via a defined class. Every axiom adds complexity and the reasoner can become slow when you start to scale up with real data if you have superfluous axioms.