There has been plenty of talk about AI taking over the world, but the truth is, we remain far from that point. Today, AI still does not have the common sense to understand human language. Natural language processing has made great strides recently, but how much does AI truly understand of what it reads? Less than we think, but there is some hope.

Helping AI to Understand: the Importance of Knowledge

To read and understand a word or phrase, you must not only know the language in which it is written (including grammatical rules, structures and conventions), but have a basic understanding of the topic being referenced. Consider the following sentence: “In 1998, a Hail Mary pass by the Vikings’ QB helped them defeat their archrival, the Bears.” You may be able to read it, but it will not make sense if you have no knowledge of American football.

So, when researchers began to develop software that could read text, they sought to replicate the same approach. Not only did they seek to embed as many linguistic rules and structures as possible into their AI engine, but they sought to enable access to the domain knowledge covered by the text.

Research advanced in this direction for many years, despite technological limitations. However, once computers became more powerful and less expensive to operate, attention largely shifted to machine learning and deep learning.

ML and Deep Learning Alone Are Not Enough

Under machine learning and deep learning, words are reformatted into “numbers” by simply applying statistics, mathematics and some brute force. Data scientists hoped they could leverage it to “teach” a computer to read and understand a document. And while great strides have been made in fields such as image recognition, we still don’t have comparable results for text understanding.

While some believe it’s just a matter of time, there is growing scepticism that this is the case, even amongst those who originally supported the approach (i.e., Geoff Hinton, Riza Berkan). The reason for this is many scenarios do not fit the specific criteria (i.e., narrow domain, abundant sample data and automation of simple tasks) necessary for success – making it nearly impossible to obtain relevant results.

In the meantime, information overload has made the benefits of machine reading and understanding increasingly visible and relevant in all fields (e.g., business, government, daily life, military, etc.). As a result, we are seeing a renewed interest in more traditional approaches based on a human-like understanding of language that leverages a rich knowledge base.

How to Improve Your AI Performance

It is clear that machine learning techniques can add value to your business. However, to improve AI-based natural language processing capabilities, we must start with an honest discussion about the limitations of this approach. In addition, we cannot run away from the inherent complexities we face when reading and understanding text.

Tasks that are complex, yet essential to human comprehension of language (e.g., building a reliable knowledge base and leveraging that knowledge to understand text), are equally complex and essential to the software replicating these actions. Thus, investing both time and effort into building a vast and deep knowledge base will undoubtedly improve performance, whether for a human or for software.

The Impact of Knowledge Graphs

In the software world, when we talk about knowledge, we tend to refer to a knowledge graph. This is a representation of the real world where concepts are defined and connected to one another by different relationship types.

Knowledge graphs can be wide or narrow, depending on the breadth of the domains they cover. They can be deep or shallow, depending on how they represent the knowledge of a certain domain from its core to its more specific elements. In addition, they are usually open and explicit, rather than a black box. This means their content and structure can be understood and directly manipulated by humans.

In a knowledge graph used to understand language, each item represents a concept and is linked with one or more other items (links represent the relations between them). Each item then has a set of attributes that describes the characteristics of words and concepts. The more expansive the knowledge graph, the better it understands the language. The better it understands the language, the more useful it becomes.

Why Knowledge is King

To begin, a knowledge graph with the aforementioned characteristics enables any natural language software to start with a higher level of accuracy. As such, the software requires less work to reach the level of performance required to be useful. This is true for any NLP software, including those based purely on machine learning.

In addition, every domain frequently introduces new concepts and knowledge. An NLP software with a robust knowledge graph can maintain its level of performance by easily and incrementally expanding the knowledge graph. This is made possible by subject matter experts who understand the structure of the repository, enabling them to easily identify where to add the new concepts.

By contrast, humans cannot understand how an ML-based NLP system works. The process is obscured in a black box. The only way to make it work is by retraining it. This means going back to square one every time there is a change to how you want the model to perform.

A Hybrid Approach Offers Best Value and Results

At this point, it should also be clear that an approach that combines the benefits of ML (e.g., simplicity, full use of the brute force made available by current cheap computer power) with more traditional systems (e.g., based on engines that rely on a rich knowledge graph) offers many practical and pragmatic advantages.

As the accurate comprehension of textual content always requires domain-specific knowledge, a knowledge graph that can be expanded upon based on the specific requirements of any use case can also offer great value.

Although a knowledge graph alone may not make AI suddenly and fully capable of understanding everything, it provides a base to build common sense into the core of any task.