Businesses using natural language processing want to determine who is talking, what they’re talking about, how they feel, and why they feel that way. The last element concerns context, which is one of the biggest challenges to overcome when training a viable NLP model. Without clear context, we humans sometimes struggle to understand what people are talking about. It’s even harder for machines.
Consider sarcasm, for example. If someone says, “That’s great. You’re a genius,” we might know the speaker is sarcastic by their level of connection with us, tone of voice, and the context of the conversation. We can train machines to detect sarcasm, but with so many nuances at play—culture, gender, personality, situation, history—a human may have to map words and context in a model.
The NLP context conundrum does not stop with sarcasm. Language is constantly evolving; the meanings of words change all the time. For example, Northwest is no longer just a direction— North West is the daughter of Kanye West and Kim Kardashian, the American celebrities. The word ‘feed,’ used as a noun, no longer refers only to the stuff horses eat; it also refers to where you might land when visiting your favorite social media platforms. Wholly new words are continually entering languages, too. Consider ‘box-off, negging, and zinester’—words added to the Oxford English Dictionary in early 2021.
The overarching challenge of NLP is to turn natural language, whether written or spoken, into mathematical values a machine can understand. This act requires distilling the many nuances and complexities of natural language into numbers or, for instance, turning things like emotions into mathematical operators. This work is no easy task, especially when training and building models with high enough accuracy rates to be viable.
Searching beyond keywords with semantic analysis
NLP is rapidly advancing in its ability to understand context, thanks to the vast and constantly growing amount of training data available. This data deluge allows machines to conduct semantic analysis to determine the context and meaning behind words and phrases.
For example, most people are now familiar with Google Search. In the old days, search engines relied strictly on keyword-based searches, which meant search results often failed to align with a user’s actual intent. The old system was also easy to abuse, with unscrupulous advertisers taking advantage of its simplicity by stuffing keywords into their website content or hiding keywords on a page by matching text and background colors.
More advanced users might be familiar with the Boolean operators AND, OR, and NOT, which all search engines still support. But adding operators to search queries is hardly a user-friendly experience. And because nearly half of all internet users now search the web using voice queries, keyword-based search and Boolean operators are becoming less practical, as they don’t map to the way people speak.
Thanks to advancements in artificial intelligence, human-to-computer interfaces are starting to offer more seamless and natural communications between people and machines. When it comes to searching the web, we now rely on keyword search plus semantic search. Semantic analysis uses NLP to determine context or, in other words, the user’s intent.
User intent often lies within information and concepts not explicitly mentioned in data, whether a written search query, a voice recording, or an email message. Here’s an example:
Your customer service is a joke. I’ve been on the blower for half an hour!
While most people can quickly pick up on the customer’s negative sentiment, a machine might have trouble doing so. Is the word ‘joke’ meant as a positive or negative? Is ‘blower’ an electric fan, hairdryer, or large aquatic mammal? In fact, ‘blower’ is a British-English slang term for ‘phone,’ which also illustrates another challenge in NLP—understanding a myriad of regional slang terms in natural language.
We’re training our systems to read better and modify better, all in the quest to interpret better. Even so, if we’re training an NLP model designed to assist with customer service, we must annotate the previous example so the machine can extrapolate context. In this case, the context is a disgruntled customer—probably based in the UK given their choice of words—unleashing their frustration on a chatbot.
How does semantic analysis work?
To overcome the NLP context conundrum, you often need an ample supply of labeled training data. It might be original text or transcribed voice recordings, but the same rules apply in semantic analysis. The teams handling your data annotation must understand the relationships between lexical terms on the word, phrase, and sentence level to ensure a high degree of accuracy and quality control.
Automated semantic analysis can work, but only after the model receives enough training data to achieve acceptable accuracy rates. In the meantime, it’s up to your data annotation team to exhaustively label the data in categories, such as synonyms, antonyms, hyponyms, and homonyms. By feeding semantically enhanced data into the NLP algorithm, you can train the machine to make more accurate predictions and progressively reduce error rates over time.
Machines may never reach the level of contextual accuracy we humans provide. Language will continue to evolve, and we will always need humans in the loop to train and refresh NLP algorithms. When the context conundrum stumps your models, our managed workforce will be here, ready to provide clarity and handle exceptions.
Our natural language processing services help you scale text and audio annotation by pairing you with an experienced, managed workforce that can understand and interpret complex, nuanced language.
