Unravelling the secrets of natural language processing

What is natural language processing?

NLP focuses on the interaction between computers and human language. It involves the ability of a computer system to analyse, interpret and generate human language in a way that is both meaningful and useful. NLP utilizes various machine learning tools, techniques and algorithms, as well as symbolic AI, to enable machines to comprehend and process natural language data, including text and speech.

By employing statistical models, machine learning and linguistic rules, NLP enables computers to perform tasks such as sentiment analysis, text classification, machine translation, chatbot development, and more.

Подпишитесь на нашу рассылку

Будьте в курсе новостей об искусственном интеллекте и связанных с ним стандартах!

Подписаться *

Almost done! 
You are only one step away from joining the ISO subscriber list. Please confirm your subscription by clicking on the email we've just sent to you. You will not be registered until you confirm your subscription. If you can't find the email, kindly check your spam folder and/or the promotions tab (if you use Gmail).

* Информационный бюллетень на английском языке

How your data will be used

Please see ISO privacy notice. This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

How does natural language processing work?

There are multiple steps along a machine learning pipeline that enable common NLP tasks. These components of NLP work together to facilitate a comprehensive understanding of human language by machines.

First is data preparation, pre-processing, reduction, indexing and encoding. The data in this case is text, which could come from a website, multiple websites or other sources. This step involves:

• Data cleaning – for example, writing a Python script to extract the text from the html of a website
• Tokenization – breaking text down into smaller units such as individual words, known as tokens. This is why you read or hear about tokens in the context of Generative AI Large Language Models (LLMs)
• Part-of-speech tagging – the process of identifying word categories, such as nouns, verbs and adjectives

Once the data has been pre-processed, a machine learning algorithm can be developed to train NLP models. This requires feeding the software with large data samples to increase the accuracy.

It can then be used to complete tasks that break down text or speech that computer programs can more easily understand, including syntax (the arrangement of words), semantics (meaning of words and sentences), pragmatics (contextual meaning) and discourse (how sentences connect in text).

Natural language processing tools

So what are some of the key tools and technologies used in NLP? Let’s look at some NLP examples.

One commonly used tool for NLP is the Natural Language Toolkit (NLTK), an open-source module built using the popular Python programming language. Thankfully, you don’t need to be an expert coder to do natural language processing with Python. Tools such as NLTK contain libraries of datasets and tutorials and provide pre-built functions and models that can be incorporated into common NLP tasks and subtasks, such as tokenization and semantic reasoning, which is the ability to reach logical conclusions based on facts extracted from text.

What is natural language processing used for?

NLP is now being used in a wide variety of everyday applications and is finding use in industries such as healthcare and finance. Here are some of the most common NLP applications and where you may have encountered natural language processing in AI:

• Chatbots and virtual assistants: AI-powered applications, such as Siri and Alexa, use NLP techniques to interact with users through natural language conversations.
• Language translation: NLP models can be trained on vast amounts of bilingual data, enabling them to accurately translate text while considering grammar rules and contextual nuances.
• Search engines: Question answering systems, such as search engines, utilize NLP algorithms to understand questions posed by users and provide relevant answers. These systems analyse the question’s context, identify key information, search for relevant documents or knowledge bases, and extract precise answers to satisfy user queries.
• Email filtering: Many people will recognize the pain of an inbox with a huge amount of unread emails. NLP is used to filter emails into different categories. Top-of-the-line spam detection technologies use NLP’s text classification capabilities to scan emails for language that indicates spam or phishing.

NLP has also become an indispensable tool in various industries, revolutionizing the way we interact with technology:

• Healthcare: NLP is crucial in the healthcare industry as it allows for efficient analysis of medical records, patient data and clinical notes. This helps improve diagnoses, identify patterns, predict outcomes and enhance overall patient care.
• Finance: NLP plays a significant role in the finance industry by automating manual tasks like analysing financial reports, news articles and customer feedback. It enables sentiment analysis, fraud detection, risk assessment and personalized financial recommendations.
• Customer service: NLP is essential for customer service departments as it enables chatbots and virtual assistants to understand and respond to customer queries in a timely manner, enhancing customer satisfaction and reducing support team workload.
• E-commerce: NLP is used in e-commerce for various purposes such as product recommendations based on user preferences and browsing history. It also helps in sentiment analysis of customer reviews to understand their opinions about products or services.
• Legal: NLP aids legal professionals by automating tasks like contract analysis and legal document review, saving time and effort.
• Education: NLP is beneficial in education as it enables intelligent tutoring systems that personalize learning experiences for students.
• Human resources: NLP helps human resource departments with tasks like resumé screening, candidate matching and sentiment analysis in employee feedback.

Challenges and limitations of natural language processing

As with any complex field, NLP comes with its fair share of challenges. The computational complexity of NLP tasks can be a significant limitation, for instance. Processing large amounts of text data requires substantial computer power and time, making it challenging to achieve real-time or near-real-time analysis. Improving the efficiency and speed of NLP algorithms is another ongoing challenge. However, challenges like these and the ones listed below also present exciting opportunities for innovation and growth.

• Limited contextual understanding and memory: NLP models often struggle to interpret or retain the meaning of words or phrases based on the context in which they are used. This can lead to misinterpretations or incorrect analysis of text data.
• Ambiguity and polysemy: Many words and phrases have multiple meanings, making it difficult for NLP models to accurately determine the intended use in a given context. This can result in inaccurate analysis or miscommunication.
• Language variations and idioms: The vast diversity of languages and their regional variations – with different dialects, idioms, slang and colloquialisms – make it challenging for NLP models to analyse and interpret text accurately across different linguistic contexts. Researchers are working to continuously update models and adapt to evolving language.
• Lack of common-sense reasoning: While humans can infer implicit information from text using their general knowledge and common sense, NLP models often lack this capability. This hinders their ability to comprehend nuanced texts or make accurate predictions based on implied information.
• Data quality and bias: The quality of data used for training NLP models plays a crucial role in their performance. Biased or incomplete datasets can lead to biased results, reinforcing existing societal biases or stereotypes.
• Ethical and privacy concerns: With the increasing use of NLP in various applications, ethical and privacy concerns have emerged. Issues such as data privacy, security and potential misuse of NLP technology raise important questions regarding responsible development and deployment of NLP systems.

Taming an unruly beast

As NLP models become integral to critical sectors like healthcare, finance and transportation, ensuring their safety, reliability and ethical use will be essential. International Standards offer a framework for consistency and quality across multiple uses, including diverse NLP applications. The creation of standards dedicated to AI, such as those developed by the group of experts in ISO/IEC JTC 1/SC 42, highlights ISO’s commitment to ensuring that AI technologies are developed and used responsibly and effectively.

• ISO/IEC 42001:2023AI management systems
• ISO/IEC 23894:2023AI — Guidance on risk management

As part of its expanded AI work programme, a joint effort is underway on natural language processing systems in collaboration with ISO/TC 37, the expert committee on language and terminology. This initiative benefits from a diverse range of AI expertise, covering both spoken and written language and involving a variety of stakeholders from around the world. The expansion of ISO’s programme of work reflects the importance of International Standards as a solution to enable responsible adoption.

The future of natural language processing

NLP stands on the brink of redefining digital communication, enhancing our ability to communicate not only with computers but with each other. Its future promises further integration with other AI fields, enhancing its capabilities. For example, the rise of neural networks in NLP is changing the way search works. Where results used to be served up from a database, neural networks now search for and serve the most relevant results to you based on your history of interaction. This will become even more accurate over time.

However, addressing legitimate concerns is crucial to ensure that this technology benefits humanity. If we can do this, through rigorous standards that are established and enforced, then NLP can help foster a future where AI and human intelligence work in harmony for collective advancement.