What is Natural Language Processing?

Natural Language Processing (NLP)

NLP is a field combining the power of computer science, computational linguistics and artificial intelligence to allow computers to understand, manipulate and analyse human natural language in a most human and useful way.

What is NLP and how can it help my organization’s coding process?

Introduction to NLP by Stanford University
Course Introduction – Stanford NLP – Professor Dan Jurafsky & Chris Manning

This domain first flowered with the publication of Alan Turing’s famous paper ‘Computing Machinery and Intelligence‘ from which the Turing test emerged. Turing affirmed that we can consider a computer as intelligent if it could hold a conversation with a human without the individual realising they were in fact interacting with a computer. For a comprehensive and hands-on introduction, refer to the Stanford Coursera course on NLP.

Levels of Linguistic Analysis

In order to accomplish this task, the computer would need to perform complex analyses on all linguistic levels shown in the diagram above. For more details see Handbook of NLP.

Phonetics and Phonology

This is the level of sounds. One must distinguish here the set of possible human sounds, which constitutes the area of phonetics proper, and the set of system sounds used in a given human language, which constitutes the area of phonology. Phonology is concerned with classifying the sounds of language and with saying how the subset used in a particular language is utilised, for instance what distinctions in meaning can be made on the basis of what sounds. ‘Seeing Speech’ – Ultrasound and MRI Images visualise sounds being produced.

Morphology

This is the level of words and endings, to put it in simplified terms. It is what one normally understands by grammar (along with syntax). The term morphology refers to the analysis of minimal forms in language which are, however, themselves comprised of sounds and which are used to construct words which have either a grammatical (walk#ed) or a lexical function (freeze#r). Lexicology is concerned with the study of the lexicon from a formal point of view and is thus closely linked to (derivational) morphology, i.e. lexical functions. For more information, see Aronoff and Fudemann ‘What is Morphology?’.

Syntax

This is the level of sentences. It is the theory of how expressions (words, phrases) are combined to form more complex expressions (phrases, sentences). concerned with the meaning of words in combination with each other to form phrases or sentences. In particular it involves differences in meaning arrived at by changes in word order, the addition or subtraction of words from sentences or changes in the form of sentences. It furthermore deals with the relatedness of different sentence types and with the analysis of ambiguous sentences (i.e. sentences which have multiple meanings).

A parser is an NLP tool which analyses syntactic structure. You can play around with the Stanford statistical parser: Stanford Parser Demo. Here’s a blog post on syntactic ambiguity in legal texts.

Semantics

This is the area of meaning. It might be thought that semantics is covered by the areas of morphology and syntax, but it is quickly seen that this level needs to be studied on its own to have a proper perspective on meaning in language. Here one touches, however, on practically every other level of language as there is lexical, grammatical, sentence and utterance meaning. Semantic Analysis demos (semantic relatedness, similarity and compositionality).

Pragmatics

The concern here is with the use of language in specific situations. The meaning of sentences need not be the same in an abstract form and in practical use. In the latter case one speaks of utterance meaning. The area of pragmatics relies strongly on the notion of speech acts which is concerned with the actual performance of language. This involves the notion of proposition – roughly the content of a sentence – and the intent and effect of an utterance. For example, the utterance “Could you please pass me the salt?” could be interpreted as a question (“Are you capable of passing me the salt?”), in which case it could be answered by “yes” or “no”. More commonly, however, this utterance is interpreted as a request and the expected response is an act, namely the passing of the salt to the person asking the question. Pragmatics can be a major source of cultural differences. ‘Why are you late? Cross-Cultural Pragmatic Study of Complaints in American English and Ukrainian‘.

Semantic/pragmatic ambiguity illustrated.

Semantic/pragmatic ambiguity illustrated.