Natural language processing of formatted documents

Abstract

Detecting and incorporating formatting characteristics within natural language processing analytics. Source documents are ingested and the markup formatting language is identified by the program. Once identified, the markup language is parsed and examined for formatting characteristics, embedded notes, comments and other metadata. The formatting characteristics of the plain text are extracted, along with the plain text, and converted into a common analysis structure (CAS), or CAS-equivalent structure, which annotates the natural language text together with its respective formatting characteristics. The CAS or CAS-equivalent structures are stored and sent to a natural language processing pipeline for further analysis via complex algorithms and rules. The natural language processing results data are curated to reflect meaningful analysis of the extracted CAS or CAS-equivalent structure.

1 . A computer-implemented method for processing text, the method comprising:
 determining, by a computer, that a natural language text is associated with one or more formatting characteristics, wherein the natural language text is unstructured;   applying, by the computer, optical character recognition (OCR) to the natural language text associated with the one or more formatting characteristics, wherein
 identifying, by the computer, that a span of the natural language text is bold type by comparing the pixel thickness of the characters of the span of the natural language text to an average pixel thickness of the natural language text; 
 identifying, by the computer, that a span of the natural language text is italics type by analyzing the angle of the pixels of the characters in the span of the natural language text; 
 identifying, by the computer, that a span of the natural language text is underlined by analyzing the number of pixels in a consistent horizontal line underneath the characters in the span of the natural language text; 
 identifying, by the computer, that a span of the natural language text is a subscript by recognizing that a numerical character is located slightly below the span of the natural language text; 
   generating, by the computer, a data structure for storage in memory comprising at least one of the one or more formatting characteristics, and a corresponding span of the natural language text;   transmitting, by the computer, the generated data structure comprising the at least one of the one or more formatting characteristics and the corresponding span of the natural language text to a natural language processing (NLP) pipeline to identify an intent of the corresponding span of the natural language text;   associating, by the computer, the one or more formatting characteristics with one or more actions, based on the intent of the corresponding span of the natural language text, the one or more actions comprising any one of:
 emphasizing the span of natural language text associated with a bold type formatting characteristic; 
 emphasizing the span of natural language text associated with an italics formatting characteristic; 
 emphasizing the span of natural language text associated with an underline formatting characteristic; 
 categorizing as a chemical formula the span of natural language text associated with a subscript formatting characteristic; 
   performing, by the computer, the one or more actions associated with the one or more formatting characteristics; and   incorporating the generated data structure into a machine learning model that learn from and makes predictions on natural language text data.