Compression of word embeddings for natural language processing systems

Abstract

Described herein are systems and methods that provide a natural language processing system (NLPS) that employs compressed word embeddings. An auto-encoder that includes encoder circuitry and decoder circuitry can be used to produce the compressed word embeddings. The decoder circuitry is trained to decompress the word embeddings with reduced or minimal differences between the original uncompressed word embeddings and the corresponding decompressed word embeddings. One or more parameters of the trained decoder circuitry are transferred to the NLPS, where the NLPS is then trained using the compressed word embeddings to improve the correctness of the responses or actions determined by the NLPS.

1 . A system, comprising:
 an auto-encoder processing unit comprising:
 encoder circuitry; and 
 decoder circuitry operably connected to the encoder circuitry; and 
   a first storage device storing computer executable instructions that when executed by the auto-encoder processing unit, performs a method comprising:
 compressing, by the encoder circuitry, one or more uncompressed word embeddings to produce one or more compressed word embeddings for use in a natural language processing system; 
 decompressing, by the decoder circuitry, the one or more compressed word embeddings to produce one or more decompressed word embeddings; and 
 a second storage device storing one or more parameters of the decoder circuitry. 
   
     
     
         2 . The system of  claim 1 , wherein the auto-encoder further comprises activation function circuitry operably connected to the encoder circuitry and the operation of compressing the one or more uncompressed word embeddings comprises compressing, by the encoder circuitry and the activation circuitry, the one or more uncompressed word embeddings to produce the one or more compressed word embeddings. 
     
     
         3 . The system of  claim 2 , wherein the auto-encoder comprises a multi-layer neural network with the encoder circuitry comprising a first layer, the activation function circuitry a second layer, and the decoder circuitry a third layer. 
     
     
         4 . The system of  claim 3 , wherein the activation function circuitry comprises a non-linear activation function. 
     
     
         5 . The system of  claim 4 , wherein the encoder circuitry comprises a first linear transformation circuit. 
     
     
         6 . The system of  claim 5 , wherein the decoder circuitry comprises a second linear transformation circuit. 
     
     
         7 . The system of  claim 3 , wherein the encoder circuitry comprises one or more parameters that are randomly initialized. 
     
     
         8 . The system of  claim 3 , wherein the encoder circuitry comprises one or more parameters that are determined through a training process. 
     
     
         9 . The system of  claim 3 , wherein the decoder circuitry comprises one or more parameters that are determined through a training process. 
     
     
         10 . A method, comprising:
 training at a first time a natural language processing system (NLPS) using uncompressed word embeddings;   training decoder circuitry in an auto-encoder processing unit with compressed word embeddings each comprising a vector of binary numbers that correspond to the uncompressed word embeddings that each comprise a vector of real numbers, the compressed word embeddings produced by encoder circuitry in the auto-encoder processing unit;   replacing one or more parameters in the NLPS with one or more parameters in the trained decoder circuitry; and   training at a second time the NLPS using the compressed word embeddings.   
     
     
         11 . The method of  claim 10 , further comprising compressing, by encoder circuitry in the auto-encoder processing unit, uncompressed word embeddings to produce the compressed word embeddings. 
     
     
         12 . The method of  claim 11 , wherein the auto-encoder processing circuitry comprises a multi-layer neural network, wherein the encoder circuitry comprises a first layer in the neural network and the decoder circuitry comprises a second layer in the neural network. 
     
     
         13 . The method of  claim 12 , wherein a third layer in the neural network comprises an activation function layer and the operation of compressing the uncompressed word embeddings comprises compressing, by encoder circuitry and the activation function layer in the auto-encoder processing unit, the uncompressed word embeddings to produce the compressed word embeddings. 
     
     
         14 . An electronic device, comprising:
 an input device for receiving a natural language input;   a storage device storing compressed word embeddings that each comprise a vector of binary numbers; and   a natural language processing system, comprising:
 a natural language understanding (NLU) circuitry operably connected to the storage device, the NLU circuitry obtaining one or more compressed word embeddings that represent at least one word in the natural language input; and 
   processing circuitry operably connected to the NLU circuitry, wherein the processing circuitry receives the compressed word embeddings, decompresses the compressed word embeddings, and processes the decompressed word embeddings to determine an action to be taken by the electronic device in response the natural language input, wherein each decompressed word embedding comprises a vector of real numbers.   
     
     
         15 . The electronic device of  claim 14 , wherein the input device comprises a microphone. 
     
     
         16 . The electronic device of  claim 14 , wherein the processing circuitry causes the determined action to be provided to an output device. 
     
     
         17 . The electronic device of  claim 16 , wherein the output device comprises a display. 
     
     
         18 . The electronic device of  claim 14 , further comprising a natural language generation (NLG) circuitry operably connected to the processing circuitry, the NLG circuitry converting the determined action into a natural language output. 
     
     
         19 . The electronic device of  claim 18 , wherein the NLG circuitry causes the natural language output to be provided to an output device. 
     
     
         20 . The electronic device of  claim 19 , wherein the output device comprises a speaker.