Detecting candidate hallucinations in outputs of a retrieval-augmented generation enhanced large language model

Abstract

The disclosure provides runtime and training methods, computing apparatus, and computer readable media for use in detecting candidate hallucinations in outputs of a Retrieval-Augmented Generation (RAG) enhanced Large Language Model (LLM) trained to retrieve documents from a closed domain knowledge base responsive to an input query, and generate an LLM output vector based on the query and any retrieved documents. The method includes inputting an LLM output vector received from an LLM to an encoder part of a Variational Autoencoder and receiving from an encoder output layer thereof an encoder output vector having values representing a distribution of the LLM output vector in a dimensionally reduced latent space. The Variational Autoencoder is trained using a training dataset of LLM output vectors generated by the LLM labelled as normal outputs of the LLM or hallucination outputs of the LLM.

1 . A method for use in detecting candidate hallucinations in outputs of a Retrieval-Augmented Generation (RAG) enhanced Large Language Model (LLM) trained to retrieve documents from a closed domain knowledge base responsive to an input query, and generate an LLM output vector based on the query and any retrieved documents, the method comprising:
 receiving from the LLM an LLM output vector representing output tokens generated by the LLM responsive to a query;   inputting the LLM output vector to an encoder input layer of an encoder part of a neural network configured as a Variational Autoencoder, the encoder input layer arranged to have nodes corresponding to the LLM output vector;   receiving from an encoder output layer of the encoder part of the Variational Autoencoder (VAE) an encoder output vector having values representing a distribution of the LLM output vector in a dimensionally reduced latent space, the encoder output layer being connected to the encoder input layer through one or more hidden layers having nodes with weights trained together with a corresponding decoder part of the Variational Autoencoder to allow the decoder part to reconstruct at a decoder output layer thereof the LLM output vector from the encoded distribution in the latent space, the Variational Autoencoder having been trained using a training dataset of LLM output vectors generated by the LLM from the retrieved documents responsive to input queries, the training dataset labelled as normal outputs of the LLM or hallucination outputs of the LLM, the training of the Variational Autoencoder thereby generating characteristic distributions of normal outputs and hallucination outputs of the LLM in the latent space based on the documents in the closed domain knowledge base;   comparing the encoder output vector with the characteristic distribution of normal outputs of the LLM that was learned and/or the distribution of hallucination outputs of the LLM; and   generating an indication of whether or not the LLM output vector is likely to be a hallucination based on the comparing.   
     
     
         2 . The method of  claim 1 , wherein the closed domain knowledge base consists of a list of specified documents or a structured data repository of finite and defined scope. 
     
     
         3 . The method of  claim 1 , wherein the trained VAE has learned the distribution and structure of the documents in the closed domain knowledge base in the dimensionally reduced latent space, the distribution and structure of the documents in the closed domain knowledge base being generated by the VAE characterizing the normal outputs of the LLM generated responsive to queries by the LLM retrieving documents from the closed domain knowledge base. 
     
     
         4 . The method of  claim 1 , wherein comparing the encoder output vector with the learned characteristic distribution of normal outputs of the LLM and/or the hallucination outputs of the LLM comprises:
 determining a metric representative of a distance between the encoder output vector and the learned characteristic distribution of normal outputs of the LLM and/or the hallucination outputs of the LLM, wherein the distance metric is indicative of dissimilarity between the encoder output vector and the distribution of normal outputs of the LLM.   
     
     
         5 . The method of  claim 4 , wherein generating an indication of whether or not the LLM output vector is likely to be a hallucination based on the comparing comprises:
 determining, based on the determined distance metric, a metric indicating whether the LLM output vector is likely to be a hallucination.   
     
     
         6 . The method of  claim 5 , wherein determining, based on the determined distance metric, a metric indicating whether the LLM output vector is likely to be a hallucination comprises:
 comparing the determined distance metric to a threshold distance value above which encoder output vector is a candidate hallucination.   
     
     
         7 . The method of  claim 1 , further comprising:
 when an indication is generated that the LLM output vector is likely to be a hallucination, performing one or more of:   providing an alert to the LLM that the LLM output vector is a candidate hallucination;   providing an instruction to the LLM to discard the LLM output vector; or   providing an instruction to the LLM to update a prompt provided to the Retrieval-Augmented Generation (RAG) enhanced Large Language Model (LLM) and re-run the query, the prompt being updated to reduce a likelihood that the LLM output vector is a candidate hallucination.   
     
     
         8 . The method of  claim 1 , further comprising generating a training dataset for the Variational Autoencoder using a prompting Large Language Model (LLM) to generate queries for the Retrieval-Augmented Generation (RAG) enhanced Large Language Model (LLM) to generate LLM output vectors, the generated LLM output vectors being used to provide a training set of labelled LLM output vectors used to train the Variational Autoencoder to determine the characteristic distribution of normal outputs of the LLM and/or the distribution of hallucination outputs of the LLM. 
     
     
         9 . The method of  claim 8 , wherein the prompting Large Language Model (LLM) is configured to search across the distribution of outputs of the LLM in the latent space. 
     
     
         10 . The method of  claim 1 , further comprising, receiving a labelled training dataset of LLM output vectors each labelled as either a normal LLM output vector or a hallucination LLM output vector, wherein the labelling is generated by one or more domain knowledge experts. 
     
     
         11 . The method of  claim 10 , further comprising:
 for each one of plural LLM output vectors of the training dataset, training the Variational Autoencoder by:
 inputting the LLM output vector to the encoder input layer of the encoder part of a Variational Autoencoder; 
 receiving from an encoder output layer of the encoder part of the Variational Autoencoder an encoder output vector having values representing a distribution of the LLM output vector in a dimensionally reduced latent space; 
 sampling values from the distribution defined by the encoder output vector to generate a decoder input vector representative of the LLM output vector in the latent space; 
 inputting the decoder input vector to a decoder input layer of the decoder part of the Variational Autoencoder; 
 receiving from a decoder output layer of the decoder part of the Variational Autoencoder a reconstructed version of the LLM output vector, the decoder output layer being connected to the decoder input layer through one or more hidden layers having nodes with weights; 
 determining a loss function characterizing a reconstruction error between the LLM output vector and the reconstructed version of the LLM output vector; and 
 using an appropriate optimization algorithm operating on the loss function, updating the connecting node weights of the hidden layers of the encoder neural network and decoder neural network to seek to minimize the loss function, 
   until the loss function converges and the Variational Autoencoder effectively reconstructs the LLM output vector.   
     
     
         12 . The method of  claim 11 , further comprising, based on the labels applied to the LLM output vectors in the training dataset:
 determining the distribution of normal outputs of the LLM in the latent space; and   determining the distribution of hallucination outputs of the LLM in the latent space.   
     
     
         13 . A computing apparatus for use in detecting candidate hallucinations in outputs of a Retrieval-Augmented Generation (RAG) enhanced Large Language Model (LLM) trained to retrieve documents from a closed domain knowledge base responsive to an input query, and generate an LLM output vector based on the query and any retrieved documents, the computing apparatus comprising:
 one or more processors; and   a memory storing instructions that, when executed by the processor, configure the apparatus to:   receive from the LLM an LLM output vector representing output tokens generated by the LLM responsive to a query;   input the LLM output vector to an encoder input layer of an encoder part of a neural network configured as a Variational Autoencoder, the encoder input layer arranged to have nodes corresponding to the LLM output vector;   receive from an encoder output layer of the encoder part of the Variational Autoencoder an encoder output vector having values representing a distribution of the LLM output vector in a dimensionally reduced latent space, the encoder output layer being connected to the encoder input layer through one or more hidden layers having nodes with weights trained together with a corresponding decoder part of the Variational Autoencoder to allow the decoder part to reconstruct at a decoder output layer thereof the LLM output vector from the encoded distribution in the latent space, the Variational Autoencoder having been trained using a training dataset of LLM output vectors generated by the LLM from the retrieved documents responsive to input queries, the training dataset labelled as normal outputs of the LLM or hallucination outputs of the LLM, the training of the Variational Autoencoder thereby generating characteristic distributions of normal outputs and hallucination outputs of the LLM in the latent space based on the documents in the closed domain knowledge base;   compare the encoder output vector with the learned characteristic distribution of normal outputs of the LLM and/or the distribution of hallucination outputs of the LLM; and   generate an indication of whether or not the LLM output vector is likely to be a hallucination based on the comparison.   
     
     
         14 . A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions for use in detecting candidate hallucinations in outputs of a Retrieval-Augmented Generation (RAG) enhanced Large Language Model (LLM) trained to retrieve documents from a closed domain knowledge base responsive to an input query, and generate an LLM output vector based on the query and any retrieved documents, wherein, when executed by one or processors of a computing apparatus, the instructions cause the computing apparatus to:
 receive from the LLM an LLM output vector representing output tokens generated by the LLM responsive to a query;   input the LLM output vector to an encoder input layer of an encoder part of a neural network configured as a Variational Autoencoder, the encoder input layer arranged to have nodes corresponding to the LLM output vector;   receive from an encoder output layer of the encoder part of the Variational Autoencoder an encoder output vector having values representing a distribution of the LLM output vector in a dimensionally reduced latent space, the encoder output layer being connected to the encoder input layer through one or more hidden layers having nodes with weights trained together with a corresponding decoder part of the Variational Autoencoder to allow the decoder part to reconstruct at a decoder output layer thereof the LLM output vector from the encoded distribution in the latent space, the Variational Autoencoder having been trained using a training dataset of LLM output vectors generated by the LLM from the retrieved documents responsive to input queries, the training dataset labelled as normal outputs of the LLM or hallucination outputs of the LLM, the training of the Variational Autoencoder thereby generating characteristic distributions of normal outputs and hallucination outputs of the LLM in the latent space based on the documents in the closed domain knowledge base;   compare the encoder output vector with the learned characteristic distribution of normal outputs of the LLM and/or the distribution of hallucination outputs of the LLM; and   generate an indication of whether or not the LLM output vector is likely to be a hallucination based on the comparison.   
     
     
         15 . Method of training a Variational Autoencoder (VAE) for use in detecting candidate hallucinations in outputs of a Retrieval-Augmented Generation (RAG) enhanced Large Language Model (LLM) trained to retrieve documents from a closed domain knowledge base responsive to an input query, and generate an LLM output vector based on the query and any retrieved documents, the method comprising:
 receiving a labelled training dataset of LLM output vectors each representing output tokens generated by the LLM responsive to a query and each labelled as either a normal LLM output vector or a hallucination LLM output vector;   for each one of plural LLM output vectors of the training dataset, training the Variational Autoencoder by:
 inputting the LLM output vector to an encoder input layer of an encoder part of a Variational Autoencoder, the encoder input layer arranged to have nodes corresponding to the LLM output vector; 
 receiving from an encoder output layer of the encoder part of the Variational Autoencoder an encoder output vector having values representing a distribution of the LLM output vector in a dimensionally reduced latent space, the encoder output layer being connected to the encoder input layer through one or more hidden layers having nodes with weights; 
 sampling values from the distribution defined by the encoder output vector to generate a decoder input vector representative of the LLM output vector in the latent space; 
 inputting the decoder input vector to a decoder input layer of a corresponding decoder part of the Variational Autoencoder; 
 receiving from a decoder output layer of the decoder part of the Variational Autoencoder a reconstructed version of the LLM output vector, the decoder output layer being connected to the decoder input layer through one or more hidden layers having nodes with weights; 
 determining a loss function characterizing a reconstruction error between the LLM output vector and the reconstructed version of the LLM output vector; and 
 using an appropriate optimization algorithm operating on the loss function, updating the connecting node weights of the hidden layers of the encoder neural network and decoder neural network to seek to minimize the loss function, 
   until the loss function converges and the Variational Autoencoder effectively reconstructs the LLM output vector,   wherein, based on the labelling of the LLM output vectors in the training dataset, the training of the Variational Autoencoder thereby generates characteristic distributions of normal outputs and hallucination outputs of the LLM in the latent space based on the documents in the closed domain knowledge base.