Re-ranking and outlier detection in an augmented semantic search system

Abstract

A system and method for augmented semantic search, including: a vector store including a set of embeddings, each representing a structured data representation of a media perspective of a media item; a query execution service including functionality to receive a search request including a query string from a client application; a query classification service including functionality to execute a first machine learning model to generate a classification object in a structured classification format; a filter extraction service including functionality to execute a second machine learning model to generate a filter object including a set of filters in a structured filter format; a recaller service including functionality to: execute an encoder model on the input query and execute a vector similarity operation on the query embedding to generate a result set; and a re-ranking service including functionality to: execute a large language model to re-rank the result set.

1 . A system for semantic search, comprising:
 a computer processor;   a vector store comprising a set of embeddings positioned within a unified vector space. wherein multiple of the set of embeddings correspond to a single media item, each representing a different structured data representation of a media perspective of the media item;   a query execution service comprising functionality to:
 receive a search request comprising a query string from a client application; 
   a query classification service comprising functionality to:
 execute a first machine learning model to generate a classification object representing classification of the query string in a structured classification format; 
   a filter extraction service comprising functionality to:
 execute a second machine learning model to generate a filter object comprising a set of filters inferred for the query string in a structured filter format; 
   a recaller service comprising functionality to:
 execute an encoder model on the input query, incorporating the classification object and the filter object, to generate a query embedding representing the input query in the unified vector space; 
 use the filter object to identify a constrained set of candidate embeddings of the vector store; and 
 execute a vector similarity operation within the unified vector space on the query embedding and the constrained set of candidate embeddings to generate a match set of embeddings; and 
   a re-ranking service comprising functionality to:
 generate a re-ranking prompt comprising the query embedding and the match set of embeddings; 
 execute a large language model using the re-ranking prompt to generate a re-ranked match set of embeddings; and 
 provide the re-ranked match set of embeddings in response to the search request. 
   
     
     
         2 . The system of  claim 1 , wherein the re-ranking service is further configured to perform outlier detection by:
 generating an outlier detection prompt comprising the query embedding, the match set of embeddings, and contextual information derived from the classification object and filter object;   executing the large language model using the outlier detection prompt to generate an outlier score for each embedding in the match set;   applying a dynamic outlier threshold based on the classification object and the distribution of outlier scores;   identifying embeddings with outlier scores exceeding the dynamic outlier threshold;   analyzing semantic relationships between identified outlier embeddings and non-outlier embeddings;   selectively excluding outlier embeddings based on both their outlier scores and their semantic distance from non-outlier embeddings; and   adjusting the re-ranked match set of embeddings to exclude the selectively excluded outlier embeddings.   
     
     
         3 . The system of  claim 1 , wherein the re-ranking service is further configured to:
 incorporate additional inputs into the re-ranking prompt, the additional inputs comprising at least one selected from a group consisting of: user profile data, historical search behavior, trending topics, and contextual information.   
     
     
         4 . The system of  claim 1 , wherein the re-ranking service is further configured to:
 incorporate the classification object and filter object into the re-ranking prompt;   generate a relevance score for each embedding based on its alignment with both the classification object and the filter object;   apply a weighted ranking algorithm balancing classification alignment and filter adherence;   prioritize embeddings matching both classification intent and filter criteria; and   ensure result diversity by considering secondary classifications when primary classification intent is met.   
     
     
         5 . The system of  claim 1 , further comprising:
 an autodata generation service comprising functionality to generate the structured data representations for each media item by:
 extracting caption data from the media item; 
 generating an autodata prompt by analyzing the caption data to identify key themes, entities, and contexts; 
 selecting relevant prompt templates based on the media item's type and content; 
 executing a third large language model using the autodata prompt to generate the structured data representation; and 
 validating the generated representation for accuracy and completeness. 
   
     
     
         6 . The system of  claim 1 , wherein:
 the query classification system executes a second large language model to generate the classification object by mapping the query to predefined classification categories;   the filter extraction system executes a third large language model to generate the filter object by identifying specific entities within the query string using boolean and range-based parameters;   the classification object informs the third large language model to refine filter granularity and resolve ambiguities.   
     
     
         7 . The system of  claim 1 , further comprising a multi-classification analyzer service configured to:
 analyze the query string to identify multiple distinct classifications;   for each identified classification:
 generate a classification-specific query embedding using the encoder model, 
 cause the recaller service to execute a separate vector similarity operation on the classification-specific query embedding to generate a classification-specific match set of embeddings, and 
 apply the re-ranking service to the classification-specific match set of embeddings to generate a classification-specific re-ranked match set; 
   merge the classification-specific re-ranked match sets into a compound result set by:
 assigning weights to each classification-specific re-ranked match set based on relevance scores derived from the query classification service, 
 normalizing ranking scores across all classification-specific re-ranked match sets, 
 interleaving results from each classification-specific re-ranked match set based on the normalized ranking scores and classification weights, and 
 applying a diversity algorithm to ensure representation from each identified classification in the compound result set; and 
   provide the compound result set in response to the search request.   
     
     
         8 . The system of  claim 1 , wherein the re-ranking service is further configured to:
 generate, for each embedding in the match set of embeddings, a confidence score indicating a likelihood of relevance to the query string;   apply a dynamic threshold to the confidence scores, wherein the dynamic threshold is adjusted based on the classification object and filter object; and   include in the re-ranked match set of embeddings only those embeddings exceeding the dynamic threshold.   
     
     
         9 . The system of  claim 1 , wherein the re-ranking service is further configured to:
 identify semantic relationships between embeddings in the match set;   cluster semantically related embeddings; and   adjust the ranking of embeddings within each cluster to ensure diversity in the re-ranked match set of embeddings.   
     
     
         10 . The system of  claim 1 , further comprising an adaptive learning module comprising functionality to:
 analyze user interactions with the re-ranked match set of embeddings;   generate labeled training examples from search interactions;   fine-tune the large language model using reinforcement learning techniques;   adjust the large language model's ranking behavior based on evolving user preferences; and   periodically evaluate the fine-tuned large language model to ensure improved performance across various domains and user segments.   
     
     
         11 . A method for semantic search, comprising:
 receiving a search request comprising a query string from a client application;   executing a first machine learning model to generate a classification object representing classification of the query string in a structured classification format;   executing a second machine learning model to generate a filter object comprising a set of filters inferred for the query string in a structured filter format;   executing an encoder model on the input query, incorporating the classification object and the filter object, to generate a query embedding representing the input query in a unified vector space;   using the filter object to identify a constrained set of candidate embeddings of a vector store comprising a set of embeddings, wherein the constrained set of embeddings is a subset of the set of embeddings positioned within the unified vector space, wherein multiple of the set of embeddings in the unified vector space correspond to a single media item, each representing a different structured data representation of a media perspective of the media item;   executing a vector similarity operation within the unified vector space on the query embedding and the constrained set of candidate embeddings to generate a match set of embeddings;   generating a re-ranking prompt comprising the query embedding and the match set of embeddings;   executing, by a computer processor, a large language model using the re-ranking prompt to generate a re-ranked match set of embeddings; and   providing the re-ranked match set of embeddings in response to the search request.   
     
     
         12 . The method of  claim 11 , further comprising:
 generating an outlier detection prompt comprising the query embedding, the match set of embeddings, and contextual information derived from the classification object and filter object;   executing the large language model using the outlier detection prompt to generate an outlier score for each embedding in the match set;   applying a dynamic outlier threshold based on the classification object and the distribution of outlier scores;   identifying embeddings with outlier scores exceeding the dynamic outlier threshold;   analyzing semantic relationships between identified outlier embeddings and non-outlier embeddings;   selectively excluding outlier embeddings based on both their outlier scores and their semantic distance from non-outlier embeddings; and   adjusting the re-ranked match set of embeddings to exclude the selectively excluded outlier embeddings.   
     
     
         13 . The method of  claim 11 , further comprising:
 incorporating additional inputs into the re-ranking prompt, the additional inputs comprising at least one selected from a group consisting of: user profile data, historical search behavior, trending topics, and contextual information.   
     
     
         14 . The method of  claim 11 , further comprising:
 incorporating the classification object and filter object into the re-ranking prompt;   generating a relevance score for each embedding based on its alignment with both the classification object and the filter object;   applying a weighted ranking algorithm balancing classification alignment and filter adherence;   prioritizing embeddings matching both classification intent and filter criteria; and   ensuring result diversity by considering secondary classifications when primary classification intent is met.   
     
     
         15 . The method of  claim 11 , further comprising:
 generating the structured data representations for each media item by:
 extracting caption data from the media item; 
 generating an autodata prompt by analyzing the caption data to identify key themes, entities, and contexts; 
 selecting relevant prompt templates based on the media item's type and content; 
 executing a third large language model using the autodata prompt to generate the structured data representation; and 
 validating the generated representation for accuracy and completeness. 
   
     
     
         16 . The method of  claim 11 , further comprising:
 executing a second large language model to generate the classification object by mapping the query to predefined classification categories; and   executing a third large language model to generate the filter object by identifying specific entities within the query string using boolean and range-based parameters, wherein the classification object informs the third large language model to refine filter granularity and resolve ambiguities.   
     
     
         17 . The method of  claim 11 , further comprising:
 analyzing the query string to identify multiple distinct classifications;   for each identified classification:
 generating a classification-specific query embedding using the encoder model, 
 causing the recaller service to execute a separate vector similarity operation on the classification-specific query embedding to generate a classification-specific match set of embeddings, and 
 applying the re-ranking service to the classification-specific match set of embeddings to generate a classification-specific re-ranked match set; 
   merging the classification-specific re-ranked match sets into a compound result set by:
 assigning weights to each classification-specific re-ranked match set based on relevance scores derived from the query classification service, 
 normalizing ranking scores across all classification-specific re-ranked match sets, 
 interleaving results from each classification-specific re-ranked match set based on the normalized ranking scores and classification weights, and 
 applying a diversity algorithm to ensure representation from each identified classification in the compound result set; and 
   providing the compound result set in response to the search request.   
     
     
         18 . The method of  claim 11 , further comprising:
 generating, for each embedding in the match set of embeddings, a confidence score indicating a likelihood of relevance to the query string;   applying a dynamic threshold to the confidence scores, wherein the dynamic threshold is adjusted based on the classification object and filter object; and   including in the re-ranked match set of embeddings only those embeddings exceeding the dynamic threshold.   
     
     
         19 . The method of  claim 11 , further comprising:
 identifying semantic relationships between embeddings in the match set;   clustering semantically related embeddings; and   adjusting the ranking of embeddings within each cluster to ensure diversity in the re-ranked match set of embeddings.   
     
     
         20 . A non-transitory computer-readable storage medium comprising a plurality of instructions for semantic search, the plurality of instructions configured to execute on at least one computer processor to enable the at least one computer processor to:
 receive a search request comprising a query string from a client application;   execute a first machine learning model to generate a classification object representing classification of the query string in a structured classification format;   execute a second machine learning model to generate a filter object comprising a set of filters inferred for the query string in a structured filter format;   execute an encoder model on the input query, incorporating the classification object and the filter object, to generate a query embedding representing the input query in a unified vector space;   use the filter object to identify a constrained set of candidate embeddings of a vector store comprising a set of embeddings, wherein the constrained set of embeddings is a subset of the set of embeddings positioned within the unified vector space, wherein multiple of the set of embeddings in the unified vector space correspond to a single media item, each representing a different structured data representation of a media perspective of the media item;   execute a vector similarity operation within the unified vector space on the query embedding and the constrained set of candidate embeddings to generate a match set of embeddings;   generate a re-ranking prompt comprising the query embedding and the match set of embeddings;   execute a large language model using the re-ranking prompt to generate a re-ranked match set of embeddings; and   provide the re-ranked match set of embeddings in response to the search request.