Extracting audiovisual features from digital components

Abstract

Systems and methods for extracting audiovisual features from images and other digital components. A data processing system can extract image data and image features from an input image. The data processing system can match the image features to the image features of a plurality of image to identify candidate images. A second image can be selected from the candidate images based on a request that the data processing system received with the input image.

1 .- 20 . (canceled) 
     
     
         21 . A system, comprising:
 a data processing system comprising at least one hardware processor coupled to memory to:   receive, from a computing device, a term and an image that is different from the term;   retrieve, from a data repository, content data for each of a plurality of content items, the content data comprising image features;   extract an image feature from the image, the image feature including at least one of an edge, a corner feature, or a ridge feature;   select a content item from the plurality of content items based at least in part on the term and on matches between the image features of the plurality of content items and the at least one of the edge, the corner feature, or the ridge feature of the image; and   transmit the content item to the computing device.   
     
     
         22 . The system of  claim 21 , comprising:
 the data processing system to receive a request from the computing device, the request comprising a query comprising the term.   
     
     
         23 . The system of  claim 21 , comprising:
 the data processing system to receive the term via an audio-based input signal detected by the computing device.   
     
     
         24 . The system of  claim 21 , comprising the data processing system to:
 select candidate content items from the plurality of content items based on matches between the image features of the plurality of content items and the at least one of the edge, the corner feature, or the ridge feature of the image; and   select the content item from the candidate content items.   
     
     
         25 . The system of  claim 24 , comprising:
 the data processing system to select the content item from the candidate content items based at least in part on the term.   
     
     
         26 . The system of  claim 21 , wherein the image is captured by a camera associated with the computing device. 
     
     
         27 . The system of  claim 21 , comprising the data processing system to:
 apply an image feature detection to the image to extract the image feature; and   apply the image feature detection to each of the plurality of content items.   
     
     
         28 . The system of  claim 21 , comprising the data processing system to:
 extract a plurality of image features from the image;   extract a second plurality of image features from each of the plurality of content items;   select candidate content items from the plurality of content items by matching a first predetermined number of the plurality of image features with a second predetermined number of the second plurality of image features; and   select the content item from the candidate content items.   
     
     
         29 . The system of  claim 21 , comprising the data processing system to:
 receive data packets comprising a request;   parse the request to identify the term and a trigger keyword corresponding to the request; and   select the content item from the plurality of content items based on the trigger keyword.   
     
     
         30 . The system of  claim 21 , comprising the data processing system to:
 generate an action data structure based on the term; and   transmit the action data structure to a service provider computing device to cause the service provider computing device to invoke a conversational application programming interface and establish a communication session between the service provider computing device and the computing device.   
     
     
         31 . A method, comprising:
 receiving, by a data processing system comprising at least one hardware processor coupled to memory, from a computing device, a term and an image that is different from the term;   retrieving, by the data processing system from a data repository, content data for each of a plurality of content items, the content data comprising image features;   extracting, by the data processing system, an image feature from the image, the image feature including at least one of an edge, a corner feature, or a ridge feature;   selecting, by the data processing system, a content item from the plurality of content items based at least in part on the term and on matches between the image features of the plurality of content items and the at least one of the edge, the corner feature, or the ridge feature of the image; and   transmitting, by the data processing system, the content item to the computing device.   
     
     
         32 . The method of  claim 31 , comprising:
 receiving, by the data processing system, a request from the computing device, the request comprising a query comprising the term.   
     
     
         33 . The method of  claim 31 , comprising:
 receiving, by the data processing system, the term via an audio-based input signal detected by the computing device.   
     
     
         34 . The method of  claim 31 , comprising:
 selecting, by the data processing system, candidate content items from the plurality of content items based on matches between the image features of the plurality of content items and the at least one of the edge, the corner feature, or the ridge feature of the image; and   selecting, by the data processing system, the content item from the candidate content items.   
     
     
         35 . The method of  claim 34 , comprising:
 selecting, by the data processing system, the content item from the candidate content items based at least in part on the term.   
     
     
         36 . The method of  claim 31 , wherein the image is captured by a camera associated with the computing device. 
     
     
         37 . The method of  claim 31 , comprising:
 applying, by the data processing system, an image feature detection to the image to extract the image feature; and   applying, by the data processing system, the image feature detection to each of the plurality of content items.   
     
     
         38 . The method of  claim 31 , comprising:
 extracting, by the data processing system, a plurality of image features from the image;   extracting, by the data processing system, a second plurality of image features from each of the plurality of content items;   selecting, by the data processing system, candidate content items from the plurality of content items by matching a first predetermined number of the plurality of image features with a second predetermined number of the second plurality of image features; and   selecting, by the data processing system, the content item from the candidate content items.   
     
     
         39 . The method of  claim 31 , comprising:
 receiving, by the data processing system, data packets comprising a request;   parsing, by the data processing system, the request to identify the term and a trigger keyword corresponding to the request; and   selecting, by the data processing system, the content item from the plurality of content items based on the trigger keyword.   
     
     
         40 . The method of  claim 31 , comprising:
 generating, by the data processing system, an action data structure based on the term; and   transmitting, by the data processing system, the action data structure to a service provider computing device to cause the service provider computing device to invoke a conversational application programming interface and establish a communication session between the service provider computing device and the computing device.