System and method for executing map-reduce tasks in a storage device

Abstract

A system and method of providing enhanced data processing and analysis in an infrastructure for distributed computing and large-scale data processing. This infrastructure uses the Hadoop™ framework to divide an application into a large number of small fragments of work, each of which may be performed on one of a large number of compute nodes. The work may involve map tasks and reduce tasks which may be used to categorize and analyze large amounts of data in distributed systems. This infrastructure includes a cluster with a master node and a plurality of slave nodes. The slave nodes may include, or may be, intelligent solid-state drives capable of executing Map-Reduce functions. The use of intelligent solid-state drives reduces the need to exchange data with a CPU in a server.

What is claimed is: 
     
         1 . An intelligent solid state drive comprising:
 a solid state drive controller;   a solid state drive host interface; and   a flash memory;   the solid state drive controller configured to be in communication with the flash memory, and comprising:   a hardware engine; and   a microcontroller;   the solid state drive configured to:   receive, through the solid state drive host interface, a data query described by Map-Reduce functions;   execute map and reduce tasks, in the solid state drive controller, to process the query; and   return a query result through the solid state drive host interface.   
     
     
         2 . An intelligent solid state drive comprising:
 a solid state drive controller;   a solid state drive host interface; and   a flash memory;   the solid state drive host interface being selected from the group consisting of: PCIe, NVMe, SATA, Fibre Channel, SAS, SCSI over PCIe, and Ethernet,   the solid state drive controller configured to be in communication with the flash memory, and comprising:   a hardware engine; and   a microcontroller;   the solid state drive configured to perform map and reduce tasks.