RELATED APPLICATION

[0001] The present application claims priority to U.S. Pat. App. Ser. No. 16/824,620, filed Mar. 19, 2020 and entitled “PROCESSING IN MEMORY METHODS FOR CONVOLUTIONAL OPERATIONS,” the entire disclosure of which is hereby incorporated herein by reference.

BACKGROUND

[0002] Generic processors may interface with memory components and caches to perform repeated calculations on stored data. Data may be loaded into a cache, the processor may then access the data, the processor may calculate a result, and then the result may be stored in memory. Processors may perform repetitive or intensive linearalgebra operations by handling matrix elements. For example, processors may perform read/write operations to fetch data, process it, and store it in memory. These generic processors may be used to implement convolutional operations.

BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Many aspects of the present disclosure can be better understood with reference to the attached drawings. The components in the drawings are not necessarily drawn to scale, with emphasis instead being placed upon clearly illustrating the principles of the disclosure. Moreover, in the drawings, like reference numerals designate corresponding parts throughout several views.

[0003] FIG. 1 is a drawing of a Processing In Memory (PIM) device according to various embodiments.

[0004] FIG. 2 is a drawing of a PIM device configured to apply filter data to a memory array using an allocation pattern according to various embodiments.

[0005] FIG. 3 is a drawing of a PIM device 100 configured to apply a data object array to a memory array 103 using an allocation pattern according to various embodiments.

[0006] FIG. 4 is a drawing of an example of a bit-serial allocation pattern for organizing data in a memory array of a PIM device. [0007] FIG. 5 is a drawing of an example of a bit-parallel allocation pattern for organizing data in a memory array of a PIM device.

[0008] FIG. 6 is a drawing of an example of a serial-parallel allocation pattern for organizing data in a memory array of a PIM device.

[0009] FIG. 7 is a flowchart illustrating varying examples of the functionality of a PIM device according to various embodiments.

[0010] FIG. 8 illustrates an example networked system that includes a PIM device, in accordance with some embodiments of the present disclosure.

DETAILED DESCRIPTION

[0011] A Processing In Memory (PIM) device is a semiconductor device that comprises one or more memory arrays and a PIM processor coupled to these arrays in-memory. The PIM processor is configured to perform operations using data stored in the cells of the memory array withoutthe need to perform time intensive input/output operations, fetch operations, or load / store operations over a memory bus. In this respect, the PIM processor may access at least some data without a buffer memory or cache or bus to perform data and compute operations.

In contrast, a host processor is coupled with one or more PIM devices over a memory bus or other link. A host processor may be a central processing unit (CPU), digital signal processor, graphics processing unit, special purpose processor, or general-purpose processor that is installed in a device or system external to the PIM device. The host processor may be installed in a computing device, lap top, mobile device, server, special purpose computer, general purpose computer.

[0012] Embodiments of the present disclosure are directed to a sequence control, which can be a part of PIM processor that is configured to store in a memory array of a PIM device with a first array of data by organizing the data of the first array into a plurality of groups. The groups are structured in the memory array according to an allocation pattern. The sequence control receives second array of data and converts at least some of the elements into a sequence for execution. The sequence control applies the sequence to sets of groups in the memory array according to the allocation pattern of the first array. The sequence is applied until there are no remaining groups in the memory array that should be addressed or until the end of sequence. A fused multiply-accumulate (FMA) unit of the PIM device is the main computational elementof the PIM processor. It may be configured to generate a dot product array of data when the second array is applied to the groups. Thus, the sequence control may be configured to effectively tesselate or otherwise convolve an array of data across a memory array to generate a plurality of dot product calculations and their results.

[0013] In some embodiments, the data stored in the memory array is the data of a data object in which a feature is to be detected. For example, the data object may be an image. In addition, the second array applied to the memory array may be a filter designed to detect or identify a feature in the data object.

[0014] In some embodiments, the data stored in the memory array is filter data designed to detect or identify a feature. In addition, the second array applied to the memory array may be at least a portion of a data object that is applied to the filter in order to detect or identify a feature.

[0015] In some embodiments, the PIM device is a special purpose device used to implementa convolutional neural network. The PIM device may be used to implement one or more layers in a convolutional neural network. The convolutional neural network may be designed for detecting: image features in image data, motion features in video streams, text patterns in textual data, statistical features in multi-dimensional complex data, scientificfeatures in scientific processes and simulations, astronomical features in astronomical data coming from the space, weatherconditions in world weather data as well as predictions based on them, words in voice audio data. The convolutional neural network may be used to detect features or characteristics in computer generated graphics, virtual reality data, and augmented reality data. Features may be detected for satellite imagery, long exposure generated graphics, time-lapse videos, slow-motion videos. The convolutional neural network may be configured to perform feature detection on graphical or visual representation of data collected from a variety of sources such as data repositories or databases. The data subject to feature detection may be data that is structured, data that is semi -structured, data that is unstructured, data objects generated from machines, data logs, real-time data generated from a remote source, data that is aggregated from a plurality of sources, data received over a network, data that has been pre-processed by external systems, data that has been subject to visual filters, or data that generated at least partially by an external computing system. Features that searched for within the data include visual patterns, horizontal lines, edges, vertical lines, various shapes, curves, angles, particular colors, orientations. In addition, simple features may be combined to formulate more complex features such as complex objects.

[0016] FIG. 1 is a drawing of a Processing In Memory (PIM) device 100 according to various embodiments. A PIM device 100 is an integrated circuit. The PIM device 100 may be a semiconductorchipordie or a die stack. The PIM device 100 may include one or more memory arrays 103. A memory array 103 comprises a plurality of rows and columns and may be defined in terms of a row-column size.

The example of FIG. 1 shows a memory array 103 having rows labeled r1 - rn and columns c1 - c n. At each row and column intersection is a memory cell configured to store an element of a data array. For example, a data array may be a single dimension array containing fourelements as follows: [A, B, C, D] where the first element is A, the second element is B, the third element is C, and the fourth element is D. The data array [A, B, C, D] may be stored in the memory array 103 such that each element of the data array is stored in a corresponding memory cell. For example, element A may be stored in cell (r1 , d ), element B may be stored in cell (r1, c2), element C may be stored in cell (r1, c3), and element D may be stored in cell (r1 , c4). Thus, in this example, the data array [A, B, C, D] is stored along the first row and occupies the first four columns. As another example, the data array [A, B,

C, D] may be stored alongthefirst column occupying thefirstfourrows. Flere, element A may be stored in cell (r1 , d ), element B may be stored in cell (r2, d ), element C may be stored in cell (r3, d ), and element D may be stored in cell (r4, d). Thus, the memory array 103 is a hardware component used to store data as a plurality of array elements addressable by rows and columns. Although only one memory array 103 is depicted in the PIM device 100 of FIG. 1 , the PIM device 100 may include several memory arrays 103 organized through out the PIM device 100. Some embodiments are directed to parallel processing using multiple arrays 103.

[0017] A data array element can be a binary bit (the bits that can have two values, e.g. “0” or Ί” encoded as low or high value of a memory cell), a multi state bit (e.g., the bits that can have multiple values, e.g. QCL NAND Flash memory can have sixteen values encoded as Vt (threshold voltage) of a floating gate transistor), or a piece of information that may have infinite range (e.g. information encoded as analog charge on a DRAM capacitor or analog resistance in a ReRAM memory cell). For example, if a data array element is a binary bit then the data array

[A, B, C, D] may be stored in the memory array 103 such that each element of the data array is stored in a corresponding memory cell as low or high value of a memory cell. For example, if a memory cell is a capacitor than low would may correspond to a capacitor having no charge (or having negative charge), and high value may correspond to a capacitor having positive charge (or having no charge). Similarly, if a data array element is a multi-state bit then the data array element may be stored in a plurality of memory cells. For example, if a memory cell is configured to store two bits and the data array element is four bits, then two memory cells are needed to store a single data array element. To illustrate, the number“9” in decimal may be expressed as thefour-bit binary number“1001”. The left part of thefour-bit binary number(two most significant bits) may be stored in one multi-state cell as the binary number“10” while the right part (two least significant bits) may be stored in another multi-state cell as the binary number T’. A data array elementmay refer to a piece of information having an infinite range such as, for example, some rational fractions, an irrational number, or other floating-pointnumbers efc. Such data array elements may be truncated or rounded and stored as a finite number of bits. Alternatively, infinite range can be represented by analog value such as analog resistance or capacitance that can store information which can be read in a form of analog current or voltage. In this respect, elements can form element-groups representing wider piece of information. For example, several binary bits may form an element-group representing an integer or a floating-point number. The meaning of data array elements and element-groups may be interchanged depending on the context.

[0018] The memory array 103 may be implemented using various types of technologies, organizations or aspects. The memory array may be defined as including both volatile and nonvolatile memory. Volatile components may be those that do not retain data values upon loss of power. Nonvolatile components may be those that retain data upon a loss of power. The memory array 103 may comprise random access memory (RAM), read-only memory (ROM), solid-state memory arrays. RAM may comprise static random-access memory (SRAM), dynamic random access memory (DRAM). The memory array 103 may comprise solid-state memory such as Flash memory, NOR Flash (e.g., Flash memory in a NOR configuration) or NAND Flash (e.g., Flash memory in a NAN D configuration). The memory array may be resistive RAM (ReRAM), cross-point memory, or cross-bar 3D memory. Each type of memory technology used to implementthe memory array may be accessed using a row, column, or other memory address. Rows may be referred to as word lines. A word line may comprise terminals of transistor gates of corresponding memory cells. Alternatively, a word line can be connected directly to memory cell matter, e.g. for resistor-like or for diode-like memory cells. Columns may be referred to as bit lines. A bit line may comprise source and/ordrains of transistors that constitute memory cells, capacitor terminals of the capacitors that constitute memory cells, resistor terminals of the resistors that constitute memory cells or diode terminals of the diodes that constitute memory cells or a combination of thereof.

[0019] Memory array 103 comprises peripheral circuity, which can be outside of it or a part of it. The peripheral circuitry includes a Fused Multiply- Accumulate (FMA) Un it 106, a sequence control 109, a buffermemory 112, a system link 115, and potentially other integrated components. The FMA unit 106 is a part of a PIM processor. The FMA unit 106 may include sense amplifiers to sense data from memory array 103 and drivers to store data back to memory array 103. The FMA unit 106 may be configured to perform dot product multiplication on data. The data may be supplied directly from the memory array 103 as well as from a sequence control 109. In some embodiments, the FMA unit 106 may be dedicated to only perform dot product matrix calculations. The FMA unit 106 may be configured to store the dot product outputas outputarray data. The FMA unit 106 may be configured to perform a mu Itiply-accu mu late operation that computes a product of the inputs and adds that product to an accumulator. In this respect, the FMA unit 106 may perform dot product calculations on two input matrices.

[0020] The sequence control 109 is a part of a PIM processor. The sequence control 109 may comprise integrated circuity dedicated to store data into the memory array 103 by organizing the data into a plurality of groups. For example, a set of groups may be organized along a single column ora set of groups may be organized along a single row.

[0021] In addition, the sequence control 109 may receive, derive or generate a second array of data and apply the second array of data to the groups according to an allocation pattern. In this respect, a second array of data plays a role of a micro-code that directs opening and closing rows and columns of memory array 103 to fetch the data in and out and the FMA unit 106 performs computations on data from memory array 103. The sequence control 109 may effectively tesselate or convolve the second array of data across the memory array 103 based on the allocation pattern. For example, the sequence control 109 may apply second array of data in several iterations to different portions of the memory array 103 following a certain allocation pattern that reflects how data is stored in memory array 103.

[0022] The PIM device 100 may also include buffer memory 112. The buffer memory may be included as part of the sequence control 109 and / or a part of FMA unit 106 or it may be external to the sequence control 109 and / or an FMA unit 106 or it may be connected to 109 and 106 via internal bus, which maybe a part of a system link 115. Alternatively, the buffer memory 112 can be a part of memory array 103 allocated specifically for buffer purposes described herein. Specifically, a part of memory array 103 allocated for buffer memory may be a part of array with faster access (e.g., having shorter path to FMA unit 106). The buffer memory 109 may comprise buffers to temporarily store data as the sequence controller 109 and FMA unit 106 perform operations. The sequence control 109 and / or FMA unit 106 may write to or read from the buffer memory 112. For example, the buffer memory 112 may be used to store second data or intermediate resultwhile the memory array 103 and/ or FMA unit 106 may be used to process first data. The sequence control 109 and / or FMA unit 106 may generate a second array based on the second data. For example, the sequence control 109 and / or FMA unit 106 may generate a second array made of elements derived from the first or second data. The second array may be applied to groups of cells of the memory array 103 according to an allocation pattern .

[0023] The PIM device 100 may also include a system link 115. The system link may provide data and/or control signals between the PIM device 100 and external systems. The system link 115 may couple to various components of the PIM device 100 such as, for example, the memory array 103, the FMA unit 106, the sequence control 109, the buffer memory 112, and other components. Thus, system link 115 may include and internal link amongst various components of PIM device 100 that allow these components to exchange data and/or control signals among each other. The system link 115 may comprise input/output ports to couple to external systems outside the PIM device 100. The system link 115 may be an Input/ Output (IO) bus such as, for example, a DDFMbusor PCIe bus. In this respect, an external system may read or write data to the memory array 103, the FMA unit 106, and buffer memory 112. In addition, external systems may transmit control signals to the sequence control 109 to program or otherwise control the sequence control 109.

[0024] An external system may include a host processor with a PCB motherboard, wherein the PIM device 100 is connected to host processor over a bus such as DDR4, DDR5 or PCIe or alike. The external system may execute an operating system, applications, libraries, scripts, or programming languages. The external system may include one or more server racks or computers or other arrangements. A server may be a single installation or may be distributed among many different geographical locations. The external system may include a plurality of computing devices that together may comprise a hosted computing resource, a grid computing resource and/or any other distributed computing arrangement. In some cases, the external system may correspond to an elastic computing resource where the allotted capacity of processing, network, storage, or other computing- related resources may vary overtime. The external system may implementoneor more virtual machines that use the resources of the computing system 110. Various software components may be executed on one or more virtual machines. The external system may also include additional PIM devices 100. In this respect, an instance of a PIM device 100 may query, control or access data in any additional PIM device 100 installed in a system.

[0025] The system link 115 may allow the PIM device 100 to couple to external systems that together combined or separately implement a convolutional neural network. For example, the PIM device 100 may implement a layerwithin a neural network or multiple layers within a neural network. For example, the PIM device 100 may be used to implement a convolution layer. The system link 115 may extract outputs of a PIM device 100 and inputthem in to different layers of the neural network located in other PIM devices or other parts of external system. A pooling layer in a neural network may obtain outputs from the system link 115 of a PIM device 100, may perform pooling operations, and may pass the result as inputs into the PIM device 100. For example, the outputdata generated by the FMA unit106 may be accessed by the system link 115 and processed externally by a pooling layer, where those results are supplied to the memory array 103 of the PIM device 100 via the system link 115 for additional processing.

[0026] While FIG. 1 provides explanation of the overall architecture,

FIGS. 2 and 3, discussed below, provide more specific examples of how data is processed in a PIM device 100 according to various embodiments.

[0027] FIG. 2 is a drawing of a PIM device 100 configured to apply filter data 204 to a memory array 103 using an allocation pattern according to various embodiments. FIG. 2 provides an example of a PIM device 100 that receives filter data 204, which maybe stored in buffer memory 112. Filter data 204 may comprise the elements of one or more filters 207a-c. A filter 207a-c may be predetermined by a system or a user. A filter 207a-c may be a convolutional filter. A filter 207a-c may be formatted as a matrix of elements where the element values are selected for purposes of searching for or identifying a feature. In thecontextof image recognition in a convolutional neural network, a filter 207a-c may contain elements designed to detect a feature such as, for example, an edge, a horizontal line, a vertical line, etc. As shown in the example of FIG. 2, the filter data 204 includes a first filter 207a, a second filter 207b, and a third filter 207c. The filter data 204 may be transferred into the PIM device 100 via a system link 115.

[0028] Further, in the example of FIG. 2, the sequence control 109 is configured to generate a filter array 210 from the filter data 204. For example, the filter array 210 may include one or more elements from filters 207a-c that comprise the filterdata 204. In this example, the filter array 210 comprises a one-dimensional array derived from elements taken from several two-dimensional arrays. For example, the upper left element [1] of the first filter 207a is the first element in the filter array 210. The upper right element [2] of the first filter 207a is the second element in the filter array 210. The lower left element [3] of the first filter 207a is the third elementin the filter array 210. The lower right element [4] of the first filter 207a is the fourth elementin the filter array 210. Then, the same pattern may be applied to the second filter 207b, and then again to the third filter 207c. Thus, in some embodiments, the elements of multiple filters 207a-c may be merged into a single filter array 210 by the sequence control 109. Alternatively, the filterdata may be delivered to sequence control 109 in this merged format.

[0029] The PIM device 100 may also store a data object 216 into the memory array 103. A data object 216 may be an image, a portion of an image, several portionstaken from several images, audio data, non-audio/visual data, or any other data object that is subject to feature identification.

[0030] The data object 216 may be represented as one or more arrays that express the information of the data object 216. The example of FIG. 2 shows the data object 216 comprising three 4x4 arrays. The data object 216 may be organized according to a format. If the data object 216 represents image data, one example of a data format is Red-Blue-Green (RGB), where the image is described with respect to a red channel, green channel, and blue channel. In some embodiments, different arrays are used to represent differentchannelsfora particularformat. For example, if the data is in an RGB format, a first array may contain data fora red channel, a second array may contain datafor a green channel, and a third array may contain datafor a blue channel. The data object 216 may be formatted in different formats other than RGB.

[0031] The data object 216 may be organized into one or more data object arrays 219a-c. A data object array 219a-c may be an array constituting a portion of the data object 216. The sequence control 109 and FMA unit 106 may be configured to store data in the memory array 103 according to the data object arrays 219a-c. In addition, the sequence control 109 and FMA unit 106 may convert the relevant elements of a data object array 219a-c into a group such as, for example, a one-dimensional array. In this example, the data object array 219a may be stored into a part of the upper third of the memory cells, the second data object array 219b, may be stored into a part of the middle third of the memory cells, and the third data object array 219c may be stored into a part of the lower third of the memory cells.

[0032] FIG. 2 shows a plurality of groups labeled N1 , N2, N3, . . . N9.

In some embodiments a group is organized as a single column-group in the memory array 103. A first group N1 is composed of the array [1 , 2, 5, 6, 17, 18, 21 , 22, 33,

34, 37, 39] A second group N2 is composed of the array [2, 3, 6, 7, 18, 19, 22, 23, 34, 35, 38, 39] Groups N3-N9 are also formed in a similar mannerby sliding a single column frame to the next column. Thus, the allocation pattern used in the example of FIG. 2 is based on a single column frame that stores the data object 216 in a plurality of groups N1-N9 in adjacent columns. Thus, the data object 216 is stored in the memory array 103 according to an allocation pattern in a mannerthat allows each group specified by the allocation pattern to be computed in an efficient mannerby the FMA unit.

[0033] As shown in the example of FIG. 2, the sequence control 109 and FMA unit 106 may apply the filter array 210 to a set of memory array column- groups N1-N9 stored in the memory array 103 according to an allocation pattern. As shown in FIG. 2, the filterarray 210 is applied to all column-groups N1-N9 of the to memory array 103, where each column-group includes at least some elements of each data object array 219a-c stored according to an allocation pattern. The FMA unit 106 may perform a dot product calculation forall column-groups N1-N9 concurrently based on the filterarray 210. For example, the FMA unit 106 may perform a dot product calculation forcolumn-group N1 and the filterarray 210. Flere, the group N1 is composed of is the array [1 , 2, 5, 6, 17, 18, 21 , 22, 33, 34, 37, 39], the filterarray 210 is composed of thearray [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] and the dot product of the two is performed as follows: 1 x 1 + 2 x 2 + 5 x 3 + 6x 4 + 17 x 5 + 18 x 6 + 21 x 7 + 22 x 8 + 33 x 9 + 34 x 10 + 37 x 11 + 39 x 12 = 2060. Concurrently with the group N1 , the FMA unit 106 may perform a dot product calculation on the next group, group N2 [2, 3, 6, 7, 18, 19, 22, 23, 34, 35, 38, 39] and the filterarray 2072 [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] as follows: 2 x 1 + 3 x 2 + 6 x 3 + 7 x 4 18 x 5 + 19 x 6 + 22 x 7 + 23 x 8 + 34 x 9 + 35 x 10 + 38 x 11 + 39 x 12 = 2138. Each computing unit of the FMA unit may perform a dot product computation for a relevant column concurrently with all other computing units. Thus, all dot products in memory array may be computed in parallel with each other.

[0034] Column-group N1 , column-group N2 as well as the remaining column-groups N3-N9 are each applied as inputs to a dot product calculation with the filterarray 210 to produce dot product calculations for each column-group. The sequence control 109 and FMA unit 106 may apply the filter array 210 to all column- groups concurrently in a series of iterations. Specifically, the sequence control 109 sequences through the filterarray 210 by applying each number as a common operand to the FMA unit 106. Each group of sense amplifiers of FMA unit 106 may load a numberfrom relevant column-group and perform multiplication and accumulation to buffer memory 112. In a first iteration the sequence control 109 applies the first multiplierfrom the filterarray 210 (e.g. the number ’) to thefirst multiplicands of all groups N1-N9or to a subset of groups of N1-N9to produce partial results. As part of this first iteration, for group N1 , the calculation is 1 x 1 = 2, for group N2, the calculation is 1 x 2 = 1 , for group N3, the calculation is 1 x 3 = 3 and so on.

[0035] In the second iteration, the sequence control 109 applies the second multiplierfrom the filterarray 210 (e.g., the number “2”) to the second multiplicands of all groups N1-N9 or a subset of groups of N1-N9 to produce partial results. As part of this second iteration, for group N1 2 x 2 = 4, for group N2, the calculation is2x 3 = 6, for group N3, the calculation is 2 x 4 = 8 and so on.

[0036] In addition, in the second iteration, the sequence control 109 accumulates these partials results to the results from the previous iteration to produce current total results. For example, for group N1, the accumulation operation is 1 + 4 = 5, for group N2, the accumulation operation is 2 + 6 = 8, and for group N3, the accumulation operation is 3 + 8 = 11. The sequence control 109 continues to iterate through multipliers of filter array 210 andapply them to groups N1-N9 to produce multiplication results and accumulate them to currenttotal results until the end of filter array 210. The results of each calculation are stored as a dot product array of data 223. For example, for group N 1 , the final accumulated dot product result is 2072, for group N2, the final accumulated dot product result is 2138 and so on. Thus, the sequence control 109 is able to produce dot product array of data 223 for all groups N1 -N 9 or for a subset of thereof concurrently by iterating through multipliers stored in filter array 210, converting them to common control signal, applying this signal to read relevant multiplicands from the memory array 103 into the FMA unit 106 and applying this control signal to the FMA unit 106 to produce fused mu Itiply-accumulate operation andto update the currenttotal result, which at the end of all iterations constitutes the dot product array of data 223. Multiple dot product arrays of data 223 can be subsequently produced and stored in the buffer memory 112 or in a memory array 103 at another set of rows or in another memory array.

[0037] FIG. 2 shows a dot product array of data 223 with elements corresponding to each individual dotproduct calculation. Elements of the dot product array of data 223 constitute a feature map, which provides information on where (or in which part of) in data object 216 the feature was detected. The highest value in thefeature map corresponds to highest probability of feature presence, and the lowest value in the feature map corresponds to lowest probability of feature presence. The feature map may be passed onto additional layers in a neural network or may be used in classification to identify a feature.

[0038] In some embodiments, the allocation pattern applied to a portion of the object data 216 using a bit-serial allocation pattern. For example, each element, such as a single numberformed by bits, from a set of groups may be structured alongthesame column ofthememory array 103. In other embodiments, the allocation pattern may be applied to the to a portion of the data object 216 using a bit-parallel allocation pattern. For example, each element, such as a single numberformed by bits, from a set of groups may be structured along the rows of the memory array column -wise. In other embodiments, the allocation pattern may be applied to the to a portion of the data object 216 using a serial-parallel allocation pattern. Examples of allocations patterns are discussed in more detail with reference to FIGS. 4-6.

[0039] The system link 115 may be used to access the dot product array data 223 and/or derivatives of the dot product array data for subsequent processing. The dot product array data 223 of the sum of its elements may express at least partly, the information of whether a feature is present in the data object 216.

[0040] The memory array 103 may be stored with additional data object data which is organized into a plurality of groups within the memory array. The additional object data may be a different image or different image data than the object data 216. The FMA unit may be configured to generate additional dot product array of data by applying the filter array 210 to the plurality of groups of the additional array. In this respect, the filter array 210 may be reused for multiple data objects as new data objects are stored into the array. In addition, a different filter array may be applied to the contents of the memory array 103. In addition, same filterarray may be applied to different memory arrays with different data objects stored in them.

[0041] In some embodiments, the PIM device 100 comprises a plurality of memory arrays 103. Each memory array 103 may store the same data object at the same time. Differentfilter arrays may be applied to each memory array 103 to effect concurrent processing of a data object with respect to different filter data.

[0042] FIG. 3 is a drawing of a PIM device 100 configured to apply a data object array to a memory array 103 using an allocation pattern according to various embodiments. While FIG. 2 depicts an example of applying a filter array to a data object 216 fixed in a memory array 103, FIG. 3, applies a data object 306 to filter data 315 fixed in a memory array. The data 306 object is applied by the sequence control 109 to the cells of the memory array 103 according to an allocation pattern. The object data 306 may be organized in a plurality of data object arrays 309a-c. A data object 306 be represented as various data object arrays 309a-c.

The sequence control 109 may be configured to generate a data object array 312 based on the data elements from one or more data object arrays 309a-c of the data object 306. The data object 306 may be subject to feature identification using a plurality of filters.

[0043] The memory array 103 may store filter data 315 according to an allocation pattern. The filter data may comprise several filter arrays 318a-c. Afilter array 318a-c may be a convolutional filter. For example, the filter data 315 may be convolutional filterdata that includes a plurality of multi -dimensional convolutional filters such as, for example, filter arrays 318a-c.

[0044] In some embodiments, the allocation pattern may be a bit-serial allocation pattern, where elements of groups of filters 318a-c are structured along a column of the memory array 103 fora bit-serial pattern. In other embodiments, the allocation pattern is a bit-parallel allocation pattern, where elements of groups of filters 318a-c are structured along the rows of the memory array 103 for a bit-parallel pattern .

[0045] The filterdata 315 is stored in the memory array 103 according to an allocation pattern. For example, multiple filter arrays may be organized in column groups M1-M9. For example, a first filter array 318a, a second filter array 318b, and a third filter array 318c may be converted intoa single dimension array constituting the column group M2. Additional filter arrays of the filterdata 315 may similarly be organized in other groups such as groups M1 and M3-M9.

[0046] The sequence control 109 may be configured to apply the data object array 312 to the groups M1-M9 stored in the memory array 103 according to an allocation pattern. For example, the data object array 312 is applied to a first group M1 to generate a dot product calculation asfollows: group M1 composed of the array [1 , 2, 5, 6, 17, 18, 21 , 22, 33, 34, 37, 39] is multiplied by the data array 312 [2, 3, 6, 7, 18, 19, 22, 23, 34, 35, 38, 39] to generate dot product result 6935. Concurrently with group M1 , the group M2 composed of the array [2, 3, 6, 7, 18, 19, 22, 23, 34, 35, 38, 39] is multiplied by the data array 312 to yield the dot product calculation 7142. In the first iteration, the sequence control 109 applies the first multiplierfrom the data array 312 (e.g., the number “2”) to the first multiplicands of all groupsM1-M9 ora subset of groups of M1-M9 to produce partial results. As part of this first iteration, for group M1 , the calculation is2x 1 = 2, for group M2, the calculation is2x 2 = 4, for group M3, the calculation is 2 x 3 = 6 and so on. In the second iteration, the sequence control 109 applies the second multiplierfrom the data array 312 (e.g., the number “3”) to the second multiplicands of all groups M1-

M9 ora subset of groups of M1-M9 to produce partial results. In the second iteration, for group M1, the calculation is3 x2 = 6, for group M2, the calculation is 3 x 3 = 9, for group M3, the calculation is 3x 4 = 12 and so on.

[0047] In addition, in thesecond iteration, the sequence control 109 accumulates these partials results to the results from the previous iteration to produce current total results. For example, for group M1, the operation is 2 + 3 = 5, for group M2, theoperation is 4 + 9 = 13, forgroupM3, the operation is 6 + 12 = 18. The sequence control 109 continues to iterate through multipliers of data array 312 and apply them to groups M1-M9 to produce multiplication results and accumulate them to currenttotal results until the end of data array 312. The results of each calculation are stored as a dot product array of data 321. For example, for group M1, the final accumulated dot product result is 6935, for group M2 the final accumulated dot product resultis 7142 and so on. Thus, the sequence control 109 is able to produce dot product array of data 321 for all groups M1-M9 or for a subset of thereof concurrently by iterating through multipliers stored in data array 312, converting them to common control signal, applying this signal to read relevant multiplicands from the memory array 103 into the FMA unit 106 and applying this control signal to the FMA unit 106 to produce fused multiply-accumulate operation and to update the currenttotal result, which at the end of all iterations constitutes the dot product array of data 321. Multiple dot product arrays of data 321 can be subsequently produced and stored in the buffer memory 112 or in a memory array 103 at an other set of rows or in another memory array.

[0048] Thus, the FMA unit 106 is configured to generate dot product array data 321 in response to applying the data object array 312 to groups within the memory array 103 according to the allocation pattern. Additional data object arrays 327 may be may include different elements of the data object 306 so that the data object 306 is represented as a plurality of data object arrays 312, 327. Each of the additional data object arrays 327 may be applied to the groups M1-M9 to generate an other dot product array data 321.

[0049] The values of the dot product array data 321 may be accumulated to generate data that may be a part of a feature map used to identify features described by filter data 315, and that may be contained in object data 306. For example, each of the additional data object arrays 327 may be applied to a particular group M1-M9 to generate corresponding dot product calculations.

[0050] In addition, the memory array 103 may be configured to store an additional set of filter arrays comprising a plurality of groups within the memory array 103. Here, additional filterdata containing newfilter arrays may be stored into the memory cells of the memory array 103. The FMA unit 106 may be configured to generate an additional dot product arrays of data by applying the data object 306 to the additional filterdata.

[0051] In some embodiments, the PIM device 100 comprises a plurality of memory arrays 103. Each memory array 103 may store the same filter data 315. Different data objects may be applied to each memory array 103 to effect con current processing of multiple data objects with respect to the same filter data. Each data object may represent an image of portions of a video stream. Consecutive images or video streams may be pipelined orbatched using multiple memory arrays 103.

[0052] FIG. 4 is a drawing of an example of a bit-serial allocation pattern for organizing data in a memory array of a PIM device 100. The memory array 103 of FIG. 4 comprises memory cells addressable by row (r1-rn) and column (c1-cn). In addition, each cell is depicted as a multi-state cell that is configured to store two bits of data (“00”, “01”, “10”, or “11”). Although multi-state cells are shown in the example of FIG. 4, the discussion of FIG. 4 is applicable to single bit cel Is, other multi-state cells, or any other type of cell configured to store data, including cells that can store data in analog form with wide dynamic range and precision.

[0053] The PIM device 100 is configured to store first data 403 in the memory array 103 according to a bit serial allocation pattern. Second data 409 is used to generate second array of 406 which includes elements from second data 412. Here, the second array of data 406 is composed of the array [1 , 5, 3, 8] in decimal format. The second array of data 406 is also expressed in a binary format as follows [0001 , 0101, 0011, 1000] Each element of the second array of data 406 is provided as an inputto the FMA unit 106 in a form of control signal where it serves as a multiplierto perform a series of iterations on multiplicands stored in the memory array 103. The first data 403 includes a first group 415 composed of the array [2, 3, 6, 7] in decimal format. The first group 415 may be expressed in binary as follows: [0010, 0011, 0110, 0111] The first group 415 may be referred to as a first array of data elements.

[0054] In one embodiment, the first data 403 is a data object while the second data includes one or more filters to be applied to the data object. Here, the second array of data 406 may be an array for a convolutional filter. In another embodiment, the first data 403 includes filter data of one or more convolutional filters while the second data 409 includes a data object. Here the second array of data comprises elements of the data object.

[0055] The sequence control 109 performs iterations of multiplications and accumulations by applying each elementfrom the second array of data 406 to the groups stored in the memory array 103. The sequence control may be implemented as a controller programmed by microcode that performs a series of dot product multiplications and accumulations on one or more multiplicands simultaneously by applying each multiplier. Each multiplier(e.g., an elementof a second array of data 406) is provided via a control signal to FMA array, or via a system lin k 115 ora memory buffer112 of the PIM device 100. In some embodiments, the second array of data 406 may be stored in the memory array 103 by the sequence control 109.

[0056] When applying a bit-serial allocation pattern, the sequence control 109 converts the first data 403 into a plurality of groups where each group is an array of elements that constitute the first data 403, such as, for example a first group 415. In this respect, each group may be formed to constitute sets of elements that together, represent the first data 403. Each group is stored bit-by-bit along a bit line. For example, the first group 415 is composed of the array [2, 3, 6, 7] in decimal format and is stored along the first bit line. For example, the bit line referenced as column c3 is the only bit line that stores the first group 415. Moreover, because each element of the first group 415 is larger than the capacity of a cell, each data array element occupies multiple cells along the same bit line. For example, the first elementof the first group 415 is the number“2” (in decimal) or “0011” (in binary).

This element is stored in cells (r1 , c3) and (r2, c3), beginning with the two least significant bits. This pattern continues such thatthefirst group 415 is stored along column c3 between rows r1 and r8.

[0057] A second group may be stored along the same bit line as the first group 415 or possibly along other bit lines. Subsequent groups are stored in a similar manneruntil thefirst data 403 is completely stored in the memory array 103.

If memory array 103 is not sufficient to store data 403 than it could be stored in anothermemory array or another part of it could be stored in memory array 103 after the computations are done on the first part. As a result, thefirst data 403 is stored in the memory comprising sets of groups that are structured according to a bit-serial allocation pattern where each set of groups is stored along a respective bit line.

[0058] Using this allocation pattern, the sequence control 109 is configured to apply the second array of data 406 to each of the plurality of sets of groups. For example, the sequence control 109 may activate all or a subset of all of bit lines or columns of the memory array 103 concurrently so that the oneor more elements (e.g. bits) within each set of groups stored on relevant bit line are provided as inputs into the relevant computing units with in FMA unit 106 along with the second array of data 406 provided by sequence control 109. For example, the sequence control 109 may open or otherwise activate the first row to select the first elements or first sets of elements from the groups within thefirstrow. This includes the bits stored across several groups in columns d -cn at the selected row. These bits are provided as inputs into relevant computing units of the FMA unit 106.

[0059] The FMA unit 106 is controlled by the sequence control 109 to perform a plurality of concurrentdot product multiplications based on data from activated columnsstored at the activated row at each iteration. The FMA unit106 may be configured as a plurality of sub-FMA units 423. For example, each sub-FMA unit426 may be dedicated to separate bit lines or columns. Flere, a first sub-FMA unit426 (unitA) performs dot product multiplications where the multiplicand is received from the first column d , a second sub-FMA unit426 (unit B) performs dot product multiplications where the multiplicand is received from the second column c2, a third sub-FMA unit 426 (u n it C) performs dot product multiplications where the multiplicand is received from the third column c3, and so on. Microcode may be used to configure and reconfigure the FMA unit 106 so that the numberof input columns to a particular sub-FMA unit426 is variable. A sub-FMA unit426 may receive a multiplicand from a plurality of columns, for example. Flowever, the example of FIG. 4 uses a bit-serial allocation pattern where each sub-FMA unit 426 is dedicated to a single column.

[0060] In a first iteration, the concurrentdot product multiplications include multiplying the elements of each column along rows r1 and r2 with the mu Itiplier “0001 ”, which is thefirst numberof the secondarray of data 406 which is [0001,0101,0011,1000] The first sub-FMA unit 423 (unitA) multiplies the number “0001” in binary with the bits stored in the cells (r2, d) and (r1, d), the second sub- FMA unit423 (unitB) multiplies the number “0001” with the bits stored in the cells

(r2, c2) and (r1, c2), the third sub-FMA unit423 (unitC) multiplies the number OOT’ with the bits stored in the cells (r2, c3) and (r1, c3), which is “0010” in binary, and so on. Thus, thefirst iteration involves several dot product calculations performed by sub-FMA units 426 in parallel for one or more activated rows. For a given iteration, multiple rows may be activated in a particular order to multiply the contents of a plurality of cells that store a single data array element. This multiplication may involve carry operations that generate intermediate values that may be stored in a buffer memory 112.

[0061] In the second iteration, the concurrent dot product multiplications include multiplying the elements of each column along rows r3 and r4 with “0101” in binary, which is based on the second number of the second array of data 406 “[0001 ,0101 ,0011 ,1000]”. Here, the first sub-FMA unit 423 (unit A) multiplies the number“0101” with the bits stored in the cells (r4, d) and (r3, d), the second sub-FMA unit 423 (unit B) multiplies the number“0101” with the bits stored at the cells (r4, c2) and (r3, c2), the third sub-FMA unit 423 (unitC) multiplies the number “0101” with the bits stored in cells (r4, c3) and (r3, c3), which is “0011” in binary, and so on. After or during the dot product multiplications of the second iteration, each sub-FMA unit 423 may accumulate the dot product multiplication results of the previous iteration(s). The FMA unit 106 may use the buffermemory 112 to store the dot product multiplication results of the previous iterations or any numbers being carried as a result of arithmetic operations. Each sub-FMA unit423 may perform a mu Itiply-accumulate operation at a given iteration. For example, in a first iteration, the third sub-FMA unit (unitC) multiplies 001” in binary with the numberstored across the cells (r2, c3) and (r1, c3), which is the binary number “0010” to generate a first result. At the second iteration, the third sub-FMA unit423 (unitC) multiplies the binary number “0101” with the numberstored across the cells (r4, c3) and (r3, c3), which is the binary number “0011” to generate a second result. The first result and second result are summed together or otherwise accumulated to generate an accumulated result. This process repeats for each iteration until the dot product multiplication for each group such as the first group415 is complete. Although decimal numbers are shown byway of example, various embodiments may use floating point numbers, numbers expressed with more multiple bits (e.g., more or less than 4-bit numbers, as shown in the example of the figures), or analog values of memory cells (e.g., analog voltages and currents) to produce relevant results (e.g., a floating point number generated from floating pointcalculations, an analog number based on current or voltage summations which can be converted to digital format or maintained in analog format and stored in respective memory cells).

[0062] Elements of the computed dot product array may constitute a feature map, which provides information on where (or in which part of) in the data object the feature was detected. The highest value in the feature map corresponds to highest probability of feature presence, and the lowest value in the feature map corresponds to lowest probability of feature presence. The feature map may be passed onto additional layers in a neural network or may be used in classification to identify a feature.

[0063] FIG. 5 is a drawing of an example of a bit-parallel allocation pattern for organizing data in a memory array of a PIM device. When applying a bit- parallel allocation pattern, the sequence control 109 converts the first data 403 into a plurality of groups such as, for example, a first group 415. Each group is an array of elements of the first data 403. Each group may be formed to constitute sets of elements that together, represent the first data 403 as a first array of data. The bits of a number in a particular group are stored in parallel along a word line or row. For example, a first number of the first group 415, the number “2” in decimal, from the group that is composed of the array [2, 3, 6, 7], is stored along the first word line or row. For example, with two bits per memory cell and with 4-bit total width, the number “2” (represented as “0010” in a 4-bit binaryformat) may occupy two cells: the cell (r1 , c3) with bits “00”, and the cell (r1 , c4) with bits “10”. A second number of group 415, the number“3”, from the group that is composed of the array [2, 3, 6, 7], is stored along the second word line or row. For example, with two bits per memory cell and with 4-bit total width, the number “3” may occupy two cells: the cell (r2, c3) with bits “00”, andthecell (r2, c4) with bits 11. Thiscontinuesforall numbers of the first group until the first group is stored in the memory array 103. Subsequent groups are stored in a similar manner, such that one or more groups may be stored in the third and fourth columns beginning on row 5 (following the first group 415) or stored along the fifth and sixth columns until the first data 403 is stored in the memory array 103. As a result, the first data 403 is stored in the memory as a first array of data comprising groups that are structured according to a bit-parallel allocation pattern.

[0064] Using this allocation pattern, the sequence control 109 is configured to apply the second array of data 406 to each of the plurality of groups. The sequence control 109 may control the calculations performed by an FMA unit 106 that is configured to include a plurality of sub-FMA units 523, where each sub- FMA unit 523 is dedicated to two columns. For example, a first sub-FMA unit 523 (unitA) may perform calculationsformultiplicands supplied bycolumnsd and c2, a second sub-FMA unit 523 (unitB) may perform calculationsformultiplicands supplied by columns c3 and c4, a third sub-FMA unit 523 (unitC) may perform calculationsformultiplicands supplied by columnsc5 and c6, and so on.

[0065] The sequence control 109 may activate each word line or row of the memory array 103 in a sequence so that the whole numbers of one or more groups with in the word line are provided as inputs into the FMA unit 106 along with the second array of data 406. For example, in a first iteration, the sequence control 109 may open or otherwise activate the first row or word line so that the first numbers of groups with in the first row or word line, such as the first group 415 and first numbers of other groups stored on the first row, are provided as inputs into the FMA unit 106. The FMA unit 106 is controlled by the sequence control 109 to perform a plurality of concurrent dot product multiplications using sub-FMA units 523 configured to operate in parallel for a given iteration. The concurrent dot product multiplications include multiplying each of thefirst numbers, such as number “2” of the first group 415 (e.g., [2, 3, 6, 7]), and anyfirst numbers of any additional groups along the selected word line with thefirst number of the second array of data 406 (e.g., [1, 5, 3, 8]), which isthe number ’. For example, for a first iteration, the second sub-FMA unit 523 (unitB) performs a dot product multiplication using the mu Itiplier “1 ” in decimal (“0001” in binary) and using the multiplicand “2” in decimal (“0010” in binary) as provided by cells (r1, c3) and (r1, c4).

[0066] For example, the sequence control 109 selects the appropriate memory cells of the memory array 103 using the allocation pattern to provide them as inputs into the FMA unit 106. The first dot product multiplication iteration includes selecting the multiplicands such as thefirst number of the first group 415 (e.g., [2, 3, 6, 7]), which isthe number“2” along with first numbers of other groups stored along the first row and selecting the multiplier as thefirst number of second array of data 406 (e.g., [1, 3, 5, 8]) which isthe number ’. The second dot product iteration includes selecting the multiplicands such as the second number of the first group 415 (e.g., [2, 3, 6, 7]) which isthe number“3” and second numbers of other groups stored along the second row and selecting the multiplier as the second number of the second array of data 406 (e.g., [1, 5, 3, 8]) which is the number“5”. After the second iteration, the process involves accumulating the dot product multiplication results with previous results. The parallel multiplication and accumulation may operate in a mannersimilarto the mannerdiscussed in the example of FIG. 4, but where a data array element is stored across a plurality of columns in parallel.

[0067] Although decimal numbers are shown byway of example, various embodiments may use floating point numbers, numbers expressed with more multiple bits (e.g., more or less than 4-bit numbers, as shown in the example of the figures), or analog values of memory cells (e.g., analog voltages and currents) to produce relevant results (e.g., a floating point number generated from floating point calculations, an analog number based on current or voltage summationswhich can be converted to digital format or maintained in analog format and stored in respective memory cells).

[0068] Elements of the computed dot product array may constitute a feature map, which provides information on where (or in which part of) in the data object the feature was detected. The highest value in the feature map corresponds to highest probability of feature presence, andthe lowest value in the feature map corresponds to lowest probability of feature presence. The feature map may be passed onto additional layers in a neural network or may be used in classification to identify a feature.

[0069] FIG. 6 is a drawing of an example of a serial-parallel allocation pattern for organizing data in a memory array of a PIM device 100. When applying a serial-parallel allocation pattern, the sequence control 109 converts the first data 403 into a plurality of groups including a first group 615, where each group is an array of elements of the first data 403. The first group 615, may be composed of the array [26, 54, 86, 90] which is represented in 8-bit binary digits as [00011010, 00110110, 01010110, 01011010] Each group is stored both in serial along a plurality of word lines and stored in parallel along a plurality of bit lines.

[0070] For example, the first number of the first group is the decimal number“26” (“00011010” in binary). This number is stored across multiple columns and multiple rows. It may be split up into four portions “00”, “01”, “10”, and“10” which, together, form the decimal number “26”. Along afirst row, a first cell (r1 , c3) and second cell (r1 , c4) may store part of the number and along a second row, a third cell (r2, c3) andfourth cell (r2, c4) store the remainder of the number. The least significant digits of the binary number “00011010” are “10” andtheymay be stored in cell (r1 , c3) while the most significantdigits are “00” and are stored in cell (r2, c4). This pattern continues until all numbers of the first group is stored in the memory array 103. Subsequent groups are stored in a similar manneruntil thefirst data 403 is stored in the memory array 103. As a result, the first data 403 is stored in the memory as a first array of data comprising groups that are structured according to a serial-parallel allocation pattern.

[0071] Using this allocation pattern, the sequence control 109 is configured to apply the second array of data 406 to each of the plurality of groups. The sequence control 109 may control the calculations performed by an FMA unit 106 that is configured to include a plurality of sub-FMA units 523. In this example, each sub-FMA unit 523 is dedicated to two columns. For example, a first sub-FMA unit 523 (unitA) may perform calculationsformultiplicands supplied bycolumnsd and c2, a second sub-FMA unit 523 (unit B) may perform calculations for multiplicands supplied by columns c3 and c4, a third sub-FMA unit 523 (unitC) may perform calculationsformultiplicands supplied by columns c5 and c6, and so on. Each element of the second array of data 406 is provided as a multiplier. In the example of FIG. 6, the second array of data is represented in 8-bit binary as the array [00000001 , 00000101 , 00000011 , 00001000] A first iteration uses as the multiplierthefirst numberin the second data array 406 which is represented binary 0000001”. A second iteration uses as the multiplierthe second numberin the second data array 406 is represented binary “00000101” and so on. This process continues until the second array of data 406 is applied to a group stored in the memory array 103.

[0072] The sequence control 109 may activate each word line or row of the memory array 103 in a sequence so that part of each numbers of one or more groupswithin theword line are provided as inputs into the FMA unit 106 along with the second array of data 406. For example, in a first iteration, the sequence control 109 may open or otherwise activate the rows r1 and r2 so that the first numbers of groupswithin word lines are provided as inputs into the FMA unit 106. The FMA unit 106 is controlled by the sequence control 109 to perform a plurality of concurrent dot product multiplications using sub-FMA units 523 configured to operate in parallel in a particular iteration. The concurrent dot product multiplications include multiplying portions of each of the first numbers, the decimal number “26” of the first group 615 (e.g., [26, 54, 86, 90]), and any first numbers of any additional groups along the selected word linewith the portions of the first number of the second array of data 406 (e.g., [1 , 5, 3, 8]), which is the number ’.

[0073] For example, for a first iteration, the second sub-FMA unit523 (unit B) performs a dot product multiplication using the multiplier 0000001” in binary (which is the number Ί” in decimal) and using the multiplicand 0011010” as provided by cells (r1, c3), (r1, c4), (r2, c3), and (r2, c4), where the multiplicand represents the decimal number “26”. As part of the first iteration, the sequence control 109 may activate rows r1 and r2 as well as all or at least a portion of the columns.

[0074] In a second iteration, the second sub-FMA unit 523 (unit B) performs a dot product multiplication using the multiplier 0000101” in binary (which is the number5 in binary) and using the multiplicand “00110110” as provided by cells (r3, c3), (r3, c4), (r5, c3), and (r5, c4), where the multiplicand represents the number “54” in decimal. The results of the two iterations are accumulated or otherwise summed. Thus, after four iterations, the dot product of multiple groups (including the first group 615) and the second array of data 406 is calculated.

[0075] In some embodiments, any iteration may include of two stages. The first stage may multiply the multiplierby the first parts of all respective multiplicands. For example, in case of the multiplicand “54” (in decimal) and the multiplier ^” (in decimal), the operation is represented as the following binary multiplication: Ό0000101’ x‘0110’ = Ό0011110’, which is 30 (in decimal). The second stage may multiply the multiplierby the second parts of all respective multiplicands, then shift bya relevant number of bits andadd to the second part. For example, in case of number“54” (in decimal) and the multiplier ^” (in decimal), the operation is represented as the following binary multiplication: Ό000010T x Ό01 T = Ό000111 T, which is 15. After shifting byfourbits, the result becomes ‘11110000’. Thereafter, this shifted value is added to the result computed in the first stage as follows: Ί 1110000’ + Ό0011110’ = 270 (in decimal). While this demonstrates one example, other embodiments are contemplated by the present disclosure. .

[0076] The allocation patterns described herein may be distributed such that multiple parts of the groups may be stored in non-adjacent memory rows. Yet these storage patterns still may be aligned along relevant columns. Although the allocation patterns presented herein may be bit-serial, bit-parallel, or serial-parallel, other allocation patterns may be used as long as theirmap is known to the FMA unit 106 and sequence control 109. For example, a diagonal allocation patterns may be used. Dynamic allocation patterns may also be used. For example, data allocation in a memory array 103 may be in bit-parallel configuration while the results may be stored in bit-serial configuration, etc..

[0077] Although decimal numbers are shown by way of example, various embodiments may use floating point numbers, numbers expressed with more multiple bits (e.g., more or less than 4-bit numbers, as shown in the example of the figures), or analog values of memory cells (e.g., analog voltages and currents) to produce relevant results (e.g., a floating point number generated from floating point calculations, an analog number based on current or voltage summationswhich can be converted to digital format or maintained in analog format and stored in respective memory cells).

[0078] Elements of the computed dot product array may constitute a feature map, which provides information on where (or in which part of) in the data object the feature was detected. The highestvalue in thefeature map corresponds to highest probability of feature presence, andthe lowest value in thefeature map corresponds to lowest probability of feature presence. The feature map may be passed onto additional layers in a neural network or may be used in classification to identify a feature.

[0079] FIG. 7 is a flowchart illustrating varying examples of the functionality of a PIM device 100 according to various embodiments. The boxes in the flowchart may represent microcode, machine code, firmware, or other software executable by a PIM processor of a PIM device 100. The boxes of the flowchart may alternatively represent steps in a method 700. The method may be performed by a PIM device 100.

[0080] At 703, the PIM device 100 stores a first array of data into a memory array 103. The first array of data may be data subject to feature identification. This may be a data object such as, for example, an image data object or other contentthat contains information needing classification or analysis. The first array of data may alternatively be comprised of elements of at least one filter such as, for example, a convolutional filter. A filter may be a filter used in neural network to identify features in a data object.

[0081] A sequence control 109 may identify elements of the first array of data and store them into corresponding cells of the memory array 103 so that each array element maps to a corresponding memory cell of the memory array 103. In addition, the sequence control 109 may organize the data stored in the memory array accordingto a plurality of groups usingan allocation pattern. The allocation pattern may be bit-serial, bit-parallel, or serial-parallel or other.

[0082] At 706, the PIM device 100 receives a second array of data.

The second array of data may be data subject to feature identification. This may be a data object such as, for example, an image data object or other content that contains information needing classification or analysis. The second array of data may alternatively be elements of at least one filter such as, for example, a convolutional filter. A filtermay be a filter used in neural network to identify features in a data object.

[0083] A sequence control 106 may derive the second array of data from one or more arrays such that the second array of data comprises elements selected from a plurality of different arrays.

[0084] At 709, the PIM device 100 applies the second array of data to groups of elements of the first array of data according to the allocation pattern. For example, a sequence control may use an allocation pattern to select different groups of elements of the first array of data. Usingan allocation pattern, the second array of data and the groups of elements of the first array of data are then inputted into an FMA unit 106 through a series of iterations. In some embodiments, the sequence control activates one or more word lines and one or more bit lines to allow the FMA unit 106 to receive multiple groups as multiplicand inputs for performing concurrent dot product calculations.

[0085] At 712 the PIM device 100 generates a dot product array of data using an FMA unit 106. The dot product array of data may be part of a feature map.

It may be retrieved using a system link 115 as part of implementing a layer in a convolutional neural network.

[0086] At 715, if there are groups remaining in thememory array 103, then the sequence controller selects the next row and/or column of the memory array 103 to identify a subsequent set of groups of elements of the first array of data. The PIM Device lOO continues to apply the second array of data to subsequent groups of elements of the first array of data until no groups remain or until a certain pointas programmed in the microcode of the sequencer or commanded to it via system link 115.

[0087] At 718, upon completing operations on data contained by a memory array 103, the PIM device 100 loads additional first array of data and/or receive additional second array of data. In this respect either the memory array 103 is updated with new data or the sequence control 109 applies different data to the memory array 103 or both. The process iterates by generating additional dot product array data.

[0088] Although the flowchart shows a specific order of execution, it is understood that the order of execution may differfrom that which is depicted. For example, the order of execution of two or more boxes may be scrambled relative to the order shown. Also, two or more boxes shown in succession may be executed concurrently or with partial concurrence. Further, in some embodiments, one or more of the boxes may be skipped or omitted. In addition, any number of counters, state variables, warning semaphores, or messages might be added to the logical flow described herein, for purposes of enhanced utility, accounting, performance measurement, or providing troubleshooting aids, etc. It is understood that all such variations are within the scope of the present disclosure.

[0089] The components carrying out the operations of the flowchart may also comprise software or code that can be embodied in any non -transitory computer-readable medium for use by or in connection with an instruction execution system such as, for example, a processor in a computing system. In this sense, the logic may comprise, for example, statements including instructions and declarations that can be fetched from the computer-readable medium and executed by the instruction execution system. In the context of the present disclosure, a “computer- readable medium” can be any medium that can contain, store, or maintain the logic or application described herein for use by or in connection with the instruction execution system.

[0090] FIG. 8 illustrates an example networked system 800 that includes a PIM device 100, in accordance with some embodiments of the present disclosure. FIG. 8 illustrates example parts of an example of a computing device 802 with is part of the networked system 800. FIG. 8 shows how such computing devices can be integrated into various machines, apparatuses, and systems, such as loT (Internet of Things) devices, mobile devices, communication network devices and apparatuses (e.g., see base station 830), appliances (e.g., see appliance 840), and vehicles (e.g., see vehicle 850).

[0091] The computing device 802 and other computing devices of the networked system 800 (e.g., see computing devices 822a, 822b, 822c, and 822d) can be communicatively coupled to one or more communication networks 820. The computing device 802 includes, for example, a bus 806, a controller 808 (e.g., a CPU), other memory 810, a network interface 812, a storage system 814, other components 816 (e.g., any type of components found in mobile or computing devices, GPS components, Input / Output (I/O) components such various types of user interface components, sensors, a camera, etc.), and a PIM device 100. The other components 816 may also include one or more user interfaces (e.g., GUIs, auditory user interfaces, tactile user interfaces, etc.), displays, different types of sensors, tactile, audio and/or visual input/output devices, additional application -specific memory, one or more additional controllers (e.g., GPU), or any combination thereof. The bus 806 communicatively couples the controller 808, the other memory 810, the network interface 812, the data storage system 814 and the other components 816, and ean couple such components to the PIM device 100 in some embodiments. For example, a system link 115 of the PIM device 100 may couple to the bus 806.

[0092] The computing device 802 includes a computer system that includes at least controller 808, other memory 810 (e.g., random access memory (RAM), read-only memory (ROM), flash memory, dynamic random-access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), static random-access memory (SRAM), cross-point or cross-bar memory, crossbar memory, etc.), the PIM device 100, and data storage system 814, which may communicate with each other via bus 806 (which can include multiple buses). In some embodiments, the PIM device 100 may not communicate over bus 806.

[0093] To put it anotherway, FIG. 8 includes a block diagram of computing device 802 that has a computer system in which embodiments of the present disclosure can operate. In some embodiments, the computer system can include a set of instructions, for causing a machine to perform at least part anyone or more of the methodologies discussed herein, when executed. In such embodiments, the machine can be connected (e.g., networked via network interface 812) to other machines in a Local Area Network (LAN), an intranet, an extranet, and/or the Internet (e.g., see network(s) 820). The machine can operate in the capacity of a server or a clientmachine in client-server network environment, as a peer machine in a peer-to-peer (or distributed) network environment, or as a server or a client machine in a cloud computing infrastructure or environment.

[0094] Controller 808 represents one or more general-purpose processing devices such as a microprocessor, a central processing unit, or the like. More particularly, the processing device can be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, single instruction multiple data (SIMD), multiple instructions multiple data (MIMD), or a processor implementing other instruction sets, or processors implementing a combination of instruction sets. Controller 808 can also be one or more special-purpose processing devices such as an ASIC, a programmable logic such as an FPGA, a digital signal processor (DSP), network processor, orthe like. Controller808 is configured to execute instructions for performing the operations and steps discussed herein. Controller 808 can further include a network interface device such as network interface 812 to communicate over one or more communication networks (such as network(s) 820).

[0095] The data storage system 814 can include a machine-readable storage medium (also known as a computer-readable medium) on which is stored one or more sets of instructions or software embodying anyone or more of the methodologies or functions described herein. The data storage system 814 can have execution capabilities such as it can at least partly execute instructions residing in the data storage system. The instructions can also reside, completely or at least partially, within at least one of the other memory 810 and the PIM device 100 and/or within the controller 808 during execution thereof by the computer system, at least one of the other memory 810 and the PIM device 100 as well as the controller 808 also constituting machine-readable storage media. The other memory 810 can be or include main memory or system memory of the computing device 802. The other memory 810 and the PIM device 100 can have execution capabilities such as it can at least partly execute instructions residing in any memory of the computing device 802.

[0096] As mentioned, the networked system 800 includes computing devices, and each of the computing devices can include one or more buses, a controller, a memory, a network interface, a storage system, and other components. Also, each of the computing devices shown in FIG. 8 and described herein can include or be a part of a mobile device or the like, e.g., a smartphone, tablet computer, loT device, smart television, smart watch, glasses or other smart household appliance, in-vehicle information system, wearable smart device, game console, PC, digital camera, or any combination thereof. As shown, the computing devices can be connected to network(s) 820 that includes at least a local to device network such as Bluetooth or the like, a wide area network (WAN), a local area network (LAN), an intranet, a mobile wireless network such as 4G or 5G, an extranet, the Internet, and/or any combination thereof. In some embodiments, as shown with the connection 819, the PIM device 100 can include at least one network interface so that it can communicate separately with other devices via communication network(s)820. For example, the system link 115 may couple to the communication network 820. In this respect, a memory module or a memory module system of the PIM device 100 may have its own network interface so that such a componentcan communicate separately with other devices via communication network(s)820.

[0097] Each of the computing devices described herein can be or be replaced by a personal computer (PC), a tablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), a cellulartelephone, aweb appliance, a server, a network router, a switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine.

[0098] Also, while a single machine is illustrated for the computing device 802 shown in FIG. 8, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform one or more of the methodologies or operations discussed herein. And, each of the illustrated computing devices as well as computing systems can each include at least a bus and/or motherboard, one or more controllers (such as one or more CPUs), a main memory that can include temporary data storage, at least one type of network interface, a storage system that can include permanent data storage, and/or any combination thereof. In some multi-device embodiments, one device can complete some parts of the methods described herein, then send the result of completion over a network to anotherdevice such that an other device can continue with other steps of the methods described herein.

[0099] While the memory, controller, and data storage parts are shown in the example embodiment to each be a single part, each part should betaken to include a single part or multiple parts that can store the instructions and perform their respective operations. The term “machine-readable storage medium” shall also be taken to include any medium that is capable of storing or encoding a set of instructions forexecution bythe machine and that cause the machine to perform any one or more of the methodologies of the present disclosure. The term “machine- readable storage medium” shall accordingly betaken to include, but not be limited to, solid-state memories, optical media, and magnetic media.

[0100] In some embodiments, a PIM device 100 may include multiple memory arrays 103. Each memory array may be configured to implement different stage of convolutional neural network. For example, the dot product array data generated from data in a first memory array 103 may be processed and stored in a second memory array 103 of the PIM device 103.

[0101] In some embodiments, the PIM device 100 may further comprise a PIM processor configured to perform in-memory video codec. The in memory codec may be used in conjunction with a neural network layer implemented using a memory array 103, FMA unit 109, and sequence control 109.

[0102] In some embodiments, the PIM device 100 may further comprise a PIM processor configured to perform pooling operations or perform Rectified Non-Linearunit(ReLU)functions in-memory. The pooling and/orReLU functions may be performed on data concurrently on the dot product array data while it is generated.

[0103] In some embodiments, the PIM device 100 may be coupled to external systems viathe system link 115 to implement a distributed convolutional neural network. For example, the PIM device 100 may communicate with a graphics processor unit (GPU), a Digital Signal Processor (DSP) or other processors.

[0104] In some embodiments, the PIM device 100 may be coupled to external systems viathe system link 115 to implement a distributed convolutional neural network. For example, the PIM device 100 may communicate with a graphics processor unit (GPU), a Digital Signal Processor (DSP) or other processors.

[0105] In some embodiments, the data provided to the PIM device 100 includes a predication mask to parallelize serial operations in the PIM device 100.

[0106] In some embodiments, PIM device 100 is configured to perform 3-dimensional convolution. For example, a data object array may be inputted into an FMA unit 106 with varying filter channels as inputs.

[0107] Disjunctive language such as the phrase “at least one of X, Y, or

Z,” unless specifically stated otherwise, is otherwise understood with the context as used in general to present that an item, term, etc., may be eitherX, Y, or Z, or any combination thereof {e.g., X, Y, and/or Z). Thus, such disjunctive language is not generally intended to, and should not, imply that certain embodiments require at least one of X, at least one of Y, or at least one of Z to each be present.

[0108] Some portions of the preceding detailed descriptions have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the ways used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. The operations are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0109] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. The present disclosure can refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system’s registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage systems.

[0110] The present disclosure also relates to an apparatus for performing the operations herein. This apparatus can be specially constructed forthe intended purposes, or it can include a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program can be stored in a computer readable storage medium, such as any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions, each coupled to a computer system bus.

[0111] The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems can be used with programs in accordance with the teachings herein, or it can prove convenient to construct a more specialized apparatus to perform the method. The structure for a variety of these systems will appear as set forth in the description below. In addition, the present disclosure is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the disclosure as described herein.

[0112] The present disclosure can be provided as a computer program product, or software, that can include a machine-readable medium having stored thereon instructions, which can be used to program a computer system (or other electronic devices) to perform a process according to the present disclosure. A machine-readable medium includes any mechanism for storing information in a form readable by a machine(e.g., a computer). In some embodiments, a machine-readable (e.g., computer-readable) medium includes a machine (e.g., a computer) readable storage medium such as a read only memory (“ROM”), random access memory (“RAM”), magnetic disk storage media, optical storage media, flash memory components, etc.

[0113] It should be emphasized that the above-described embodiments of the present disclosure are merely possible examples of implementations set forth for a clear understanding of the principles of the disclosure. Many variations and modifications may be made to the above-described embodiment(s) without departing substantially from the spirit and principles of the disclosure. All such modifications and variations are intended to be included herein within the scope of this disclosure and protected by the following claims.