FIELDS

CROSS-REFERENCE TO RELATED APPLICATIONS [0001] This application claims the benefit of U.S. Provisional Application No. 62/892,110, filed August 27, 2019, which is hereby incorporated by reference in its entirety.

BACKGROUND

[0002] Embodiments of the present disclosure relate to remote sensing, and more specifically, to imagery-based boundary identification for agricultural fields.

BRIEF SUMMARY

[0003] According to embodiments of the present disclosure, methods of and computer program products for agricultural field boundary identification are provided. A time series of surface reflectance rasters for a geographic region is received. For each of the surface reflectance rasters, at least one index raster is determined, yielding at least one time series of index rasters. The at least one time series of index rasters is divided into a plurality of consecutive time windows. The at least one time series of index rasters is composited within each of the plurality of time windows, yielding a composite index raster for each of the at least one time series of index rasters in each of the plurality of time windows. The composite index rasters are segmented into a plurality of spatially compact regions of the geographic region. A plurality of polygons is generated from the plurality of spatially compact regions, each of the plurality of polygons corresponding to an agricultural field in the geographic region. [0004] In some embodiments, the time series of surface reflectance rasters comprises satellite data. In some embodiments, the time series of surface reflectance rasters spans a growing season in the geographic region. In some embodiments, receiving the time series of surface reflectance rasters comprises determining surface reflectance from uncorrected reflectance data.

[0005] In some embodiments, the at least one index raster comprises a normalized difference vegetation index raster. In some embodiments, the at least one index raster comprises a land surface water index raster. In some embodiments, the at least one index raster comprises a mean brightness raster. In some embodiments, determining the at least one index raster comprises downsampling the surface reflectance rasters.

[0006] In some embodiments, the plurality of consecutive time windows correspond to early, mid-, and late phases of a growing season in the geographic region. In some embodiments, compositing comprises averaging the at least one time series of index rasters within each of the plurality of time windows.

[0007] In some embodiments, segmenting comprises filling missing pixels in the composite index rasters. In some embodiments, filling missing pixels comprises applying linear interpolation to the composite index rasters. In some embodiments, segmenting comprises normalizing the composite index rasters. In some embodiments, segmenting comprises denoising. In some embodiments, denoising comprises applying a spatial low pass filter. In some embodiments, segmenting comprises graph-based segmentation. In some embodiments, segmenting comprises Felzenszwalb segmentation.

[0008] In some embodiments, generating the plurality of polygons comprises applying spatial smoothing to the plurality of spatially compact regions. BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0009] Fig. 1 illustrates a field delineation pipeline according to embodiments of the present disclosure.

[0010] Fig. 2 illustrates an image preprocessing method according to embodiments of the present disclosure.

[0011] Fig. 3 illustrates an image segmentation method according to embodiments of the present disclosure.

[0012] Figs. 4A-B illustrate exemplary image segmentations according to embodiments of the present disclosure.

[0013] Fig. 5 illustrates a segmentation postprocessing method according to embodiments of the present disclosure.

[0014] Figs. 6A-B illustrate exemplary image segmentations according to embodiments of the present disclosure.

[0015] Figs. 7A-B illustrate exemplary field boundaries according to embodiments of the present disclosure.

[0016] Fig. 8 illustrates a method for agricultural field boundary identification according to embodiments of the present disclosure.

[0017] Fig. 9 depicts a computing node according to an embodiment of the present disclosure.

DETAILED DESCRIPTION

[0018] Farm fields represent a fundamental spatial unit of agriculture. Observation, analysis, and modeling of agricultural characteristics at the farm field level requires specification of their geographic boundaries. However, there is no standard reference data set for farm field boundaries.

[0019] While field boundaries can be created by manual digitization, whether by on-site data collection or manual markup of aerial or satellite imagery coupled with on-site verification, this process is not scalable to large areas. Individual US counties can contain thousands of fields, and the US as a whole contains millions of fields. Manually digitizing hundreds or thousands of fields would be prohibitively time consuming. Moreover, agricultural activity often shifts from season to season, utilizing different parts of a given field. Accordingly, the geographic definition of the field must be allowed to change over time, putting a temporal constraint on the validity of a given field identification.

[0020] As used herein, a field boundary refers to a spatially compact (that is, closed and bounded) unit of the landscape that exhibited an approximately uniform pattern temporally and spectrally for a given growing season. In particular, field boundary delineates a bounded area with common crop type and management. A field boundary may differ from what visual inspection of a single image might suggest. For example, visual inspection is likely to be strongly affected by the presence of roads, streams, paths, or other such boundaries, irrespective of whether a crop was actually grown on the land in question.

[0021] Various image processing algorithms may be used to partition images into coherent spatial units (segmentation) based on detection of boundaries in the image. In this way, satellite data at sufficient spatial resolution may be used to create farm field boundaries. However, a given static image does not contain sufficient information to produce an accurate boundary. For example, at any given time, two adjacent fields might appear as one field in an image. Similarly, satellite data contain generally multiple spectral reflectance bands, which can be combined algebraically to produce indices, but no single index is guaranteed to provide distinguishing power to resolve between fields. Even if satellite information alone were sufficient to delineate fields, a general segmentation approach would not delineate fields alone, but all image content with a discemable boundary.

[0022] Accordingly, there is a need for field boundary detection approaches that leverage spectral, temporal, and spatial information in remote sensing imagery to create high confidence boundaries suitable for downstream processes, free of non-field boundaries. [0023] The present disclosure provides systems, methods, and computer program products for automated identification of field boundaries in remote sensing imagery that exploits spectral, temporal, and spatial patterns in the data to create a geospatial data set ( e.g ., polygonal features) indicative of field boundaries.

[0024] With reference now to Fig. 1, an exemplary field delineation pipeline is illustrated according to embodiments of the present disclosure. In various embodiments, the field delineation pipeline includes three sequential steps: preprocessing 101, in which precursor images are created to enable an accurate characterization of boundaries; segmentation 102, in which an appropriate combination of filters is applied to the imagery prior to segmentation; and post-processing 103, in which contextual and geometric screening is performed to remove boundaries that are determined to not be fields.

[0025] Referring to Fig. 2, an exemplary preprocessing method is illustrated according to embodiments of the present disclosure. It will be appreciated that the quality and particular characteristics of the input imagery used to create boundaries is important to accurate results from the delineation algorithms provided herein. [0026] Remote sensing data are retrieved from one or more datastore 201. In various embodiments, remote sensing data comprise satellite data including surface reflectance at a plurality of resolutions, at a plurality of times. In some embodiments, the datastore is the NASA Harmonized Landsat-Sentinel2 (HLS) product archive. HLS takes advantage of the complementary overpass times of Landsat and Sentinel2, to provide denser coverage in time, but with uniform radiometric and geospatial characteristics. In some embodiments datastore 201 contains uncorrected reflectance data, which is converted to surface reflectance prior to use ( e.g ., by cloud mapping and atmospheric correction). It will be appreciated, however, that a variety of alternative satellite systems are suitable for providing data as set out herein.

[0027] In various embodiments, remote sensing data are fetched and stored in a local cache 202 for further processing. It will be appreciated, however, that in some embodiments, data may be read directly from a local datastore, or may be streamed directly from a remote data store without the need for local caching.

[0028] In some embodiments, the Geospatial Intelligence Production Solution (GIPS) is used for data retrieval. GIPS is an open source solution that provides a uniform interface to a wide range of satellite, weather, and other geospatial data sources. However, it will be appreciated that a variety of alternative APIs and platforms may be used to retrieve suitable satellite data.

[0029] The remote sensing data is processed to compute 203 one or more indices 204 for each point in time for which data is available at each pixel of the input images. In some embodiments, surface reflectance images (e.g., from HLS) are processed to create a three- band product consisting of normalized difference vegetation index (NDVI), land surface water index (LSWI), and mean brightness (BRGT). These three indices represent the three principal axes of variability of optical data, and may be referred to as greenness, wetness, and brightness. In the example shown, each of three indices contains a plurality of snapshots in time. Each snapshot is a raster, or image, whose pixel intensity indicates the index value.

[0030] In alternative embodiments, different indices are selected, resulting in a different number of bands. For example, in some embodiments, the brightness band described above is omitted. Brightness, greenness, and wetness are generally the most dominant modes of variability for optical remote sensing bands. However, it will be appreciated that a variety of different combinations of bands and specific computation of bands may be used for field delineation according to the present disclosure. For example, Enhanced Vegetation Index (EVI) or EVI2 may be used in place of NDVI.

[0031] Omitting the brightness band reduces the number of subfield segments by 13%- 43%, depending on terrain, resulting in a weighted mean decrease of 22% over an exemplary selection of tiles. Further comparison to manual field boundaries may be made using Intersection Over Union (IOU), which is a measure of agreement between two sets of polygons. It refers to the ratio of the intersected area of the polygons to the union. For the two groups of fields (hand drawn, and auto-delineated) this measure is assymetric with respect to each set. GDF1 is used herein to refer to the mean comparison of each manually delineated field to autodelineated fields. GDF2 is used herein to refer to the mean comparison of each autodelineated field to manual fields. In the present example, GDF2 IOU increased by 13%-84%, yielding a weighted mean increase of 33%. GDF1 IOU was about the same.

[0032] Remote sensing data may be available on an irregular schedule, for example due to orbital periods of a given constellation. The HLS source images are provided irregularly in time, and may contain gaps which propagate into the indices. To address this variability, in some embodiments, the index images are composited 205 within pre- specified time windows, enabling delivery of a small number of high-value variables for use in the downstream algorithms. It will be appreciated that various techniques may be used to composite the source images prior to index computation. However, compositing the index images is advantageous as it reduces noise and lowers the dimensionality of the problem, thereby enabling more efficient computation.

[0033] In some embodiments, the predetermined time windows correspond to phases of the growing season. In some embodiments, the time windows correspond to the early growing season, the mid-season, and the late growing season for a given crop. In an exemplary embodiment, a first window spans April and May, a second window spans June and July, and a third window spans August and September. It will be appreciated that these exemplary windows are calibrated to the northern hemisphere, and would be transposed by six months for use in the southern hemisphere. It will also be appreciated that while these windows are suitable for the continental US, they may be shortened or lengthened for certain crops at certain higher or lower latitudes.

[0034] In various embodiments, a user is able to define the number of time windows, and the start and end date of each window separately. This approach allows the delineation of field boundaries for each specific season, or at multiple times within a season, capturing potential changes in the use of the land. For example, a field could be farmed in its entirety for a cash crop, then part of the field subsequently could be used for a cover crop. Similarly, different indices may be used for different conditions or different geographies. [0035] In various embodiments, compositing 205 comprises performing a temporal linear interpolation to reduce potential bias from having the distribution of measurements in time significantly different for different places. In some such embodiments, linear interpolation is performed between available observations, which due to clouds and overpass constraints, may not be evenly distributed in time. After interpolation, for each pixel, the average in time within a window is taken. In an exemplary embodiment in which three indices are assessed over three time windows, the result is a nine band (3 indices x 3 windows) image stack 206.

[0036] It will be appreciated that the above process may be performed for a global data set, or only for certain areas of interest. In some embodiments, the resulting image stack is downsampled to a predetermined resolution in order to limit the overall storage size necessary to maintain the image stacks. In some embodiments, the target resolution is 0.15 degrees. This resolution allows for storage of a global dataset while providing sufficient resolution for further downstream processing.

[0037] Referring to Fig. 3, an exemplary segmentation method is illustrated according to embodiments of the present disclosure. As set out below, once the image data are available from the preprocessing stage, a series of numerical image processing steps are performed. The combination of pre-segmentation steps (gap-filling, scaling, noise filtering) contribute to the reliability of the segmentation results.

[0038] At 301, gaps are filled in the available multi-temporal multi-index imagery. Even after compositing, some data sets contain residual missing pixels which must be addressed. In some embodiments, gap-filling comprises applying linear interpolation to gap fill these residual missing values. The post-compositing gaps are typically very small (1-10 pixels), making linear interpolation sufficient.

[0039] At 302, the images corresponding to each index are normalized. This addresses the potential for each variable to have a different dynamic range. In some embodiments, normalizing comprises rescaling. In some embodiments, normalizing comprises quantizing. In some embodiments, all bands are normalized by subtracting the mean and dividing the result by its standard deviation. [0040] At 303, the images are filtered. In some embodiments, filtering comprises applying a denoising filter. In some embodiments, the denoising filter is the scikit-image restoration. denoise bilateral filter for spatial and variable-wise smoothing. This filter removes noise by applying a spatial low pass filter that does not smooth over features that appear consistently in the nine bands.

[0041] In various embodiments, an edge-preserving, denoising filter is used (such as those provided by scikit-image restoration). Such filters average pixels based on their spatial closeness and radiometric similarity. In various embodiments, spatial closeness is measured by the Gaussian function of the Euclidean distance between two pixels and a configurable standard deviation value (denoted sigma spatial in scikit-image restoration). A larger value of the standard deviation for range distance results in averaging of pixels with larger spatial differences. In various embodiments, the standard deviation value is 0.1, 0.5, 0.8, or 0.9. A value of 0.5 or lower results in situations in which the auto- delineated fields may have four times or more polygons than manually delineated fields, which is undesirable.

[0042] In various embodiments, spatial closeness is measured by the Gaussian function of the Euclidean distance between two color values and a configurable standard deviation value (denoted sigma color in scikit-image restoration). A larger value of the standard deviation for range distance results in averaging of pixels with larger radiometric differences. In various embodiments, the image is converted using the img as float function (of the scikit-image restoration library) and thus the standard deviation is in respect to the range [0, 1] If the value is none, the standard deviation of the image is used. In various embodiments, the standard deviation value is none, 0.3, 0.5, 0.8. In testing, changing this parameter did not change GDF1 IOU, GDF2 IOU, or the resulting number of field polygons. [0043] At 304, image segmentation is performed. In some embodiments, segmentation is performed by graph-based image segmentation. In some embodiments, the Felzenszwalb method for efficient graph-based image segmentation is used. In some embodiments, segmentation is implemented using the scikit-image segmentation. felzenszwalb algorithm. This exemplary algorithm creates a single layer representing raster classes with labels such that the labeled classes are spatially compact parcels of land. The resulting parcels correspond to distinct units of the landscape, ready for postprocessing to generate vector features of farm fields.

[0044] In various embodiments, such as those implemented with the segmentation. felzenszwalb algorithm, the observation level is configurable via a scale value. Higher scale generally means fewer, larger segments. Segment size within an image can vary greatly depending on local contrast, so scale is scene dependent.

However, in some embodiments a consistent scale value is applied for all scenes. Exemplary scale values include scale: 550, 600, 650, 700, and 750. Across these ranges, the max difference in GDF1 IOU is only about 1-2%. There is not a clear dominant scale that produces significantly better results across all test tiles. GDF2 IOU is more strongly and inversely related to scale (-5-6% difference between 550 and 750).

[0045] In various embodiments, such as those implemented with the segmentation. felzenszwalb algorithm, the diameter (standard deviation) of a Gaussian kernel used for smoothing the image prior to segmentation is configurable. This value may be denoted as sigma. Exemplary values of sigma include 0.5, 0.8, and 0.9. Across this parameter range, the maximum difference in GDF1 IOU is about 1%. However, the number of polygons varies significantly, with higher sigma (smoothing) values correlated with fewer, larger segments. It is preferable to use higher values that still maintain segments that do not cross field boundaries, such as 0.9. [0046] Referring to Figs. 4A-B, exemplary segmentations with variable sigma values are illustrated. In Fig. 4A, a sigma value of 0.8 is used. In Fig. 4B, a sigma value of 0.9 is used. As shown, a high value results in larger contiguous segments.

[0047] In various embodiments, such as those implemented with the segmentation. felzenszwalb algorithm, the minimum component size is configurable, which is enforced using post processing. This value may be denoted as min size. Exemplary values for this value include 500, 600, and 700. Across this parameter range, the maximum difference in both GDF1 IOU and GDF2 IOU is about 1 -2%. There is also an insignificant change in the number of polygons. In all cases, min size = 700 has the best IOU, but not by a significant margin.

[0048] Referring to Fig. 5, an exemplary postprocessing method is illustrated according to embodiments of the present disclosure.

[0049] At 501, the segmented images are polygonised to create candidate field polygons in vector format based on raster classes. In some embodiments, the segmentated images are polygonised using GDAL polygonise.

[0050] At 502, the polygons are spatially cleaned. Spatial cleaning includes checking topological validity, fixing broken geometries (such as non-closed polygons), and removing complex shapes that are unlikely to represent all or part of a farm field. In some embodiments, complex shapes are identified by computing the area of the convex hull surrounding the polygon, divided by the area of the polygon. In some embodiments, perimeter to area ratio is used. Additional suitable metrics include: eccentricity (the maximum of the set of shortest distances from each vertex, to all vertices in the polygon); equivalent diameter (the diameter of the smallest circle containing the polygon); perimeter to area ratio; and the ratio of minor axis to major axis of the smallest ellipse containing the polygon. Heuristics may be applied to evaluate whether a polygon falls within the reasonable range of field geometries. For example, polygons outside of predetermined size thresholds may be discarded. Similarly, polygons with too high an aspect ratio may be discarded. Across all parameters tested, the max difference in GDF1 and GDF2 IOU is about 3-4%. Larger opening values result in fewer polygons, however, parts of legitimate fields may be missed. Larger values also result in rounded field edges [0051] In some embodiments, spatial cleaning comprises applying spatial smoothing such as a buffer and reverse buffer cycle. In various embodiments, the buffer size (or opening) is configurable. The buffering removes morphologically inconsistent pieces of polygons, such as long, narrow strips between fields that erroneously connect two distinct fields.

The larger the opening, the more artifacts are removed, but additional edges of the remaining field boundaries are rounded. Accordingly, there is a tradeoff between boundary accuracy/fidelity and problematic artifacts. Exemplary buffer size values include 1, 5, 10, and 20 meters.

[0052] Referring to Figs. 6A-B, exemplary segmentations using variable buffer size are shown. In Fig. 6A, a buffer size of 1 is used. Region 601 is an example of geometry that should be removed during the spatial cleaning step. Fig. 6B shows the result after spatial cleaning with a buffer size of 20. Region 601 is removed, and the remaining fields appear with rounded corners.

[0053] At 503, the polygons are screened to remove non-crop records. In some embodiments, screening comprises comparing the polygons against a reference layer of crop data, such as the USGS Crop Data Layer (CDL). In some embodiments, polygons that lie outside known croplands are discarded.

[0054] At 504, the resulting field polygons are stored for further use, such as visualization. In some embodiments, the field polygons are organized in tiles for efficient retrieval of relevant data for a given problem. In some embodiments, the field polygons are stored with additional metadata, such as a historical crop type or other attributes derived from remote sensing data or drawn from additional data layers.

[0055] Referring to Figs. 7A-B, exemplary field boundaries are illustrated according to an embodiment of the present disclosure. Fig. 7A shows a multitemporal/multispectral image tile such as would result from the preprocessing stage described above. Fig. 7B shows the same image tile with automatically delineated fields superimposed.

[0056] Referring to Fig. 8, a method for agricultural field boundary identification is illustrated according to embodiments of the present disclosure. At 801, time series of surface reflectance rasters for a geographic region is received. At 802, for each of the surface reflectance rasters, at least one index raster is determined, yielding at least one time series of index rasters. At 803, the at least one time series of index rasters is divided into a plurality of consecutive time windows. At 804, the at least one time series of index rasters is composited within each of the plurality of time windows, yielding a composite index raster for each of the at least one time series of index rasters in each of the plurality of time windows. At 805, the composite index rasters are segmented into a plurality of spatially compact regions of the geographic region. At 806, a plurality of polygons is generated from the plurality of spatially compact regions, each of the plurality of polygons corresponding to an agricultural field in the geographic region.

[0057] Referring now to Fig. 9, a schematic of an example of a computing node is shown. Computing node 10 is only one example of a suitable computing node and is not intended to suggest any limitation as to the scope of use or functionality of embodiments described herein. Regardless, computing node 10 is capable of being implemented and/or performing any of the functionality set forth hereinabove. [0058] In computing node 10 there is a computer system/server 12, which is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and/or configurations that may be suitable for use with computer system/server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices, and the like.

[0059] Computer system/server 12 may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types. Computer system/server 12 may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.

[0060] As shown in Fig. 9, computer system/server 12 in computing node 10 is shown in the form of a general-purpose computing device. The components of computer system/server 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including system memory 28 to processor 16.

[0061] Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, Peripheral Component Interconnect (PCI) bus, Peripheral Component Interconnect Express (PCIe), and Advanced Microcontroller Bus Architecture (AMBA).

[0062] Computer system/server 12 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by computer system/server 12, and it includes both volatile and non-volatile media, removable and non-removable media.

[0063] System memory 28 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and/or cache memory 32. Computer system/server 12 may further include other removable/non-removable, volatile/non-volatile computer system storage media. By way of example only, storage system 34 can be provided for reading from and writing to a non-removable, non-volatile magnetic media (not shown and typically called a "hard drive"). Although not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk ( e.g ., a "floppy disk"), and an optical disk drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media can be provided. In such instances, each can be connected to bus 18 by one or more data media interfaces. As will be further depicted and described below, memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the disclosure. [0064] Program/utility 40, having a set (at least one) of program modules 42, may be stored in memory 28 by way of example, and not limitation, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data or some combination thereof, may include an implementation of a networking environment. Program modules 42 generally carry out the functions and/or methodologies of embodiments as described herein.

[0065] Computer system/server 12 may also communicate with one or more external devices 14 such as a keyboard, a pointing device, a display 24, etc.; one or more devices that enable a user to interact with computer system/server 12; and/or any devices ( e.g ., network card, modem, etc.) that enable computer system/server 12 to communicate with one or more other computing devices. Such communication can occur via Input/Output (EO) interfaces 22. Still yet, computer system/server 12 can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and/or a public network (e.g., the Internet) via network adapter 20. As depicted, network adapter 20 communicates with the other components of computer system/server 12 via bus 18. It should be understood that although not shown, other hardware and/or software components could be used in conjunction with computer system/server 12. Examples, include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0066] The present disclosure may be embodied as a system, a method, and/or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure. [0067] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non- exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media ( e.g ., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0068] Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.

[0069] Computer readable program instructions for carrying out operations of the present disclosure may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user’s computer, partly on the user’s computer, as a stand-alone software package, partly on the user’s computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user’s computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field- programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0070] Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

[0071] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.

[0072] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

[0073] The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

[0074] The descriptions of the various embodiments of the present disclosure have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.