TECHNICAL FIELD

[001] The present disclosure generally relates to computerized systems and methods for database migration. In particular, embodiments of the present disclosure relate to systems and methods for improved migration between live real-time databases employing dual-writing, serial cloning, and consistency verification while minimizing downtime and automating traffic handoffs to a migrated database.

BACKGROUND

[002] Database migration is the process of migrating data between different databases. When a migration is finished, the data in an original (or origin) database is copied, and in some instances restructured, in a migrated (or target) database. The database migration process may also include a rerouting of instructions (e.g., write or read instructions) that are redirected from the original database to the migrated database. In some scenarios, the database migration may be homogeneous where the original and migrated databases are of the same database management system. In other scenarios, the database migration may be heterogenous where the original and migrated databases are of different database management systems. The database migration can include processes of database replication (also known as database streaming) and a database transfer. The database replication copies data in the original database into the migrated database. The database transfer would move, replace, and/or swift resources between databases. [003] Database migration comes with certain risks, which can be particularly problematic when migrating live or real time databases (i.e., connected databases that handle workloads in constant flux, are extremely time-sensitive, and/or are concurrently being accessed by clients). During database migrations there are risks of data loss (e.g., some of the data may not migrate over from the source system), change in semantics (e.g., semantics errors can occur creating inconsistencies), extended downtime (e.g., when the data migration process takes longer than expected preventing access to critical data), and data corruption (e.g., unwanted data can be migrated into the new system, leading to potential crashes and errors). Particularly relevant for live database migration, additional risks must be considered. For example, migrating a database that is connected to the cloud and actively accessed and modified by users presents additional challenges of application stability risks (e.g., an application relying on the database may become unstable for improper development or improper coding resulting in blackout periods or failures), orchestration issues (e.g., when the processes of data migration are not performed in order), interference problems (e.g., when multiple stakeholders make use of the application during the migration process simultaneously) and/or improper application parameterization (e.g., the target system can become incompatible with data migration programs).

[004] Moreover, database migration has become more difficult and critical with the rise on cloud computing. Nowadays, databases — and the information they store — can be one of the most valuable assets for a business. The entire operation of a business may rely on accessing, updating, and modifying databases. For certain business, this information must be available persistently and in real-time. Downtime can cause business disruption, customer dissatisfaction, and missed opportunities. Further, current databases can be massive, with often terabytes or even petabytes of information, and need to have a very high query throughput to sustain large client bases with high database demands. For example, databases supporting streaming services not only have very large datasets but they also need to be delivered quickly. This combination makes live or real-time database migration particularly complicated because it becomes necessary to migrate large databases, while they are operating (both providing and receiving new data), with minimum downtime, and having a low error tolerance.

[005] The disclosed systems and methods for database migration address one or more problems set forth above and/or other problems in the prior art.

SUMMARY

[006] One aspect of the present disclosure is directed to a system for database migration. The system includes one or more processors and one or more memory devices storing instructions that, when executed by the one or more processors, configure the one or more processors to perform operations. The operations may include cloning the online database, with online traffic, to a first clone database, generating export files from the clone database and importing them to a migration database; performing a row comparison of the first clone database and the migration database after importing the export files (the row comparison being performed by calculating shard keys for a selected number of rows in the online database and the migration database), and initializing a double write operation replicating writing of the online database on the migration database to capture incremental data. The operations may also include cloning the online database to a second clone database after performing the row comparison, identifying catch-up data by comparing the second clone database with the incremental data, and updating the migration database to include the catch-up data.

[007] Another aspect of the present disclosure is directed to a method for database migration. The method may include, cloning the online database to a first clone database, generating export files from the first clone database and importing them to a migration database, performing a row comparison of the first clone database and the migration database after importing the export files (the row comparison being performed by calculating shard keys for a selected number of rows in the online database and the migration database), and initializing a double write operation replicating writing of the online database on the migration database to capture incremental data. The method may also include cloning the online database to a second clone database after performing the row comparison, identifying catch-up data by comparing the second clone database with the incremental data, and updating the migration database to include the catch-up data.

[008] Yet another aspect of the present disclosure is directed to a system for migration of an online database. The system may include a first database coupled to a network, a second database, at least one processor coupled to the first database and the second database, and at least one memory device storing instructions. When these instructions are executed by the at least one processor, they configure the at least one processor to: clone the first database to a first clone database, generate export files from the first clone database and import them to the second database, perform a row comparison of the first clone database and the second database after importing the export files (the row comparison being performed by calculating shard keys for a selected number of rows in the first database and the second database), and initialize a double write operation replicating writing of the first database on the second database to capture incremental data. The instructions may also configure the processor to: clone the first database to a second clone database after performing the row comparison, identify catch-up data by comparing the second clone database with the incremental data, update the second database to include the catch-up data, and performing a second row comparison between the second clone database and the second database to determine whether data is consistent. The instructions may also configure the processor to: upon determining the data is consistent, pause the double write operation and switch online traffic from the first database to the second database.

BRIEF DESCRIPTION OF THE DRAWINGS

[009] FIG. 1A is a schematic block diagram illustrating an exemplary embodiment of a network including computerized systems for communications enabling shipping, transportation, and logistics operations, consistent with the disclosed embodiments.

[010] FIG. 1 B depicts a sample Search Result Page (SRP) that includes one or more search results satisfying a search request along with interactive user interface elements, consistent with the disclosed embodiments.

[011] FIG. 1 C depicts a sample Single Display Page (SDP) that includes a product and information about the product along with interactive user interface elements, consistent with the disclosed embodiments.

[012] FIG. 1 D depicts a sample cart page that includes items in a virtual shopping cart along with interactive user interface elements, consistent with the disclosed embodiments. [013] FIG. 1 E depicts a sample order page that includes items from the virtual shopping cart along with information regarding purchase and shipping, along with interactive user interface elements, consistent with the disclosed embodiments.

[014] FIG. 2 is a diagrammatic illustration of an exemplary fulfillment center configured to utilize disclosed computerized systems, consistent with the disclosed embodiments.

[015] FIG. 3 is a schematic block diagram of an exemplary system, consistent with disclosed embodiments.

[016] FIG. 4 is a block diagram of an exemplary database migration system, consistent with disclosed embodiments.

[017] FIG. 5 is a flow diagram of an exemplary process for database migration, consistent with disclosed embodiments.

[018] FIG. 6 is a flow diagram of an exemplary process for migrated data consistency verification, consistent with disclosed embodiments.

[019] FIG. 7 is a flow diagram of an exemplary process for migration of a live database, consistent with disclosed embodiments.

[020] FIG. 8 is a flow diagram of an exemplary process for database migration verification, consistent with disclosed embodiments.

[021] FIG. 9 is a flow diagram of an exemplary process for online database migration, consistent with disclosed embodiments.

[022] FIG. 10 is a schematic representation of a database migration process using a clone database, consistent with disclosed embodiments.

[023] FIG. 11 is a timing diagram of an exemplary sequence for database migration, consistent with disclosed embodiments. DETAILED DESCRIPTION

[024] The following detailed description refers to the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the following description to refer to the same or similar parts. While several illustrative embodiments are described herein, modifications, adaptations and other implementations are possible. For example, substitutions, additions, or modifications may be made to the components and steps illustrated in the drawings, and the illustrative methods described herein may be modified by substituting, reordering, removing, or adding steps to the disclosed methods. The following detailed description is not limited to the disclosed embodiments and examples.

[025] Literals used to reference individual elements in the figures, e.g., A or B, do not specify the number of an element or the total number of elements. Instead, they are variable references that indicate a variable element number and a variable number of total elements. For example, literal B does not indicate that the element with the “B” literal is the 2 ^nd one. Instead, B is a variable reference that could indicate any integer number.

[026] Embodiments of the present disclosure are directed to systems and methods for database migration. The disclosed systems and methods enable fast and stable migration of databases using one or more clone databases and concurrent writing to facilitate data migration and traffic transfer. The disclosed systems and methods may be applicable for live databases that are used in real-time operations to handle workflows in constant flux. For example, the disclosed methods and systems may be used for databases supporting online applications of e- commerce that have constant changes and high traffic demands. The disclosed systems and methods address issues of traditional database migration process. Traditional systems frequently suffer from longer than acceptable database downtime. Current online application frequently require almost constant availability and any downtime may be detrimental for operations. Database down time may result in, for example, customer dissatisfaction and poor retention. Disclosed embodiments provide systems and methods for live database migration with minimal or no downtime.

[027] Moreover, disclosed systems and methods may improve quality of data migration employing catch up operations to verify all data has been migrated. The migration of live or online databases can be cumbersome as the dynamic interactions in an original database may result in differences between the original and the migrated database. Because it is undesirable to disconnect a live database, data in an online database may be modified during the migration process. These concurrent changes may create inconsistencies between the original and the migrated database. The disclosed systems and methods address these issues using methods for multiple cloning of origin databases and the application of one or more catch up and verification processes to make sure that the migration results complete data transfer and/or replication.

[028] Some of the disclosed embodiments are applicable to cloud-based databases. For example, some of the disclosed embodiments are applicable to cloud-based relational database services. In such embodiments, the disclosed systems and methods may include functions of data allocation, automatic replication, and SQL compatibility.

[029] The disclosed systems and methods may also minimize downtime during migration through the use of database cloning. Some of the disclosed embodiments include cloning of database by creating a point-in-time copy of a production or online database. In such embodiments, the online application may continue regular operations during migration operations, maintaining the same data structure and content. These embodiment of the disclosed systems and methods enable concurrent cloning of database to minimize downtime. For example, databases may be cloned simultaneously and/or in sequence, get assigned data structures that replicate origin databases, and execute server operations. Accordingly, certain embodiments of the disclosed systems and methods enable database migration with little to no downtime.

[030] Moreover, disclosed systems and methods may allow efficient migration even for databases with high traffic loads and/or large datasets. Current online applications are frequently data intensive and may require heavy data usage and transfer. Migration of these online or live databases raises additional challenges. For example, database migration through traditional systems may be too long and cumbersome as replication operations that may cause delays and/or data misses. Accordingly, for certain database migration it is not possible to use traditional methods without disrupting operations. The disclosed systems and methods address these issues by providing a method that allows transfers of databases with high traffic loads and/or large datasets while minimizing downtime and improving data accuracy.

[031] In some embodiments, the disclosed systems and methods facilitate data migration employing double writing features and catch up operations. Traditional migration of live databases may require pauses in live traffic to prevent changes during migration periods. These pauses, however, may disrupt operations and cause malfunctioning systems and/or customer dissatisfaction. The disclosed systems and methods address these issues by employing dual write features and clustering system that allow migration without requiring pauses on the databases. Some of the disclosed systems and methods may combine clustering algorithms with double writing operations to segregate data changes that can then be replicated in migrated databases without requiring disconnection of the original database. Further, some of the disclosed systems and methods may include testing of dual-write connections and operations to minimize data inaccuracies during migration.

[032] Moreover, disclosed systems and methods may facilitate automation of the migration processes. Some embodiments of the disclosed systems and methods may include automated systems that coordinate multiple online tools to perform database migration. For example, some of the disclosed systems and methods may automate the capturing of data from databases using, for example, automated importation of SQL files. Further, the disclosed systems and methods may configure tools to automatically import SQL files and/or metadata to other databases during the migration process, without requiring user interaction. Further, the disclosed systems and methods may also facilitate automated and/or concurrent data verification. In some embodiments, the disclosed systems and methods may include tools that verification processes in the background and in clusters of datasets to identify inconsistencies.

[033] Furthermore, some embodiments of the disclosed systems and methods may address issues of poor data consistency during migration. Particularly for large dynamic data sets, like those of databases supporting online applications, the data migration processes may result in data inconsistencies and/or corruption. Some of the disclosed systems and methods address these issues using tools for data verification coupled with catch up tools in specific ranges to fix them. For example, some of the disclosed embodiments may use tools that compare data sets to identify inconsistencies and corrupted files. These tools may use tools to check accuracy and inconsistencies after data migration. These methods may include Source Data Verification (SDV) or the use of hash-based verification, such as using MD5 file checksums. In these embodiments, data verification may help to automatically and concurrently determine whether data was accurately translated when data is transferred from one source to another, is complete, and supports processes in the new system. Further, embodiments of the disclosed systems and methods also support employing catch-up tools for concurrent fixing of inconsistencies and data clustering mechanisms to identify areas of disparity and forestall erroneous data loss.

[034] The disclosed systems and methods may also facilitate migration of frequently and highly used databases that support online applications. For example, online applications related to e-commerce often require frequent and fluent access to pricing databases. These type of databases handle very large sets of information, are dynamically updated or modified, and have little to no tolerance to downtime. Particularly in view of recent practices, like dynamic pricing, quick and reliable access to databases is critical for business operations. Migration of these type of pricing databases is thus complicated and critical. The disclosed systems and methods allow efficient migration of the databases, with minimal downtime, and employing tools for clustering, verification, and cloning, that minimize data loss and/or reduce data corruption.

[035] Reference will now be made to the disclosed embodiments, examples of which are illustrated in the accompanying drawings. [036] FIG. 1A shows a schematic block diagram of system 100 illustrating an exemplary embodiment of a system including computerized systems for communications enabling shipping, transportation, and logistics operations. As illustrated in FIG. 1A, system 100 may include a variety of systems, each of which may be connected to one another via one or more networks. The systems may also be connected to one another via a direct connection, for example, using a cable. The depicted systems include a shipment authority technology (SAT) system 101 , an external front-end system 103, an internal front-end system 105, a transportation system 107, mobile devices 107A, 107B, and 107C, seller portal 109, shipment and order tracking (SOT) system 111 , fulfillment optimization (FO) system 113, fulfillment messaging gateway (FMG) 115, supply chain management (SCM) system 117, workforce management system 119, mobile devices 119A, 119B, and 119C (depicted as being inside of fulfillment center (FC) 200), 3 ^rd party fulfillment systems 121 A, 121 B, and 121 C, fulfillment center authorization system (FC Auth) 123, and labor management system (LMS) 125.

[037] SAT system 101 , in some embodiments, may be implemented as a computer system that monitors order status and delivery status. SAT system 101 may also monitor data, including output (such as a number of packages shipped during a particular time period) and input (such as the number of empty cardboard boxes received for use in shipping). SAT system 101 may also act as a gateway between different devices in system 100, enabling communication (e.g., using store- and-forward or other techniques) between devices such as external front-end system 103 and FO system 113.

[038] External front-end system 103, in some embodiments, may be implemented as a computer system that enables external users to interact with one or more systems in system 100. For example, in embodiments where system 100 enables the presentation of systems to enable users to place an order for an item, external front-end system 103 may be implemented as a web server that receives search requests, presents item pages, and solicits payment information. For example, external front-end system 103 may be implemented as a computer or computers running software such as the Apache HTTP Server, Microsoft Internet Information Services (IIS), NGINX, or the like. In other embodiments, external frontend system 103 may run custom web server software designed to receive and process requests from external devices (e.g., mobile device 102A or computer 102B), acquire information from databases and other data stores based on those requests, and provide responses to the received requests based on acquired information.

[039] In some embodiments, external front-end system 103 may include one or more of a web caching system, a database, a search system, or a payment system. In one aspect, external front-end system 103 may include one or more of these systems, while in another aspect, external front-end system 103 may include interfaces (e.g., server-to-server, database-to-database, or other network connections) connected to one or more of these systems.

[040] An illustrative set of steps, illustrated by FIGS. 1 B, 1 C, 1 D, and 1 E, will help to describe some operations of external front-end system 103. External frontend system 103 may receive information from systems or devices in system 100 for presentation and/or display. For example, external front-end system 103 may host or provide one or more web pages, including a Search Result Page (SRP) (e.g., FIG. 1 B), a Single Display Page (SDP) (e.g., FIG. 1 C), a Cart page (e.g., FIG. 1 D), or an Order page (e.g., FIG. 1 E). A user device (e.g., using mobile device 102A or computer 102B) may navigate to external front-end system 103 and request a search by entering information into a search box. External front-end system 103 may request information from one or more systems in system 100. For example, external front-end system 103 may request information from FO System 113 that satisfies the search request.

[041] External front-end system 103 may prepare an SRP (e.g., FIG. 1 B) based on the information. The SRP may include information that satisfies the search request. For example, this may include pictures of products that satisfy the search request. The SRP may also include respective prices for each product, or information relating to enhanced delivery options for each product, promised delivery date (PDD), weight, size, offers, discounts, or the like. In some embodiments, the SRP may also include delivery options, cutoff times for delivery options and/or hypermedia elements requesting user input. External front-end system 103 may send the SRP to the requesting user device (e.g., via a network).

[042] A user device may then select a product from the SRP, e.g., by clicking or tapping a user interface, or using another input device, to select a product represented on the SRP. The user device may formulate a request for information on the selected product and send it to external front-end system 103. In response, external front-end system 103 may request information related to the selected product. For example, the information may include additional information beyond that presented for a product on the respective SRP. This could include, for example, shelf life, country of origin, weight, size, number of items in package, handling instructions, cutoff time for dawn or first time deliveries, or other information about the product. The information could also include recommendations for similar products (based on, for example, big data and/or machine learning analysis of customers who bought this product and at least one other product), answers to frequently asked questions, reviews from customers, manufacturer information, pictures, or the like.

[043] External front-end system 103 may prepare an SDP (Single Display Page) (e.g., FIG. 10) based on the received product information, location of the customer device, and availability of delivery options. The SDP may also include other interactive elements such as a “Buy Now” button, a “Add to Cart” button, a quantity field , a picture of the item, or the like. The SDP may further include a list of sellers that offer the product. The list may be ordered based on the price each seller offers such that the seller that offers to sell the product at the lowest price may be listed at the top. The list may also be ordered based on the seller ranking such that the highest ranked seller may be listed at the top. The seller ranking may be formulated based on multiple factors, including, for example, the seller’s past track record of meeting a promised PDD. External front-end system 103 may deliver the SDP to the requesting user device (e.g., via a network).

[044] The requesting user device may receive the SDP which lists the product information. Upon receiving the SDP, the user device may then interact with the SDP. For example, a user of the requesting user device may click or otherwise interact with a “Place in Cart” button on the SDP. This adds the product to a shopping cart associated with the user. Alternatively, or additionally, the user may interact with the SDP by providing instructions for delivery. The user device may transmit this request to add the product to the shopping cart to external front-end system 103.

[045] External front-end system 103 may generate a Cart page (e.g., FIG. 1 D). The Cart page, in some embodiments, lists the products that the user has added to a virtual “shopping cart.” A user device may request the Cart page by clicking on or otherwise interacting with an icon on the SRP, SDP, or other pages. The Cart page may, in some embodiments, list all products that the user has added to the shopping cart, as well as information about the products in the cart such as a quantity of each product, a price for each product per item, a price for each product based on an associated quantity, information regarding promised delivery date (PDD), a delivery method, a shipping cost, user interface elements for modifying the products in the shopping cart (e.g., deletion or modification of a quantity), options for ordering other product or setting up periodic delivery of products, options for setting up interest payments, user interface elements for proceeding to purchase, or the like. A user at a user device may click on or otherwise interact with a user interface element (e.g., a button that reads “Buy Now”) to initiate the purchase of the product in the shopping cart. Upon doing so, the user device may transmit this request to initiate the purchase to external front-end system 103. In some embodiments, the Cart page may include text box inputs, interactive icons, or recommendation messages for each product delivery.

[046] External front-end system 103 may generate an order page (e.g., FIG. 1 E) in response to receiving the request to initiate a purchase. The order page, in some embodiments, re-lists the items from the shopping cart and requests input of payment and shipping information. For example, the order page may include a section requesting information about the purchaser of the items in the shopping cart (e.g., name, address, e-mail address, phone number), information about the recipient (e.g., name, address, phone number, delivery information), shipping information (e.g., speed/method of delivery and/or pickup), payment information (e.g., credit card, bank transfer, check, stored credit), user interface elements to request a cash receipt (e.g., for tax purposes), or the like. External front-end system

103 may send the Order page to the user device.

[047] The user device may enter information on the order page and click or otherwise interact with a user interface element that sends the information to external front-end system 103. From there, external front-end system 103 may send the information to different systems in system 100 to enable the creation and processing of a new order with the products in the shopping cart. In some embodiments, external front-end system 103 may be further configured to enable sellers to transmit and receive information relating to orders.

[048] Internal front-end system 105, in some embodiments, may be implemented as a computer system that enables internal users (e.g., employees of an organization that owns, operates, or leases system 100) to interact with one or more systems in system 100. For example, in embodiments where SAT system 101 enables the presentation of systems to enable users to place an order for an item, internal front-end system 105 may be implemented as a web server that enables internal users to view diagnostic and statistical information about orders, modify item information, or review statistics relating to orders. For example, internal front-end system 105 may be implemented as a computer or computers running software such as the Apache HTTP Server, Microsoft Internet Information Services (IIS), NGINX, or the like. In other embodiments, internal front-end system 105 may run custom web server software designed to receive and process requests from systems or devices depicted in system 100 (as well as other devices not depicted), acquire information from databases and other data stores based on those requests, and provide responses to the received requests based on acquired information. [049] In some embodiments, internal front-end system 105 may include one or more of a web caching system, a database, a search system, a payment system, an analytics system, an order monitoring system, or the like. In one aspect, internal front-end system 105 may include one or more of these systems, while in another aspect, internal front-end system 105 may include interfaces (e.g., server-to-server, database-to-database, or other network connections) connected to one or more of these systems.

[050] Transportation system 107, in some embodiments, may be implemented as a computer system that enables communication between systems or devices in system 100 and mobile devices 107A-107C. Transportation system 107, in some embodiments, may receive information from one or more mobile devices 107A-107C (e.g., mobile phones, smart phones, PDAs, or the like). For example, in some embodiments, mobile devices 107A-107C may include devices operated by delivery workers. The delivery workers, who may be permanent, temporary, or shift employees, may utilize mobile devices 107A-107C to effect delivery of packages containing the products ordered by users. For example, to deliver a package, the delivery worker may receive a notification on a mobile device indicating which package to deliver and where to deliver it. Upon arriving at the delivery location, the delivery worker may locate the package (e.g., in the back of a truck or in a crate of packages), scan or otherwise capture data associated with an identifier on the package (e.g., a barcode, an image, a text string, an RFID tag, or the like) using the mobile device, and deliver the package (e.g., by leaving it at a front door, leaving it with a security guard, handing it to the recipient, or the like). In some embodiments, the delivery worker may capture photo(s) of the package and/or may obtain a signature using the mobile device. The mobile device may send information to transportation system 107 including information about the delivery, including, for example, time, date, GPS location, photo(s), an identifier associated with the delivery worker, an identifier associated with the mobile device, or the like. Transportation system 107 may store this information in a database (not pictured) for access by other systems in system 100. Transportation system 107 may, in some embodiments, use this information to prepare and send tracking data to other systems indicating the location of a particular package.

[051] In some embodiments, certain users may use one kind of mobile device (e.g., permanent workers may use a specialized PDA with custom hardware such as a barcode scanner, stylus, and other devices) while other users may use other kinds of mobile devices (e.g., temporary or shift workers may utilize off-the- shelf mobile phones and/or smartphones).

[052] In some embodiments, transportation system 107 may associate a user with each device. For example, transportation system 107 may store an association between a user (represented by, e.g., a user identifier, an employee identifier, or a phone number) and a mobile device (represented by, e.g., an International Mobile Equipment Identity (IMEI), an International Mobile Subscription Identifier (IMSI), a phone number, a Universal Unique Identifier (UUID), or a Globally Unique Identifier (GUID)). Transportation system 107 may use this association in conjunction with data received on deliveries to analyze data stored in the database in order to determine, among other things, a location of the worker, an efficiency of the worker, or a speed of the worker.

[053] Seller portal 109, in some embodiments, may be implemented as a computer system that enables sellers or other external entities to electronically communicate with one or more systems in system 100. For example, a seller may utilize a computer system (not pictured) to upload or provide product information, order information, contact information, or the like, for products that the seller wishes to sell through system 100 using seller portal 109.

[054] Shipment and order tracking system 111 , in some embodiments, may be implemented as a computer system that receives, stores, and forwards information regarding the location of packages containing products ordered by customers (e.g., by a user using devices 102A-102B). In some embodiments, shipment and order tracking system 111 may request or store information from web servers (not pictured) operated by shipping companies that deliver packages containing products ordered by customers.

[055] In some embodiments, shipment and order tracking system 111 may request and store information from systems depicted in system 100. For example, shipment and order tracking system 111 may request information from transportation system 107. As discussed above, transportation system 107 may receive information from one or more mobile devices 107A-107C (e.g., mobile phones, smart phones, PDAs, or the like) that are associated with one or more users (e.g., a delivery worker) or a vehicle (e.g., a delivery truck). In some embodiments, shipment and order tracking system 111 may also request information from workforce management system (WMS) 119 to determine the location of individual products inside of a fulfillment center (e.g., fulfillment center 200). Shipment and order tracking system 111 may request data from one or more of transportation system 107 or WMS 119, process it, and present it to a device (e.g., user devices 102A and 102B) upon request.

[056] Fulfillment optimization (FO) system 113, in some embodiments, may be implemented as a computer system that stores information for customer orders from other systems (e.g., external front-end system 103 and/or shipment and order tracking system 111 ). FO system 113 may also store information describing where particular items are held or stored. For example, certain items may be stored only in one fulfillment center, while certain other items may be stored in multiple fulfillment centers. In still other embodiments, certain fulfillment centers may be designed to store only a particular set of items (e.g., fresh produce or frozen products). FO system 113 stores this information as well as associated information (e.g., quantity, size, date of receipt, expiration date, etc.).

[057] Fulfillment messaging gateway (FMG) 115, in some embodiments, may be implemented as a computer system that receives a request or response in one format or protocol from one or more systems in system 100, such as FO system 113, converts it to another format or protocol, and forward it in the converted format or protocol to other systems, such as WMS 119 or 3 ^rd party fulfillment systems 121 A, 121 B, or 121 C, and vice versa.

[058] Supply chain management (SCM) system 117, in some embodiments, may be implemented as a computer system that performs forecasting functions. For example, SCM system 117 may forecast a level of demand for a particular product based on, for example, a past demand for products, an expected demand for a product, a network-wide past demand, a network-wide expected demand, a count products stored in each fulfillment center 200, expected or current orders for each product, or the like. In response to this forecasted level and the amount of each product across all fulfillment centers, SCM system 117 may generate one or more purchase orders to purchase and stock a sufficient quantity to satisfy the forecasted demand for a particular product. [059] Workforce management system (WMS) 119, in some embodiments, may be implemented as a computer system that monitors workflow. For example,

WMS 119 may receive event data from individual devices (e.g., devices 107A-107C or 119A-119C) indicating discrete events. For example, WMS 119 may receive event data indicating the use of one of these devices to scan a package. As discussed below with respect to fulfillment center 200 and FIG. 2, during the fulfillment process, a package identifier (e.g., a barcode or RFID tag data) may be scanned or read by machines at particular stages (e.g., automated or handheld barcode scanners, RFID readers, high-speed cameras, devices such as tablet 119A, mobile device/PDA 119B, computer 119C, or the like). WMS 119 may store each event indicating a scan or a read of a package identifier in a corresponding database (not pictured) along with the package identifier, a time, date, location, user identifier, or other information, and may provide this information to other systems (e.g., shipment and order tracking system 111).

[060] WMS 119, in some embodiments, may store information associating one or more devices (e.g., devices 107A-107C or 119A-119C) with one or more users associated with system 100. For example, in some situations, a user (such as a part- or full-time employee) may be associated with a mobile device in that the user owns the mobile device (e.g., the mobile device is a smartphone). In other situations, a user may be associated with a mobile device in that the user is temporarily in custody of the mobile device (e.g., the user checked the mobile device out at the start of the day, will use it during the day, and will return it at the end of the day).

[061] WMS 119, in some embodiments, may maintain a work log for each user associated with system 100. For example, WMS 119 may store information associated with each employee, including any assigned processes (e.g., unloading trucks, picking items from a pick zone, rebin wall work, packing items), a user identifier, a location (e.g., a floor or zone in a fulfillment center 200), a number of units moved through the system by the employee (e.g., number of items picked, number of items packed), an identifier associated with a device (e.g., devices 119A- 119C), or the like. In some embodiments, WMS 119 may receive check-in and check-out information from a timekeeping system, such as a timekeeping system operated on a device 119A-119C.

[062] 3 ^rd party fulfillment (3PL) systems 121A-121 C, in some embodiments, represent computer systems associated with third-party providers of logistics and products. For example, while some products are stored in fulfillment center 200 (as discussed below regarding FIG. 2), other products may be stored off-site, may be produced on demand, or may be otherwise unavailable for storage in fulfillment center 200. 3PL systems 121A-121 C may be configured to receive orders from FO system 113 (e.g., through FMG 115) and may provide products and/or services (e.g., delivery or installation) to customers directly. In some embodiments, one or more of 3PL systems 121A-121 C may be part of system 100, while in other embodiments, one or more of 3PL systems 121A-121 C may be outside of system 100 (e.g., owned or operated by a third-party provider).

[063] Fulfillment Center Auth system (FC Auth) 123, in some embodiments, may be implemented as a computer system with a variety of functions. For example, in some embodiments, FC Auth 123 may act as a single-sign on (SSO) service for one or more other systems in system 100. For example, FC Auth 123 may enable a user to log in via internal front-end system 105, determine that the user has similar privileges to access resources at shipment and order tracking system 111 , and enable the user to access those privileges without requiring a second log in process. FC Auth 123, in other embodiments, may enable users (e.g., employees) to associate themselves with a particular task. For example, some employees may not have an electronic device (such as devices 119A-119C) and may instead move from task to task, and zone to zone, within a fulfillment center 200, during the course of a day. FC Auth 123 may be configured to enable those employees to indicate what task they are performing and what zone they are in at different times of day.

[064] Labor management system (LMS) 125, in some embodiments, may be implemented as a computer system that stores attendance and overtime information for employees (including full-time and part-time employees). For example, LMS 125 may receive information from FC Auth 123, WMA 119, devices 119A-119C, transportation system 107, and/or devices 107A-107C.

[065] The particular configuration depicted in FIG. 1 A is an example only. For example, while FIG. 1A depicts FC Auth system 123 connected to FO system 113, not all embodiments require this particular configuration. Indeed, in some embodiments, the systems in system 100 may be connected to one another through one or more public or private networks, including the Internet, an Intranet, a WAN (Wide-Area Network), a MAN (Metropolitan-Area Network), a wireless network compliant with the IEEE 802.11a/b/g/n Standards, a leased line, or the like. In some embodiments, one or more of the systems in system 100 may be implemented as one or more virtual servers implemented at a data center, server farm, or the like.

[066] FIG. 2 depicts a fulfillment center 200. Fulfillment center (FC) 200 is an example of a physical location that stores items for shipping to customers when ordered. Fulfillment center (FC) 200 may be divided into multiple zones, each of which are depicted in FIG. 2. These “zones,” in some embodiments, may be thought of as virtual divisions between different stages of a process of receiving items, storing the items, retrieving the items, and shipping the items. So, while the “zones” are depicted in FIG. 2, other divisions of zones are possible and the zones in FIG. 2 may be omitted, duplicated, and/or modified in some embodiments.

[067] Inbound zone 203 represents an area of FC 200 where items are received from sellers who wish to sell products using system 100 (FIG. 1A). For example, a seller may deliver items 202A and 202B using truck 201 . Item 202A may represent a single item large enough to occupy its own shipping pallet, while item 202B may represent a set of items that are stacked together on the same pallet to save space.

[068] In some embodiments, as shown in FIG. 2, one or more of the sections of FC 200 may include positioning sensors 217. Positioning sensors 217 may include a plurality of sensors that may be used to determine the position of products within the FC and track their movement through the FC. In such embodiments, positioning sensors 217 may be configured to both track the position of products in the FC and estimate movement between different sections. For instance, positioning sensors 217 may be configured to capture and/or store historic data of products' location and time, to determine movement between the different regions of FC 200. Other systems may use this information for determining distances or estimated times between storing zones and packing zones.

[069] As shown in FIG. 2 the positioning sensors 217 may include sensors 217A in packing zone 211 , sensors 217B in picking zone 209, and sensors 217C in drop zone 205. In some embodiments, more sensors may be placed in different regions of FC 200 with the goal of tracking and capturing the position of items FC 200 and improve the accuracy of estimated deliveries or maximize the availability of delivery options. In some embodiments, positioning sensors 217 may include optical sensors, like cameras. In other embodiments, positioning sensors 217 may include wireless sensors, like radiofrequency, Bluetooth, or WiFi sensors. Additionally, or alternatively, positioning sensors 217 may include weight sensors.

[070] A worker will receive the items in inbound zone 203 and may optionally check the items for damage and correctness using a computer system (not pictured). For example, the worker may use a computer system to compare the quantity of items 202A and 202B to an ordered quantity of items. If the quantity does not match, that worker may refuse one or more of items 202A or 202B. If the quantity does match, the worker may move those items (using, e.g., a dolly, a handtruck, a forklift, or manually) to buffer zone 205. Buffer zone 205 may be a temporary storage area for items that are not currently needed in the picking zone, for example, because there is a high enough quantity of that item in the picking zone to satisfy forecasted demand. In some embodiments, forklifts 206 operate to move items around buffer zone 205 and between inbound zone 203 and drop zone 207. If there is a need for items 202A or 202B in the picking zone (e.g., because of forecasted demand), a forklift may move items 202A or 202B to drop zone 207.

[071] Drop zone 207 may be an area of FC 200 that stores items before they are moved to picking zone 209. A worker assigned to the picking task (a “picker”) may approach items 202A and 202B in the picking zone, scan a barcode for the picking zone, and scan barcodes associated with items 202A and 202B using a mobile device (e.g., device 119B). Such event may update a real time location system that updates a database to specify the item has been moved into the FC. The picker may then take the item to picking zone 209 (e.g., by placing it on a cart or carrying it) and the real time location system may request the position of storage for the new item. [072] Picking zone 209 may be an area of FC 200 where items 208 are stored on storage units 210. In some embodiments, storage units 210 may include one or more of physical shelving, bookshelves, boxes, totes, refrigerators, freezers, cold stores, or the like. In some embodiments, picking zone 209 may be organized into multiple floors. In some embodiments, workers or machines may move items into picking zone 209 in multiple ways, including, for example, a forklift, an elevator, a conveyor belt, a cart, a handtruck, a dolly, an automated robot or device, or manually. For example, a picker may place items 202A and 202B on a handtruck or cart in drop zone 207 and walk items 202A and 202B to picking zone 209.

[073] A picker may receive an instruction to place (or “stow”) the items in particular spots in picking zone 209, such as a particular space on a storage unit 210. For example, a picker may scan item 202A using a mobile device (e.g., device 119B). The device may indicate where the picker should stow item 202A, for example, using a system that indicate an aisle, shelf, and location. In some embodiments, the location to stow item 202A may be determined based on predictive algorithms that attempt to maximize the availability of special delivery options, such as dawn deliveries. The device may then prompt the picker to scan a barcode at that location before stowing item 202A in that location. Alternatively, a wireless sensor or a camera coupled with image recognition, may store the location of the time. In some embodiments, the device may send (e.g., via a wireless network) data to a computer system such as WMS 119 in FIG. 1A indicating that item 202A has been stowed at the location by the user using device 119B.

[074] Once a user places an order, a picker may receive an instruction on device 119B to retrieve one or more items 208 from storage unit 210. In some embodiments, as further described in connection with FIG. 11 , the picker may receive instructions through a placement or storing guide to stow the products. The picker may retrieve item 208, scan a barcode on item 208, and place it on transport mechanism 214. While transport mechanism 214 is represented as a slide, in some embodiments, transport mechanism may be implemented as one or more of a conveyor belt, an elevator, a cart, a forklift, a handtruck, a dolly, a cart, or the like. Item 208 may then arrive at packing zone 211.

[075] Packing zone 211 may be an area of FC 200 where items are received from picking zone 209 and packed into boxes or bags for eventual shipping to customers. In packing zone 211 , a worker assigned to receiving items (a “rebin worker”) will receive item 208 from picking zone 209 and determine what order it corresponds to. For example, the rebin worker may use a device, such as computer 119C, to scan a barcode on item 208. Computer 119C may indicate visually which order item 208 is associated with. This may include, for example, a space or “cell” on a wall 216 that corresponds to an order. Once the order is complete (e.g., because the cell contains all items for the order), the rebin worker may indicate to a packing worker (or “packer”) that the order is complete. The packer may retrieve the items from the cell and place them in a box or bag for shipping. The packer may then send the box or bag to a hub zone 213, e.g., via forklift, cart, dolly, handtruck, conveyor belt, manually, or otherwise.

[076] Hub zone 213 may be an area of FC 200 that receives all boxes or bags (“packages”) from packing zone 211 . Workers and/or machines in hub zone 213 may retrieve package 218 and determine which portion of a delivery area each package is intended to go to and route the package to an appropriate camp zone 215. For example, if the delivery area has two smaller sub-areas, packages will go to one of two camp zones 215. In some embodiments, a worker or machine may scan a package (e.g., using one of devices 119A-119C) to determine its eventual destination. Routing the package to camp zone 215 may include, for example, determining a portion of a geographical area that the package is destined for (e.g., based on a postal code) and determining a camp zone 215 associated with the portion of the geographical area.

[077] Camp zone 215, in some embodiments, may include one or more buildings, one or more physical spaces, or one or more areas, where packages are received from hub zone 213 for sorting into routes and/or sub-routes. In some embodiments, camp zone 215 is physically separate from FC 200 while in other embodiments camp zone 215 may form a part of FC 200.

[078] Workers and/or machines in camp zone 215 may determine which route and/or sub-route a package 220 should be associated with, for example, based on a comparison of the destination to an existing route and/or sub-route, a calculation of workload for each route and/or sub-route, the time of day, a shipping method, the cost to ship the package 220, a PDD associated with the items in package 220, a delivery option, or the like. In some embodiments, a worker or machine may scan a package (e.g., using one of devices 119A-119C) to determine its eventual destination. Once package 220 is assigned to a particular route and/or sub-route, a worker and/or machine may move package 220 to be shipped. In exemplary FIG. 2, camp zone 215 includes a truck 222, a car 226, and delivery workers 224A and 224B. In some embodiments, truck 222 may be driven by delivery worker 224A, where delivery worker 224A is a full-time employee that delivers packages for FC 200 and truck 222 is owned, leased, or operated by the same company that owns, leases, or operates FC 200. In some embodiments, car 226 may be driven by delivery worker 224B, where delivery worker 224B is a “flex” or occasional worker that is delivering on an as-needed basis (e.g., seasonally). Car

226 may be owned, leased, or operated by delivery worker 224B.

[079] FIG. 3 is a block diagram of an exemplary system 300, consistent with disclosed embodiments. In system 300, a data migration (DM) system 320 may include servers, computer modules, and/or data processing centers configured to manage consumer and product data, control database backup processes, and/or deploy recovery data. System 300 may include, in addition to DM system 320, online resources 340, client devices 350, third-party systems 360, FC/Warehouse systems 310, and database 380. In some embodiments, as shown in FIG. 3, components of system 300 may be connected to a network 370. However, in other embodiments components of system 300 may be connected directly with each other, without network 370. For example, databases 380 may be directly coupled to DM system 320.

[080] In some embodiments, DM system 320 may connect to other elements of system 300 to provide a platform for database migration. Thus, in some embodiments, DM system 320 may have access, store, and/or curate information of system 300. For example, in an e-commerce context, DM system 320 may be connected to internal databases, external databases, application interfaces, and/or cloud services to manage and coordinate migration processes, execute verification operations, and arrange (or re-arrange) data structures. DM system 320 may communicate with databases through both External Front End System 103 and Internal Front End System 105. DM system 320 may also be coupled to Seller Portal 109. In certain embodiments, DM system 320 may be configured to handle database operations and for example, clone databases and handle traffic related to the database. In such embodiments, DM system 320 may be configured to store and deploy data necessary to generate the SDP of FIG. 10. DM system 320 may also coordinate security protocols and permissions to access or modify data.

[081] DM system 320 may include a controller 323 that may be coupled to tools and databases in system 300. Controller 323 may also be configured to generate clusters of data, store data, verify data, and perform catch-up operations. Controller 323 may also manage dual-write, replication, and/or cloning of databases. For example, as further discussed below in connection with FIGs. 4 and 5, controller 323 may perform operations to deploy new database schema onto production, trigger applications to write data in a new database, write to the new database while keeping the source database as a primary, and perform double writing operations.

[082] Further, in certain embodiments controller 323 may also perform functions of encryption and decryption of data. For example controller 323 may implement encryption and decryption engines that may be used for encryption and decryption of database clones or snapshots as they are migrated. Controller 323 may also encrypt traffic pipes to avoid corruption during the migration. In some embodiments, to improve security of the migration process, controller 323 may include engines or kernels for Advanced Encryption Standard(AES) for 128 bit, 192 bit, and/or 256 bit encryption. In some embodiments, controller 323 may include one or more known processing devices, such as, but not limited to, microprocessors from the Pentium™ or Xeon™ family manufactured by Intel™, the Turion™ family manufactured by AMD™, or any of various processors from other manufacturers. However, in other embodiments, controller 323 may be a plurality of devices coupled and configured to perform functions consistent with the disclosure. In some embodiments, controller 323 may be coupled to one or more databases and a clustering system, as further discussed below with respect to FIG. 4. [083] Further, DM system 320 may include local databases 325 that store and manage certain information of system 300. For example, local databases 325 may include, for example, Oracle™ databases, Sybase™ databases, or other relational databases or non-relational databases, such as Hadoop™ sequence files, HBase™, or Cassandra™. In some embodiments, local databases 325 may include computing components (e.g., database management system, database server, etc.) configured to receive and process requests for data stored in memory devices of the database(s) and to provide data from the database(s). In other embodiments, controller 322 may control operations of local databases 325 and local databases 325 may be storage units. In some embodiments, local databases 325 may include NoSQL databases such as HBase, MongoDB™ or Cassandra™. Further, local databases 325 may include relational databases such as Oracle, MySQL and Microsoft SQL Server.

[084] Local databases 325 may store data that may be used by other elements of system 300. For example, local databases 325 may store data that may be requested by client devices 350 or online resources 340. Data stored in local databases 325 may include any suitable data associated with delivery orders and/or deliverers that can be used to select a pricing algorithm and to determine and/or adjust payout to deliverers. Data stored in local databases 325 may also include financial information about users, user credentials, and/or user preferences. While FIG. 3 illustrates local databases 325 within DM system 320, in some embodiments local databases 325 may be located separately (virtually or physically) from DM system 320. For example, in some embodiments, local databases 325 may be external storage units that can be connected or disconnected by users. In some embodiments, local databases 325 may act as clone databases during migration processes performed by controller 323. For example, local databases 325 may be configured to clone one or more of external databases 380 during a migration process. Further, local databases 325 may be configured as cache databases to, for example, capture catch-up data during data migrations.

[085] In some embodiments, DM system 320 may be implemented with one or more of the components of system 100 (FIG. 1A). For example, DM system 320 may be implemented as part of internal front-end system 105, FO system 113, SCM system 117, SAT system 101 , and/or WMS 119 (FIG. 1A). In other embodiments, DM system 320 may be implemented with a database server having a client-server model for database access. In such embodiments, DM system 320 may manage information about order data, click data, and/or product data that is collected through External Front End System 103 (FIG. 1A). Alternatively, or additionally, DM system 320 may be implemented as an embedded database within a specific application. For example, DM system 320 may be embedded as part of SCM 117 (FIG. 1A). In some embodiments, DM system 320 may control access to local databases 325 through a front-end I back-end assignments of tasks in which the front end runs on client devices 350 — which displays requested data — while the back end runs on the server and handles tasks such as data analysis and storage.

[086] In some embodiments, DM system 320 may request and provide resources through application programing interfaces (APIs) that respond to query languages. APIs for DM system 320 specify filter conditions and parameters. The APIs may also provide methods for authentication and to pass or share credentials. Through APIs, DM system 320 may allow client devices 350 (or other elements of system 300) to request a data migration. Through APIs, DM system 320 may also expose migrated databases and/or request information from external databases. Further, through APIs, DM system 320 may read the database after accessing through known credentials. DM system 320 may respond to API request by fetching the records from the relevant tables in the database, wrap them and tag them, and return files the program as a returning parameter or sends to an address as passed in the parameters.

[087] DM system 320 may implement data management using proprietary database applications like Oracle, DB2, Informix, and Microsoft SQL Server. Alternatively, or additionally, DM system 320 may be implemented with free software database applications including PostgreSQL; and under the GNU General Public License.

[088] System 300 may include, besides DM system 320, online resources 340, client devices 350, third-party systems 360, FC systems 310, and databases 380. In some embodiments, as shown in FIG. 3, components of system 300 may be connected to a network 370. However, in other embodiments components of system 300 may be connected directly with each other, without network 370. For example, databases 380 may be directly coupled to DM system 320.

[089] Online resources 340 may include one or more servers or storage services provided by an entity such as a provider of webpage hosting, networking, cloud, or backup services. In some embodiments, online resources 340 may be associated with hosting services or servers that store web pages for authentication services, Domain Name System (DNS), or landing pages. In other embodiments, online resources 340 may be associated with a cloud computing service. In yet other embodiments, online resources 340 may be associated with a messaging service, such as Apple Push Notification Service, Azure Mobile Services, or Google Cloud Messaging. In such embodiments, online resources 340 may handle the delivery of messages and notifications related to functions of the disclosed embodiments.

[090] Moreover, online resources 340 may include redirect servers that can be configured by DM system 320 to switch online traffic between different elements of system 300. Further, online resources 340 may include cloud services for retrieving data from a database, duplicating data, or clustering database spaces. In some embodiments, online resources 340 may be programmable by DM system 320. For example, redirect servers in online resources 340 may be programmed by DM system 320 to redirect database queries before, during, and after a database migration. Alternatively, or additionally, online resources 340 may be configurable for encryption, decryption, and/or authentication processes. As further discussed in connection with, for example, FIG. 11 , the disclosed systems and methods may coordinate different operations for database migration and data verification.

[091] Client devices 350 may include one or more computing devices configured to perform one or more operations consistent with disclosed embodiments. For example, client devices 350 may include a desktop computer, a laptop, a server, a mobile device (e.g., tablet, smart phone, etc.), a set-top box, a gaming device, a wearable computing device, or other type of computing device. In some embodiments, client devices 350 may include the user devices 102A/102B (FIG. 1A) and be operated as part of system 100. In other embodiments, however, client devices 350 may be independent from system 100. Client devices 350 may include one or more processors configured to execute software instructions stored in memory, such as memory in client devices 350, to perform operations to implement the functions described below. For example, client devices 350 may access information in local databases 325 through DM system 320. Further, client devices 350 may be configured to perform operations according to instructions transmitted by DM system 320. Further, client devices 350 may be configured to trigger operations form DM system 320. For example, client devices 350 may initiate a database migration and/or monitor migration statuses. Further, in certain embodiments client devices 350 may receive data verification reports and/or request verification reports during or after a database migration.

[092] In some embodiments, client devices 350 may be configured for wired and/or wireless communications and may include software that when executed by a processor performs internet-related communication (e.g., TCP/IP) and content display processes. For instance, client devices 350 may execute browser software that generates and displays interfaces with product information. Thus, client devices 350 may execute applications that allow client devices 350 to communicate with components over network 370 and display content in interfaces via display devices included in client devices 350.

[093] The disclosed embodiments are not limited to any particular configuration of client devices 350. For instance, a client device 350 may be a mobile device that stores and executes mobile applications to perform operations that provide functions offered by DM system 320 and/or online resources 340.

[094] Databases 380 may include one or more computing devices configured with appropriate software to store, manage, provide, and modify data. Databases 380 may include physical databases. Further, databases 380 may include cloudbased database instances. Databases 380 may include, for example, Oracle™ databases, Sybase™ databases, or other relational databases or non-relational databases, such as Hadoop™ sequence files, HBase™, or Cassandra™.

Databases 380 may include computing components (e.g., database management system, database server, etc.) configured to receive and process requests for data stored in memory devices of the database(s) and to provide data from the database(s). Additionally, or alternatively, databases 380 may include cloud databases that are built and accessed through a cloud platform. In such embodiments, databases 380 may be managed through other elements of system 300. For example, databases 380 may be managed by controller 323. Further, cloud-based databases 380 may be configured to support relational databases (including MySQL and PostgreSQL) and NoSQL databases (including MongoDB and Apache CouchDB) and to be accessed through APIs managed by third-party systems 360 or through controller 323.

[095] While databases 380 are shown separately, in some embodiments databases 380 may be included in, or otherwise related to, DM system 320 or online resources 340. For example, databases 380 may be implemented as local databases 325 in certain embodiments. In some embodiments, databases 380 may store information related to the operation of system 300. For example, databases 380 may include pricing data, be connected to online applications (e.g., through online resources 340).

[096] Third-party systems 360 may include one or more elements of system 100. For example, third-party systems 360 may include 3PL systems 121A- 121 C (FIG. 1 ). Additionally, or alternatively, third-party systems 360 may include one or more servers or storage services provided by an entity related to DM system 320, such as a provider of services or a fulfillment center. Third-party systems 360 may also be connected to system 300 via network 370, but in other embodiments third- party systems 360 may include direct connections with some elements of system 300. For example, to minimize delays or network congestion, third-party systems 360 may be connected in a private network with DM system 320. Further, third-party systems 360 may be configured to provide and/or request information from DM system 320, or other elements of system 300. In some embodiments, while third- party systems 360 may also be coupled to network 370, they may not be clients of DM system 320. Instead, third-party systems 360 may include systems that include information of users or clients of DM system 320. For example, third-party systems 360 may include servers of delivery contractors, which may be used by DM system 320 when a product delivery involves a third-party contractor. In such embodiments, DM system 320 may be configured to generate backups of data accessible from third-party systems 360.

[097] FC/Warehouse systems 310 may include sensors and processors for determining and/or storing the location of products within an FC or warehouse and periodically update databases (such as local databases 325). For example, FC/Warehouse systems 310 may include sensors 217A-217C (FIG. 2). Alternatively, or additionally, FC/Warehouse systems 310 may include cameras that capture images of shelves and use image recognition methods to identify products and determine the position of products in the FC. Further, FC/Warehouse systems 310 may be coupled to scan devices and track the positioning of products in an FC by monitoring scanning events of products. Moreover, FC/Warehouse systems 310 may be in communication with DM system 320 to provide information that facilitates estimating available inventory. In some embodiments, FC/Warehouse systems 310 may be configured to provide periodic updates of available inventory to DM system 320. For example, FC/Warehouse systems 310 may be configured to send available inventory to DM system 320 at the end of every day. [098] Network 370 may be any type of network configured to provide communications between components of system 300. For example, network 370 may be any type of network (including infrastructure) that provides communications, exchanges information, and/or facilitates the exchange of information, such as the Internet, a Local Area Network, near field communication (NFC), or other suitable connection(s) that enables the sending and receiving of information between the components of system 300. In some embodiment network 370 may be wired, wireless, or a combination. In other embodiments, one or more components of system 300 may communicate directly through a dedicated communication link(s). In yet other embodiments, network 370 may include multiple networks, organizing for example a network or networks.

[099] It is to be understood that the configuration and boundaries of the functional building blocks of system 300 have been defined herein for the convenience of the description. Alternative boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed. Alternatives (including equivalents, extensions, variations, deviations, etc., of those described herein) will be apparent. Such alternatives fall within the scope of the disclosed embodiments.

[0100] FIG. 4 is a block diagram of an exemplary database migration system 400, consistent with disclosed embodiments. In some embodiments, migration system 400 may be part of system 300. For example, controller 323 in system 400 may be the same controller discussed as part of DM system 320 (FIG. 3). However, in other embodiments, migration system 400 may be independent from system 300.

[0101] Migration system 400 may include a controller 323. As shown in FIG. 4, controller 323 may coupled to a clone database (clone DB) 406 and a database clustering system (DB clustering system) 420. Additionally, system 400 may include a migration tool 402 and an online database (online DB) 404.

[0102] In some embodiments, controller 323 may be part of DM system 320. In other embodiments, however, controller 323 may be implemented as an independent element from DM system 320. Controller 323 may include a series of tools for database migration. These tools may be cloud-based tools or local applications. The tools may include a data expert tool 412, a data loader tool 414, a verification tool 416, and a catch-up tool 418.

[0103] Data export tool 412 may include software and/or hardware for exporting data from one or more databases. For example, data export tool 412 may be configurable to export data from online DB 404 and/or clone DB 406. Data export tool 412 may export data stored in a distributed SQL database platform or CSV data files and can be used to make a logical full back up or export. Data export tool 412 may be configurable through interfaces of controller 323 and may be used to export data in a database to SQL files, to CSV files, or other export-type file.

[0104] In some embodiments, data export tool 412 may also be configured for filtering data that do not get exported. For example, data export tool 412 may by default export the tables of the entire database. Through interfaces of controller 323, data export tool 412 may be configurable to exclude certain tables in the database, such as tables in the system databases. Additionally, or alternatively, data export tool 412 may perform filtering operations based on metadata in the tables or keywords, identifiers, or markups in the data itself. For example, data export tool 412 may filter specific database or tables by specifying a table filter.

[0105] In some embodiments, data export tool 412 may perform export operations through concurrency. For example data export tool 412 may be configured to perform concurrent operations in transmitting or exporting data to DB clustering system 420. In such embodiments, data export tool 412 may be configured to specify the maximum number of records (or the number of rows in the database) for a single file and enable concurrency in the table to improve the speed of exporting large tables. Additionally, or alternatively, data export tool 412 may verify consistency during data operations. For example, data export tool 412 may be configurable to control the way in which data is exported for consistency assurance. In such embodiments, data export tool 412 may get a snapshot of a certain timestamp by default (i.e. --consistency snapshot), and then using the snapshot for consistency. In such embodiments, data export tool 412 may use certain parameters in the snapshot to specify the timestamp to be backed up.

[0106] Data loader tool 414 may include software and/or hardware for importing data into one or more databases. For example, through data loader tool 414, controller 323 may load data into DB clustering system 420. Data loader tool 414 may be configured as a tool for multi-threaded loading of data and backups. Data loader tool 414 may operate by connecting to a host server, or database, identifying privileges required for writing and/or loading data, and initiate the data transfer. For example, data loader tool 414 may open ports and establish connections to domain socket files to then deliver data to specific locations in a database and to preserve certain structures. Further data loader tool 414 may be configurable to perform one or more data corrections. For example, data loader tool 414 may deduplicate table names, defragment certain portions of data, and/or overwrite data that may have been corrupted. Further, in certain embodiments data loader tool 414 may generate output files reporting exported data, data structures, and any modifications. Additionally, or alternatively, data loader tool 414 may be configurable to transfer metadata information in the tables of a database and metadata of the data load, including, for example, start and end time of the load as well as the master binary log positions, separated files, table schemas, and binary logs.

[0107] Verification tool 416 may include hardware and/or software for verification of data before, during, and/or after the database migration process. Verification tool 416 may be configured for data verification across different environments. In some embodiments, verification tool 416 may include a framework to connect to a large number of online data sources that aid in verification processes. Verification tool 416 may perform multi-leveled data validation functions, from the table level all the way to the row level. For example, verification tool 416 may be configurable to perform table level verification, perform operations like table row count, group by row count, column aggregation, randomized filters, and limits. Alternatively, or additionally, verification may be configurable to perform column, row, or SQL level verification. For example, verification tool may perform operations to verify schema/column data type, execute hash comparison, and/or run custom queries on different data sources.

[0108] In some embodiments, verification tool 416 may be configurable through interfaces of controller 323 to perform concurrent operations. For example, verification tool 416 may be connected to both data and target databases to run range- or block-based verification. During such preliminary or concurrent verification, verification tool 416 may verify based on counting number of columns in the source table and verify it matches on the target table. Additionally, or alternatively, verification tool 416 may perform concurrent verification through specific columns, based on labels, or customized validations. [0109] Catch-up tool 418 may include software and/or hardware for fixing data gaps between target and source databases. Catch-up tool 418 may be programable through interfaces of controller 323 and perform operations for double writing in different databases, creating cache or transitory databases to store refreshed and/or gap data, and perform required iterations to confirm verifications. In some embodiments, catch-up tool 418 may use an incremental copy approach where modified data are periodically copied from the source to target environment utilizing multiple passes of the source data. In such embodiments, catch-up tool 418 may turn on double write to write incremental data in the target environment, create a new database block or instance to retrieve the incremental data, and run verifications and catch-up operations to transfer incremental data from the source to the target environment. In such embodiments, catch-up tool 418 may migrate incremental changes to the data are processed with each subsequent pass. And in certain embodiments, catch-up tool may break incremental data in clusters for faster verification and placing in the target database.

[0110] In some embodiments, catch-up tool 418 may additionally, or alternatively, apply a dual write approach, simultaneously storing, and modifying two databases. In some embodiments, catch-up tool 418 may perform certain tests before initializing dual-writing operations. For example, controller 323 may engage catch-up tool 418 when applying a database schema to DB clustering system 420 and then use verification tool 416 to confirm the dual writing operation is applying the same data structure and writing data concurrently.

[0111] Moreover, in certain embodiments catch-up tool 418 may perform a concurrent monitoring of modifications or entries in the source or origin database so it need not compare data periodically but identify gap data as it is written in the source database. In such embodiments, catch-up tool 418 may be coupled to an event information stream from the source data that allows the tool to monitor activities and scan changes in stored information. In such embodiments, catch-up tool 418 not just identifies change gap data, but places a continuous stream of change notifications with scanned data that are undergoing change to then make determinations of whether they have been captured through the dual-write operation. In such embodiments, catch-up tool 418 may enable data migrations to be performed even as the data sets are undergoing active change.

[0112] As shown in FIG. 4, in certain embodiments controller 323 (and the tools in controller 323) may be connected to DB clustering system 420. DB clustering system 420 may include a database clustering system that stores and organizes data through generalized sharding. In some embodiments, DB clustering system 420 may encapsulate a shard-routing logic. In such embodiments, DB clustering system 420 may allows application code and database queries to the distribution of data onto multiple shards. For example, DB clustering system 420 may split and merge shards as data is inputted into the database. In such embodiments, DB clustering system 420 may select a pre-defined clustering size and arrange data in the different shards based on the data size. Moreover, in some embodiments DB clustering system 420 may include a database clustering system exposed, or exposable, to queries routed through application programing interface (API) calls. In this way, DB clustering system 420 may be a migration database that handles online queries after migration. Further, DB clustering system 420 may be connected to online resources. For example, DB clustering system 420 may be connected to online resources 340.

[0113] In some embodiments, DB clustering system 420 may provide solutions for quick transfer of data of live data bases with high traffic and large datasets. For example, DB clustering system 420 may deploy, size, and manage large clusters of database instances. DB clustering system 420 may manage multiple types of databases including MySQL, Percona, and MariaDB databases. Further DB clustering system 420 may be architected to work as effectively on a public or private cloud architecture as on dedicated hardware. In some embodiments, DB clustering system 420 may be configured to share it information while keeping application changes to a minimum. For example, as information is loaded in DB clustering system 420, one or more clusters may be setup to share with external or internal sources. DB clustering system 420 may also communicate with cloud-based databases and deploy individual database instances that can be managed with independent structure files.

[0114] In certain embodiments, DB clustering system 420 may include several server processes, command-line utilities, and web-based utilities, supported by a consistent metadata store. For example, during the migration process controller 323 may setup DB clustering system 420 as the target database that will be used to support online operations. In such embodiments, DB clustering system 420 may be setup a database server, using a database application that provide database services to other computer programs or to computers, as defined by the clientserver model. Moreover, in some embodiments DB clustering system 420 may act as the target or migration database during a migration process.

[0115] As shown in FIG. 4, in additional to being coupled to DB clustering system 420, controller 323 may also be connected to a clone DB 406. Clone DB 406 may include hardware and/or software that creates a new cluster with the same volume and the same data as an original database. Clone DB 406 may be a cloudbased database instance with its associated data volume associated with an original database. In some embodiments, clone DB 406 may be a provisioned clone from a database cluster. Alternatively, or additionally, clone DB 406 may be serverless and provisioned with the same rules as the original database(s).

[0116] Clone DB 406 may have the same behavior as the original database. For example, clone DB 406 may have the same data structure, database schema, and data arrangement. Further, clone DB 406 may store the same metadata and verification data.

[0117] In embodiments with large data sets, clone DB 406 may use a copy and write protocol for efficient replication of data in clone DB 406. In such embodiments, clone DB 406 may store data in pages in and underlying storage volume of clone DB 406. Further clone DB 406 may clone the data using the same segment on the physical storage media as the source segment. Clone DB 406 may also adjust demanded storage on the fly. For example, during cloning of clone DB 406, the database may determine a storage that is equal to that of the origin database. If additional space is required, clone DB 406 may create new pages as the data is being replicated in the new database. Further, clone DB 406 may reference original pages for data routing and API responses.

[0118] In some embodiments, and as further discussed below in connection with FIG. 5, controller 323 may create clone DB 406 using one or more of local databases 325. Controller 323 may create clone DB 406 replicating the original or source database in a new location. Additionally, or alternatively, controller 323 may create a provisioned clone DB 406 from a provisioned previously provisioned database instance. Further, clone DB 406 may be created as an encrypted database to improve security. Moreover, while clone DB 406 is shown as a single database in FIG. 4, in some embodiments clone DB 406 may include multiple databases that may be associated through one or more administration servers.

[0119] Online DB 404 may be similar to clone DB 406. Online DB 404, however, may be connected to online traffic. In certain embodiments, online DB 404 may be a serverless on-demand database system. In such embodiments, online DB 404 may be configured to receive and process API calls requesting data from online DB 404.

[0120] In some embodiments, online DB 404 may be a cloud-based database instance. In other embodiments, however, online DB 404 may be a database connected to online traffic. For example, online DB 404 may be one or more of local databases 325 connected to network 370. Moreover, online DB 404 may be implemented as a specific purpose database, with a specific capacity range and supporting targeted applications.

[0121] In some embodiments, online DB 404 and clone DB 406 may be one or more of databases 380. In other embodiments, online DB 404 and clone DB 406 may be local databases 325. In yet other embodiments, online DB 404 and clone DB 406 may be a combination of databases 380 and local databases 325. For example, while online DB 404 may be setup as a one of databases 380, external to DM system 320 and connected to network 370 to support online operations, clone DB 406 may be setup as one of local databases 325.

[0122] As shown in FIG. 4, online DB 404 and DB clustering system 420 may be connected to a migration tool 402. Migration tool 402 may include hardware and/or software that connects to elements in migration system 400 to interface with client devices and coordinate database migration processes. For example, migration tool 402 may interface with client devices 350 (FIG. 3) and receive and/or process instructions for migration of databases. In turn, migration tool 402 may configure controller 323, or the tools within controller 323, to operate on online DB 404, create clone DB 406, and migrate data to DB clustering system 420. For example, migration tool 402 may receive instructions to transfer data from one type of database to another, or from a database to another type of data repository such as a data warehouse or data lake. In such scenarios, migration tool 402 would configure controller 323, DB clustering system 420, to export, load, and then catch-up gap data so that data in online DB 404 migrates to DB clustering system 420. Migration tool 402 may also coordinate database replication instructions to facilitate data migration and, as explained above, minimize downtime and improve data accuracy through concurrent verification. As further discussed in FIGs. 5-11 , through migration tool 402 migration system 400 may facilitate migration of databases.

[0123] Migration tool 402 may be configured as a migration application, accessible by clients or administrators to automate migration processes and configure and execute migrations. In some embodiments, migration tool 402 may provide a graphical designer or mapping tool. Further, migration tool 402 may be equipped with log-based reports, and tools to adjust migration targets, filters, and data structure modifications.

[0124] It is to be understood that the configuration and boundaries of the functional building blocks of system 400 have been defined herein for the convenience of the description. Alternative boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed. Alternatives (including equivalents, extensions, variations, deviations, etc., of those described herein) will be apparent. Such alternatives fall within the scope of the disclosed embodiments. [0125] FIG. 5 is a flow diagram of an exemplary process 500 for database migration, consistent with disclosed embodiments. In some embodiments, elements of system 300 may perform process 500. For example, as disclosed in the steps description below, DM system 320 may perform process 500. In some embodiments, controller 323 may be configured to perform the operations in process 500. Alternatively, or additionally, online resources 340 and/or third-party systems 360 may perform process 500, or parts of process 500. Further, in other embodiments system 100, or parts of system 100, may perform process 500. For instance, internal front end system 105, SCM system 117, and/or SAT 101 may perform process 500. Further, in other embodiments system 400 may implement process 500. For example, migration tool 402, controller 323, and/or DB clustering system 420 may perform operations in process 500.

[0126] The description of process 500 below illustrates an embodiment in DM system 320 performs steps of process 500. However, as previously discussed, other elements of system 300 may also be configurable to perform one or more of the steps in process 500. For example, online resources 340 may perform one or more steps of process 500.

[0127] In step 502, DM system 320 may apply a database schema to a target database, which may be setup as a database clustering system. For example, DM system 320 may apply a database schema to one of local databases 325. In some embodiments system DM system 320 may apply a database schema to DB clustering system 420 through controller 323. In some embodiments, as further discussed in connection with FIG. 9, exporting data in step 504 may include generating a clone database. To minimize downtown and avoid operation disruption, DM system 320 may initiate process 500 with a cloning operation that allows DM system 320 to perform data exports and verification without disturbing the online database (like online DB 404).

[0128] In step 504, DM system 320 may export data from a database. For example, DM system 320 may export data in one or more databases 380 to SQL files and/or CSV files. In some embodiments, DM system 320 may export data of online DB 404 through controller 323. For example, as discussed in connection with FIG. 4, controller 323 may include an export tool 412. In such embodiments, DM system 320 may configure and execute operations for exporting data from online DB 404 using the configurations and/or functions of export tool 412.

[0129] In step 506, DM system 320 may import files to the target base, which may be setup as a database clustering system. For example, DM system 320 may import files to one of local databases 325. In some embodiments, DM system may import the data exported in step 504 to DB clustering system 420 through controller 323. For example, as discussed in connection with FIG. 4, controller 323 may include data loader tool 414. In such embodiments, DM system 320 may configure and execute operations for loading data to DB clustering system 420 using the configurations and/or functions of data loader tool 414.

[0130] In step 508, DM system 320 may verify data consistency of importation. For example, DM system 320 may verify the consistency and/or accuracy of the data imported into the target database in step 506. In some embodiments, DM system 320 may verify the data imported in step 506 to DB clustering system 420 through controller 323. For example, as discussed in connection with FIG. 4, controller 323 may include verification tool 416. In such embodiments, DM system 320 may configure and execute operations for verifying the import operations into the DB clustering system 420. [0131] While FIG. 5 shows step 508 following step 506, some embodiments may perform concurrent verification of data consistency. As discussed in connection with FIG. 4, verification tool 416 may operate during importation by, for example, operating on specific clusters of DB clustering system 420 while the other clusters are completed. The concurrent verification may include higher-level verification, as discussed above. In other embodiments, however, step 510 may include both concurrent and sequential verification.

[0132] In step 510, DM system 320 may turn on double write operations. For example, as discussed above in connection with FIG. 4, controller 323 may include tools for double writing that captures incremental data in two database spaces. The incremental data may be received from online resources 340 and/or client devices 350. For example, a user through a client device 350 may issue database queries to update pricing data for a product or multiple products. In such embodiments, the changes requested by the user may be replicated in both online DB 404 and DB clustering system 420. Alternatively, or additionally, the instructions updating online DB 404 may be received from online resources 340 and /or migration tool 402, as user make changes while DM system 320 performs the migration. In step 510 DM system 320 may initiate the double writing operations to replicate data modifications in an online database into the target database.

[0133] In step 512, DM system 320 may create a new database clone from the online database. For example, DM system 320 may create a new clone database to capture more recent data and data modifications in the online database. In such embodiments, the new database may retrieve more refreshed data. In some embodiments, DM system 320 may use controller 323 to create new clones in new cloud instances. In some embodiments, the new or second clone of the online database to a second clone database may be rewritten over the first clone. For example, step 512 may include rewriting over a previous clone database. In other embodiments, however, the clone databases may be independent.

[0134] In step 514, DM system 320 may identify and load a delta of datasets or modifications between the clone generated in step 512 from the point that the double write operations started in step 510. For example, DM system 320 may identify data that has changed since the start of the double write to capture incremental data. In some embodiments, DM system 320 may identify the data through data verification. And DM system 320 may load the delta, or catch-up data, through a synchronization of tables from the new clone database to the DB clustering system 420.

[0135] In step 516, DM system 320 may run a new consistency verification. For example, after loading the data delta to DB clustering system 420 in step 514, DM system 320 may run a new verification process similar to the one performed in step 508 but this time using a live database (e.g., online DB 404) dataset for verification. Through step 516, for example, DM system 420 verifies that the catchup data loaded in step 514 results in the same data in online DB 404 as in the target DB clustering system 420. This step allows DM system 320 to verify that the data migration has been completed and that the two databases have the same content and structure.

[0136] In step 520, DM system 320 may determine whether there are any inconsistencies in the migration. Based on the results of the verification from step 516, DM system 320 may determine if there are inconsistencies between origin and target databases. If in step 520 DM system 320 determines that there are data inconsistencies (step 520: Yes), DM system 320 may return to step 512, create a new DB clone to then identify, load the catch up data (step 516), and then perform a new verification (520). In some embodiments, this iterative process may be replicated multiple times until the verification process is successful. If DM system 320 determines that there are no data inconsistencies (step 520: No), DM system 320 may continue to step 522.

[0137] In step 522, DM system 320 may determine the migration process has completed the data catch-up and/or that the migration has resulted in complete replication of the databases. In some embodiments, as further discussed in connection with FIG. 9, the culmination of the catch-up operations to load data deltas may then continue to a substitution of online traffic from the original database to a migrated, or target, database. For example, in some embodiments traffic that was originally routed to online DB 404 may transfer to DB clustering system 420. Moreover, DM system 320 may perform additional operations, like generating notifications to client devices 350 and/or updating API processing based on the data migration.

[0138] FIG. 6 is a flow diagram of an exemplary process 600 for migrated data consistency verification, consistent with disclosed embodiments. In some embodiments, elements of system 300 may perform process 600. For example, as disclosed in the steps description below, DM system 320 may perform process 600. In such embodiments, controller 323, and more specifically verification tool 416, may be configured to perform the operations in process 600. Alternatively, or additionally, online resources 340 and/or third-party systems 360 may perform process 600, or parts of process 600. Further, in other embodiments system 100, or parts of system 100, may perform process 600. For instance, internal front end system 105, SCM system 117, and/or SAT 101 may perform process 600. Further, in other embodiments system 400 may implement process 600. For example, migration tool 402, controller 323, and/or DB clustering system 420 may perform operations in process 600. In some embodiments, process 600 may be part of process 500. For example, process 600 may be executed as part of step 516 in process 500 (FIG. 5). In other embodiments, process 600 may be independent from process 500.

[0139] The description of process 600 below illustrates an embodiment in DM system 320 performs steps of process 600. However, as previously discussed, other elements of system 300 may also be configurable to perform one or more of the steps in process 600. For example, online resources 340 may perform one or more steps of process 600. In such embodiments, online resources 340 may communicate and execute operations as dictated by controller 323.

[0140] In step 602, DM system 320 may initiate a verification process. For example, as part of process 500 DM system 320 may initiate data verification during importation of data, cloning, and/or data verification. In some embodiments, as shown in FIG. 6, the verification may be based on a row comparison verification. In other embodiments, however, the verification may be based on table-level and/or row verification. In such embodiments, DM system 320 may perform operations like table row count; group by row count; column aggregation; randomized filters; and filter limits. Moreover, in step 602 DM system may be configurable to perform column or SQL level verification. For example, verification tool may perform operations to verify schema/column data type, execute hash comparison, and/or run custom queries on different data sources. Moreover, in certain embodiments — and to improve resource allocation during migration — DM system 320 may perform the row verification over a selected number of rows in online databases. These rows may be selected based on a data range identified by a user (e.g., through migration tool 402). In such embodiments, DM system 320 may perform row comparison of clone and migration databases by comparing hash of column values in each row of the first cloned database and the migration database.

[0141] In step 610, DM system 320 may determine whether it found any inconsistencies in the data. For example, DM system 320 may determine whether all rows between two databases pass the verification tests (e.g., using hash functions). If DM system 320 determines that there are inconsistencies as result of the verification (step 610: Yes), DM system 320 may continue to step 612.

[0142] In step 612, DM system 320 may identify the range of inconsistencies. For example, DM system 320 may identify a range of rows that has inconsistent datasets. In step 614, DM system 320 may attempt to resolve the inconsistencies by running catch-up loading for the specific data range. For example, as discussed in connection with FIG. 5, DM system 320 may create clone databases that are then used to load delta data. In step 614, DM system may run these processes (e.g., steps 512 and 514) but limited to the specific ranges identified in step 612 to resolve inconsistencies. As shown in FIG. 6, after re-running catch-up loading for the range with inconsistent data, DM system 320 may return to step 602 and re run the verification.

[0143] However, if in step 610 DM system 320 determines that there are no inconsistencies (step 610: No), DM system 320 may continue to step 622 and turn off dual writing operations. As discussed, in connection with FIG. 5, DM system 320 may use dual writing operations to, for example, write incremental data in a target database, such as DB clustering system 420. When the verification is completed and no inconsistencies are discovered, DM system 320 may stop the double writing operation. [0144] In step 624, DM system 320 may re-run a catch-up verification for a pre-determined time window to verify no additional data was modified in the online database after the double write cutoff. As online databases may have changes, even during small time windows, DM system 320 may re-run a catch-up verification during a time window to identify any data changes that may have from the last successful verification and that may have not been captured through the double write operation. If any missing data is identified, DM system 320 may load the catch up data.

[0145] In step 626, DM system 320 may switch traffic to the target database. For example, DM system 320 may switch traffic from online DB 404 to DB clustering system 420. For example, DM system 320 may adjust database addresses, servers, and/or API processors to route data request from/to the original databases to the migrated database.

[0146] In step 628, DM system 320 may liberate space of the clone databases. For example, as discussed in connection with FIG. 5, DM system 320 may create clone databases in local databases 325, such as clone DB 406, that are used for verification, export, and manipulation of data during the migration process. DM system 320 may also use local databases 325 for clones for catch-up operations that synchronize target databases with refreshed data. In step 628, DM system 320 may liberate space in these clone databases by, for example, freeing up space in local databases 325.

[0147] FIG. 7 is a flow diagram of an exemplary process for migration of a live database, consistent with disclosed embodiments. In some embodiments, elements of system 300 may perform process 700. For example, as disclosed in the steps description below, DM system 320 may perform process 700. In such embodiments, controller 323 may be configured to perform the operations in process 700. Alternatively, or additionally, online resources 340 and/or databases 380 may perform process 700, or parts of process 700. Further, in other embodiments system 100, or parts of system 100, may perform process 700. For instance, internal front end system 105, SCM system 117, and/or SAT 101 may perform process 700. Further, in other embodiments system 400 may implement process 700. For example, migration tool 402, controller 323, and/or DB clustering system 420 may perform operations in process 700.

[0148] The description of process 700 below illustrates an embodiment in DM system 320 performs steps of process 700. However, as previously discussed, other elements of system 300 may also be configurable to perform one or more of the steps in process 700. For example, online resources 340 may perform one or more steps of process 700. In such embodiments, online resources 340 may communicate and execute operations as dictated by controller 323.

[0149] In step 702, DM system 320 may apply a database schema and turn on double write to replicate incoming data modifications to a DB clustering system. For example, DM system 320 may apply a database schema from online DB 404 to DB clustering system 420. Further in step 704 DM system 320 may turn on double writing to test replication of data through the double writing operation. In some embodiments, as shown in FIG. 7, steps 702 and 704 may be performed before generating the export files. In such embodiments, DM system 320 may apply a database schema to the migration database; perform a test of the double write operation; and pause the double write operation before export files are generated and/or the live database is cloned. [0150] In step 706, DM system 320 may create a first clone database from an online database. For example, though controller 323 DM system 320 may create a clone of online DB 404 to the clone DB 406, as discussed in FIG. 4.

[0151] In step 708, DM system 320 may export data from clone DB 406 to transfer files. For example, through controller 323 and/or data export tool 412, DM system 320 may export data from clone DB 406 into transfer files. The transfer files may include SQL files. In other embodiments, however, the transfer files may include CSV files or other type of format.

[0152] In step 710, DM system 320 may import transfer files to a target database with a clustering system. For example, through controller 323 and/or data loader tool 414, DM system 320 may import transfer files to DB clustering system 420.

[0153] In step 712, DM system 320 may create a new clone that replicates the original database at a later point, after the initial import of files into the target database. For example, DM system 320 may create a new database clone in one of local databases 325 with refreshed data. As further discussed in connection with FIG. 5, DM system 320 may perform catch-up operations to identify and then load data that was not captured during initial cloning. Through sequential cloning of the online database and concurrent or iterative verification, DM system 320 may complete the migration without data losses and without downtime.

[0154] In step 714, DM system 320 may run catch-up programs to complete transfer of new data. For example, DM system 320 may perform operations for identifying delta of information not captured since the last transfer and load it in the target database. The catch-up programs may also include operations performed by catch-up tool 418 (FIG. 4). Moreover, catch-up programs may also include concurrent and/or sequential verification of data, as discussed in steps 514-520 (FIG. 5).

[0155] In step 716, DM system 320 may perform a final synchronization between origin and target databases. For example, DM system 320 may select a time window to perform traffic analysis and range verification between the databases and port any inconsistent data from the origin database to the target database.

[0156] In step 718, DM system may transfer traffic from the origin live database to the migrated or target database. For example, DM system 320 may switch traffic from online DB 404 to DB clustering system 420. In some embodiments, DM system 320 may adjust database addresses, servers, and/or API processors to route data request from/to the original databases to the migrated database, or instruct other devices to do the same.

[0157] FIG. 8 is a flow diagram of an exemplary process for database migration verification, consistent with disclosed embodiments. In some embodiments, elements of system 300 may perform process 800. For example, as disclosed in the steps description below, DM system 320 may perform process 800. In such embodiments, controller 323 may be configured to perform the operations in process 800. Alternatively, or additionally, online resources 340, client devices 350, and/or databases 380 may perform process 800, or parts of process 800. Further, in other embodiments system 100, or parts of system 100, may perform process 800. For instance, internal front end system 105, SCM system 117, and/or SAT 101 may perform process 800. Further, in other embodiments system 400 may implement process 800. For example, migration tool 402, controller 323, and/or DB clustering system 420 may perform operations in process 800. [0158] The description of process 800 below illustrates an embodiment in DM system 320 performs steps of process 800. However, as previously discussed, other elements of system 300 may also be configurable to perform one or more of the steps in process 800. For example, online resources 340 may perform one or more steps of process 800. In such embodiments, online resources 340 may communicate and execute operations as dictated by controller 323.

[0159] In some embodiments, to perform process 800 DM system 320 may perform a series of operations to first execute a query using sequential operations of query, shard determinations, and row comparison. For example, DM system 320 may execute queries using instructions such as:

SELECT

PK,

SHARD_KEYS, crc32(CONCAT_WS(<ALL_COLUMNS>)) as crc

FROM <TABLE>

WHERE ('PK' >= 1 AND 'PK' < 1001 )

ORDER BY 'PK'

[0160] In step 802, DM system 320 may initiate a verification operation. For example, as part of process 700, DM system 320 may verify data after (or during) export/import operations (e.g., in step 710) and/or identify missing data for a catchup operation (e.g., in step 714). In such scenarios, DM system 320 may initiate a data verification by, for example, engaging or triggering operations by verification tool 416.

[0161] As shown in FIG. 8, the verification process may be performed with concurrent operations in origin and target databases. For example, verification tool 416 may be configured to operate in both origin and target databases after the data migration and perform parallel operations to determine data inconsistencies. In such embodiments, steps 804-808 may be executed over data in an origin database. For example, controller 323 may perform steps 804-806 on online DB 404. In contrast, steps 810-812 may be performed on data in a migration or target database. For example, controller 323 may perform steps 810-814 on DB clustering system 420.

[0162] In step 804, DM system 320 may read tables in a database. In some embodiments, as discussed above, to minimize downtime DM system 320 may perform the verification operations over a clone database. In this way, DM system 320 may avoid disrupting operations, particularly for databases that support online applications. Thus, in step 804 DM system 320 may read tables in a clone database. For example, DM system 320 may configure controller 323 to read tables in clone DB 406.

[0163] In step 806, DM system 320 may calculate a shard key for each row in database to determine a shard. Shard keys may include single indexed field and/or multiple fields covered by a compound index that determines the distribution of the collection's documents among a cluster's shards. DM system 320 may Calculate Shard Key's through a has value and use this as a partition. In calculating shard keys, DM system 320 may calculate hash values and/or a preconfigured label in the source config of each row. In some embodiments, the calculated shard keys can be an application and the metric name, which define a group of shards to use for that application. Additionally, or alternatively, DM system 320 may calculate a hash from each row in database objects and compare them with a shard identifier representing the shard. In such embodiments, DM system 320 may create a log of index data on the shard. DM system 320 may use different methods for the calculation of the shard keys, including but not limited to DJB2 (32-bit), SDBM (32-bit), LoseLose (32-bit), FNV-1/FNV-1a (32-bit), CRC16 (16-bit), and/or Murmur2/Murmur3 (32-bit).

[0164] In parallel to steps 804-806, DM system 320 may be performing steps 810-812 on the target database. In some embodiments, steps 810-812 may correspond to steps 804-806 but being applied on the target database. For example, while DM system 320 may perform the verification of steps 804-806 on clone DB 406, DM system 320 may perform the verification of steps 810-812 on DB clustering system 420. Similar to step 804, in step 810 DM system 320 may read tables in DB clustering system 420. And, like in step 806, DM system 320 may calculate a shard key for each row to determine a shard in DB clustering system 420. Moreover, DM system 320 may determine shard keys by using the SHA1 algorithm, creating hash values and/or shard keys. For example, DM system 320 may use UnicodeEncoding class to convert strings into an array of bytes that are hashed by using the SHA256 class.

[0165] In step 814, DM system 320 may compare column values and/or shards calculated in steps 804-812. For example, DM system 320 may compare shard keys, shards, and hash values using iterative operations through each byte of the values to makes comparisons.

[0166] In step 816, DM system 320 may determine whether records are equal. For example, in step 816 DM system 320 may determine whether target and the origin database have the same content and data structure based on the comparison of shards, shard keys, and/or hash values. If DM system determines that the records are not equal (step 816: No), DM system 320 may continue to step 818.

[0167] In step 818, DM system 320 may determine weather the different records (e.g., different rows) have a newer modification after a double write operation. Because different records may have been modified after the execution of the dual write, the different records may need no correction as they would reflect the most recent changes. Thus, records that have a newer modification after the double write operation (e.g., after step 510 was executed) could be up-to-date and the differences need not be addressed or corrected. Thus, if in step 818 DM system 320 determines that the different record has a newer modification time, DM system 320 may continue to step 822 and ignore the row or record with the different hash or the table with the different shard. After ignoring the record with the newer modification, DM system 320 may continue to step 816 and once again determine whether the compared records are equal. However, if in step 818 DM system 320 determines that the different record do not have the newer modification time, DM system 320 may continue to step 820 and indicate that the verification has been unsuccessful. If different records do not have a newer modification, it means that the dual-write process has not captured the differences and that there is an issues with the data export or import. In such scenarios, verification tool 416 may trigger a signal indicating improper verification and, for example, indicate inconsistencies that get addressed by DM system 320 as discussed in connection with FIG. 5 (see e.g., step 520).

[0168] However, if in step 816 DM system 320 determines that the records are equal (step 816: Yes), DM system 320 may continue to step 824. In step 824, DM system 824 may determine that the data transfer was successful. In such embodiments, verification tool 416 may trigger a signal indicating proper verification and, for example, indicate no inconsistencies were found in the migration and continue the migration process, as described in FIG. 5. [0169] In some embodiments, DM system 320 may perform process 800 through a sequence of first determining whether rows in the first cloned database have the same shard key as corresponding rows in the migration database. In response to determining one or more rows in the first cloned database have different shard key from corresponding rows in the migration database, DM system 320 may determine whether the row in the migration database has a later modification time. Then, DM system 320 may ignore rows in the first cloned database with different shard key from corresponding rows in the migration database. And DM system 320 may determine data consistency when shard keys of unignored rows are the same. Though this sequence DM system 320 may perform a data verification process.

[0170] FIG. 9 is a flow diagram of an exemplary process for online database migration, consistent with disclosed embodiments. In some embodiments, elements of system 300 may perform process 900. For example, as disclosed in the steps description below, DM system 320 may perform process 900. In such embodiments, controller 323 may be configured to perform the operations in process 900. Alternatively, or additionally, online resources 340 and/or databases 380 may perform process 900, or parts of process 900. Further, in other embodiments system 100, or parts of system 100, may perform process 900. For instance, internal front end system 105, SCM system 117, and/or SAT 101 may perform process 900. Further, in other embodiments system 400 may implement process 900. For example, migration tool 402, controller 323, and/or DB clustering system 420 may perform operations in process 900.

[0171] The description of process 900 below illustrates an embodiment in DM system 320 performs steps of process 900. However, as previously discussed, other elements of system 300 may also be configurable to perform one or more of the steps in process 900. For example, online resources 340 may perform one or more steps of process 900. In such embodiments, online resources 340 may communicate and execute operations as dictated by controller 323.

[0172] In step 902, DM system 320 may clone the live database. For example, through controller 323, DM system 320 may export files from online DB 404 to generate clone DB 406. In some embodiments, online DB 404 (which is cloned) may be handling online traffic. In certain embodiments, the cloning of step 904 may facilitate data migration with little to now down time.

[0173] In step 904, DM system 320 may generate export files from one or more clone databases and importing them to a migration database. For example, through controller 323 DM system 320 may generate export files from clone DB 406 and then load them into DB clustering system 420. In some embodiments, DM system 320 may use data export tool 412 to generate export files from clone DB 406. DM system 320 may also employ data loader tool 414 for importing or loading the export files to DB clustering system 420. As further explained in connection with FIGs. 4-5, the export files may include SQL files, CSV files, and/or different type of export files. Further, as discussed in connection with FIG. 4, data loader tool 414 may perform loading operations in parallel while generating multiple clusters.

[0174] In step 906, DM system 320 may perform a row comparison of the first clone database and the migration database. As shown in FIG. 9, step 906 may be performed after step 904. That is step 906 may be executed after importing the export files into the target database. In other embodiments, however, step 906 and step 904 may be executed concurrently. For example, as discussed in connection with FIG. 4, the data verification processes may occur while data is being loaded into DB clustering system 420. [0175] In step 906, DM system 320 may perform a data verification by calculating shard keys for a selected number of rows in the clone database and the migration database. For example, as further discussed in connection with FIG. 8, DM system 320 may perform data verification by reading tables in both clone DB 406 and DB clustering system 420, calculating shard keys, hash values, and/or determining shards, to then compare records and make determinations of whether or not they are equal.

[0176] In step 908, DM system 320 may initialize a double write operation replicating writing of the online database on the migration database to capture incremental data. As discussed in connection with FIG. 5, double writing operations may allow capturing incremental data. In some embodiments, the double write operation of step 908 may have been used before during initialization. For example, as discussed for process 700 (FIG. 7), DM system 320 may apply database schema at the beginning of the process and then turn it off. The initial double write operation may be used for testing and quality control. For example, DM system 320 may perform the dual writing operation and then perform data verification to confirm the dual operation is working and replicating incremental data in online DB 404 and DB clustering system 420. As discussed in connection withs FIG. 5 and 11 , the double writing operation may replicate data modification in a live database (like online DB 404) to the target or migration database (like DB clustering system 420).

[0177] In step 910, DM system 320 may clone the online database to a second clone database. For example, as discussed in connection with FIG. 5, DM system 320 may generate a new clone with refreshed data that may enable the identification of catch-up or delta data. In some embodiments, DM system 320 may use the same tools and procedures for the generation of the second clone in step 910 as for the generation of the first clone in step 902.

[0178] In step 912, DM system 320 may identify catch-up data by comparing the clone database of step 910 with the incremental data written after initiating the dual write operation. As discussed in connection with FIG. 5, DM system 320 may perform operations for identifying delta data sets that have not been loaded in the migration or target database. For example, DM system 320 may identify catch-up data (which is some embodiments may be identified based on the modification time being since the double write time).

[0179] In step 914, DM system 320 may update the migration database to include the catch-up data. For example, through controller 323 DM system may load data identified in step 912 into the DB clustering system 420. Also, as discussed in connection with FIG. 5, DM system 320 may perform iterative data verification until no inconsistencies are found between clone database and the target or migration database. In some embodiments, the update catch-up operation in step 914 may be performed by selecting a group of rows in the new clone database comprising at least a portion of the catch-up data, generating a temporary file comprising the group of rows, and synchronizing the migration database with the second cloned database by inserting the temporary file in a position consistent with the group of rows in the second cloned database. In this way the, DM system 320 may efficiently replicate delta or catch up data identified in an online database (like online DB 404) into the target database.

[0180] In step 916, DM system 320 may determine whether the migrated data is consistent across databases. For example, DM system 320 may employ the same operations of verification discussed in connection with FIG. 8 to determine if data in online DB 404 and DB clustering system 420 are consistent. If DM system 320 determines that the migrated data is not consistent (step 916: No), DM system 320 may return to step 910 and clone the live database to a new clone database. For example, in some embodiments, after updating the migration database to include the catch-up data, DM system 320 may perform a second row comparison between a second clone database and the migration database to determine whether data is consistent. And in response to determining the data is in not consistent, DM system may identify inconsistent rows, generate a temporary file comprising the inconsistent rows, and updating the migration database by inserting the temporary file in positions corresponding to the inconsistent rows. However, if DM system 320 determines that the migrated data is consistent (step 916: Yes), DM system 320 may continue to step 918.

[0181] In step 918, DM system 320 may stop or pause the double write operation. And in step 920, DM system 320 may switch online traffic. For example, DM system 320 may coordinate addresses, server tables, and/or API processors to route data request from/to the online DB 404 to DB clustering system 420. Thus, in certain embodiments, in response to determining the data is consistent DM system 320 may pause the double write operation and switch online traffic from the online database to the migration database. Moreover, in certain embodiments after pausing the double write operation, DM system 320 may perform a final synchronization between the second cloned database and the migration database.

[0182] FIG. 10 is a schematic representation 1000 of a database migration process using a clone database, consistent with disclosed embodiments. Schematic representation 1000 provides an example of a database migration showing how some of databases in the migration process would reflect certain information at different stages of the process.

[0183] In a first time 1010, online DB 404 may have records 1012. These records have been replicated into DB clustering system 420, which has records 1014. Time 1010 may be the state after DM system 320 has exported files from online DB 404 and imported the files to DB clustering system 420.

[0184] In a second time 1020, records in online DB 404 have changed, there are more, and online DB 404 now has records 1022. The records 1024 in the DB clustering system have not changed. Time 1020 may be when DM system creates the first clone (see e.g., FIG. 9, step 902). Thus, clone DB 406 may have the same records as online DB 404 in time 1020.

[0185] In a third time 1030, records in online DB 404 have changed again, there is an additional record and online DB 404 now stores records 1032. For time 1030 the dual-write operation may have been started. For example, as discussed in connection with FIGs. 5, 7, and 9 a dual write operation to replicate incremental data may have been initiated. Accordingly, the new record is also present in records 1034 of the DB clustering system. The new record, however, is not available in the records 1036 of clone DB 406 because the new record was a modification after cloning and before the dual-write operation.

[0186] In a fourth time 1040, there are no changes in the records 1042 in online DB 404. However, records 1044 in DB clustering system 420 have been updated to include missing records that were captured in the cloning operation but had not been captured in the initialization. As shown in FIG. 10, some of the data is missing from the initialization but available through clone DB 406. In time 1040, as further discussed in connection with FIGs. 5, 7, and 9, data from clone DB 406 may be exported to data files that get imported into DB clustering system. Thus, records 1044 capture the missing data from records 1046.

[0187] In a fifth time 1050, there are no changes in the records in 1054 of DB clustering system 420 and there are no changes on the records 1056 of clone DB 406. However, there are changes that are changes in records 1052 of online DB 404. These changes were not replicated into DB clustering system 420. This may occur when the dual write operation has been turned off (e.g., after running a successful verification) but new data modifications applied to online DB 404. As discussed in connection with FIGs. 6-7, DM system 320 may stop dual-writing operations after completing importing and verifying data transfers. However, as online DB 404 is a live database that may have multiple records changes, even within a small time-frame, records may have changed (even if during a small period of time).

[0188] In sixth time 1060, records 1062 in online DB 404 and records 1064 in DB clustering system 420 remain the same. However, records 1066 in clone DB 406 have been updated and replicate records 1062. As discussed in connection with FIG. 6, DM system 320 may perform a re-run catch up for a predetermined time window. Thus, in time 1060 clone DB 406 may include refreshed data from online DB 404. Although FIG. 10 shows the updated records in the same memory space in clone DB 406, in some embodiments the space may be different. For example, clone DB 406 may refer to multiple databases, like local databases 325 or databases 380 and each clone may be written in a different space.

[0189] In seventh time 1070, records 1072 in online DB 404 and records 1076 in clone DB 406 remain the same. However, records 1074 in DB clustering system

420 have been updated to include the missing record after the turn off of the dual- write. Similar to the process used in time 1050, data in clone DB 406 may get replicated into DB clustering system 420.

[0190] FIG. 11 is a timing diagram of an exemplary sequence 1100 for database migration, consistent with disclosed embodiments. As shown in FIG. 11 , sequence 1100 may involve multiple elements of system 400, including online DB 404, DB clustering system 420, and clone DB 406. However, other elements of system 400 may be involved in sequence 1100. For example, while not shown, controller 323 may perform one or more operations in sequence 1100. Further, other of the disclosed systems and elements may perform sequence 1100. For example, in some embodiments system 300 may perform sequence 1100. In such embodiments, DM system 320, databases 380, and/or online resources 340 may perform one or more operations of sequence 1100.

[0191] In step 1102, online DB 404 may communicate with DB clustering system 420 to apply a schema and/or test double write operations. In some embodiments, the communication between online DB 404 and DB clustering system 420 may happen through controller 323 in DM system 320. For example, as discussed in connection with FIG. 7, DM system 320 may perform the operations for getting a database schema from online DB 404, applying the schema, and turn on double write to the DB clustering system 420.

[0192] In step 1104, online DB 404 may communicate with clone DB 406 to generate a clone. In some embodiments, step 1104 may execute only if the dualwriting operation test in step 1102 is successful. Further, in some embodiments, as those disclosed in connection with FIGs. 5 and 7, DM system 320 may coordinate the generation of the clone database. [0193] In step 1106, files in clone DB 406 may be exported. For example, as described in connection with FIGs. 4-5, export tool 412 in controller 323 may generate export files, which may include SQL, CSV, or other type of files. In step 1108, clone DB 406 may communicate with DB clustering system 420 to load the exported data files. For example, as discussed in connection with FIGs. 4-5, data loader tool 414 may load data exported in step 1106 to DB clustering system 420.

[0194] In step 1110, clone DB 406 and DB clustering system 420 may communicate to perform a new verification of consistency, similar to that of steps 1110 and 1118.

[0195] In step 1112, online DB 404 and DB clustering system 420 may communicate to have dual writing operation. As discussed in connection with FIGs. 5 and 7, DM system 320 may initiate operations dual writing from online DB 404 to DB clustering system 420 and capture incremental data.

[0196] In step 1114, online DB 404 and clone DB 406 may communicate to generate a new clone. In some embodiments, the second clone DB may be the same as the first clone database from step 1104. In other embodiments, however, the second cloned DB may be simply part of the clone DB 406 architecture. For example, clone DB 406 may be implemented with local databases 325. In such embodiments, the second clone databases may be different instances in local databases 325.

[0197] In step 1116, clone DB 406 and DB clustering 420 may communicate to identify and load catch-up data. For example, as discussed in connection with FIGs. 5 and 9, clone DB 406 and DB clustering system 420 may identify delta data to the DB clustering system 420 from the last double write time. This data delta or catch-up data may be loaded to the DB clustering system 420. [0198] In step 1118, clone DB 406 and DB clustering system 420 may communicate to verify data consistency. For example, like in step 1110, and as further discussed in connection with FIG. 8, clone DB 406 and DB clustering system 420 may communicate to determine whether the data is consistent between the different databases.

[0199] In step 1120, clone DB 406 and DB clustering system 420 may communicate to insert certain records in DB clustering system 420 to complete the catch up. For example, as further discussed in connection with FIGs. 5 and 7, DB clustering system 420 may be synchronized with the latest records in the second clone DB 406 using by inserting into the table selected records that had not been ported to the DB clustering system 420 (e.g., where the table indicates NULL values).

[0200] In step 1122, online DB 404 may communicate again with clone DB 406 to provide a limited window replication. For example, as further discussed in connection with FIG. 6, in some embodiments after the verification process and cutoff of the dual writing operation, a new timed verification may be executed with a pre-determined time window. The time window may, for example, be based on a specific number of minutes. In step 1124, clone DB 406 and DB clustering system 420 may communicate to perform a new verification of consistency, similar to that of steps 1110 and 1118.

[0201] In step 1126, online DB 404 and DB clustering system 420 may communicate to switch traffic. As discussed in connection with FIGs. 6 and 7, DM system 320 may coordinate addresses, server tables, and/or API processors to route data request from/to the online DB 404 to DB clustering system 420. In such embodiments, online DB 404 and DB clustering system 420may exchange addresses and/or other information for the traffic handoff.

[0202] Another aspect of the disclosure is directed to a non-transitory computer-readable medium storing instructions that, when executed, cause one or more processors to perform the methods, as discussed above. The computer- readable medium may include volatile or non-volatile, magnetic, semiconductor, tape, optical, removable, non-removable, or other types of computer-readable medium or computer-readable storage devices. For example, the computer-readable medium may be the storage unit or the memory module having the computer instructions stored thereon, as disclosed. In some embodiments, the computer- readable medium may be a disc or a flash drive having the computer instructions stored thereon.

[0203] It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed system and related methods. Other embodiments will be apparent to those skilled in the art from consideration of the specification and practice of the disclosed system and related methods. It is intended that the specification and examples be considered as exemplary only, with a true scope being indicated by the following claims and their equivalents.

[0204] While the present disclosure has been shown and described with reference to particular embodiments thereof, it will be understood that the present disclosure can be practiced, without modification, in other environments. The foregoing description has been presented for purposes of illustration. It is not exhaustive and is not limited to the precise forms or embodiments disclosed. Modifications and adaptations will be apparent to those skilled in the art from consideration of the specification and practice of the disclosed embodiments. Additionally, although aspects of the disclosed embodiments are described as being stored in memory, one skilled in the art will appreciate that these aspects can also be stored on other types of computer readable media, such as secondary storage devices, for example, hard disks or CD ROM, or other forms of RAM or ROM, USB media, DVD, Blu-ray, or other optical drive media.

[0205] Computer programs based on the written description and disclosed methods are within the skill of an experienced developer. Various programs or program modules can be created using any of the techniques known to one skilled in the art or can be designed in connection with existing software. For example, program sections or program modules can be designed in or by means of .Net Framework, .Net Compact Framework (and related languages, such as Visual Basic, C, etc.), Java, C++, Objective-C, HTML, HTML/AJAX combinations, XML, or HTML with included Java applets.

[0206] Moreover, while illustrative embodiments have been described herein, the scope of any and all embodiments having equivalent elements, modifications, omissions, combinations (e.g., of aspects across various embodiments), adaptations and/or alterations as would be appreciated by those skilled in the art based on the present disclosure. The limitations in the claims are to be interpreted broadly based on the language employed in the claims and not limited to examples described in the present specification or during the prosecution of the application. The examples are to be construed as non-exclusive. Furthermore, the steps of the disclosed methods may be modified in any manner, including by reordering steps and/or inserting or deleting steps. It is intended, therefore, that the specification and examples be considered as illustrative only, with a true scope and spirit being indicated by the following claims and their full scope of equivalents. [0207] Thus, the foregoing description has been presented for purposes of illustration only. It is not exhaustive and is not limiting to the precise forms or embodiments disclosed. Modifications and adaptations will be apparent to those skilled in the art from consideration of the specification and practice of the disclosed embodiments.

[0208] The claims are to be interpreted broadly based on the language employed in the claims and not limited to examples described in the present specification, which examples are to be construed as non-exclusive. Further, the steps of the disclosed methods may be modified in any manner, including by reordering steps and/or inserting or deleting steps.