TECHNICAL FIELD

The present disclosure relates generally to methods for creating workflow instructions; and more specifically, to methods for creating workflow instructions for assembling of products from demonstrative content.

BACKGROUND

Generally, workflow instructions enable execution of tasks such as assembling systems, operating systems, and so forth. Notably, such tasks include assembly of one or more devices/components/products that are designed to operate in conjunction to produce a result. Specifically, the workflow instructions provide a step-by-step guide for performing the task to assemble a product including the one or more devices. Moreover, the workflow instructions are recorded for various configurations of the product. Furthermore, several steps of the workflow instructions may be altered depending on the configuration of the product to provide unambiguous instructions.

However, recording workflow instructions, that provide accurate guide of the task, is challenging owing to multiple possible configurations of the product. Moreover, the product may have identical configurations wherein the employed one or more devices have different identification number (such as model number, device identification code, and so forth). Consequently, the workflow instructions must provide a clear guide to a user so as to eliminate confusion and speculation while performing steps for assembling of the product.

Conventionally, the workflow instructions provide a guide through the plurality of configurations of the product by way of including a plurality of branch points. Typically, the plurality of branch points adds distinguishing steps for auxiliary configurations of the product, wherein the auxiliary configurations are modifications to the basic configuration of the product. However, such branch points make the workflow instructions difficult to understand by the user. Additionally, such workflow instructions comprising the branch points may lead to misinterpretations of the workflow instructions. Further users might assemble the same product with different ways. These "individual" ways might not be documented at all or might be documented partially. Recently, the workflow instructions for assembling the product are provided by a skilled person manually recording written instructions. However, such workflow instructions may not provide detailed information pertaining to the one or more devices or operations involved in each step while performing the task. Furthermore, the extracted workflow instructions are not optimized and may comprise errors. More recently, wearable cognitive assistance systems are employed to identify the one or more devices correctly. However, such wearable cognitive assistance systems rely majorly on pre-defined scripts (e.g. description of work steps, sequence of completion status) or pre-defined object positions in a pre-defined working environment. Consequently, such wearable cognitive assistance systems require rigorous manual labour in defining each new step in the workflow instructions. Furthermore, a labelled training dataset is typically required to generate the workflow instructions by employing the demonstrative content if object recognition is used. If the system relies on markers, labelled training dataset is not necessarily needed. Subsequently, the demonstrative content or the wearable cognitive assistance systems lacks competence in generating the set of instructions owing to massive labour cost and increased time and effort. Therefore, in light of the foregoing discussion, there exists a need to overcome the aforementioned drawbacks associated with conventional methods of creating the workflow instructions.

SUMMARY The present disclosure seeks to provide a method of creating workflow instructions for assembling of a product. The present disclosure seeks to provide a solution to the existing problem of tedious and time-consuming methods for creating workflow instructions. An aim of the present disclosure is to provide a solution that overcomes at least partially the problems encountered in prior art and provides a method for unsupervised creation of the workflow instructions from the demonstrative content.

In one aspect, an embodiment of the present disclosure provides a method of creating workflow instructions for assembling of a product, the method comprising obtaining a demonstrative content depicting the assembling of the product;

identifying, in the demonstrative content, a functioning member assembling the product, wherein the functioning member executes operations for assembling of the product;

identifying, in the demonstrative content, a plurality of object candidates;

tracking the operations executed by the functioning member, on at least one of the plurality of object candidates to determine a first object and a second object, from the plurality of object candidates, wherein the first object and the second object are being assembled to form the product;

associating a timeline with the tracked operations;

organizing the tracked operations based on the timeline; creating the workflow instructions based upon the organized tracked operations and the determined first and second objects.

Embodiments of the present disclosure substantially eliminate or at least partially address the aforementioned problems in the prior art and enables creation of the workflow instructions without using labelled data that eliminates human intervention and time consumed for creation thereof.

Additional aspects, advantages, features and objects of the present disclosure would be made apparent from the drawings and the detailed description of the illustrative embodiments construed in conjunction with the appended claims that follow.

It will be appreciated that features of the present disclosure are susceptible to being combined in various combinations without departing from the scope of the present disclosure as defined by the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The summary above, as well as the following detailed description of illustrative embodiments, is better understood when read in conjunction with the appended drawings. For the purpose of illustrating the present disclosure, exemplary constructions of the disclosure are shown in the drawings. However, the present disclosure is not limited to specific methods and instrumentalities disclosed herein. Moreover, those in the art will understand that the drawings are not to scale. Wherever possible, like elements have been indicated by identical numbers. Embodiments of the present disclosure will now be described, by way of example only, with reference to the following diagrams wherein : FIGs. 1A, IB, 1C, ID and IE are example views of image frames of demonstrative content depicting assembling of a product, in accordance with an embodiment of the present disclosure;

FIGs 2A and 2B collectively are a flow chart depicting steps of a method for creating workflow instructions for assembling of a product, in accordance with an embodiment of the present disclosure;

FIGs. 3 and 4 illustrate exemplary implementations of the method for creating workflow instructions for assembling of a product, in accordance with an embodiment of the present disclosure. In the accompanying drawings, an underlined number is employed to represent an item over which the underlined number is positioned or an item to which the underlined number is adjacent. A non-underlined number relates to an item identified by a line linking the non-underlined number to the item. When a number is non-underlined and accompanied by an associated arrow, the non-underlined number is used to identify a general item at which the arrow is pointing.

DETAILED DESCRIPTION OF EMBODIMENTS

The following detailed description illustrates embodiments of the present disclosure and ways in which they can be implemented. Although some modes of carrying out the present disclosure have been disclosed, those skilled in the art would recognize that other embodiments for carrying out or practicing the present disclosure are also possible.

In one aspect, an embodiment of the present disclosure provides a method of creating workflow instructions for assembling of a product, the method comprising obtaining a demonstrative content depicting the assembling of the product; identifying, in the demonstrative content, a functioning member assembling the product, wherein the functioning member executes operations for assembling of the product; identifying, in the demonstrative content, a plurality of object candidates; tracking the operations executed by the functioning member, on at least one of the plurality of object candidates to determine a first object and a second object, from the plurality of object candidates, wherein the first object and the second object are being assembled to form the product; associating a timeline with the tracked operations; organizing the tracked operations based on the timeline; creating the workflow instructions based upon the organized tracked operations and the determined first and second objects. The aforesaid method of creating workflow instructions for assembling of a product provides a method for generating workflow instructions from a demonstrative content. Beneficially, the method described herein enables unsupervised generation of the workflow instructions from the demonstrative content. The method includes analysing the demonstrative content to determine object candidates and steps performed on the object candidates while executing a task for assembling the product. Typically, each of the determined steps from the demonstrative content describes a semantic workflow model for executing the task. Thus the semantic workflow model describes the determined and ordered steps for executing the task. The determined and ordered steps are obtained from the demonstrated content. The method described herein employs techniques such as object detection and tracking for unsupervised determination of object candidates and steps performed on the object candidates. Furthermore, such techniques substantially eliminate the need for manual labour in defining the object candidates using pre-labelled datasets and classifiers, thereby making the method less tedious, economical, user-friendly and less time- consuming. Moreover, the semantic workflow model describes a semantic information relating to each step from the demonstrative content. Subsequently, features associated with the steps (such as number and order of steps, initiation step, completion step, activity involved in each step, and so forth) are described semantically, thereby removing confusion and ambiguity owing to multiple configuration of the object candidates while assembling the product.

In an additional or further embodiment the method further comprises updating the created workflow instructions based upon the organized tracked operations and the determined first and second objects to include a second set of organized tracked operations and a determined second set of objects. The second set of objects might comprise a third object and a fourth object. As an example some of the steps of the method might be repeated to produce more complex workflow instructions. In one embodiment steps of tracking the operations, associating a time line, organizing the tracked operations and creating the workflow are repeated for number of times to update workflow which was generated in the previous round.

The method further employs semantic information for error detection in assembling of the product, progress control in assembling of the product and optimizing the workflow for assembling of the product. The method enables generation of large variance in workflow instructions thereby enabling creation of customized workflow instruction, wherein the workflow instructions provide detailed information associated with the object candidates thereby eliminating any misinterpretation of information or instruction. Additionally, the method described herein may be employed for creation of workflow instructions for multiple operations such as maintenance processes, surgical processes, sports training, gaming, interaction between humans and machines, clinical trials, medical trainings, interactive computer-generated programs, and so forth.

It will be appreciated that the aforesaid method is not limited to creating workflow instructions for assembling of a product. The method may be employed to create workflow instructions for several operations performed on the product such as disassembling of the product, maintenance of the product, error detection of the product, and so forth. Moreover, the method may be employed to create workflow instructions for multiple products. It will be appreciated that the operations performed on the multiple products may or may not be similar. Moreover, the method may be employed to create workflow instructions for various configurations of a singular product.

Throughout the present disclosure, the term "product" refers to an article comprising a plurality of objects that are assembled to obtain the article. In an example, the plurality of objects may be detachably attached to each other to form the product. In another example, the plurality of objects may be operatively coupled to form the product.

Examples of the product include, but are not limited to, electrical instruments (such as motors, electrical gears, generators, heaters, fans, light panels, solar panels, vacuum cleaners, measuring instruments, and so forth), automobile instruments (such as gears, pallet trucks, dent pulling systems, pneumatic transfer systems, testing equipment, maintenance equipment, and so forth), mechanical instruments (such as drills, chain saw, vices, pullers, and so forth), electronic instruments (such as computers, printers, mobile devices, communication devices, amplifiers, washing machines, cameras, refrigerators, microwaves, air- conditioner, and so forth). Throughout the present disclosure, the term "workflow instructions" refers to a set of instructions that describes operational process for execution of a task. Specifically, the operational process is defined by way of a series of operations (namely, steps) that are performed sequentially or parallelly. Moreover, the workflow instructions define the steps for execution of the task such that the task passes from initiation to completion. In other words, the workflow instructions provide detailed information relating to steps, that when performed, enables completion of the task from the initiation. Pursuant to embodiments of the present disclosure, the workflow instructions refer to steps that describe operational processes associated with assembling of the product. Specifically, the workflow instructions describe the process for assembling of the product by employing one or more constituent parts of the product. It will be appreciated that the workflow instructions define the operational processes for assembling of the product depending on a configuration of the product. In an embodiment, the workflow instructions may be provided in form of a textual manual. In another embodiment, the workflow instructions may be provided in form of a pictorial manual, wherein the steps of the workflow instructions are representation in form of pictures.

Furthermore, the method for creating workflow instructions for assembling of the product comprises obtaining the demonstrative content depicting the assembling of the product. Throughout the present disclosure, the term "demonstrative content" refers to an illustrative information that exhibits or explains a task. Specifically, the demonstrative content is employed for creating workflow instructions for completion of the task exhibited therein. Moreover, the demonstrative content is recorded and stored digitally. In an example, the demonstrative content may be recorded using a camera and further stored in a digital library of a computer executing the aforesaid method of creating workflow instructions. In accordance with an embodiment of the present disclosure, the demonstrative content refers to an illustrative information that exhibits assembling of the product. Subsequently, workflow instructions for assembling of the product is created by employing the demonstrative content. Optionally, the demonstrative content comprises at least one of: a set of images, a video. Specifically, demonstrative content exhibits the task of assembling of the product by way of the set of images and/or the video. In an embodiment, the demonstrative content may comprise the video, wherein the video describes the steps for assembling of the product by employing moving visual image frames. In another embodiment, the demonstrative content may comprise of the set of images, wherein each image in the set of images depicts a step for assembling of the product. In an embodiment, the video may be converted to a plurality of image frames for creating workflow instructions therefrom. More optionally, the demonstrative content may further comprise audio therein. In an example, the demonstrative content may describe workflow instructions for assembling of a fan using a video. In another example, the demonstrative content may describe workflow instructions for assembling of a plastic toy (namely, Lego toy) using a set of images. The method for creating workflow instructions comprise identifying the functioning member assembling the product in the demonstrative content, wherein the functioning member executes operations for assembling of the product. Throughout the present disclosure, the term "functioning member" refers to an entity performing the steps of the task depicted in the demonstrative content. Pursuant to embodiments of the present disclosure, the functioning member is operable to perform the steps of assembling the product. It will be appreciated that the functioning member executes the steps for assembling the product such that each step is depicted in the demonstrative content. Furthermore, the functioning member may be implemented as a human, arms of a human, a robot, arms of a robot, and so forth. In an example, the functioning member may perform steps for assembling of a fan. In another example, the functioning member may perform steps for assembling of a toy helicopter. It will be appreciated that the functioning member performs the steps for assembling of the product in the demonstrative content. Subsequently, identifying the functioning member in the demonstrative content enables tracking of the steps performed in the demonstrative content for assembling of the product. The method comprises identifying the plurality of object candidates in the demonstrative content. Throughout the present disclosure, the term "object candidate" refers to articles, items, areas or regions in the demonstrative content that may or may not constitute the product. Specifically, the plurality of object candidates are identified to identify therefrom objects that constitute the product. More specifically, each of the plurality of object candidates is a potential possibility of being an object constituting the product. In an example, the product may be a toy helicopter. In such example, the demonstrative content depicting the assembling of the toy helicopter may comprise articles such as a table, a tumbler, a rotor blade, a helicopter body, screws, a plant and so forth. Consequently, such articles are identified as the plurality of object candidates. Notably, the objects constituting the toy helicopter may only include the rotor blade, the helicopter body and the screws.

Optionally, identifying the plurality of object candidates in the demonstrative content comprises: identifying, from the demonstrative content, at least one object region;

identifying, from the at least one object region, the plurality of object candidates. The term "object region" refer to an area in the demonstrative content, wherein the area has definable characteristics. Typically, the at least one object region is identified by partitioning the demonstrative content into regions such that the regions are similar according to a set of predefined criteria. Such predefined criteria for identification of the plurality of object regions may be intensity, colour, texture, and so forth. It will be appreciated that the at least one object region may not have fixed and/or continuous boundaries. In an embodiment, the at least one object region may be identified by extracting edges and contours from an image frame in the demonstrative content. Beneficially, the at least one object region enhances comprehension of the at least one object candidate from the demonstrative content and further increases efficiency by improving comprehension and reducing the computational time.

Furthermore, the plurality of object candidates are identified from the at least one object region. Typically, features associated with the at least one object region are extracted to identify the at least one object candidate by employing a plurality of techniques. In an embodiment, Scale-Invariant feature transform technique may be employed for extraction of the features associated with the plurality of object candidates.

More optionally, identifying the plurality of object candidates comprises using at least one of: edge detection technique, contour extraction technique, feature extraction technique. Typically, the techniques are employed for extraction of the features associated with the plurality of object candidates, so as to identify the plurality of object candidates from the at least one object region. Furthermore, such techniques for identifying the plurality of object candidates subdivide or partition the at least one object region into multiple meaningful candidates or set of pixel candidates with respect to a particular application. Such candidates or the set of pixel candidates are identified as the plurality of object candidates based on measurements taken from the demonstrative content. Additionally, the measurement factors might be grey level, colour, texture, dept, and motion. Optionally, the plurality of object candidates identified from the at least one object region may collectively cover the entire demonstrative content.

In an embodiment, the edge detection technique may be employed for identifying the plurality of object candidates. Typically, an edge in an image is a significant local change in the image intensity. Furthermore, the local change may be identified due to a discontinuity in the image intensity or due to a first derivative of the image intensity. Subsequently, the 'edge detection technique' is operable to identify the plurality of object candidates by identifying the edges of the at least one potential object candidate comprised in at least one object region. It will be appreciated that the edge detection techniques employ edge detectors based on features (such as image gradient, image intensity, and so forth) to find boundaries of the plurality of object candidates so as to determine edges of the plurality of object candidates. Examples of model employed for edge detection technique include, but are not limited to, Sobel, Prewitt, Roberts, Canny, LoG, Expectation-Maximization algorithm, OSTU algorithm, and Genetic algorithm.

In an embodiment, the contour extraction technique may be employed for identifying the plurality of object candidates. Typically, the contours depict curvatures and junctions in an image thereby providing crucial shape information. Pursuant to embodiment of the present disclosure, the 'contour extraction technique' depicts a shape of the plurality of object candidates thereby enabling identification thereof. Optionally, the plurality of object candidates is identified by the contour extraction technique. Typically, contour points are extracted from a potential object candidate and the contour points are with contour points of a reference object candidate, wherein the reference object candidate may be described previously in an object dictionary, as discussed in detail later herein. Beneficially, the contour extraction technique enables simplified identification of the plurality of object candidates owing to smooth change in contour points within the potential object candidate. Examples of algorithm employed for contour extraction technique may include, but are not limited to, Square Tracing algorithm, Moore-Neighbour Algorithm, Radial Sweep algorithm, Theo Pavlidis' algorithm, and Curvature scale space algorithm. Optionally, the edge detection technique may be performed prior to the contour extraction technique for identifying the plurality of object candidates.

In an embodiment, the feature extraction technique may be employed for identifying the plurality of object candidates. Typically, the 'feature extraction technique' concerns with extracting various attributes (namely, features) of a potential object candidate for identification thereof. Such features include colour, texture, shape, and so forth. Examples of model employed for feature extraction technique include, but are not limited to, Harris interest point detector, Scale Invariant Feature Transform (SIFT) method, Speed-up Robust Feature (SURF) method, Histogram of Gradient (HOG) method, Random Sample consensus (RANSAC) method, Deep/Convolutional Neural Networks method. In an example, the SIFT method may be employed for identifying the plurality of object candidates. In such case, features of the potential object candidate is compared with features of a reference object candidate. It will be appreciated that the reference object candidate may be described previously in an object dictionary, as discussed in detail later herein. In an embodiment, the feature extraction technique may be performed independently for identifying the plurality of object candidates. In another embodiment, the feature extraction technique may be performed in conjunction with at least one of the edge detection technique and the contour extraction technique. Optionally, the plurality of object candidates is identified by employing machine learning algorithms. Throughout the present disclosure, the term " machine learning algorithms" refer to a category of algorithms employed by a software application to perform a task. Typically, the machine learning algorithms allows the software application to become more accurate in predicting outcomes and/or performing tasks, without being explicitly programmed. It will be appreciated that the software application refers to a set of instructions that are executable by a computer or other digital system. The software application configures the computer or the digital system to perform the method described herein. Specifically, the machine learning algorithms are employed to artificially train the software application to enable them to automatically identify the plurality of object candidates from the demonstrative content and improve accuracy from experience, without being explicitly programmed.

More optionally, the machine learning algorithms are trained using unsupervised dataset. The 'unsupervised dataseV are unlabelled training datasets or unstructured training information. Specifically, the unlabelled training datasets are neither classified nor labelled. More specifically, the machine learning algorithms employing unsupervised datasets are not trained with a desired output. Alternately, the unsupervised machine learning algorithms analyse the unlabelled training datasets provided for training and further interprets the training datasets to sort the training datasets without using predefined labels. Furthermore, optionally, examples for algorithms employed for unsupervised training of the machine learning algorithms, include, but not limited to: k-means clustering, k-NN, Dimensionality Reduction, Singular Value Decomposition, Distribution models, Hierarchical clustering, Mixture models, Principal Component Analysis, and autoencoders. Optionally, the method further comprises filtering out weak object regions. Specifically, the 'weak object regions' refer to bounded regions that have low probability of containing the plurality of object candidates. In an embodiment, the weak object regions may refer to a region that covers more than a threshold area of a given image frame in the demonstrative content. In an example, the threshold area is two-third of the given image frame in the demonstrative content. In another embodiment, the weak object regions may refer to regions having lesser number of characteristic attributes (such as edges, contours, and features) than a specified threshold. In a first example, the specified threshold may be four SIFT feature points. In a second example, the specified threshold may be ten SIFT feature points. Beneficially, filtering the weak object regions enhances the accuracy and reliability in identifying the plurality of object candidates. Optionally, the method further comprises filtering a noise from the demonstrative content prior to identifying the plurality of object candidates. Typically, the term "noise" refers to a disturbance in the demonstrative content. Thus, the noise becomes a limiting factor in a quality of the demonstrative content. It will be appreciated that the noise in the demonstrative content may be an unwanted sound (namely, audio disturbance), an unwanted pixel pattern in the demonstrative content, and/or an unwanted electrical disturbance. Beneficially, filtering the noise eliminates a possibility of misinterpretation of information in the demonstrative content. Moreover, filtering the noise facilitates extraction of useful information by cancelling relatively unintelligible information. Furthermore, pre-processing the demonstrative content for filtering the noise facilitates coherent processing of the demonstrative content in identifying the plurality of object candidates owing to enhanced quality thereof. More optionally, filtering the noise enhances the quality of the demonstrative content by improving factors such as amplitude, frequency, pixel, image grains, contrast, resolution, brightness, sharpness, blurriness, audio, and so forth. Moreover, improving such factors escalates the efficiency of the method thereby removing risks associated with errors due to fallacious identification of the plurality of object candidates.

The method comprises tracking the operations executed by the functioning member on the at least one of the plurality of object candidates to determine the first object and the second object, from the plurality of object candidates, wherein the first object and the second object are being assembled to form the product. In other words, from the plurality of object candidates, the object candidates employed by the functioning member to assemble the product are identified as objects. Specifically, the 'first object and the 'second object refer to constituent parts of the product. It will be appreciated that the product may have multiple constituent parts (namely, more than two constituent parts) and the terms "first", "second" and like terms herein do not denote any specific role or order or importance, but rather are used to distinguish one object from another. In other words, the first object and the second object can act as a primary object and a secondary object, respectively, at a given point of time, and can act as a secondary object and a primary object, respectively, at another point of time.

Furthermore, the operations executed by the functioning member, on the at least one object candidate are tracked. Specifically, the movements of the functioning member to execute operations are traced (namely, tracked) to determine the first object and the second object. It will be appreciated that the functioning member executes the operations for assembling the first object and the second object to assemble the product. In an example, the first object may be placed inside the second object to form the product. In such case, the tracked operation executed by the functioning member may be the action of placing the second object inside the first object.

Optionally, the demonstrative content may comprise a video, wherein the video depicts operations for assembling of the product. It will be appreciated that a functioning member (implemented as hands of a human) and plurality of object candidates are identified from the demonstrative content. Furthermore, the operations executed by the functioning member are tracked by recognizing an operational state for each of the operations executed. Typically, the operational state refers to an interaction between the functioning member and the plurality of object candidates for assembling. In an example, the operational state for each of the operations executed may be defined by way of an idle operational state, a plugging operational state, a searching operational state, and a pose changing operational state. Typically, the 'idle operational state' refers to a status of the operation, wherein no functioning member interacts with the plurality of object candidates Furthermore, the 'plugging operational state' refers to a status of operation, wherein an action of attaching a given object candidate to another given object candidate, from the plurality of object candidates, is performed for assembling of the product through interaction of the functioning member with the given object candidates. The 'searching operational state' refers to a status of operation, wherein a view to look for the plurality of object candidates is changed. Moreover, the 'pose changing operational state' refers to a status of operation, wherein a pose of the product for assembling is changed. It will be appreciated that each of the operations executed contains an action that causes change in the product, a change in the pose of the product, or a change in the operational state of the assembly process. Furthermore, creation of the workflow instructions is implemented by tracking movements of the functioning member and relative positions of the functioning member with the plurality of object candidates and/or the product for assembling . Optionally, the object candidates associated with each of the hands of the human is identified and stored . More optionally, the operational state for each of the operations executed are represented by way of Boolean expressions.

In an embodiment, Boolean expression is used for representing the operational state for each of the operations executed by the functioning member. In such case, a Boolean value for the product for assembling is denoted as 7 . Furthermore, a Boolean value for the functioning member is denoted as 'β' and a Boolean value for the operational state is denoted as 'C Subsequently, a Boolean expression for the idle operational state (namely, Ci) is given by:

A A B → C ;

A Boolean expression for the plugging operational state (namely, C2) is given by :

A A B→ C ₂ ;

A Boolean expression for the searching operational state (namely, C3) is given by :

D→ C ₃ ; Wherein a Boolean value 'D' represents a status of operation when the product for assembling is not in the view. Additionally, in such case, the functioning member is not in the view.

A Boolean expression for the pose changing operational state (namely, C ₄ ) is given by: A A B→C ₄ ;

Optionally, the operations executed by the functioning member on the at least one object candidate is tracked using multiple instance learning (MIL) technique. Specifically, the MIL technique is a framework wherein training examples are provided in form of labelled bags rather than labelled instances. More specifically, the MIL technique tracks the plurality of object candidates by selecting a set of characteristic attributes (such as edges, contours, and features). In an example, the MIL technique selects a set of features as the characteristic attributes, from a plurality of features associated with a given object candidate from the plurality of object candidates for tracking. Typically, the plurality of features consists of computationally efficient features. Furthermore, each of the plurality of features is used for construction of weak classifier. Moreover, the set of features with high characteristic power is used to form strong classifier. Typically, the set of features from the plurality of features associated with the given object candidate are selected in a small amount of time so as to ensure fast and accurate tracking of the given object candidate. Optionally, the method further comprises creating an object dictionary comprising information related to the first object and the second object. The term "object dictionary" refers to a database comprising a list of multiple objects that further gives information relating to each of the multiple objects. Specifically, examples of information relating to each of the multiple objects given by the object dictionary may include, but are not limited to, meaning, usage, language, at least one equivalent object candidate in a same language, at least one equivalent object candidate in different language, characteristic attributes (such as edge points, feature points, contour points), origin, pronunciation, and identification characteristics. More optionally, the list of multiple objects may be arranged in an alphabetical order. Furthermore, optionally, the object dictionary may be recorded in a specified language. Additionally, the object dictionary may have a plurality of entries associated with a given object from the multiple objects. Each of the plurality of entries comprise different characteristic information relating to the given object. The term 'database' as used herein relates to an organized body of digital information regardless of the manner in which the data or the organized body thereof is represented. Optionally, the database may be hardware, software, firmware and/or any combination thereof. For example, the organized body of related data may be in the form of a table, a map, a grid, a packet, a datagram, a file, a document, a list or in any other form. The database includes any data storage software and systems, such as, for example, a relational database like IBM DB2 and Oracle 9. Optionally, the database may be operable to support relational operations, regardless of whether it enforces strict adherence to the relational model, as understood by those of ordinary skill in the art. Additionally, the database is populated by data elements. Furthermore, the data elements include information relating to the at least one object candidate that may be stored in cells of a database. Optionally, identifying the plurality of object candidates is based upon information in the object dictionary. It will be appreciated that the object dictionary comprises a list containing information relating to the multiple objects. Specifically, the information relating to multiple objects may comprise information relating to the plurality of object candidates. More specifically, a potential object candidate may be identified by comparing characteristic attributes of the reference object candidate with the potential object candidate. Furthermore, information relating to the first object and the second object from the plurality of object candidates is extracted from the object dictionary to create the workflow instructions. In an example, information relating to each of the at least one object candidate may be recorded in the object dictionary. In another example, the object dictionary may not have information relating to the at least one object candidate recorded therein. In such case, the at least one object candidate is recorded in the object dictionary. Optionally, the method comprises detecting an error in the tracked operations performed by the functioning member. In an example, the method for assembling the product detects an error by extracting RGB- Local Binary Pattern (RGB-LBP) features of the assembled product. Moreover, such RGB-LBP features are used for describing an appearance of the assembled product. Furthermore, the assembled product may be matched with an output of the previous operation described in the demonstrative content using the RGB-LBP feature.

Optionally, tracking the operations executed by the functioning member comprises extracting semantic information related to the tracked operations, wherein the workflow instructions are created further based on the extracted semantic information. Typically, the semantic information related to the tracked operations provide a mechanism for assigning semantic labels or captions to the demonstrative content. Moreover, such semantic labels are assigned based on low-level measurable features (such as colour, composition, colour texture, and so forth) and/or accompanying data (such as date of tracked operation, time of tracked operation, presence and distribution of open spaces, and so forth). It will be appreciated that the low-level measurable features and the accompanying data are objectively measured by analysis of the demonstrative content. Furthermore, the semantic information establishes a link between operations executed by the functioning member that provides a meaning for the operations executed. Optionally, the link may be a unidirectional link. Optionally, the semantic information may further include high-level semantic labels, which corresponds to cognition and represents the tracked operations such that the workflow instructions depict customised meaning to a user. More optionally, the object dictionary may comprise pre-existing semantic labels associated with the multiple objects. In an embodiment, similarities may be found between the tracked operations performed on at least one of the plurality of object candidates and the multiple objects in the object dictionary. In such case, the semantic information related to the tracked operations may have been extracted from the object dictionary, as previously applied to acquired objects that are now stored therein. In another embodiment, the semantic information related to the tracked operations may have been extracted from other processes such as manual annotation or manual execution. Furthermore, an association may be maintained between the extracted semantic information, the measurable features and the accompanying data to facilitate automatic extraction of semantic information for new demonstrative content.

Furthermore, optionally, the semantic information may comprise a number of relation types. Examples of the relation types include, but are not limited to, cause, domain, hypernym, location, manner, material, means, modifier, part, possessor, purpose, synonym, time, typical object candidate, and typical subject candidate.

In an example, semantic information relating to each operation executed by the functioning member may be extracted to create the workflow instructions for assembling of the product. It will be appreciated that the operations executed by the functioning member are performed for assembling of the product. In such case, the semantic information may be described by way of a set of two element tuple. Notably, 'tuple' refers to an ordered and unchangeable collection of values. Specifically, the set of two element tuple contains two specified values separated by a comma. Pursuant to embodiments of the present disclosure, the two specified variables may be 'operation type' and 'object name list'. In such case, the operation type defines a given operational state for each of the operations executed by the functioning member. Moreover, the object name list comprises a list of objects involved in performing the given operational state. In an exemplary implementation, the demonstrative content may be a video. It will be appreciated that video is implemented by way of a plurality of image frames. Subsequently, a given object region is identified from the at least one of object region, by extracting edges and contours from a given image frame comprising the given object region. Furthermore, the feature extraction technique is executed on the given object region to extract a first set of features associated with the given object region. It will be appreciated that the given image frame may comprise plurality of object regions. Subsequently, a first set of features associated with each of the at least one of object region is extracted (using a feature extraction technique namely, SIFT) and further weak object regions having low probability of containing the at least one object candidate is filtered out based on the first set of features associated with the plurality of object regions. Furthermore, upon obtaining the first set of features in the given object region having a value higher than a threshold, plurality of object candidates comprised in the given object region is identified. Specifically, the pluralty of object candidates are identified from the given object region upon obtaining the first set of features higher than the threshold value, wherein the threshold value is 10 SIFT features. Furthermore, information related to the identified plurality of object candidates is stored in an object dictionary thereby updating the object dictionary. It will be appreciated that the object dictionary is created when no existing object dictionary is updated. Feature extraction technique for object detection is further conducted in the given object region to recognize the plurality object candidates previously stored in the object dictionary thereby eliminating false identification of the plurality of object candidates. In an example, an identified object candidate is previously stored in the object dictionary. Moreover, the identified object candidate may be represented by way of a bounding box. Consequently, the bounding box of the identified object candidate (namely, reference bounding box), previously stored in the object dictionary is utilised to refine the bounding box of the identified object candidate described upon identification thereof. Furthermore, the refined bounding box (namely, detected bounding box) of the identified object candidate is tracked simultaneous to the process of refining, using MIL tracking technique. It will be appreciated that the detected bounding box is refined based on techniques (namely, feature matching, 2- dimensional transformation, and so forth). The reference bounding box and the detected bounding box is divided into a plurality of sub regions. Additionally, the reference bounding box and the detected bounding box is used for calculating a distance information, wherein the distance information enables selection of a bounding box (from amongst the reference bounding box and the detected bounding box) that proffers greater accuracy and efficiency in identifying a second set of features associated with the identified object candidate. The equation employed for calculating the distance information is given as: aDd{x,y) + (1 -a)Dt{x,y) < θ{α ε [0, 1],β ε [0, 1]) Furthermore, in the above equation,

Dd(x,y) is a normalized distance between a sub region centred at (x,y) in the detected bounding box and a corresponding sub region in the reference bounding box;

Dt(x,y) is a normalized distance for the tracked bounding box obtained from tracking the detected bounding box; and

Θ is a threshold value used for controlling acceptable dissimilarity or distance between the sub regions. Furthermore, the bounding box proffering higher accuracy is added to the object dictionary accompanied with the second set of features associated with the identified object candidates.

In an embodiment, various movements and alterations to assemble the product, owing to the movements, are tracked (namely, traced). Furthermore, tracking such movements throughout the demonstrative content provides additional information for identification of the objects (namely, constituent parts) of the product. It will be appreciated that the plurality of object candidates may be moved throughout the demonstrative content. However, each of the plurality of object candidates may not be moved specifically for assembling the product. Consequently, tracking various movements of the plurality of the object candidates enables identification of movements having a utility in the method for assembling of the product.

In an exemplary embodiment, the demonstrative content may be a video implemented by a plurality of image frames. Furthermore, the method includes analysing the demonstrative content for unsupervised identification of the object candidates. Specifically, various movements and alterations performed in the demonstrative content, for assembling of the product, are tracked to identify the objects. Additionally, information stored in the object dictionary is used for tracking the movements and alterations performed in the demonstrative content. Simultaneously, the functioning member is identified from the demonstrative content. Moreover, a given object region is identified from the at least one object regions, by extracting edges and contours from a given image frame comprising the given object region. Furthermore, techniques (such as edge detection technique, contour extraction technique, feature extraction technique, and so forth) are executed on the given object region to identify object candidates from each of the plurality of image frames in the demonstrative content. Additionally, an information stored in the object dictionary and operations executed by the functioning member are used for identifying the plurality of object candidates. Subsequently, the operations executed by the functioning member and the movements and alterations performed in the demonstrative content are analysed to identify the first object and the second object (namely, constituent parts) of the product. Moreover, information related to the identified first and second object is stored in the object dictionary. It will be appreciated that tracking the operations performed by the functioning member and relative position of the functioning member to the objects (namely, the first object and the second object) enables recognition of operations executed at each step. Typically, each step contains a series of actions that causes a change in the product (such as pose change, state change). In an example, the change in product may be determined using Boolean expressions. Subsequently, the workflow for assembling of the product is determined. Furthermore, after tracking the operations executed by the functioning member, the semantic information related to the tracked operations are extracted. Moreover, the workflow instructions are created based on the extracted semantic information.

The method comprises associating the timeline with the tracked operations. Specifically, the timeline represents a period of time, wherein the tracked operations executed by the functioning member are performed in the period of time. Moreover, associating the timeline with the tracked operations enables identifying a systematic flow of the operations executed by the functioning member. Furthermore, optionally, the demonstrative content may be implemented by way of a plurality of images, wherein the timeline may be associated with each of plurality of image frames in the demonstrative content, wherein the plurality of image frames comprises the tracked operations. In an embodiment, a timestamp may be associated with each of the tracked operation in the demonstrative content. In such case, the timestamps describe the timeline for each of the tracked operation.

Furthermore, the method comprises organizing the tracked operations based on the timeline. Specifically, the tracked operations executed by the functioning member is organized so as to create a course of continuous and systematic instructions for performing the operations. Moreover, organising the tracked operations based on the associated timeline enables structuring a first operation that may be executed simultaneous to a second operation. Consequently, organizing the tracked operations comprised in each of the plurality of image frame in the demonstrative content, based on the timeline, creates a structure for workflow instructions for assembling of the product. In an embodiment, each of the image frame may be arranged based on an associated timestamp.

The method comprises creating the workflow instructions based upon the organized tracked operations and the determined first and second objects. Specifically, the tracked operations executed by the functioning member on the first and the second object is described so as to generate the workflow instructions. In an embodiment, the tracked operations may be described by way of textual information to create the workflow instructions. In another embodiment, the tracked operations may be described by way of pictorial information to create the workflow instructions. In yet another embodiment, the tracked operations may be described by way of pictorial information, wherein the pictorial information is further explained using textual information to create the workflow instructions.

Optionally, the workflow instructions may be described by way of the pictorial information for a demonstrative content comprising a video. In such case, the pictorial information may be extracted from the demonstrative content. In an example, one or more image frames of the demonstrative content comprising the video may be employed for creating the pictorial workflow information. In such case, regions containing the functioning member is detected. Furthermore, such regions containing the functioning member is removed from the one or more image frames. Examples of techniques employed for removing the regions containing the functioning member are Oxford Hands, EgoHands, EgoYouTube-Hands (EYTH), HandOverFace (HOF), EgoHands +, GeorgiaTech Egocentric Activity (GTEA), and so forth. Beneficially, removing the regions containing the functioning member enables creation of intelligible workflow instructions described by way of pictorial information.

In an example, the product may be a wardrobe. In such case, the functioning member may be a human arm, wherein the human arm executes operations for assembling the wardrobe. Furthermore, a video is obtained depicting the assembling of the wardrobe. It will be appreciated that the video is implemented by way of a plurality of image frames. Moreover, the video depicts operations for assembling the wardrobe depicted in one or more image frames of the plurality of the image frames. The video is tracked to determine at least one object candidate using SIFT technique, wherein the at least one object candidate is employed for assembling the wardrobe. The at least one object candidate employed for assembling the wardrobe includes a floor panel, a head panel, three side panels, two door leaf panels, and a pair of knobs for operating the door leaf panels. Furthermore, the video is tracked using MIL technique to determine operations executed by the functioning member for assembling the wardrobe. A timestamp in minutes is associated with the tracked operations depicted in the one or more image frames for assembling the wardrobe. The tracked operations depicted in the video at corresponding timestamp includes:

(00:01) : Arranging a first side panel perpendicular to the floor panel. (00:05) : Arranging a second side panel adjacent to the first side panel, perpendicular to the floor panel.

(00: 10) : Arranging a third side panel adjacent to the second side panel, perpendicular to the floor panel. (00: 15) : Arranging a first door leaf panel adjacent to the first side panel, perpendicular to the floor panel.

(00: 20) : Arranging a second door leaf panel adjacent to the third side panel, perpendicular to the floor panel. (00: 25) : Arranging the head panel parallel to the floor panel separated by the three side panels and the door leave panels.

(00: 30) : Arranging a first knob from the pair of knobs on the first door leaf.

(00: 32) : Arranging a second knob from the pair of knobs on the second door leaf.

Subsequently, each of the one or more image frames are arranged depending on the timestamp associated with the tracked operations depicted in the plurality of image frames. Furthermore, a workflow instruction is created based upon the organized tracked operations. Specifically, the workflow instructions describe the tracked operations executed by the functioning member. Moreover, the workflow instructions specify an identification character (such as an identification number, a model number, a colour, a shape, and so forth) for the at least one object candidate employed for each of the tracked operation for assembling the wardrobe.

In one aspect, an embodiment of the present disclosure provides a system of creating workflow instructions for assembling of a product, the system comprising a data processing arrangement, wherein the data processing arrangement is configured to: - receive a demonstrative content depicting the assembling of the product; identifying, in the demonstrative content, a functioning member assembling the product, wherein the functioning member executes operations for assembling of the product; identify, in the demonstrative content, a plurality of object candidates; track the operations executed by the functioning member, on at least one of the plurality of object candidates to determine a first object and a second object, from the plurality of object candidates, wherein the first object and the second object are being assembled to form the product; associate a timeline with the tracked operations; organise the tracked operations based on the timeline; create the workflow instructions based upon the organized tracked operations and the determined first and second objects. In further embodiment the system might be set up to allow updating the created workflow instructions by performing some of the steps more than once. In the further embodiment the system is configured to update the created workflow instructions based upon the organized tracked operations and the determined first and second objects to include a second set of organized tracked operations and a determined second set of objects. Example of the second set of objects is a third object and a fourth object. In other works some of the steps might be repeated to produce more complex workflow instructions. In one example embodiment steps of tracking the operations, associating a time line, organizing the tracked operations and creating the workflow are repeated for number of times to update workflow which was generated in the previous round.

In one aspect of the present disclosure an assembly line wherein workflow instructions are created using described a methods. According to an optional embodiment the created workflow according to above methods or system can be used to program autonomous robots to perform a manufacturing task. In such an example a human assembles a product and the created workflow instructions are used to program an assembly line with robots. Yet on other example the workflow instructions for an automatic assembly factory is provided with the method of the present disclosure. In further embodiment an assembly line comprises a system according to embodiments of present disclosure.

DETAILED DESCRIPTION OF THE DRAWINGS Referring to FIGs. 1A, IB, 1C ID and IE there is shown example views of image frames 100A, 100B, lOOC, 100D and 100E of demonstrative content depicting assembling of a product, in accordance with an embodiment of the present disclosure. The demonstrative content comprises a video, wherein the video is implemented by the image frames 100A, 100B, lOOC, 100D and 100E. Furthermore, the image frames depict the operational states that describe interaction between a functioning member 102 and objects 104 and 106.

Referring to FIG. 1A, IB and IE, there are shown image frames 100A, 100B and 100E respectively, comprising the functioning member 102 and the object 104. Moreover, the image frames 100A, 100B and 100E depict a plugging operational state.

Referring to FIG. 1C, there is shown an image frame lOOC comprising the object 104. Furthermore, the image frame lOOC depicts a pose changing operational state. Referring to FIG. ID, there is shown an image frame 100D comprising the object 106. Furthermore, the image frame 100D depicts a searching operational state. FIGs. 1A, I B, 1C, ID and IE are merely examples, which should not unduly limit the scope of the claims herein. A person skilled in the art will recognize many variations, alternatives, and modifications of embodiments of the present disclosure. For example, the demonstrative content may comprise a set of images and depict operational state in the demonstrative content.

Referring to FIGs 2A and 2B illustrated is a flow chart depicting steps of a method for creating workflow instructions for assembling of a product, in accordance with an embodiment of the present disclosure. The method is depicted as a collection of steps in a logical flow diagram, which represents a sequence of steps that can be implemented in hardware, software, or a combination thereof.

At a step 202, a demonstrative content depicting the assembling of the product is obtained. At a step 204, a functioning member assembling the product in the demonstrative content is identified. Furthermore, the functioning member executes operations for assembling of the product.

At a step 206, at least one object candidate in the demonstrative content is identified . At a step 208, the operations executed by the functioning member on the at least one object candidate is tracked to determine a first object and a second object, from the at least one object candidate. Furthermore, the first object and the second object are being assembled to form the product. At a step 210, a timeline is associated with the tracked operations.

At a step 212, the tracked operations are organized based on the timeline. At a step 214, the workflow instructions are created based upon the organized tracked operations and the determined first and second objects.

The steps 202 to 214 are only illustrative and other alternatives can also be provided where one or more steps are added, one or more steps are removed, or one or more steps are provided in a different sequence without departing from the scope of the claims herein. Further the method can comprise iterating one or more of the steps. In deed initial workflow instructions might be created first and the created initial workflow instructions might be updated by repeating one or more of the steps.

Referring to FIG. 3 illustrated is an exemplary implementation of the method for creating workflow instructions for assembling of a product, in accordance with an embodiment of the present disclosure. The method is depicted as a collection of steps in a logical flow diagram, which represents a sequence of steps that can be implemented in hardware, software, or a combination thereof.

At a step 320, a demonstrative content 302 is obtained and a first object and a second object comprising the product are identified from the demonstrative content. The step 320 is implemented by way of steps 322, 324, 326, and 328, as shown in the figure.

At the step 322, a functioning member is identified in the demonstrative content 302, wherein the functioning member executes operations for assembling of the product. At the step 324, a plurality of object candidates are identified in the demonstrative content 302, wherein techniques (such as edge detection technique, contour extraction technique, feature extraction technique, and so forth) are executed to identify the plurality of object candidates from the demonstrative content 302. Additionally, an information stored in the object dictionary 304 and operations executed by the functioning member are used for identifying the plurality of object candidates.

At the step 326, various movements and alterations performed in the demonstrative content 302, for assembling of the product, is tracked. Additionally, an information stored in the object dictionary 304 is used for tracking the movements and alterations performed in the demonstrative content 302.

At the step 328, the operations executed by the functioning member on the plurality of object candidates and the movements and alterations performed in the demonstrative content 302, for assembling of the product, are analysed to identify the first object and the second object of the product.

At a step 340, a workflow for assembling of the product is identified in the demonstrative content 302, wherein the workflow is determined based upon the operations executed by the functioning member.

At a step 360, a semantic information related to the tracked operations are extracted for creating the workflow instructions for assembling of the product. The steps 320 to 360 are only illustrative and other alternatives can also be provided where one or more steps are added, one or more steps are removed, or one or more steps are provided in a different sequence without departing from the scope of the claims herein. The steps might be also repeated to iterate the workflow instructions. Referring to FIG. 4 illustrated is an exemplary implementation of the method for creating workflow instructions for assembling of a product, in accordance with an embodiment of the present disclosure. The method is depicted as a collection of steps in a logical flow diagram, which represents a sequence of steps that can be implemented in hardware, software, or a combination thereof.

At a step 410, a demonstrative content 402 is obtained and a functioning member is identified in the demonstrative content 402, wherein the functioning member executes operations for assembling of the product.

At a step 420, at least one object region is identified in the demonstrative content 402. Typically, the at least one object candidate is identified by extracting edges and contours from the demonstrative content 402.

At a step 430, a feature extraction technique is executed on the at least one object region to extract a first set of features associated therewith.

At a step 440, weak object regions are filtered out, wherein the weak object regions are identified based on the first set of features. Moreover, a plurality of object candidates are identified based upon the first set of features, from the at least one object region. At a step 450, the first set of features relating to the identified at least one object region and the plurality of object candidates are stored in an object dictionary, thereby updating the object dictionary. It will be appreciated that the object dictionary is created when no existing object dictionary is updated. At a step 460, the feature extraction technique is further conducted on the at least one object region to identify the plurality of object candidates previously stored in the object dictionary thereby eliminating false identification of the plurality of object candidates. Furthermore, the plurality of object candidates previously stored in the object dictionary (namely, reference object candidates) are utilised to refine the identified plurality of object candidates described upon identification thereof.

At a step 470, the identified plurality of object candidates are tracked. At a step 480, a distance is calculated between the identified plurality of object candidates (obtained at the step 460) and the plurality of object candidates previously stored in the object dictionary (obtained at the step 470). Furthermore, the distance is calculated to select a plurality of object candidates (from amongst the identified plurality of object candidates and the plurality of object candidates previously stored in the object dictionary) that proffers greater accuracy and efficiency in identifying a second set of features associated with the plurality of object candidates. It will be appreciated that the plurality of object candidates proffering higher accuracy is updated in the object dictionary accompanied with the second set of features associated therewith.

The steps 410 to 480 are only illustrative and other alternatives can also be provided where one or more steps are added, one or more steps are removed, or one or more steps are provided in a different sequence without departing from the scope of the claims herein.

Modifications to embodiments of the present disclosure described in the foregoing are possible without departing from the scope of the present disclosure as defined by the accompanying claims. Expressions such as "including", "comprising", "incorporating", "have", "is" used to describe and claim the present disclosure are intended to be construed in a nonexclusive manner, namely allowing for items, components or elements not explicitly described also to be present. Reference to the singular is also to be construed to relate to the plural.