Generation of test models from Behavior Driven Development scenarios based on Behavior Driven Development step defini tions and similarity analysis using Neuro Linguistic Program ming and machine learning mechanisms

The present invention pertains to a computer-implemented method for automated verification of a software program in a behavior-driven development environment. The present inven tion further pertains to a device comprising a processor con figured to perform such a method. The present invention par ticularly pertains to Behavior Driven Development (BDD) .

In recent years, BDD has emerged as an agile software devel opment approach for the specification and execution of auto mated acceptance tests of software programs. BDD was intro duced by Dan North in 2006 to simplify Test-Driven Develop ment (TDD), see for example Brandel et al . , "Drei Methoden, ein Ziel: Testautomatisierung mit BDD, MBT und KDT im Ver- gleich, " Softwaretechnik-Trends , 35(3), 2015. TDD is a soft ware development methodology which essentially states that for each unit of software, a software developer must define specific test sets for the unit first, then implement the unit and finally verify that the implementation of the unit makes the tests succeed. BDD combines Test-Driven Development (TDD) , Object-Oriented Analysis (OOA) , Object-Oriented Design (OOD) and Domain-Driven Design (DDD) to provide a unified language and approach for handling such a software develop ment process from requirements analysis to implementation.

BDD is largely facilitated through the use of a simple do main-specific language (DSL) using natural language con structs (e.g., English-like sentences) that can express the behavior and the expected outcomes of the software. This 'ubiquitous language' can be understood and jointly used by quality managers, domain experts, software developers and customers. BDD employs a semi-formal format for behavioral specification of the software, which is borrowed from user story specifications from the field of object-oriented analy sis and design.

To this end, each software unit is decomposed into so-called scenarios, each scenario testing one individual aspect of the software. Each scenario is in turn divided into test steps, which describe a desired outcome of the respective aspect of the software starting from given initial conditions and run ning through predefined events. Each scenario with its test steps is formulated as a natural language script, which can later be translated into executable test scripts in an auto mated way. The executable test scripts can then be executed as automated tests for testing the software for its correct implementation. The software requirements within the test scripts are usually written in "given-when-then" sentences based on the ubiquitous language of domain-driven design.

This is intended to facilitate the transition between the language used to define the domain-driven requirements and the programming language used to implement them.

One test automation framework widely used for automated ac ceptance tests written in BDD style is called Cucumber, which comprises a plain language parser called Gherkin. The desired behavior of the software is formulated within Gherkin in a declarative way:

-GIVEN (precondition/initial conditions) ...

-WHEN (event/action/trigger) ...

-THEN (effect to be observed/system response) ...

Such descriptive languages are semi-formal with the capital words (GIVEN, WHEN, THEN) serving as pre-designated keywords. Due to the simple grammar and the natural language keywords, the BDD requirements can be understood and manually executed by technical testers. Cucumber runs through these keywords and processes them step by step, thereby mapping every non capital phrase following these keywords to a parameterized function call. Traditionally, Ruby scripts were used for this purpose within Cucumber, which replace the test steps by au tomated program calls and thus make the BDD description auto matically executable. However, Cucumber now supports a varie ty of different programming languages through various imple mentations, including Java and C# .

BDD is easy to understand and straightforward to implemented. However, in large and complex use cases, the approach with its textual and manually created scenarios may lack the man ageability to handle a large number of scenarios and complex tests sets while ensuring completeness and consistency. For the development of complex systems, approaches like Model Based Testing (MBT) or Keyword Based Testing (KBT) are often seen as more appropriate. In particular, MBT approaches allow reviewing and verifying the completeness and consistency of even complex test scenarios using a visual representation of the scenarios, e.g. using diagrams in Unified Modelling Lan guage (UML) . However, MBT has to be individually embedded in to the existing development and test process for each soft ware component .

Against this background, it is an object of the present in vention to find solutions with improved convenience and au tomatization for the verification of complex software packag es .

This object is achieved by a method having the features of claim 1, a data processing system having the features of claim 10 a computer program product having the features of claim 11 and a data storage medium having the features of claim 12.

According to an aspect of the invention, a computer- implemented method for automated verification of a software program in a behavior-driven development environment compris es receiving, with a data processing system, test scenarios, each test scenario defining an expected behavior of the soft ware program in consecutive test steps, which are formulated in a domain-specific language using natural language phrases and which describe a desired outcome of the software program for predefined events based on given initial conditions; im porting test step definitions from the behavior-driven devel opment environment; determining for each test step of the test scenarios if the test step matches with one of the test step definitions on basis of the natural language phrases of the test step; assigning all matched test steps to the corre sponding test step definitions; applying natural language processing, NLP, on the natural language phrases of any test steps remaining unmatched, wherein the NLP provides a confi dence level for each unmatched test step to correspond to one of the test step definitions; assigning any unmatched test step to the corresponding test step definition when the con fidence level surpasses a first predefined matching probabil ity; and at least one of: generating graphical test models for the test scenarios on basis of the assigned test step definitions; and generating executable test scripts for the test scenarios on basis of the assigned test step defini tions .

According to another aspect of the invention, a data pro cessing system comprises a processor configured to perform a method according to the invention.

According to yet another aspect of the invention, a computer program comprises executable program instructions configured to, when executed, perform the method according to the inven tion.

According to yet another aspect of the invention, a non transient computer-readable data storage medium comprises ex ecutable program instructions configured to, when executed, perform the method according to the invention.

The non-transient computer-readable data storage medium may comprise, or consist of, any type of computer memory, in par ticular semiconductor memory such as a solid-state memory.

The data storage medium may also comprise, or consist of, a CD, a DVD, a Blu-Ray-Disc, an USB memory stick, a memory card (e.g. an SD card) or the like.

According to yet another aspect, the invention provides a da ta stream representing, or configured to generate, executable program instructions configured to, when executed, perform the method according to the invention.

One idea of the present invention is to provide the means to utilize the benefits of BDD with its easy to use and natural language based scenarios while maintaining the required man ageability for large, complex development projects. To this end, the proposed solution automatically assigns the test steps of each scenario with already existing test step defi nitions of a test automation framework (from the integrated BDD development environment) . If a literal and/or unambiguous matching is not possible, e.g. because the respective scenar io was written in a different style and/or uses different wording, then the NLP algorithm is used to find a best match of the respective test step among the existing test step def initions. If the probability of this best match is high enough to provide a correct/likely fit between test step and test step definition, e.g. if it has a matching probability of at least 80% or 90% or more, then the test step is as signed to the respective test step definition.

The present approach allows an efficient mapping of BDD step phrases to a test automation framework of the integrated BDD development environment and supports the structured develop ment of the necessary framework code. Furthermore, it facili tates the automated generation and synchronization of a graphical test model from BDD scenarios so that the ad vantages both from BDD and MBT methods may be utilized even for large and complex development projects. The graphical test models may be used to visualize, review and modify test cases so that the consistency and completeness of the BDD scenarios can be ensured. For example, missing scenarios may be identified based on a test model review. As another exam- pie, similar scenarios may be combined into a single scenar io. Furthermore, the ability to use MBT techniques adds an additional abstraction level and supports the change manage ment. The executable test scripts may be automatically gener ated directly based on the assigned test step definitions and/or after a verification of the scenario (s) on basis of the generated test model.

Advantageous embodiments and improvements of the present in vention are found in the subordinate claims.

According to an embodiment, the method further may comprise updating, when the confidence level is above the first prede fined matching probability, the respective test step defini tion on basis of the natural word phrases of the respective test step. Hence, the existing phrase pattern definitions from the BDD test automation framework may be adapted to in clude alternative and/or modified test step definitions cor responding to the matched test steps. The first predefined matching probability may be set to a high confidence value of 80% or more so that there is a high probability for a match between the test step and the test step definition.

According to an embodiment, the method may further comprise adding, when the confidence level is below a second prede fined matching probability, a test step definition to the be havior-driven development environment corresponding to the respective test step. Hence, in case that the confidence lev el is lower than this reference probability, which may be for example 50% or similar, it is decided that the test step does not match to any existing test step definition. Instead, the test step is used to define a new test step definition, which is then added to the BDD test automation framework and may be used further.

According to an embodiment, a user verification may be re quested if the confidence level is below the first predefined matching probability but above a second predefined matching probability. For example, the first predefined matching prob- ability may be set to 80% or 90% and the second predefined matching probability may be set to 50%. If the confidence level is above the first predefined matching probability, then the test step is considered to match the respective test step definition, which may then be updated based on the for mulation of the test step. If the confidence level is below the second predefined matching probability, then the test step does not match any of the existing definitions and hence may be used to define a new one. However, in the intermediate range between 50% and 80% (or 90%) , the situation may be un clear, i.e. the test step may or may not match one of the ex isting definitions. In that case a user input may be required to settle the further procedure, that is if a new definition is introduced, if an existing definition is updated or if the scenario is discarded, etc.

According to an embodiment, the method may further comprise feeding the user verification to a machine learning algorithm of the NLP. For example, commonality criteria of the NLP may be adjusted based on the verification results, e.g. reduce the relevance of certain phrases, identify invalid commonali ties and/or commonalities not yet identified. Then, the com monality detection accuracy may be compared in future execu tions to check whether the optimized criteria did improve the accuracy of the NLP. In the long run, this may reduce the ef fort for manual validation and improve the accuracy of the commonality detection over time. After the required training and optimization of the NLP engine, the algorithm of the in vention may detect and match phrases in a fully unattended and automated way.

According to an embodiment, generating the graphical test models may comprise combining similar test scenarios on basis of test steps assigned to the same test step definition.

According to an embodiment, generating the graphical test models may comprise identifying test data within the test scenarios based on the natural language phrases. According to an embodiment, the graphical test models may comprise unified modeling language diagrams, i.e. UML dia grams .

According to an embodiment, the method may further comprise comparing the graphical test models with the test scenarios to determine if the graphical test models are in compliance with the expected behavior of the software program. Hence, based on the generated graphical test model, missing and/or incorrect scenarios may be identified.

The invention will be explained in greater detail with refer ence to exemplary embodiments depicted in the drawings as ap pended .

The accompanying drawings are included to provide a further understanding of the present invention and are incorporated in and constitute a part of this specification. The drawings illustrate the embodiments of the present invention and to gether with the description serve to explain the principles of the invention. Other embodiments of the present invention and many of the intended advantages of the present invention will be readily appreciated as they become better understood by reference to the following detailed description. The ele ments of the drawings are not necessarily to scale relative to each other. In the figures, like reference numerals denote like or functionally like components, unless indicated other wise .

Fig. 1 shows a device with a processor performing a method according to an embodiment of the invention.

Fig. 2 shows a schematic flow diagram demonstrating as pects of the method of Fig. 1.

Fig. 3 shows a schematic flow diagram demonstrating as pects of the method of Fig. 1. Fig. 4 shows an example for graphical test models de rived with the method of Fig. 1.

Although specific embodiments are illustrated and described herein, it will be appreciated by those of ordinary skill in the art that a variety of alternate and/or equivalent imple mentations may be substituted for the specific embodiments shown and described without departing from the scope of the present invention. Generally, this application is intended to cover any adaptations or variations of the specific embodi ments discussed herein.

Figure 1 shows a data processing system 10 with a processor 11 performing a method M according to an embodiment of the invention. Certain aspects of the method M are exemplified in Figs . 2 and 3.

Besides the processor 11, the data processing system 10 may include the usual components like an accessible memory, a storage unit, an input unit, an output unit and so on (not shown) . The processing unit 10, as used herein, means any type of computer or computational circuit, such as, but not limited to, a microprocessor unit, a microcontroller, a graphics processing unit, a digital signal processing unit, or any other type of processing circuit.

The method M provides automated verification of a software program in a behavior-driven development environment, e.g. an integrated development environment, which may comprise a BDD test automation framework like Cucumber, SpecFlow, Behave or similar having a library with BDD scripts and phrase pat tern/test step definitions.

The method M comprises under M0 receiving, with the data pro cessing system 10, test scenarios 1, e.g. by importing them from the BDD development environment. Each test scenario 1 defines an expected behavior of the software program in con secutive test steps 2. The test scenario 1 and thus the test steps 2 are formulated in a domain-specific language using natural language phrases and describe a desired outcome

("THEN") of the software program for predefined events

("WHEN") based on given initial conditions ("GIVEN") . The test scenarios 1 thus represent a specification and/or re quirements of the software program in a chronological se quence .

As a simple example, a registering/login software may com prise the following (schematic) scenarios, wherein the key words GIVEN, WHEN, THEN each define a respective test step 2:

- Scenario 1: successful login

GIVEN a user has entered valid credentials

WHEN clicking on login

THEN start screen is shown

- Scenario 2: wrong password

GIVEN a user has entered invalid credentials

WHEN pressing login

THEN an error message is shown

- Scenario 3: unregistered user

GIVEN a not registered user has entered some credentials

WHEN press login

THEN an error message is shown

- Scenario 4: Registration

GIVEN a not registered user

WHEN click on register

THEN the registration dialog is shown

The method M further comprises under Ml importing test step definitions 3 from the behavior-driven development environ ment. For the example above, such existing test step defini tions 3 may look like this (formulated in an arbitrary pro gramming language, e.g. Jave or C#) :

@Given(" a user has entered [* credentials] $") public void

enter_credentials (UCred argl)

@When ("''Click on Registration button$")

public void

click_on_registration ()

@Then ( "^Start screen is shown$")

public void

verify_start_screen_shown ()

@Then (" Ehhoh message is shown$")

public void

verify_error_msg_shown ()

@Then ("''Registration screen is shown$")

public void

verify_registration_shown ()

Next, the method M comprises under M2 determining for each test step 2 of the test scenarios 1 if the test step 2 match es with one of the test step definitions 3 on basis of the natural language phrases of the test step 2. The method M further comprises under M3 assigning all matched test steps 2 to the corresponding test step definitions 3.

In the example from above, matched steps may comprise:

- GIVEN a user has entered valid credentials

GIVEN a user has entered invalid credentials

@Given(" a user has entered [* credentials ]$" )

- WHEN clicking on login

WHEN press login @When ( " Click on Login button$")

@When ( " Press Login$")

However, a literal one-to-one matching may not be possible for all test steps 2. For example, "WHEN pressing login" is different form "WHEN press login" due to the different usage of the word "press". However, both test steps 2 are similar and thus natural language processing (NLP) may be used to recognize these similarities. In a similar vein, "GIVEN a not registered user has entered some credentials" is similar to:

@Given(" a user has entered [* credentials ]$" )

and "WHEN click on Register" is similar to:

@When ( " Click on Registration button$")

To identify these similarities, the method M further compris es under M4 applying NLP on the natural language phrases of any test steps 2 remaining unmatched. The NLP provides a con fidence level for each unmatched test step 2 to correspond to one of the test step definitions 3. The method M further com prises under M5 assigning any unmatched test step 2 to the corresponding test step definition 3 when the confidence lev el surpasses a first predefined matching probability.

Fig. 3 shows an example, where the NLP is called under M4 as soon as one of the test steps 2 cannot be matched under step M2, M3. The NLP provides a confidence level, which is then compared with two predefined matching probabilities, a high probability of 90% and a low probability of 50%. If the con fidence level is above 90%, then the corresponding test step 2 is assigned to the respective test step definition 3 under M5. The test step definitions 3 may be updated by including the assigned test step 2 in repository of test step defini tions 3 within the BDD framework (cf. reference sign T1 in Fig. 2) . In case that the confidence level is below 50%, a new test step definition 3 is added to the BDD test automa tion framework that corresponds to the not yet existing test step 2 (cf . reference sign T2 in Fig. 2) . If the confidence level is between 50% and 90%, a user veri fication is requested (cf. middle quadratic box in Fig. 1), which is then used under T3 as input for a machine learning algorithm of the NLP to improve the accuracy of the NLP in future runs. For example, the results of the assignments may be analyzed with respect to commonalities not identified, e.g. "Select" should be equal to "Click", or invalid common alities, e.g. "Start screen" should be different from "Regis tration screen". Furthermore, the relevance of certain words or phrases may be reduced, e.g. in case of irrelevant words like "some". By optimizing the NLP engine, manual interven tions for the NLP application may be reduced and/or complete ly avoided in the consecutive runs of the method M.

In the example from above, the assignment of test steps 2 and test step definitions 3 may look like this after running the NLP :

- GIVEN a user has entered valid credentials

GIVEN a user has entered invalid credentials

GIVEN a not registered user has entered some credentials @Given(" a user has entered [* credentials ]$" )

- WHEN clicking on login

WHEN press login

WHEN pressing login

@When ( " Click on Login button$")

@When ( " Press Login$")

- WHEN click on Register

@When ( " Click on Registration button$")

- THEN start screen is shown

@Then ( " Start screen is shown$")

- THEN an error message is shown

@Then ( " Error message is shown$") - THEN the registration dialog is shown

@Then (" '“'Registration screen is shown$")

Updated test step definitions 3 may comprise:

@When ( " Click on Login button$")

@When ( " Press Login$")

@When ( " dressing Login$")

public void click_on_login ( )

Next the method M comprises under M6 generating graphical test models 4 for the test scenarios 1 on basis of the as signed test step definitions 3. The graphical test models 4 may be represented, for example, by Unified Modelling Lan guage diagrams. Here, similar scenarios 1 may be combined on basis of the assignment of test steps 2 to test step defini tions 3. An example is shown in Fig. 4, where two graphical test models 3 are generated from the above example, namely the case where a user has entered credentials and the case where a user has not entered credentials. The credentials may be invalid or valid or the user may not be registered at all. These different test data (user credentials: invalid, not registered, valid) may be identified by the NLP based on the natural language phrases and may be used to combine similar test scenarios 1, as it is shown in Fig. 4.

As indicated in Fig. 2, the method M may comprise under T4 comparing the graphical test models 4 with the test scenarios 1 to determine if the graphical test models 4 are in compli ance with the expected behavior of the software program. For example, based on these graphical test models 4, missing sce narios 1 may be identified, e.g. a user with expired creden tials or a registration attempt of an already registered us er. Moreover, optimized test scenarios 1 may be generated, e.g.:

- Scenario 1: successful login

GIVEN a user has entered [valid] credentials

WHEN click on login button

THEN start screen is shown - Scenario 2: wrong password

GIVEN a user has entered [invalid] credentials

WHEN click on login button

THEN an error message is shown

The method M further comprises under M7 generating executable test scripts 6 for the test scenarios 1 on basis of the as signed test step definitions 3. To this end, existing BDD tools and frameworks may be utilized, e.g. Cucumber or simi lar .

As a result, the method M described above provides the means to utilize the benefits of BDD (with easy to use, natural language based scenarios) while maintaining the required man ageability for large, complex development projects. The gen eration and synchronization of a test model from BDD scenari os allows to utilize the advantages both from BDD and MBT methods, especially for large complex development projects. The model based review and generation of test cases ensures the consistency and completeness of the BDD scenarios. The ability to use MBT techniques to add an additional abstrac tion level bridges the gap between RE focused usage of BDD and a BDD based test automation approach. The automated step matching using machine learning allows an efficient mapping of BDD step phrases to a test automation framework and sup ports the structured development of the necessary framework code .

In the foregoing detailed description, various features are grouped together in one or more examples or examples with the purpose of streamlining the disclosure. It is to be under stood that the above description is intended to be illustra tive, and not restrictive. It is intended to cover all alter natives, modifications and equivalents. Many other examples will be apparent to one skilled in the art upon reviewing the above specification. The embodiments were chosen and described in order to best explain the principles of the invention and its practical ap plications, to thereby enable others skilled in the art to best utilize the invention and various embodiments with vari- ous modifications as are suited to the particular use contem plated. Many other examples will be apparent to one skilled in the art upon reviewing the above specification.