The invention relates to a method of protocol compliance testing.

BACKGROUND

Interoperability of devices from different manufacturers may require each product to comply with a respective industry wide communication protocol standard. A certification and conformance to a protocol standard ensures such devices can communicate reliably so that the overall system can function correctly. Certified products may allow customers to implement cost effective multi vendor solutions and may reduce interoperability issues.

However the current certification process can be costly and time consuming if the device is not checked for compliance with standards before approaching the certification body as the device may need to go through the testing cycles a number of times. Testing for conformance before certification therefore may lower the cost and time required for achieving certification. With pre certification conformance, the product should already be compliant with the test procedures executed by certification organizations. Thus, the total hours spent or iterations spent for official certification will be minimised, reducing the usually high cost of official certification.

Each different protocol may have a system for pre certification

compliance testing, but they are each specific to a single protocol i.e. no multiprotocol support, there may be no support for automation or unattended testing, manual & expert intervention may be required and/or there may be no support for dynamic addition of test cases, protocol data like objects and services without changing source code. SUMMARY OF THE INVENTION

In general terms, the invention proposes that different devices can be automatically tested for compliance with respective protocols. This may have the advantage that a user does not have to use separate systems and test setups to test for compliance for each protocol. The system may be dynamic and flexible allowing enhancements to protocols supported, additions of new protocols, updating of existing protocols, customization and use in a range of configurations.

In a specific expression of the invention, there is provided a method of compliance testing as claimed in claim 1.

One or more implementations of the invention are given in claims 2 to 15.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more example embodiments of the invention will now be described, with reference to the following figures, in which:

Figure 1 is a schematic drawing of a compliance testing system

according to an example embodiment,

Figure 2 is a flow diagram of a run test activity,

Figure 3 is a flow diagram of a method of compliance testing according to an example embodiment, Figure 4 is a data flow diagram of the data exchange between system components,

Figure 5 is a screen shot of a user interface,

Figure 6 is a screen shot of a configuration device interface,

Figure 7 is a screen shot of a run test interface, and

Figure 8 is a screen shot of a detailed report.

DETAILED DESCRIPTION

A system 100 for pre certification compliance testing according to the example embodiment is shown in Figure 1. The system 100 can test a device for a specific protocol, in-house, in isolation and/or in an integrated environment. A person may not be an expert to perform the compliance testing. It may support multiple protocols like BACnet, CIP protocol suite, Profibus, Mod bus etc.

The system 100 generally includes a set of test suites 102, a framework 104 to implement each test suite and a set of interfaces 106 to communicate with various types of devices.

The test suites 102 are XML based and may be inbuilt as a part of conformance test or user created. XML may be used for representing test suites to allow flexibility and due to its ability to express hierarchical constructs. XML based test suite may include a large repository of prebuilt test cases that can be used for standard compliance testing and provide inputs to execute custom testing.

Metadata and test scripts in XML may make the architecture extensible. Also XML allows the user to make changes in data and test scripts easily and offline. Here metadata refers to the xml definition of protocol services and data structure used for communication with the device.

The protocol configuration files are the database repository of the application. The files contain all the configuration data for different protocols, protocol version, protocol objects & services supported, the protocol stack structure & related info.

The framework 104 includes a user interface 108, a test scheduler 110, an XML interpreter 112, a logger 116, an adaptation layer 120, and a reporting module 114.

The user interface 108 has an Integrated Development Environment (IDE) look and feel with a workflow wizard to guide the user through various steps to perform an automation compliance test or a manual test. The interface 108 is dynamic and displays appropriate controls depending on the selected protocol. Protocol specific representation of configuration screens is

represented in xml format in test suites 102.

The test scheduler 110 executes the automated & manual tests as per the priority set by the user. Test scheduler is responsible for tracking the time taken by each test case or test clause and feeds this information to the reporting component 114. Test scheduler 110 is also responsible to update the interface 108 indicating the current state of test suit execution.

The XML interpreter 112 is used to convert the configuration data, test parameter data & protocol meta-data into object model that can directly be interpreted by the application. The interpreter 112 shall use factories to build protocol specific object model. For example the factory method is a creational pattern that helps to model an interface for creating an object which at creation time can let its subclasses decide which class to instantiate. The Factory Pattern is responsible for "Manufacturing" an Object. It helps instantiate the appropriate Subclass by creating the right Object from a group of related classes. The Factory Pattern promotes loose coupling by eliminating the need to bind application-specific classes into the code.

The reporting component 114 provides functionality for various types of reports, short report, detailed report, PDF format, etc. Also a result database records all results of the tests.

The logger 116 module logs all the activities logs, communication logs, debug logs & error logs in the application.

The adaptation layer interface 118 may be abided(implemented) by the protocol implementation so that the protocol tests can be executed. The adaptation layer interface 118 and adaptation layer implementation 120 together allows for pluggability of protocols.

The set of interfaces 106 includes for each interface a protocol adaption layer and protocol stack implementation layer 120, a protocol driver layer 122 and a physical port 124.

The protocol adaptation layer 120 converts protocol independent test data into format required for a particular protocol stack. The protocol stack implementation layer is the actual implementation of the protocol. The adaption layer interface 118 provides facade interface and hide details of a protocol stack. Hence the protocol stack can be a third party component. The protocol stack is a specific implementation of a Protocol specification using custom code and the utility classes and is part of Protocol Stack component.

The test suite 102 and set of interfaces 106 may include provision for a plurality of protocols, for example:

1. CI P Protocol Suite

a. Ethernet/IP

b. Control Net

c. Device Net

2. BACnet

3. Serial Modbus and Modbus TCP

4. Profibus Suite

a. Profibus-DP (V0, V1 and V2)

b. ProfiNet (Basic and SafetySIL3)

c.

5. LONWorks

6. Other industrial protocols

The test suite 102 and set of interfaces 106 may also include the following features:

· Development of meta-data pattern, object model (composite) pattern, Protocol Pattern, State machine pattern to reduce code size and effort by finding commonality across protocols. Framework is provided with default set of layer services implementation and default encoding / decoding features implemented, these features can be reused or extended to suit new protocol implementation need. This interface based and abstract framework makes it possible to develop plug and play based implementation for protocol stack implementation layer 120 and protocol driver layer 122.

Loose coupling of Protocol stack with engine framework using the set of interfaces 106. The set of interfaces 106 will be defined by the framework and implemented by the Protocol Stacks., thus allowing for pluggability and loose coupling

Multi-threading environment to support asynchronous communication The above mention system may be executed on a personal computer. The user interface may include a screen and an input device, such as a key board and mouse. Alternatively a touch screen may be used.

A method of compliance testing 300 according to the example

embodiment is shown in Figure 3.

At 302 to test a device, a user will first connect a device to the respective port 124.

At 304 the user can configure the device by setting the IP, whether manual or auto. An example of device configuration is shown in Figure 6.

At 306 the user selects which protocol to use. For each protocol a workspace is created. A workspace is a context in which the user will perform his tasks. A workspace is associated with a protocol and is tied to that protocol. A user will create a workspace for a device of a protocol that he/she wants to test. An example workspace for a single device is shown in Figure 7. The user may also manage multiple devices under a single workspace.

At 308 the user has the option to set device configuration for the protocol objects. This may include items like communication properties, protocol revision, device capabilities, etc. The user may also select reporting

configuration, like log files, reports, etc. The workspace created may be saved by the user for future use.

A user may modify workspace properties by opening an already created workspace or changing properties of currently open workspace.

At 310 a user may select to execute the automated test suite for the configured device in his/her workspace. The user may not have to provide any additional information other than what is needed as external files which may be already created or already provided. The tests are as per the sequence specified in test suite without user intervention. The results are available for view to the user even at a later time. An example of test execution is shown in Figure 7.

A user may want to perform manual tests. In that case, a user may select an available test, provide input data for that test and execute it.

The XML repository may maintain the protocol specifications and test scripts for multiple protocols. By choosing the protocol through the user interface 108, end user will be able to get the device tested against any of the protocols available in the system based on the user selection. At 312, the pre populated protocol specification is loaded into an object model.

At 314 test scripts are fetched from the pre populated database and parsed into the test objects in a way that can be used by the system. At 316, the protocol stack runs the scripts and sends the protocol service requests to the device. An example of the test run activity is shown in Figure 2. At 318 the device responses are returned back to the Test Engine.

Example data flows are shown in Figure 4.

At 320 the results are stored in the database.

At 322 user may choose to view the test results stored in the database.. An example report is given in Figure 8. This may make it easier to analyze the device's level of conformance to a protocol specification by providing

communication logs in form of reports.

An exhaustive set of tests for the protocol conformance may be

customised and/or configured as desired.

One or more embodiments have been described in relation to

compliance testing. Other applications include:

a. A network simulator for load testing, or testing for the performance of the assembly.

b. A protocol converter or gateway in communication between different network types.

c. A protocol analyzer or a communication analyzer and can maintain logs for various communications within the system.

d. A device monitor for a large system.

Advantages of one or more embodiments may include:

· Service of pre-compliance which reduces cost and schedule for

introducing new products.

• Shorter cycle to get compliance for new protocols or to develop a new tool.

• Reduce analysis time to develop a new protocol stack. • Common reusable asset repository which reduces implementation time.

• Deployment of products with multiple features and protocols increases the need for testing of these products multifold depending on the number of features supported. There are challenges of protocol testing for different kinds of products (Communication, Industrial, Storage,

Consumer and Automotive). Conformance testing is essential for products supporting multiple protocols. This tool can act as a single interface for multiple Protocol testing.

• Predictive analysis gives first hand idea to developers & no need for expert for analysis reduces cost of labour.

• Simple user interface for writing test case reduces cost on expensive labour to write scripts.

• Automated test case execution again reduces cost for having a person to monitor it.

• Total cost of ownership of Software simulators would be lesser as

compared to hardware simulators.

• No need for different sets of tools for different protocols testing reduces cost on tools.

• Devices can be tested during all the development phases, in-house & in isolation.

• A single tool for testing different types of protocol implementations

• Ability to conduct an exhaustive test using a particular protocol as

defined by conformance test plan. • Ability to build a custom test plan using the exhaustive list of test cases built in the system.

• Automatic test case execution by just configuring device & run test on press of a single button.

· Comprehensive Reports for developers.

• Communication Logs for the test result.

• Predictive analysis for the errors encountered

While example embodiments of the invention have been described in detail, many variations are possible within the scope of the invention as claimed as will be clear to a skilled reader.