Login| Sign Up| Help| Contact|

Patent Searching and Data


Title:
MODEL-BASED TEST SYSTEM SECURITY
Document Type and Number:
WIPO Patent Application WO/2021/041233
Kind Code:
A1
Abstract:
A method performed by a computing system that collects information on transactions in a telecommunication system includes receiving an event object. The event object includes parameters associated with a telecommunication event. The method further includes classifying the event object, using a classification mechanism, as one of a plurality of Key Performance Indicators (KPIs) by using the parameters, including considering events from multiple protocols for classifying event objects. The method further includes, after classifying the event object, inferring relationships between a set of inputs and a set of outputs, the set of inputs including information types from the event object and the set of outputs including KPIs.

Inventors:
RATHORE HEENA (US)
SAMANT ABHAY (US)
Application Number:
PCT/US2020/047449
Publication Date:
March 04, 2021
Filing Date:
August 21, 2020
Export Citation:
Click for automatic bibliography generation   Help
Assignee:
HILLER MEASUREMENTS LLC (US)
International Classes:
G01R31/26; G06F11/34; G06N20/00
Foreign References:
US20110314331A12011-12-22
US20130125204A12013-05-16
Attorney, Agent or Firm:
CLEMENTS, Calmann, J. et al. (US)
Download PDF:
Claims:
WHAT IS CLAIMED IS: 1. A method comprising: with a computing system integrated with a test system, identifying a plurality of elements of the test system that are approved for use within the test system, the test system being configured to interface with a device under test and a network; with the computing system, collecting information associated with the elements during operation of the test system, the information including inputs and outputs of the plurality of elements; with the computing system, using the information to create a model of behavior for each of the plurality of elements using a machine-learning function; and with the computing system, after the test system is connected to a secure network, periodically comparing current behavior of each of the plurality of elements with the model. 2. The method of claim 1, further comprising, detecting that a new element has been added to the test system. 3. The method of claim 2, further comprising, in response to detecting the new element, determining whether the new element is of a same type as one of the plurality of elements. 4. The method of claim 3, further comprising, in response to determining that the new element is of the same type as one of the plurality of elements, authorizing the new element for use with the test system. 5. The method of claim 4, further comprising, comparing behavior of the new element with the model. 6. The method of claim 1, further comprising, in response to detecting a difference between the model and behavior of one of the plurality of elements, alerting a user.

7. The method of claim 1, further comprising, in response to detecting a difference between the model and behavior of one of the plurality of elements, disabling the one of the plurality of elements. 8. The method of claim 1, wherein one of the plurality of elements corresponds to a test instrument. 9. The method of claim 8, wherein the test instrument comprises one of: a digital multimeter and an oscilloscope. 10. The method of claim 1, wherein one of the plurality of elements corresponds to an interface module. 11. A test system comprising: an interface module; a plurality of test instruments; and a computing system comprising machine readable instructions that when executed by a processor, cause the computing system to: determine that each of the plurality of test instruments is approved for use within the test system; collect information associated with the plurality of test instruments during operation of the test system, the information including inputs and outputs of the plurality of test instruments; use the information to create a model of behavior for each of the plurality of test instruments using a machine-learning function; and with the test system connected to a secure network, periodically compare current behavior of each of the plurality of elements with the model. 12. The system of claim 11, wherein the computing system is further to detect that a new test instrument has been added to the test system.

13. The system of claim 12, wherein, in response to detecting the new test instrument, the computing system is further to determine whether the new test instrument is of a same type as one of the plurality of elements. 14. The system of claim 13, wherein, in response to determining that the new element is of the same type as one of the plurality of test instruments, the computing system is configured to authorize the new test instrument for use with the test system. 15. The system of claim 14, wherein the computing system is further to compare behavior of the new test instrument with the model. 16. The system of claim 11, wherein, in response to detecting a difference between the model and behavior of one of the plurality of test instruments, the computing system is further to alert a user. 17. The system of claim 11, wherein, in response to detecting a difference between the model and behavior of one of the plurality of test instruments, the computing system is further to disable the one of the plurality of test instruments. 18. A computer program product comprising machine readable instructions stored on a non-transitory computer readable media, the computer program product comprising: code for identifying a plurality of elements of the test system that are approved for use within the test system, the test system being configured to interface with a device under test and a network; code for collecting information associated with the elements during operation of the test system, the information including inputs and outputs of the plurality of elements; code for using the information to create a model of behavior for each of the plurality of elements using a machine-learning function; and code for periodically comparing current behavior of each of the plurality of elements with the model.

19. The computer readable program product of claim 18, wherein the plurality of elements includes test instruments, which comprise one of: a digital multimeter and an oscilloscope. 20. The computer readable program product of claim 18, wherein one of the plurality of elements corresponds to an interface module.

Description:
MODEL-BASED TEST SYSTEM SECURITY Inventors Names: Heena Rathore, Abhay Samant BACKGROUND 5 [0001] Various electronic components employ intensive testing to ensure proper functionality and to determine areas of improvement for the electronic components. One example of such an electronic component is a Command Navigation Information (CNI) module. Such a module may be used, for example, in an aircraft to process command, navigation, and information data. Testing such a component involves connecting the 10 component to a test system. The component may be referred to as a device under test (DUT). The test system includes a number of test instruments that may provide various inputs to the DUT. The test instruments also measure outputs of the DUT. It may also contain control lines to command the DUT and the test instruments. [0002] Test systems may be taken to the facility of an entity that wishes to run tests on 15 a particular electronic component. The testing process may involve using large amounts of data that are stored as test configurations. The test configurations are typically stored on the enterprise level server (of the entity, for example) to be transferred to the test system during test setup. And, entities may have an interest in obtaining the raw data generated from the tests on the component. Thus, it is desirable to exchange information between the 20 test system and an entity’s enterprise server. However, many entities have highly secure networks and desire that the test system implement proper security protocols and features if the test system is to be connected to the enterprise network. BRIEF DESCRIPTION OF THE DRAWINGS 25 [0003] The present disclosure is best understood from the following detailed description when read with the accompanying figures. [0004] Fig.1 is a diagram showing an illustrative test system, according to one example of principles described herein. [0005] Fig.2 is a flowchart showing an illustrative method for testing security of a test 30 system, according to one example of principles described herein. [0006] Figs. 3A and 3B are diagrams showing creation of a model for behavior of the test system, according to one example of principles described herein. [0007] Fig.4 is a diagram showing modifications to a test system, according to one example of principles described herein. [0008] Fig.5 is a diagram showing an illustrative computing system that may perform functions described herein. DETAILED DESCRIPTION [0009] The following disclosure provides many different embodiments, or examples, for implementing different features of the invention. Specific examples of components and arrangements are described below to simplify the present disclosure. These are, of course, merely examples and are not intended to be limiting. In addition, the present disclosure may repeat reference numerals and/or letters in the various examples. This repetition is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and/or configurations discussed. [0010] As mentioned above, it may be desired that the test system implements proper security protocols and features if the test system is to be connected to the enterprise network. Connecting the test system may provide several advantages. Specifically, connecting the test system to the enterprise network may allow for quick and efficient transfer of the test configuration files to the test system. And, it allows for quick and efficient transfer of raw testing data from the test system to the enterprise server. Conventionally, test systems would only save final measurement results and transfer them to the enterprise server. This is accomplished via an external storage device, as connecting the test system to the enterprise server may present a security risk. [0011] According to principles described herein, a test system can be designed with additional security features to mitigate the risk associated with connecting the test system to the enterprise network. Specifically, before and while the test system is connected to the network, each element within the test system may be vetted to make sure it is secure and has not been compromised. Such elements may include a computing module, an interface module, and the test instruments. The test instruments may include, for example, digital multimeters, signal generators, and oscilloscopes. These vetted elements, which may be referred to as whitelisted elements, may then be used to test a DUT. During testing of the DUT, the behavior of the elements may be monitored. [0012] To monitor the behavior of the elements of the test system, data associated with each of the elements as well as the DUT itself may be used as an input to a machine learning function. For example, a machine learning function may have multiple input nodes. Each node may correspond to a different element of the test system. During testing of a particular DUT, the machine-learning function will create a model of the behavior of the test system. This model may then be used as a baseline for the correct behavior of the particular elements in the test system during further operation. For example, during subsequent testing of the DUT, the behavior of the test system may be compared with the model. This subsequent testing may be done with the test system connected to the enterprise network. If there is a deviation from the model behavior of the test system, then steps can be taken to provide additional security. For example, the test system may be disconnected from the network. Or, an administrator may be notified to examine whether the deviation may indicate a security compromise. [0013] Using techniques described herein, a test system is provided with additional security features that may notify an administrator that the test system has been compromised. With this additional security, there is less risk associated with connecting the test system to an enterprise network. Thus, the benefits of connecting to the network can be realized without additional security risk. [0014] Fig.1 is a diagram showing an illustrative test system 102. According to the present example, the test system 102 includes several elements, including a computing system 110, an interface module 112, and test instruments 114a, 114b. In one example, the elements of the test system may be mounted on a wheel-based rack. Thus, the test system 102 may be portable so as to be transported to various facilities. [0015] The test system 102 may be configured to run several types of tests on a particular electronic component, which will be referred to as a device-under-test (DUT) 104. The DUT 104 may be, for example and as mentioned above, a CNI module for an aircraft. The scope of embodiments is not limited to a CNI, as DUT 104 may also be other complex electrical components that may perform sensitive operations and thus should undergo various testing to ensure proper functionality. [0016] In some examples, it may be desirable to connect a test system to a network 106 so that data can be easily exchanged between the test system and a server 108 or other computing device. For example, an organization (i.e., a corporation, government agency, or other entity) may create a particular electronic component that they wish to have tested. That organization may hire a separate company to come and test that electronic component with a test system 102. The organization may have a series of tests to be applied to the DUT 104 by the testing system 104. Those tests may, for example, simulate real world inputs that the DUT 104 expects to experience in normal operation. The test files for these tests may relatively large. Thus, it may be time consuming to put such tests on multiple portable storage drives such as flash drives and move them to the test system 102. By connecting to the server 108 through the network 106, such test files may be transferred from the server more efficiently and without having to make copies of the test files for storage on portable drives. Furthermore, the raw data obtained by the test system during the tests may be desired by the organization. This data may be used for analysis to improve the design of the DUT 104. [0017] The server 108 may be an enterprise server associated with the organization. The server 108 may include a storage device, or may have access to a remote storage device, that stores the test files for use by the test system 102. The server 108 may also store raw data received from the test system 102. The enterprise server may be managed by network administrators at the enterprise level. It may have security built into it. [0018] The interface module 112 may include a user interface for the test system 102. The user interface may include various input devices such as a keyboard or mouse. The interface module 112 may also include various knobs, buttons, or switches for controlling various aspects of the test system 102. The interface module 112 may also include output devices such as a display. [0019] The computing system 110 of the test system 102 may include the hardware and/or software to manage the test system 102. In one example, the computing system 110 may include a processor and a memory as will be discussed in further detail in the text accompanying Fig. 5. The memory of the computing system 110 may store machine readable instructions that perform the methods described herein, such as monitoring the behavior of the test instruments 114a, 114b. [0020] The test instruments 114a, 114b may include various pieces of hardware that are designed to perform various tests on the DUT 104. In one example, the test instruments 114a, 114b may include a signal generator to generate specific signals (e.g., sinusoidal, square wave) at various amplitudes and frequencies for application to an input of the DUT 104. The test instruments 114a, 114b may also include an oscilloscope for measuring the output signals from the DUT 104. The test instruments 114a, 114b may include a digital multimeter for measuring voltage, current, impedance, or other characteristics of the DUT 104. [0021] Fig.2 is a flowchart showing an illustrative method 200 for testing security of a test system, according to one embodiment. The method 200 includes two phases 201, 203. The first phase 201 involves performing a series of tests on a DUT without being connected to a secure network. The first phase 201 is used to learn the behavior of the elements of the test system. The first phase 201 may be referred to as a training phase for a machine-learning function. The second phase 203 is used for operating the test system after the behavior of the elements have been learned. The test system may be connected to a secure environment such as an enterprise network during the phase. The second phase 203 may be referred to as a machine-learning testing phase. [0022] According to the present example, the method 200 includes a process 202 for whitelisting the elements of the test system. By whitelisting elements of the test system, only approved elements may be used. Any element that has not been whitelisted may not be allowed to operate within the test system. Before a particular element is whitelisted, it may be vetted to make sure that it does not include any malware or has been comprised in any way. [0023] The method 200 further includes a process 204 for using the elements of the test system 102 to test the DUT. For example, the test instruments may provide various test signals to the DUT. The test instruments may also take measurements from the output of the DUT. As the tests on the DUT are being performed, the information associated with each of the elements may be monitored. [0024] The method 200 further includes a process 206 for collecting input and output information from the elements during testing of the DUT. This information may then be stored on a memory associated with the computing system (e.g., 110) of the testing system. The input from an element may be, for example, the data received from the DUT. An output of an element may be, for example, a signal provided to the DUT. [0025] The method 200 further includes a process 208 for building a model of the elements using the information. This model may be built using a machine-learning function. Machine-learning functions take a set of inputs and a set of outputs and derive a model that relates the inputs to the outputs. The model can then be used to predict a particular output given a particular input. More detail on the machine-learning function will be described below with the text accompanying Fig.3. [0026] After the model has been created, the test system may be used in the second phase 203. In the second phase 203, the method 200 further includes a process 210 for connecting the test system to a secure network. [0027] The method 200 further includes a process 212 for comparing the behavior of the elements of the test system to the model. For example, as a test instrument such as test instrument 114a is applying a signal to the DUT, the computing system may monitor the behavior of the test instrument 114a to make sure that it is behaving as expected. If the test instrument 114a were compromised, the test instrument 114a may behave oddly. For example, the test instrument 114a may have a small computing system integrated therewith that may run a set of machine-readable instructions on a processor to calibrate the test instrument as desired. Thus, the test instrument 114a may have a memory array that is used for storing calibration data and software/firmware for calibration routines. It may be the case that that processing system associated with the test instrument may have its memory array overwritten with malware. In such case, the test instrument may mostly work but exhibit slightly different behavior. In other words, if the calibration data is compromised it may not be able to calibrate correctly and may exhibit odd behavior. The computing system 110 of the test system may notice this odd behavior by comparing it to the model to see if an expected output is produced. [0028] Additionally, a test instrument 114a, 114b may have identification data that identifies itself as a particular instrument to the test system. In some cases, a test instrument may be compromised by having its identity altered. For example, a signal generator may be altered to present as an oscilloscope or some other instrument. However, if such is the case, the instrument may not behave in accordance with the model. Thus, the computing system of the test system may detect this odd behavior using principles described herein. [0029] In some examples, additional instruments may be added to the test system after the initial whitelisting and learning phase 201. For example, a manufacturer of a particular test instrument may introduce a newer version of a particular test instrument with additional features. To avoid having to start over with the training process, a particular new test instrument, after being vetted, may be inserted or integrated into the test system. Furthermore, in some examples, the DUT itself may be modeled. In other words, the DUT itself may be considered one of the elements in the model and may be monitored to ensure it has not been compromised. [0030] The method 200 then includes a process 214 for comparing the behavior of the newly added element to the expected behavior of a similar element (e.g., the legacy version of the element) as defined by the model. [0031] The method 200 further includes a process 216 for taking action in response to detecting differences between the model and detected behavior. The model defines expected behavior for a particular element. If there is no departure from expected behavior, then the test system can be allowed to continue operating as normal. However, if there is a departure from expected behavior for a particular test instrument, various steps may be taken to address that detected departure. In one example, the steps taken to address the detected departure may be to disconnect or disable the connection to the network. In some examples, an administrator may be notified of the departure from ideal or expected behavior. Thus, the administrator can decide if the departure warrants any steps. In some examples, the decision regarding which steps to take may depend on the severity or degree to which the behavior of a test instrument departs from its expected behavior as indicated by the model. For example, minor deviations from the expected behavior may warrant simply logging the departure for later review. More significant deviations may warrant immediately notifying an administrator. Further significant deviations may warrant disconnecting the test system from the network and/or discontinuing testing. [0032] Fig.3A is a diagram showing creation of a model 308 for behavior of the test system, according to one embodiment. According to the present example, a machine- learning function 304 is used to create the model 308 of behavior for elements of the test system. The machine-learning model receives input data from various sources, the inputs 302a, 302b, 302c may correspond to different elements of the test system. For example, one input 302a may correspond to the interface module, one input 302b may correspond to a first test instrument (e.g., 114a), and another input 302c may correspond to a second test instrument 114b. [0033] Various types of machine-learning functions may be used. In one example, the machine-learning function 304 is based on Artificial Neural Networks (ANN), which derives its inspiration from the biology of human brain neurons. ANN models may not require regular updates and replacement. Instead, they may be capable of maintaining themselves and even learning during changing conditions. An ANN machine-learning function includes a number of nodes. The nodes may be positioned within different layers. [0034] Fig.3B is an example of a layered machine-learning function such as an ANN machine-learning function. The machine learning function may include L total layers, including an input layer, an output layer, and L-2 hidden layers. In the present example, the machine-learning function includes 4 layers, including an input layer 312 (with nodes 312a, 312b, 312c, 312d), an output layer 318 (with node 318a), and 2 (4-2) hidden layers 314, 316 (with nodes 314a, 314b, 314c and 316a, 316b, respectively). Each node in the ANN represents a mathematical function. Each function can be unique or repeated across nodes. A function is generally not repeated across different layers. [0035] Each layer can have its unique mathematical function as an activation function. The choice of activation function is an aspect of the output layer as it defines the format that predictions take. For the output layer, a logistic sigmoid activation function may be used. The sigmoid function is used for classification, as it has characteristics like nonlinearity, differentiability and the (0,1) range to give us a probability of return values. Use of hyperbolic tangential function (Tanh) provides finer gradients as the range lies in [- 1,1] compared to that of sigmoid function which is [0,1]. The activation function for hidden layers is chosen as tanh(X) which makes the data centered around 0 with high derivative values. Tanh function is also non-linear and differentiable. Its output is in the (- 1,1) range and its maximum derivative is one which allows us to pass error through the layers. [0036] The hidden layers (e.g., 314, 316) have a role in increasing the accuracy of the model. The number of hidden layers may be chosen as 2 because it can represent an arbitrary decision boundary to arbitrary accuracy with rational activation functions and can approximate any smooth mapping to any accuracy. [0037] In one example, the input nodes 312a, 312b, 312c, 312d correspond to different elements of the test system 102, including the test instruments or the DUT. For example, node 312a may correspond to a stimulus generator type of test instrument such as a signal generator. Node 312b may correspond to a measuring instrument such as an oscilloscope of DMM. The relationship between node 312a and 312b (i.e., the relationship between the signal generator and oscilloscope), may be defined by one of the hidden layer nodes, such as node 314a. [0038] While the test system is running and comparing current behavior against the model, it can be determined whether the oscilloscope is performing as expected responsively to the output of the signal generator. If the oscilloscope is not performing correctly in relation to the signal generator, it is possible that either the oscilloscope or signal generator has been compromised. By comparing each of the nodes in the model to the behavior of the real system, a specific node and thus its corresponding element may be identified as being potentially compromised. The system may then take action accordingly as described above. [0039] Fig.4 is a diagram showing modifications to a test system 102, according to one embodiment. As described above, it may be desirable to upgrade test instruments or add new test instruments. According to the present example, test instrument 402a is an upgraded or newer version of rest instrument 402b. Thus, as described in process 210 above, the behavior of test instrument 402a may be compared to the model for test instrument 114b. In some examples, a new test instrument, such as test instrument 402b may not have a previous counterpart. Thus, such a new test instrument 402b may not be whitelisted until it undergoes a training phase to build a model of its behavior. [0040] Fig.5 is a diagram showing an illustrative computing system that may perform functions described herein, such as the functions described above in the text accompanying Fig.2, according to one embodiment. In other words, the computing system 500 may be used to perform the functions associated with the test system 102. Other functions described herein may also be performed by computing systems such as computing system 500. According to certain illustrative examples, the computing system 500 includes a memory 504 which may include software 506 and a data store 508. The processing system 500 also includes a processor 510, a network interface 514, and a user interface 512. [0041] The memory 504 may be one of several different types of memory. Some types of memory, such as solid-state drives, are designed for storage. These types of memory typically have large storage volume but relatively slow performance. Other types of memory, such as those used for Random Access Memory (RAM), are optimized for speed and are often referred to as “working memory.” The various types of memory may store information in the form of software 506 and data in the data store 508. [0042] The computing system 500 also includes a processor 510 for executing the software 506 and using or updating the data 508 stored in memory 504. The software 506 may include an operating system and any other software applications a user may wish to install. In some examples, the computing system 500 may be associated with a user. In such case, the software 506 may be an application to render web content, such as a browser. The software 506 may include machine readable instructions of a computer program product that when executed, perform the functions described above in accordance with the text accompanying Fig.2. [0043] The user interface 512 may include a number of input devices such as a mouse, touchpad, or touchscreen that allow the user to interact with the computing system 500. The user interface 512 may also include a number of different types of output devices such as a monitor or a touchscreen. The user interface allows the user to interact with the processing system 500 in a manner as described above. [0044] The network interface 514 may include hardware and software that allows the processing system 500 to communicate with other processing systems over a network 516. The network interface 514 may be designed to communicate with the network 516 through hardwire media such as Ethernet, coaxial, fiber-optic, etc. The network interface 514 may also be designed to communicate with the network 516 using wireless technologies. [0045] Some examples of processing systems described herein may include non- transitory, tangible, machine readable media that include executable code that when run by one or more processors may cause the one or more processors to perform the processes of methods as described above. Some common forms of machine-readable media that may include the processes of methods are, for example, floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and/or any other medium from which a processor or computer is adapted to read. [0046] The foregoing outlines features of several embodiments so that those skilled in the art may better understand the aspects of the present disclosure. Those skilled in the art should appreciate that they may readily use the present disclosure as a basis for designing or modifying other processes and structures for carrying out the same purposes and/or achieving the same advantages of the embodiments introduced herein. Those skilled in the art should also realize that such equivalent constructions do not depart from the spirit and scope of the present disclosure, and that they may make various changes, substitutions, and alterations herein without departing from the spirit and scope of the present disclosure.