USING TIME-CORRELATED VIDEO AND NETWORK ACTIVITY

TECHNICAL FIELD

The present disclosure relates to the field of application debugging, more particularly, to the mobile application debugging and profiling.

PRIOR ART

Modern applications, would they be native or browser-based apps have their logic distributed across code running on a device such as mobile phone, tablet or regular personal computer, multiple network servers and numerous 3rd party services which provide authentication, tracking or additional infrastructure mechanism.

As a result, troubleshooting of applications during development and QA process is very complicated, as people who catch a bug need to elaborate what had happened to the user on the screen, and possibly try to re-create it, which takes time and often might not be possible due to the root cause being very often in network responses from one or many mentioned services.

While capturing network traffic and screen is technically possible, it requires complicated setup, often involving multiple computers, and results are inherently uncorrected, making capture time consuming, error-prone and therefore impractical.

In order to capture network traffic "wireshark" or "tcpdump" network capturing and filtering tools could be used, which captures network packets from given network interface. However, due to security and sandboxing requirements, this tool is not operational on mobile devices, hence an environment incorporating a separate computer which would bypass network traffic from mobile device is required, which is a relatively labor-intensive task to accomplish. .

SUMMARY

In some embodiments the method of mobile device screen, network activity, profiling information capture, sharing and analysis comprising (Fig.7): capturing profiling information about mobile application under test on a mobile device and binding said profiling information to said mobile device's system time; recording said mobile application screen, encoding and storing said video stream to said mobile device system's binding time into first cyclic buffer; redirecting said mobile application network traffic from said mobile device to a remote computer system, recording network activity of the mobile application on said remote computer system into second cyclic buffer and binding said network activity to said remote computer system' s time; sending said video stream stored in said first cyclic buffer and said profiling information to said remote computer system; preparing an interactive report on said remote computer system contains system time correlated said profiling information, said video stream and information about said network activity of said mobile application.

In some embodiments, remote computer system includes network capture and network analysis functionality.

In some embodiments, at least two remote computer systems are used where the first computer system is used for network traffic data collection (capturing) and the other for analysis and generating reports.

In some embodiments mobile application may consist of multiple processes, including cases where certain functions are achieved by temporarily switching to secondary application, providing the said functions such as social network login. In some embodiments mobile application screen is recorded on the mobile device and the video stream is encoded using a hardware video codec of the said mobile device.

In some embodiments profiling information comprising: CPU load, RAM allocation, bandwidth utilization and other OS kernel exposed metrics, system and application processes logs.

In some embodiments the video stream is encoded using video codec such as H264, or H265, or WebP or Bink.

In some embodiments the first cyclic buffer is stored in said mobile device

RAM.

In some embodiments the mobile application screen is recorded on the computer system using the operating system installed on the computer, using either digital signal interface such as HDMI or video compressed stream interface such as AirPlay.

In some embodiments first cyclic buffer is stored in the computer system

RAM.

In some embodiments the video stream and profiling information about the mobile application is sent from said mobile device to said remote computer system;

In some embodiments the video stream is sent from the computer system to the remote computer system, and the profiling information about the mobile application is sent from the said mobile device to said remote computer system;

In some embodiments, cyclic buffer may be logically mapped into a nonvolatile storage such as NAND memory.

In some embodiments the method of mobile device screen, network activity, profiling information capture, sharing and analysis comprising (Fig.8): capturing profiling information about mobile application under test on a mobile device and binding said profiling information to said mobile device's system time; recording said mobile application screen, encoding and storing said video stream to said mobile device system' s binding time into first cyclic buffer; redirecting said tested mobile application network traffic from said mobile device to said remote computer system in order to record network activity of the said mobile application on said remote computer system into second cyclic buffer and binding said network activity to said remote computer system' s time; sending said video stream stored in said cyclic buffer, said profiling information to said remote computer system to prepare an interactive report on said remote computer system contains system time correlated said video stream, said profiling information, and said information about mobile application network activity.

In some embodiments the method of mobile device screen, network activity, profiling information capture, sharing and analysis comprising: capturing profiling information about mobile application under test on a mobile device and binding said profiling information to said mobile device's system time; recording said mobile application screen, encoding and storing said video stream to said mobile device system' s binding time into first cyclic buffer; recording network activity of the mobile application on said mobile application into second cyclic buffer and binding said network activity to said mobile application system's time; preparing an interactive report on said mobile application contains system time correlated said profiling information, said video stream and information about said network activity of said mobile application.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig.1 - embodiment where mobile device and remote server are used. Fig. 2 - profiling application and application under test. Fig. 3 - cyclic buffer. Fig.4 - embodiment where computer system, mobile device and remote server are used.

Fig.5 - redirection of a network traffic via Capture Server.

Fig.6 - interactive report.

Fig. 7, 8 - embodiments of described method

DETAILED DESCRIPTION

Various embodiments of the technical solution are discussed in detail below. While specific implementations are discussed, it should be understood that this is done for illustration purposes only. A person with ordinary skill in the relevant art will recognize that other components and configurations may be used without parting from the spirit and scope of the invention.

This technical solution in its different embodiments can be implemented as a computer method, in the form of a system or a machine -readable medium containing instructions for using the said method.

In this technical solution the system means a computer system, PC (personal computer), CNC (computer numeric control), PLC (programmable logic controller), computerized control systems and any other devices that can perform defined, clearly determined sequence of operations (actions, instructions).

Command processing device means an electronic unit or integral circuit (microprocessor) that executes machine instructions (programs).

Command processing device reads and executes machine instructions (programs) from one or more data storage devices. Data storage devices include but are not limited to hard drives (HDD), flash memory, ROM (read-only memory), solid-state drives (SSD), optical drives. Program means a sequence of instructions intended for execution by computer control device or command processing devices.

Non-volatile memory, nonvolatile memory, NVM or non-volatile storage is computer memory that can retrieve stored information even after having been power cycled. Examples of non-volatile memory include read-only memory, flash memory, ferroelectric RAM (F-RAM), most types of magnetic computer storage devices (e.g. hard disks, floppy disks, and magnetic tape), optical discs.

In cryptography and computer security, a root certificate is an unsigned or a self-signed public key certificate that identifies the Root Certificate Authority (CA).

The method of mobile device screen, network activity, profiling information capture, sharing and analysis comprising:

Before performing the method, a mobile device user must install mobile profiling application 201 (Fig. 2) on the mobile device 101 (Fig. 1). A CA Root self-signed root certificate will be installed during installation on the mobile device (in some embodiments it's not required). A user must then launch profiling application 201 to connect to the remote computer system 103 via network 102 (Fig. l) for subsequent redirection of network traffic of the tested application 202 (Fig.2). A user should then launch the tested application 202 and turn profiling application 201 on. capturing profiling information about mobile application under test on a mobile device and binding said profiling information to said mobile device ^' s system time;

To perform profiling, application 201 must record certain profiling information (utilization of memory, CPU etc.) of application 202. In the process of operation, application 202 can create system log files. In some embodiments profiling information comprising: CPU load, RAM allocation, bandwidth utilization and other OS kernel exposed metrics, system and application processes logs recording said mobile application screen, encoding and storing said video stream to said mobile device system ^' s binding time into first cyclic buffer;

After starting the video recording of a mobile application under test 202 (Fig.2), mobile device screen capture is performed by capturing mobile device 101 video buffer and coding it into video motion codec such as H264 by mobile device CPU hardware acceleration video coding functionality integrated in it and timestamped

In some embodiments, a video motion codec is implemented as separate logical core to the CPU.

In some embodiments, a video motion codec is implemented on CPU as a subprogram or a library.

Received coded video stream packets are continuously stored into cyclic buffer (Fig.3) according to such packet time stamp.

Buffer possesses fixed size and is stored in RAM to eliminate memory fragmentation. It maintains an index between timestamps and elementary video stream packet locations within buffer, which enables rapid playback, seek and video exporting.

In some embodiments, cyclic buffer may be logically mapped into a nonvolatile storage such as NAND memory. redirecting said mobile application network traffic from said mobile device to a remote computer system, recording network activity of the mobile application on said remote computer system into second cyclic buffer and binding said network activity to said remote computer system ^' s time

Remote computer system 103 (Fig. 1) receives application under test 202 network traffic (Fig.2), analyzes and stores it in a circular buffer allocated in said remote computer system' s 103 RAM.

In some embodiments, cyclic buffer may be logically mapped into nonvolatile storage such as HDD or SDD.

In order to analyze the mobile application traffic, the following technique is implemented. It should be noted, that because modern smartphones are based on fully featured operating systems such as Linux (Android) and BSD (iOS), the described technique could also be implemented on the mobile device itself without much modification, which may become practical in the future when mobile device processing power and available RAM would increase.

In some embodiments a separate capture and network traffic analysis server is used.

Traffic Interception

In order to divert all network (TCP, IP and UDP) traffic from mobile handsets, utilizing one of virtual private network protocols, such as IPSEC, Open VPN, PPP, L2TP and others, widely supported on majority of modern mobile devices across various platforms including Android, iOS, Windows Phone, and others, is a natural choice.

By either using an OS -provided API for bespoke VPN tunnel, utilizing a 3rd party application enabling VPN access, or OS -provided configuration mechanism, we establish a network tunnel between mobile device 501 (Fig. l) and capture server 502, which would make an operating system to send all network packets to capture server. In some embodiments, instead of utilizing a VPN tunnelling, a network mobile device is connected to may be configured in such manner that all network traffic would naturally be routed thru capture server, i.e. declaring it as default router for mobile devices.

In some embodiments, an application under test may interact with remote servers using specially designed hostnames. This method requires intercepting traffic to be transmitted in an encrypted manner, over TLS (HTTPS) protocol, and implies the fact that server name transmission is part of TLS handshake protocol. One would need to configure a Domain Name Service (DNS) special domain which would resolve host name queries into capture server address. Then, if application developers would change in the application code the hostname of target server of api.server.com into api-server-com.special.domain.com where special.domain.com is a capture server special domain zone. When receiving such requests, capture server naturally possesses the destination server by parsing TLS server name during TLS handshake process.

Traffic Analysis

On capture server, we could configure operating system networking stack in such a manner that capture server would become an internet gateway for the mobile device, by means of performing Network Address Translation (NAT) for the TCP and UDP connections made by mobile device. This procedure is the same for cases when mobile device is connecting via VPN or it is a network setup to route network data thru this server.

In order to analyze network traffic from mobile device, we could utilize one of the standard methods, by manipulating operating system network stack configuration: a) capture all packets passing thru outbound network interface of capture host on low-level, physical network interface level (i.e. Ethernet). In order to assemble the data into meaningful client-server conversation flow, an intercepting application would utilize knowledge of underlying transport protocols functioning, such as TCP sequence numbers which explicitly order individual packets sent over network. b) make an operating system transparently proxy all outbound connections to our application, which would then initiate another connection towards destination and perform effectively "proxying" of data; this latter method is called "transparent proxying".

There are multiple ways one may capture network traffic from a mobile handset, some may be preferred in specific environments.

Secured Data Interception

Mobile application under test 501 (Fig.5) initiates a TCP or UDP connection by means of stateful protocol, such as HTTPS to target server, using standard OS means. In order to fulfill application request, mobile operating system has to issue at least the following commands:

1. Perform domain name to IP address resolution of a target server

2. Open TCP or UDP connection to a well-known port, as defined by the protocol standard

An operating system of capture server 502 (Fig.5), acting as a networking gateway for the mobile device under test 501, would be configured to terminate the above mentioned connections to local TCP or UDP ports, served by Capture Server 502 (Fig.5). Capture Server 502 would use the acquired knowledge of prior DNS (from DNS-server 505) resolution requests (destination server address response, mapped to domain name) and IP payload information about original destination address, would open another connection to the destination server 506, impersonating the mobile application under test 501, and would proxy incoming data from 501 to destination server 506, while being able to perform data recording and time stamping.

In case Transport Layer Security would be required by application protocol in question, Capture Server 502 would, by either knowing the prior DNS resolution requests from mobile device 501 or by utilizing Server Name fields in the TLS protocol, or by performing a TLS handshake in advance with destination server IP and learning the domains it serves, would perform TLS handshake with mobile device 501 by providing a server certificate, signed by self-signed CA root. As CA root is installed as trusted root CA certificate source on mobile device, an mobile application under test 501 would normally complete TLS handshake assuming it is directly communicating with destination server 506.

In some embodiments, a method of dynamic hostnames and special DNS zone may be utilized for the same purpose without need to install a self-signed root CA certificate to mobile device under test.

Therefore, Capture Server operating system 503 (Fig.5) is configured to fulfill the following parameters:

1. Accept VPN connections from remote clients (if VPN is used for network capture), OR

2. Route the traffic of client subnet / IP addresses to public network, if transparent routing is used as a method for network capture

3. Perform 'port forwarding' or "proxying" of all client traffic for supported protocol ports to local ports, served by Capture Server. I.e. all requests to port 80 (HTTP) would get terminated to port 2080 on Capture Server machine.

Data Collection In order to perform data collection, we could utilize either a continuous recording to permanent media such as file-system backed by hard drives (HDD) or solid state drives (SSD), or keep it in volatile memory. In both cases, it might be beneficial to periodically purge older data which is likely to be of no interest to the client, in order to maintain a continuous cyclic buffer.

As capture server communicates with the screen capturing software, it would maintain at least the same time-frame for network data as screen buffer. Each client therefore is enjoying its own resource constraints and is unaffected by other client's collection activities happening on this server.

Capture server may employ some advanced methods to better utilize constrained environments, such as deciding to only keep part of the request data or to store larger request data to non-volatile memory. As an example, only up to certain amount of data being kept per individual request, which may be of help while debugging i.e. video-centric applications where one is interested in API calls and not really interested in caching streamed video data in full. Alternatively a longer requests might be stored to nonvolatile memory outright, without affecting the RAM buffer.

Data Analysis

Application level protocols, such as HTTP, Websocket, DNS, and others are formally defined by numerous standards, making it feasible to both automatically detect and parse their contents into human readable form.

In some embodiments captured network data 504 decoded in accordance to the RFC and other specifications.

In order to correlate network protocol events with other data collected on mobile device (such as screen capture or CPU load sampling, memory usage sampling or system log entries capture) we assign a timestamp to both individual network packets received and upper level parsed structures which are part of the application-level protocol, such as timestamp an HTTP request was fully sent to server and timestamp when response to this request has been obtained. An accurate and thorough timestamping is crucial for correlation analysis such as understanding the delay from server response reception to screen refresh in mobile application, which allows to discriminate between network-specific issues and application data processing ones.

Such a thorough timestamping from application level, parsed form of the protocol down to individual packets allow us to perform numerous quantitative analysis in order to detect various anomalies or malfunctions automatically via a set of rules, possibly extendable by the user. Examples of such rules might include detection of whether there is an HTTP pipelining in place between client and server by detecting multiplexed overlapping requests / responses over single TCP channel, identifying tolerance of client to network delays, identifying bottlenecks in Transport Layer Security handshakes, etc.

Timestamping

As there could be multiple individual devices involved in overall data capture (i.e. mobile handset screen being captured from desktop PC, device system log being fetched directly from device include device timestamps, and network capture on a dedicated capture server - possibly in the cloud, it is important to maintain timestamp accuracy between them. We assume all parties maintain a correct internal clock using network time protocol (NTP).

In order to perform correction between the timestamps observable on i.e. capture server to the point in time where this same packet was sent or received on a mobile device, we measure the communication link between the said mobile device and said remote system latency - i.e. by sending ICMP Ping packets between them. The derived latency value could later on be applied to capture -server observable timestamps to convert them into mobile device-observable timestamps. sending said video stream stored in said first cyclic buffer and said profiling information to said remote computer system

Once the required testing of application 202 (Fig. 2) is completed, the user must activate application 201 and select the video clip of the recorded application screen to be exported. Application 201 will then send the user-selected video clip, log files, and profiling information (i.e. CPU, RAM) to the remote computer system 103 (Fig. 1). preparing an interactive report on said remote computer system contains system time correlated said profiling information, said video stream and information about said network activity of said mobile application

Interactive report consists of: a) video recording of mobile handset screen, b) network requests view, c) system logs, d) any other collectable metrics such as RAM, CPU usage.

In some embodiments, network requests could be presented in a table format.

In some embodiments, network requests could be presented in form of Gantt chart, with optional grouping by various parameters such as destination hostname, TCP connection, transmitted content type and other meaningful parameters.

The interactive report shown in Fig. 6 consists of the mobile device time- synchronized screen capture 601 (Fig. 6), network request filter 602(Fig.6), request timeline 603 (Fig.6) in form of Gantt chart where network requests 607 (Fig.7) such as individual transactions within HTTP protocol are aligned horizontally by time, and grouped first by server name or address 606 (Fig.6), and then by logical data stream such as TCP connection, information from log files, and other metrics.

When starting the video stream, it is possible to view the network traffic information in request timeline 603, along with basic data on ongoing processes containing information about protocols, sources, time, date, etc. The information is arranged as a table with certain fragments color highlighted. An extended information about individual request 604 may be obtained by selecting any individual request in the table. An accurate positioning for a specific point in time may be achieved by either manipulating a time control 605 which seeks video 601 and request timeline 603 to render said point in time.

These and other aspects of the present invention will become apparent to those skilled in the art by a review of the preceding detailed description. Although a number of salient features of the present invention have been described above, the invention is capable of other embodiments and of being practiced and carried out in various ways that would be apparent to one of ordinary skill in the art after reading the disclosed invention. Therefore, the above description should not be considered to be exclusive of these other embodiments. Also, it is to be understood that the phraseology and terminology employed herein are for the purposes of description and should not be regarded as limiting.