Cross-reference

[0001] The present application claims priority to Russian Patent Application No 2015116942, filed May 06, 2015, entitled "SYSTEM AND METHOD FOR TESTING WEB PAGES" the entirety of which is incorporated herein.

Background

[0002] Websites have many functions and can be used in various contexts and ways. For example, a website may be a personal website, an e-commercial website, a government website, or a non-profit organization website. Here, e-commerce generally refers the exchanging of products and services by consumers or businesses via the Internet. In a familiar example, a consumer is "shopping online" by searching and selecting products or services through online catalogues or inventories, then using a credit card to complete the purchase. The Internet greatly facilitates conducting e-commerce, in part, because it uses standardized techniques for exchanging information.

[0003] Many standards have been established for exchanging information over the Internet, such as electronic mail and the World Wide Web ("WWW"). The WWW service may allow a server computer system (i.e., web server or website) to send various information of web pages to a remote client computer system for display thereon. Each resource (e.g., each client computing device or web page) of the WWW is uniquely identifiable by a Uniform Resource Locator ("URL"). To view a specific web page, a client computer system may specify a URL for that web page in, e.g., a HyperText Transfer Protocol ("HTTP") request. The request is subsequently forwarded to a web server that supports that web page. In response, the web server sends the requested web page to the client computer system for displaying on a browser, which is typically a special-purpose application program installed on the client computer system that effects the requesting of web pages and the displaying of web pages.

[0004] Currently, web pages are generally defined using HyperText Markup Language ("HTML"). HTML may provide a standard set of tags that define how a web page is to be displayed. In one example, when a user indicates to a browser associated with a client computer system to display a web page, the browser sends a request to a server computer system requesting a transfer to the client computer system a HTML document that defines the web page. The HTML document may contain various tags that control the displaying of text, graphics, controls, and URLs of other web pages available on that server computer system or other server computer systems. Web pages may be generally defined by static HTML documents that are created and stored at a server computer system and then retrieved whenever requested by one or more client computer systems.

[0005] In tracking changes of web page related information and data, and online user activity, various tags of web pages may be activated at any time during a web page loading. In addition, online publishers or website owners may work with third parties that provide tags of various types and for various purposes. However, due to constant dynamic and voluminous changes to information and data associated with web page tags, checking data validity and consistence of a web site can be time-consuming, laborious, and expensive because of the number of web pages that may be used by the website, the complex interrelation of web pages, and the high level of skill needed by a web page developer. As the number of tags increases, delayed loading and processing of web pages, and inconsistent, erroneous or inoperability of web pages may significantly impact user experience, thereby degrading the effectiveness of online advertising or marketing efforts. Moreover, online publishers or website owners may face a number of other issues with managing the tags of various types and purposes (including obsolete tags), supporting tags with new or changing features, and/or improving efficiency in managing ever-increasing numbers of tags which may be of different types. Therefore, there is a need for efficient web page testing.

Summary

[0006] Disclosed are systems, methods and computer program products for testing web pages. In one example aspect, a system may be configured to perform automatic testing of the various web pages hosted by a web server (e.g., e-commerce web server) in order to verify correctness of these web pages and to detect any errors in the layout and/or content of the web pages. The testing may be periodically performed on all hosted web pages, only on newly added web pages, or only on newly updated web pages (e.g., web pages that have updated content). The testing of web pages may be performed using sets of test rules and templates, which are automatically generated by the system based on analysis of previously verified web pages. For example, the system may be configured to automatically analyze HTML elements of a plurality of verified web pages of a same category of products/goods/services and generate a set of test template identifying common web page elements and a set of test rules specifying relationships between web page elements. Specifically, the test rules and templates may cover elements of the web pages, which are similar, or different, or same elements with changeable content associated with these elements. These sets of test rules and templates are then used to test other web pages of the same category to verify that they operate properly, thereby facilitating accuracy and consistency of the website with improved user experience.

[0007] In one example, a method for testing web pages, the method comprising: obtaining two or more reference web pages, each web page comprising a plurality of web page elements; analyzing, by a hardware processor, the web page elements of the two or more reference web pages to identify similarities and relationships among the web page elements; generating, by the hardware processor, a set of test templates identifying at least a plurality of elements common to the two or more reference web pages; generating, by the hardware processor, a set of test rules specifying at least logical relationships between web page elements of each reference web page; and verifying, by the hardware processor, compliance of a test web page with the set of test template and the set of test rules.

[0008] In another example, analyzing the web page elements of the two or more reference web pages to identify similarities and relationships among the web page elements further comprises: generating one or more regular expressions for one or more web page elements; and using the regular expressions to identify similar elements among the two or more reference web pages.

[0009] In another example, a regular expression may comprise a plurality of characters and operators for defining a set of one or more strings in a form of an expression or a pattern.

[0010] In another example, a regular expression may comprise the longest common prefix, longest common suffix and longest common substring of text content of a web page element.

[0011] In another example, a regular expression may be configured to match a selected percentage of text content of a web page element.

[0012] In another example, the method may further include identifying a category of the two or more reference web pages, and associating the generated set of test rules and set of test templates with the category. [0013] In another example, verifying compliance of a test web page with the set of test template and the set of test rules may further comprise: identifying a category of the test web page; identifying a set of test rules and a set of test templates associated with the category of the test web page; and applying the set of test rules and the set of test templates to the test web page.

[0014] In another example, identifying a set of test rules and a set of test templates associated with the category may further comprise: identifying in a hierarchical tree structure of a plurality of nodes, wherein each node is associated with a set of test rules and a set of test templates, at least one set of test rules and a set of test templates associated with the category of the test web page; traversing the hierarchical tree structure from the identified node to the root node, and identifying sets of test rules and sets of test templates associated with each traversed node; retrieving a plurality of identified sets of test rules and sets of test templates associated with each traversed node to generate a complete set of test rules and test templates; and using the complete set of test rules and test templates for verifying the test web page.

[0015] In another example, a system for testing web pages, comprising: at least one hardware processor configured to: obtain two or more reference web pages, each web page comprising a plurality of web page elements; analyze the web page elements of the two or more reference web pages to identify similarities and relationships among the web page elements; generate a set of test templates identifying at least a plurality of elements common to the two or more reference web pages; generate a set of test rules specifying at least logical relationships between web page elements of each reference web page; and verify compliance of a test web page with the set of test template and the set of test rules.

[0016] In another example, a non-transitory computer readable medium comprising computer executable instructions for testing of web pages includes instructions for: obtaining two or more reference web pages, each web page comprising a plurality of web page elements; analyzing the web page elements of the two or more reference web pages to identify similarities and relationships among the web page elements; generating a set of test templates identifying at least a plurality of elements common to the two or more reference web pages; generating a set of test rules specifying at least logical relationships between web page elements of each reference web page; and verifying compliance of a test web page with the set of test template and the set of test rules. [0017] The above simplified summary of example aspects serves to provide a basic understanding of the invention. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all aspects of the invention. Its sole purpose is to present one or more aspects in a simplified form as a prelude to the more detailed description of the invention that follows. To the accomplishment of the foregoing, the one or more aspects of the invention include the features described and particularly pointed out in the claims.

Brief Description of the Drawings

[0018] The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate one or more example aspects of the invention and, together with the detailed description, serve to explain their principles and implementations.

[0019] Fig. 1 is a diagram illustrating an example network architecture for testing of web pages.

[0020] Fig. 2 is a block diagram illustrating an example system for testing of web pages.

[0021] Fig. 3 is a diagram illustrating an example aspect of analyzing two pre-verified web pages to determine test rules and templates for testing of web pages.

[0022] Fig. 4 shows an example hierarchical tree structure for storing and managing test rules and templates for testing of web pages.

[0023] Fig. 5 is a flow diagram illustrating an example method for testing of web pages.

[0024] Fig. 6 is a flow diagram illustrating another example method for testing of web pages.

[0025] Fig. 7 is a diagram illustrating an example aspect of a general-purpose computer system on which are implemented the systems and methods for testing of web pages.

Detailed Description

[0026] Example aspects of the present invention are described herein in the context of systems, methods and computer program products for testing of web pages. Those of ordinary skill in the art will realize that the following description is illustrative only and is not intended to be in any way limiting. Other aspects will readily suggest themselves to those skilled in the art having the benefit of this disclosure. Reference will now be made in detail to implementations of the example aspects as illustrated in the accompanying drawings. The same reference indicators will be used to the extent possible throughout the drawings and the following description to refer to the same items.

[0027] Fig. 1 is a diagram depicting that an example multi-tier web-based system 100 employing a client-server architecture may be deployed to implement the disclosed systems and methods for testing web pages. The networked system 100 of, e.g., an e-commerce website, a social media website, a service-provider website (financial services, travel services), government websites, or non-profit organization or institute websites may include a client side comprising various electronic user devices 102, such as a mobile device, a desktop computer, a laptop, etc. In one aspect, a device 102 may include at least an application module 104 (including, e.g., third party applications using an Application Program Interface (API)) and a web browser 106 (e.g., Firefox, Internet Explorer, Google Chrome, Opera, and Safari) for accessing a network-based marketplace or publication systems on a server side via a network 108, such as the Internet, via a wired or wireless connection. In the context of e-commerce, server-side functionalities may include, e.g., providing items or products for sale over the Internet from various remote sellers via the network 108 to the devices 102.

[0028] In one aspect, the application module on device 102 may also include a third party application having access to the networked system 100 via a programmatic interface provided by API server 112 or third party server 120. For example, the API server 112 or third party server 120 may, utilizing information retrieved from the application server(s) 114, support one or more features or functions on a website hosted by one or more third parties to each device 102. The third party website(s) may provide one or more promotional, marketplace, or payment functions that are supported by the relevant applications of the networked system 100. It should be appreciated that, in addition to primarily access the network 100, the application module 104 together with the web browser 106 on each device 102 may also be used to access information provided by various web servers in private networks or files in file systems executing on each device.

[0029] The term "module" as used herein means a real-world device, apparatus, or arrangement of modules implemented using hardware, such as by an application specific integrated circuit (ASIC) or field-programmable gate array (FPGA), for example, or as a combination of hardware and software, such as by a microprocessor system and a set of instructions to implement the module's functionality, which (while being executed) transform the microprocessor system into a special-purpose device. A module can also be implemented as a combination of the two, with certain functions facilitated by hardware alone, and other functions facilitated by a combination of hardware and software. In certain implementations, at least a portion, and in some cases, all, of a module can be executed on the processor of a general purpose computer (such as the one described in greater detail in Fig. 6 below). Accordingly, each module can be realized in a variety of suitable configurations, and should not be limited to any particular implementation exemplified herein.

[0030] As shown in Fig. 1, the server side of network 100 may employ an N-tier application architecture (N is a selected integer) including, e.g., one or more web servers 110, Application Programming Interface (API) servers 112, application servers 114, database servers 116 for connecting with and managing various databases 118. Such an N-tier server architecture may not only allow a user interface on each device 102 (e.g., a thin client configured as a computer or a computer program for data exchange and presentation while depending heavily on another computer (its server) to fulfill its computational roles), logic (e.g., rules for data transformation and management of web page related information in accordance with aspects of the present application disclosed below) to be implemented on the server side, computer data storage and data access to be developed and maintained as independent modules on same or separate platforms (e.g., database server 116 and database(s) 118), but also allow any of the N tiers to be upgraded or replaced independently in response to changes in system development, management, and usage requirements or technologies.

[0031] A front end web server 110 may host one or more services, such as mobile commerce, electronic funds transfer, supply chain management, Internet marketing, online transaction processing, electronic data interchange (EDI), inventory management systems, and automated data collection systems to various user devices 102. In one aspect, the front end web server 110 may be configured to process (e.g., receiving and responding to) HTTP requests from each user device 102 for HTML pages, provide security service via, e.g., a secure sockets layer, by verifying username and password received from each user device 102, and process certificates and private/public key information required for authentication, credit card processing and other secure information. The web server 110 may also be configured to facilitate file transfer from server to server, and provide indexing of site contents and keyword search capability of the site contents. Moreover, the web server 110 may log files of user visits including time, duration and sources, and provide email services including sending, receiving and storing email messages. With respect to the served contents (static and dynamic), the web server 110 may also be configured to provide site management tools including calculating and displaying key site statistics (e.g., unique visitors, page requests, and origin of HTTP requests).

[0032] In addition, the web page server 110 may store HTML, or other code required for generating a web page with all the header and body information as a complete web page or template. With the aid of dynamic page generation tools, the web server 110 may store the contents of one or more web pages as objects in a database, rather than hard-coding the contents in its HTML. The generated web pages include the HTML code that controls the appearance of the web page and other code (e.g., paired delimiters) that controls web page semantics. During the processing of a particular web page or template, the web server 110 may locate paired delimiters and interpret each character string therebetween. These character strings may define, e.g., a dynamic tag. Although web pages are typically coded in HTML, web pages may be created or coded using other formats using XML, VoiceXML, or other markup languages.

[0033] The web server 110, API server 112 and third party server 120 may be directly coupled to, or may use programmatic web interfaces to connect with one or more application server(s) 114, which host one or more applications to provide specific functionalities required of a website. The application server(s) 114 may be coupled to one or more back-end database servers 116 that facilitate access to one or more database(s) 118. As such, the application server(s) 114 may be a middleware server program that allows front-end, browser-based clients to retrieve data from back-end databases and legacy systems such as large server computers or mainframes. In one aspect, web server 110 and application server(s) 114 may be merged to collectively expose back-end business logic through various protocols, including HTTP, and manage security, transaction processing, resource pooling, and messaging. However, in extensive e-commerce operations where load balancing is a concern, multiple application servers and several separate web servers may be implemented in accordance with an N-tier network architecture. [0034] The application server(s) 114 may implement various applications (not shown) to provide crucial functions and services to users who access the networked system 100. In the context of e-commerce, the application server(s) 114 may coordinate with database servers 116 and databases(s) 118 to store and manage customer and product descriptions and price information, provide an online catalog/database display of product description and prices, store and deliver streaming media content (text, audio or video), provide online transaction processing including accepting orders, processing payments, conducting online auctions, and linking with other marketplaces associated with the web server 110, API server 112 and third party server 120, maintain web-enabled advertisements including customized and personalized display of advertisement elements based on consumer behavior and characteristics, create/store mailing lists and manage Internet e-mail marketing campaigns. Whether merged together or implemented separately, the application server(s) 114 may also be configured to monitor and control access to main web server 110 and implement firewall protection.

[0035] It should be appreciated that, although the system 100 in Fig. 1 has been shown to employ a client-server architecture, alternative network architectures may be employed including a distributed, or peer-to-peer, architecture system.

[0036] In one example aspect, system 100 may be configured to perform automatic testing of the various web pages hosted by the web server 110 and stored in the database 118 in order to verify correctness of these web pages and to detect any errors in the layout and/or content of the web pages. The testing may be periodically performed on all web pages, only on newly added web pages or only newly updated web pages (e.g., web pages that have updated content). The testing of web pages may be performed using sets of test rules and templates, which are automatically generated by the system based on analysis of previously verified web pages. For example, the system 100 is configured to automatically analyze HTML elements of a plurality of verified web pages of a same category of products/goods/services and generate test templates identifying common web page elements and test rules specifying relationships between web page elements. Specifically, the rules and templates may cover elements of the web pages, which are the same and substantial similar, or different, or same elements with changeable content associated with these elements. These test rules and templates are then used to test other web pages of the same category to verify that they operate properly, thereby facilitating accuracy and consistency of the website with improved user experience.

[0037] Fig. 2 depicts an example system 200 for testing web pages that may be implemented in connection with the network architecture 100. In one example aspect, an application server 210 may be configured to generate sets of test rules and templates for testing and verification of web pages hosted by a web server 208. To that end, the application server 210 may include a web page selector 218 configured to request/retrieve via an interface 214 from a web page database 228 of the web server 208 two or more pre-verified reference web pages of a website (e.g., an e-commerce website) hosted by the web server 208. The web server 208 may be operated, for example, by an online retailer/service. Moreover, depending upon the load balancing among different server operations, the web server 208 and application server 210 may be merged or implemented separately in different aspects.

[0038] In one example aspect, the requested pre-verified reference web pages should be associated with the same category of goods or services. This is necessary because web pages of the same category have similar layout and contain similar text content, while web page of different categories may have different layout and contain different text content. Thus, for example, all web pages that sell digital cameras will have similar content, such as type of optical sensor technology, type of lens, image resolution, digital and analog zoom information, shutter speed, etc. In contrast, web pages that sell shoes will have different layout and content, such as shoe color selection window, shoe size selection window, multiple images of the product in different colors, etc. However, it should be noted that despite many differences, web pages of different categories of products will also have some similar content, such as product price, product description, customer reviews, related product listing, shipping information, etc.

[0039] In response to the request from the application server 210, the web server 208 may identify two pre-verified reference web pages of the same category (e.g., digital cameras) using for example URLs of these web pages. Thus, for example, a URL of one webpage may be "market.yandex.ru/search/goods/cameras/cameral.htm" and a URL of another similar webpage may be "market.yandex.ru/search/goods/cameras/camera2.htm". Analysis of the URL of these web pages by the web server 208 indicates that they are both associated with products in the same product category - "cameras". Since these two pre-verified reference web pages are associated with the same category (e.g., category of products), then the layout and content of these web pages may be similar. Therefore, these two web pages may be used to generate test rules and templates for testing of other web pages of the same category.

[0040] Having received the selected pre-verified reference web pages from the web server 208, the application server 210 may use web page element data analyzer 220 to perform analysis of the layout and/or content of these web pages in order to identify similar and different elements of these web pages as well as the logical relationships between the identified elements. Fig. 3 shows examples of retrieved HTML document 306 and 308, which when executed by a browser 206 of a user device 202 generate web pages 302 and 304. Specifically, each HTML document 306 or 308 may comprise a set of codes in a text file format for controlling the presentation and display of information associated with various web page elements on a computing device (e.g. device 202 in Fig. 2). Examples of information that may be displayed on web pages 302 and 304 through the use of the HTML may include textual information, images, video, audio, and other executable programs. In one aspect, the web page element data analyzer 220 may analyze HTML tags, attributes and the text associated with each web page element of web pages 302 and 304 to determine the similarity or differences between elements of these two web pages.

[0041] In HTML syntax, most elements are written with a start tag and an end tag, with the content in between. A HTML tag may be composed of the name of the element, surrounded by angle brackets. An end tag also has a slash after the opening angle bracket, to distinguish it from the start tag. For example, a paragraph <p> tag may be used to structure textual information of a web page into different paragraphs. Each paragraph of text may go in between an opening <p> and a closing </p> tag. Example common HTML tags may include heading tag <hl> - <h6>, bold <b>, anchor <a>, image <img>, division <div>, table <table>, button <button>, different types of input <input>, list of items <select>, video <video>, etc. Each tag may contain, within its opening bracket, a plurality of attributes defining, for example, visual characteristics of the text associated with the tag, such as text font type, font color, font size, background color, dimensions, etc. Between the brackets a tag may contain text, which may be reproduced by the browser 206 for the display on the user device 202. [0042] Provided below a portion of a simplified example code of the HTML document 306 that, when executed by a Web browser 206, generates for display on the user device 202 web page 302. The code of the HTML document 308 may be substantially similar.

<HTML>

<HEAD>

<TITLE>Digital Cameras 1</TITLE>

<META name= keywords content="Yandex, market, camera" >

</HEAD>

<BODY>

<TABLE>

<TR>

<TD> <A href= "gallery/camera l.php"> <IMG id=camera_picture2 alt="Viewl "

src="images/prospective_view_cameral.gif" align=left border=0 width=50> </Ax/TD>

<TD> <A href="gallery/camera2.php"> <IMG id=camera_picture2 alt="View2"

src="images/prospective_view_camera2.gif" align=left border=0 width=50> </Ax/TD>

<TD> <A href="gallery/camera3.php"> <IMG id=camera_picture3 alt="View3"

src="images/prospective_view_camera3.gif" align=left border=0 width=50> </Ax/TD>

<TD> <video src="Cameral.mpg" poster="Cameral_video_image.jpg"x/TD>

</TR>

</TABLE>

<TABLE>

<TR>

<p>Product Name for Camera l</p>

<p>Manufacturer Name for Camera l</p>

<p>Customer Reviews for Camera l</p>

<p>Specification for Camera l</p>

</TR>

<TABLE>

<TR>

<p>Color:</p>

<p>Model Number:</p>

<p>Camera Lens Size:</p> <p>Dimensions:</p>

<p>Digital Zoom:</p>

</TR>

</TABLE>

</BODY>

</HTML>

[0043] In one example aspect, the web page element data analyzer 220 configured to analyze the HTML codes 306 and 308 of the web pages 302 and 304 and identify similar, different and related elements therein. Here, an HTML or web page element may refer to an individual component of an HTML document or web page, which has been parsed into a Document Object Model (DOM), including, but not limited tags, attributes and text. In one example aspect, the web page element data analyzer 220 may analyze the HTML path to each of the elements of the web pages 302 and 304 and associate each address of the web page, for example, market.yandex.ru/shop/goods/camera/cameral (the link guides to the web page of a digital camera), with specific list of elements, which are relevant for the web page of the specific category of product/goods/services (e.g., cameras). In one example aspect, the web page element data analyzer 220 retrieves all of the HTML or web page elements associated with the selected pre- verified reference web pages from the web page database 228 and other servers/databases 236 (e.g., the API server 112 and third party server 120 in Fig. 1).

[0044] Once all elements of the selected pre-verified reference web pages 302 and 304 have been obtained, the web page element data analyzer 220 may parse the HTML codes 306 and 308 and extract some or all text portions of the identified HTML elements. The web page element data analyzer 220 then compares extracted text portions of the two reference web pages to determine similar and different elements therein. For example, with reference to Fig. 3, analysis of web pages 302 and 304 reveals that these two pages have a number of similar elements, including, but not limited to three prospective views and a video of a camera, as well as common text fields, such as, product name, manufacturer name, customer reviews, list price, specification, which includes color, model number, camera lens size, dimensions and digital zoom. In one example aspect, the layout of these elements on the web pages 302 and 304 may be different, but it may not be relevant because the web page element data analyzer 220 analyzes the similarities on the content of these web pages (e.g., text) rather than the organization of the web page documents. In another example aspect, the web page element data analyzer 220 may also compare organization of the web pages 302 and 304 by, for example, comparing different types of tags and their respective attributes, such as table tag, form tag, input tag, style tag, etc., which define the structure of the web page.

[0045] In one example aspect, the web page element data analyzer 220 may use simple regular expressions for comparison and identification of similar text elements in the two or more web pages 302 and 304. Generally, regular expressions may refer to patterns for specifying and recognizing text strings during the processing semi-structured or unstructured data sets. Regular expressions may be used in flexible matching and pattern-based lookup, and pattern-based filtering. In operation, regular expressions may be employed to search and match data as a function of a predefined pattern or set of patterns. As such, patterns may employ a specific syntax by which particular characters or strings are selected from a body of text. More specifically, the expressions may comprise constants and operators that denote sets of strings and operations over these sets, respectively. Using the specific syntax of a regular expression or other pattern language, advanced text pattern matching may be performed.

[0046] Referring to Fig. 2, the web page element data analyzer 220 may perform textual pattern matching utilizing regular expressions via regular expression engine (not shown) for providing textual pattern matching as a function of an input regular expression. Such a regular expression engine may be either a text-directed engine or a regex-directed engine, wherein a text-directed engine is a deterministic finite automation (DFA) and a regex-directed engine is a non-deterministic finite automation (NFA). In either instance, the regular expression engine may receive, retrieve, or otherwise obtain as input a regular expression and a string of textual data (indicated by HTML, for example) for which to identify matches. Regular expressions may comprise a plurality of normal characters and operators that describe a set of one or more strings in the form of an expression or pattern in processing input text. By way of example, where the input text is a string "Digital Zoom" and the pattern corresponds to "zoom." In this case, the regular expression engine may search the string incrementally (e.g., "d," "i," "g," "i," "t," "a," "1," "z," "o," "o", "m") until it finds the first letter "z." Subsequently, the engine will look for the remaining letters "oom" to ultimately match the pattern. [0047] The character strings under analysis may be forwarded to a string parser that is implemented by the web page element data analyzer 220, or another module executable by the processor 216 of the application server 210. In one aspect, string parser may automatically identify the most common prefixes and suffixes in, e.g., equivalent and corresponding rows of texts in web pages 302 and 304, due to their recurring presence in a statistically significant portion of a group of character strings. More specifically, in response to receiving a number of different character strings, the web page element data analyzer 220 may analyze each individual text string to determine whether a unique substring of characters is located at the beginning and/or end of the strings and occurs with a frequency that is statistically significant enough in comparison with a user selected numerical measurement to conclude that the unique substring may be considered to be a deliberate sequence of characters that is a prefix or suffix and not a randomly occurring sequence of characters.

[0048] In one particular aspect, the web page element data analyzer 220 may extract text elements (e.g., strings of text) of the same block of the reference web pages (e.g., pages 302 and 304), and build for them simple regular expressions, so that X% of the texts match the expressions. The X% may be specified at 90% for example. In one example aspect, a simple regular expression may be used to identify similar elements in the following manner: the web page element data analyzer 220 takes two sets of rows of text, and searches for the biggest common prefix, the biggest common suffix, and the biggest common substring inside: like "camera lens size", or substring "lens size", or substring "lens". Thus, the specific position of these elements may collectively indicate a string of the letters forming the regular expression like "****" (stands for the main word "lens" of 4 letters), and "****" (represents the word "size" of 4 letters) and finally ''*****" (stands for "camera"). Generally speaking, the expression may look like "****"(lens) AND "****"(size) AND ''*****" (camera). In the current example, AND (logical conjunction operator) is a logical operator meaning that the words "lens", "size" and "camera" may be presented together in any sequence. Alternatively, the web page element data analyzer 220 may find only two words of the expression (say "lens" and "size"). In the particular case, the web page element data analyzer 220 may still decide that web page element meets the expression, however, missing one word. [0049] In another example aspect, when the prefix, suffix and substring may be too short to effectively construct a workable and useful regular expression, the given set of strings may be analyzed by the web page element data analyzer 220 to determine character classes contained in the string. For example, the character classes may include letters of one or more known languages (such as English or Russian), numerical digits, printable American Standard Code for Information Interchange ("ASCII") characters, certain "Unicode" characters (which might be encoded in the Unicode Transformation Format "UTF-8" for example), and/or characters that are allowable in URLs (e.g., alphanumerical characters), reserved characters (e.g., "/", "?", "@", "=", "&" "%") and special characters (e.g., "$", "+", "!", "*"). Having determined character classes contained in the text strings, the web page element data analyzer 220 may capture a pattern of a common text elements as an expression, e.g., "first from 3 to 12 are English small characters, then the punctuation symbols, and then from 7 to 9 digits."

[0050] In another example aspect, the web page element data analyzer 220 may determine a simple regular expression the following way. The web page element data analyzer 220 may extract elements of the same block of the reference web pages and extract numerical information from the block. For example, for a date of announcement of the product or a date of the product release on the market. A first block may have a date (say, a year of the product announcement) "1999" and a second may have a date of (the year of the product release) "2017". The web page element data analyzer 220 may create a regular expression like {4} [0-9], which means that there are four digits in a row, the digits may be any varying zero to nine. Alternatively, the web page element data analyzer 220 may create a regular expression like {2} (19l20)[0-9] meaning that there may be sets of digits, the first one either "19" or "20" followed by two digits, the digits may be any varying zero to nine.

[0051] In one example aspect, the web page element data analyzer 220 passes the information about HTML elements identified in the pre-verified reference web pages to a rules generator 222 and template generator 224. The template generator 224 is configured to create a set of test templates for a specific category of web pages that contain listing of, for example, substantially similar (i.e., common), different, or the same elements with dynamic and changeable content embedded therein. A set of templates may include one or more templates. A template may be implemented as an XML-like file. Thus, for example, one set of test templates generated from pre-verified reference web pages 302 and 304, which are associated with the category of "cameras", may include the following common text elements: product name, manufacturer name, customer reviews, list price and specification, color, model number, camera lens size, dimensions and digital zoom. In addition, another set of test templates may be generated that includes identifiers of images of three prospective views of a camera and a video of the camera. Thus, any verified web page of the same category should have at least the same text elements, and, in one particular aspect, images and videos embedded therein.

[0052] In one example aspect, the rules generator 222 is configured to generated sets of test rules indicating logical, contextual and/or positional relationships between elements of the web page templates, as well as the compatibility between elements. A set of rules may include one or more rules. The rules may be implemented using conditional expressions (e.g., "if/then" statements) using, for example XML, Java, JavaScript or other programming languages. In one example aspect, one set of test rules may specify logical or contextual relationships between web page elements. Thus, for example, with reference to Fig. 3, the "specifications of camera 1" element of web page 302 must include at least "color", "model number", "camera lens size", "dimensions", and "digital zoom" elements. In another example aspect, another set of test rules may determine whether a specific web page element should be presented on the web page (e.g., a search bar). In another example aspect, another set of rules may determine that a collection of web page elements are cohesive and should be presented on every page and together (e.g., the search bar and the "about store element"). In yet another example aspect, another set of rules may determine that two or more web page elements cannot be shown on the same web page (i.e., exclusive of one another).

[0053] In one example aspect, the generated sets of rules and templates are passed to the web page verifier 226, which uses these rules and templates to tests the compliance and consistency of other web pages of the same category, such as newly added web pages or newly updated web pages (e.g., web pages that have updated content). First, the web page verifier 226 may identify the category of the web page that requires testing by, for example, analyzing a URL of the tested web page. Thus, if a URL of the tested web page is

"market.yandex.ru/search/goods/cameras/camera3.htm", the web page verifier 226 may determine that the web page is associated with the category of digital camera and appropriate set of test rules and templates should be used for testing of this web page. During testing of a web page, the web page verifier 226 may check compliance of the web page elements with the sets of tests rule and templates. Particularly, the web page verifier 226 may parse the HTML code of the web page, identify text elements contained therein, and compare of the identified text elements to those stored in the associated test template. In one example aspect, simple regular expressions may be used to compare text elements of the web page with the information contained in the test template. In addition, the web page verifier 226 may apply associated test rules to verify presence of logical relationships between the elements of the tested web page. Such comparison may be performed against a selected threshold value (e.g., X% difference) or a set of thresholds (e.g., one or more upper thresholds and lower threshold indicating different value sets/ranges). In one example, no or minor difference (e.g., < 5%) between elements of the tested web page and elements of the set of test templates may result in the web page verifier 226 marking the web page as "verified". However, if the web page fails the verification process due to, for example, a large difference (e.g., > 5%) between the text elements of the tested web page and elements of the set of test templates, or violation of some logical or contextual relationship between text elements required by the test rules, then the web page verifier 226 may submit this web page for review to a human analyst.

[0054] In one example aspect, the generated sets of test rules and templates may be saved in a local database(s) 212 of the application server 210. Alternatively, under a horizontal scaling approach, a back-end database server and associated database 228 may be employed to allow entry, storage and retrieval of the generated rules and templates. In an aspect, as shown in Fig. 2, the back-end database server and associated database 228 may execute a rule manager 230, template manager 232, and an optional web page element data module 234 for efficiently organizing the generated test rules, templates and relevant web page element data obtained from the application server 210 using any suitable data structure such as relational, object-oriented, operational, NoSQL databases. In the case of a relational database system, the rule manager 230 may store and manage obtained rules using tables and each rule may be uniquely identified with a compound identifier. A portion of such a compound identifier may operate to identify the particular table in the database 228 with which the rule is associated. Accordingly, when an identifier of a rule is called upon external to the database, such as within higher-level API code, the associated table for that rule may be easily determined from the identifier itself. In retrieval operations, this allows a desired rule to be efficiently accessed from the table on which it resides. In addition, knowledge of the originating table for a rule may permit other information relating to the rule to be easily tracked and learned (e.g., based on knowledge of the various tables in the database and the types of data contain therein). Each of the template manager 232 and optional web page element data module 234 may manage information stored therein in a similar manner.

[0055] In one example aspect, the generated sets of test rules and templates may be associated with different categories of products/services/goods to assure quick search and retrieval of appropriate test rules and templates by the template manager 232 and rules manager 230 during the testing and verification of web pages. For example, for an e-commerce website, which sells different products, one set of rules and templates may be associated with a "digital camera" category of web pages, and a different set of test rules and templates may be associated with a "laser printers" category of web pages. Thus, when a new test web page is received, the web page verifier 226 determines it category and requests rules manager 230 and template manager 232 to provide test rules and templates associated with this category. The rules manager 230 and template manager 232 retrieve from the database 228 complete sets of test rules and templates associated with that category of products and return them to the web page verifier 226. Thus, in this implementation, the sets of rules and templates may be organized and stored in a database 228 linearly (i.e., one complete set of test rules and templates for each category of products/goods/services), which may be suitable for websites that have few different web pages categories and, therefore, have few different sets of rules and templates.

[0056] However, for websites that have large number of different categories and subcategories of products/goods/services and corresponding web pages, like large e-commerce websites (e.g., Yandex market, Amazon, and eBay), a hierarchical tree organization and storage of test rules and templates may be more suitable to allow quick search and retrieval of complete sets of test rules and templates without storing many identical copies of subsets of test rules and templates for each subcategory of web pages. For example, an e-commerce website may have a hierarchical tree organization of product categories 400 show in Fig. 4. Root node 414 represents top category of "electronics & computers" products. Child node 410 represents a subcategory of "camera, photo & video" products. Child node 412 represents a subcategory of "printers & accessories" products. Grandchild node 406 represents a subcategory of "digital SLRs". Grandchild node 408 represents a subcategory of "camcorders". Grandgrandchild node 404 represents subcategory of "lenses". Node 202 represents, for example, a web page for a Nikon 50mm f/1.4G Lens for Nikon Digital SLR Cameras.

[0057] In one example aspect, each category and subcategory of hierarchical tree structure 400 may have different sets of test rules and templates associated therewith. For example, the top category of "electronics & computers" products 414 may have a set of test rules 1/templates 1 associated therewith. The set of rules 1 and templates 1 may identify web page elements common to all subcategories of web pages, including for example, web pages of all types of printers and printer accessories, cameras, and other photo and video equipment and accessories. Such templates may include "description of the product" element, "list price" element, "customer review" element, and other text elements common to all web pages of the website. Next, the two sibling subcategories of "camera, photo & video" products 410 and "printers & accessories" products 412 may have different sets of test rules 2a/templates 2a and test rules 2b/templates 2b, associated respectively therewith. For example, the set of rules 2a/templates 2a may identify web page elements (and their respective relationships) common to all subcategories of camera, photo and video equipment web pages. Such templates may include, for example, a "product image" element, "product video" element, "product color" element, "accessories" element, and other text elements common to all web pages of camera, photo and video equipment. Next, the two sibling subcategories of "digital SLRs" products 406 and "camcorders" products 408 may have yet another different sets of test rules 3a/templates 3a and test rules 3b/templates 3b, associated respectively therewith. For example, the set of rules 3a/templates 3a may identify web page elements (and their respective relationships) common to all subcategories of digital SLR cameras web pages. Such templates may include, for example, a "image sensor" element, "lens size" element, "iso range" element, and other text elements common to all web pages of digital SLR cameras. Finally, the last subcategory of "lenses" products 404 may have yet another different set of test rules 4/templates 4 associated therewith. For example, the set of rules 4/templates 4 may identify web page elements (and their respective relationships) common to all lenses for digital SLR cameras web pages. Such templates may include, for example, a "lens mount type" element, "lens size" element, "focal length" element, "minimum focus" element, and other text elements common to all web pages of lenses for digital SLR cameras. Thus, in one example aspect, a complete set of test templates and rules for testing of the web page 402 will include all subsets of templates and rules from all parent categories of products, including a set of rules 1 and templates 1, a set of rules 2a and templates 2a, a set of rules 3a and templates 3 a and a set of rules 4 and templates 4. It should be noted that in case of a conflict between two or more elements of a child and parent templates (e.g., an element of the template 3a being incompatible with another element of the template 2a under certain conditions), an appropriate rule 2a (or 3a) may be generated to exclude incompatible element(s) from the complete set of templates for web page 402.

[0058] This above-described hierarchical tree organization of sets of test rules and templates improves scalability of the rules and template management system (i.e., database server 228, rules manager 230 and template manager 232) while minimized data storage requirements, because hierarchical/distributed storage of sets of rules and templates minimized duplication of sets of rules and templates common to all categories of web pages.

[0059] In an example aspect, the above-described hierarchical tree organization of sets of test rules and templates may be used to test web pages in the following manner. With reference to Figs. 2 and 4, first, the web page verifier 226 may identify the category of the web page that requires testing by, for example, analyzing a URL of the tested web page. Thus, if a URL of the tested web page is "market.yandex.ru/search/goods/cameras/SLRcamera3.htm", the web page verifier 226 may determine that the web page is associated with the category of "digital SLRs" products and appropriate sets of test templates and rules should be used for testing of this web page. The web page verifier 226 will request from the rule manager 230 and template manager 232 the appropriate sets of test rules and templates for the category of digital SLRs cameras. The rules manager 230 and/or template manager 232 may identify in the hierarchical tree structure of sets of rules and templates 400 a node 406 associated with the subcategory of "digital SLRs", and retrieve a set of test rules 3a and templates 3a contained therein. The rules manager 230 and/or template manager 232 may then traverse the path of connected nodes starting from the node 406, to the parent node 406, and to the root node 414, and retrieve all sets of test rules and templates associated with the traversed nodes, including a set of test rule 2a and templates 2a, and a set of test rules 1 and templates 1. The rules manager 230 and/or template manager 232 then return the retrieved complete set of test rules 1, 2a and 3 a, and a complete set of test templates 1, 2a and 3a to the web page verifier 226. The web page verifier 226 may in turn parse the HTML code of the test web page, identify text elements contained therein, and check the compliance of the identified text elements to the complete sets of test rules and templates received from the rule manager 230 and/or template manager 232. In one example aspect, the web page verifier 226 may simple regular expressions to compare text elements of the web page with the information contained in the sets of test rules and templates. In addition, the web page verifier 226 may apply associated test rules to verify presence of logical relationships between the elements of the tested web page. Such comparison may be performed against a selected threshold value (e.g., X% difference) or a set of thresholds (e.g., one or more upper thresholds and lower threshold indicating different value sets/ranges). In one example, no or minor difference (e.g., < 5%) between elements of the tested web page and elements of the set of test templates may result in the web page verifier 226 marking the web page as "verified". However, if the web page fails the verification process due to, for example, a large difference (e.g., > 5%) between the text elements of the tested web page and elements of the set of test templates, or violation of some logical or contextual relationship between text elements required by the test rules, then the web page verifier 226 may submit this web page for review to a human analyst.

[0060] Fig. 5 depicts an example method for testing web pages according one aspect. The method 500 may be implemented by the systems 100 and 200 of Figs. 1 and 2. At step 502, the method 500 may include receiving, via, for example, the processor of the application server 210 of Fig. 2, two or more reference web pages, each reference web page comprising a plurality of HTML elements. At step 504, the method 500 may include analyzing the HTML elements of the two or more reference web pages to identify similarities and relationships among the analyzed elements. At step 506, the method 500 may include generating a set of test template defining at least elements common to the two or more reference web pages. At step 508, the method 500 may include generating a set of test rules specifying logical relationships between HTML elements of each reference web page. At step 510, the method 500 may include verifying compliance of a test web page with the set of test template and the set of test rules.

[0061] Fig. 6 depicts an example method for testing web pages according another aspect. The method 500 may be implemented by the systems 100 and 200 of Figs. 1 and 2. At step 602, the method 600 may include identifying a category of a test web page. At step 604, the method 600 may include identifying a set of test rules and a set of test templates associated with the category of the test web page. At step 606, the method 600 may include identifying in a hierarchical tree structure of a plurality of nodes, wherein each node is associated with a set of test rules and a set of test templates, at least one set of test rules and a set of test templates associated with the category of the test web page. At step 608, the method 600 may include traversing the hierarchical tree structure from the identified node to the root node, and identifying sets of test rules and sets of test templates associated with each traversed node. At step 610, the method 600 may include retrieving a plurality of identified sets of test rules and sets of test templates associated with each traversed node to generate a complete set of test rules and test templates. At step 610, the method 600 may include using the complete set of test rules and test templates for verifying the test web page.

[0062] Fig. 7 depicts one example aspect of a computer system 5 that may be used to implement the disclosed systems and methods for testing web pages. The computer system 5 may include, but not limited to, an application server (e.g., server 210), a web server, a personal computer, a notebook, tablet computer, a smart phone or other type of data processing device. As shown, computer system 5 may include one or more hardware processors 15, memory 20, one or more hard disk drive(s) 30, optical drive(s) 35, serial port(s) 40, graphics card 45, audio card 50 and network card(s) 55 connected by system bus 10. System bus 10 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus and a local bus using any of a variety of known bus architectures. Processor 15 may include one or more Intel® Core 2 Quad 2.33 GHz processors or other type of microprocessor.

[0063] System memory 20 may include a read-only memory (ROM) 21 and random access memory (RAM) 23. Memory 20 may be implemented as in DRAM (dynamic RAM), EPROM, EEPROM, Flash or other type of memory architecture. ROM 21 stores a basic input/output system 22 (BIOS), containing the basic routines that help to transfer information between the modules of computer system 5, such as during start-up. RAM 23 stores operating system 24 (OS), such as Windows® 7 Professional or other type of operating system, that is responsible for management and coordination of processes and allocation and sharing of hardware resources in computer system 5. Memory 20 also stores applications and programs 25. Memory 20 also stores various runtime data 26 used by programs 25. [0064] Computer system 5 may further include hard disk drive(s) 30, such as SATA HDD, and optical disk drive(s) 35 for reading from or writing to a removable optical disk, such as a CD-ROM, DVD-ROM or other optical media. Drives 30 and 35 and their associated computer- readable media provide non-volatile storage of computer readable instructions, data structures, applications and program modules/subroutines that implement algorithms and methods disclosed herein. Although the exemplary computer system 5 employs magnetic and optical disks, it should be appreciated by those skilled in the art that other types of computer readable media that can store data accessible by a computer system 5, such as magnetic cassettes, flash memory cards, digital video disks, RAMs, ROMs, EPROMs and other types of memory may also be used in alternative aspects of the computer system 5.

[0065] Computer system 5 further includes a plurality of serial ports 40, such as Universal Serial Bus (USB), for connecting data input device(s) 75, such as keyboard, mouse, touch pad and other. Serial ports 40 may be also be used to connect data output device(s) 80, such as printer, scanner and other, as well as other peripheral device(s) 85, such as external data storage devices and the like. System 5 may also include graphics card 45, such as nVidia® GeForce® GT 240M or other video card, for interfacing with a display 60 or other video reproduction device, such as touch-screen display. System 5 may also include an audio card 50 for reproducing sound via internal or external speakers 65. In addition, system 5 may include network card(s) 55, such as Ethernet, WiFi, GSM, Bluetooth or other wired, wireless, or cellular network interface for connecting computer system 5 to network 70, such as the Internet.

[0066] In various aspects, the systems and methods described herein may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the methods may be stored as one or more instructions or code on a non-transitory computer- readable medium. Computer-readable medium includes data storage. By way of example, and not limitation, such computer-readable medium can comprise RAM, ROM, EEPROM, CD- ROM, Flash memory or other types of electric, magnetic, or optical storage medium, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a processor of a general purpose computer.

[0067] In the interest of clarity, not all of the routine features of the aspects are disclosed herein. It will be appreciated that in the development of any actual implementation of the invention, numerous implementation-specific decisions must be made in order to achieve the developer's specific goals, and that these specific goals will vary for different implementations and different developers. It will be appreciated that such a development effort might be complex and time-consuming, but would nevertheless be a routine undertaking of engineering for those of ordinary skill in the art having the benefit of this disclosure.

[0068] Furthermore, it is to be understood that the phraseology or terminology used herein is for the purpose of description and not of restriction, such that the terminology or phraseology of the present specification is to be interpreted by the skilled in the art in light of the teachings and guidance presented herein, in combination with the knowledge of the skilled in the relevant art(s). Moreover, it is not intended for any term in the specification or claims to be ascribed an uncommon or special meaning unless explicitly set forth as such.

[0069] The various aspects disclosed herein encompass present and future known equivalents to the known modules referred to herein by way of illustration. Moreover, while aspects and applications have been shown and described, it would be apparent to those skilled in the art having the benefit of this disclosure that many more modifications than mentioned above are possible without departing from the inventive concepts disclosed herein.