BACKGROUND OF THE INVENTION This application is related to Israel Patent Application Number "METHOD AND SYSTEM FOR INTERCEPTING A APPLICATION PROGRAM INTERFACE"filed 14 November 1999.

Modern computers are designed with the requirements of high-level languages in mind. The most essential technique for structuring computer programs introduced by high-level langages, is the procedure or the function.

Procedures or functions are computer programs. A procedure call or a function call is a high-level abstraction that alters the flow of the calling program execution. In contrast with the more traditional "jump"or"goto"instructions, which also alter the flow of execution, a procedure or a function, after the execution of its own code, returns control to the instruction immediately following the call. To implement procedure or function calis in the manner described, a memory device called a stack is utilized.

A stack is a contiguous block of memory containing data. Its size is dynamically adjusted by the operating system routines at run time. The data is inserted to and removed from the stack by Central Processing Unit (CPU) utilizing Assembler langage instructions such as"push"or"pop".

The stack consists of logical stack frames or Procedure Activation Records that are inserted into the stack when a function is called and removed from the stack when the said function returns control to the calling program. The stack frame itself contains parameters to the called function, local variables, pointers to recover the previous stack frame, and the return address of the calling computer program. The return address is the instruction pointer of the calling program at the time of the function call.

Induced buffer overflow or buffer overflow attack is known in the art. Buffer overflow attacks exploit the lack of bounds checking on the size of input being stored in a buffer array. Arrays are predefined allocated memory devices within a computer system : By writing data intentionally past the end of an allocated array, an attacker can make arbitrary changes to data stored adjacent to the said array. The most SUBSTITUTE SHEET (RULE 26) WO 01/37094 common data structure to be corrupted in this fashion is the stack.

Therefore this type of attack is also known as stack smashing.

The prevalent form of buffer overflow exploitation is to attack buffers allocated on the stack. Such attacks attempt to achieve two mutually dependent objectives. One such objective is inserting an attack code in the form of an executable binary code native to the attacked machine. Another such objective is to change the return address to point to the attacker's supplied code now residing within said stack memory. Such attacker's supplie code may be utilized to gain enhanced privileges over said computer system.

The programs that are attacked using this technique are usually high privilege utilities or daemons that run under the user-id root to perform essential services. The effect of a successful buffer overflow attack is to provide the attacker non-authorized root privileges. Gaining root privileges within a computer system allows non-authorized users access privileged resources.

As the maximum length of the overflowing data string can be only the current depth of the stack, the inserted attack code should be short in terms of code length. Writing data outside the stack limit will result in an exception condition that will prevent the attack code to execute. Therefore, the buffer overflow attacker will be forced to write SUBSTITUTE SHEET (RULE 26) WO 01/37094 short code and will have to use high-fevef System calls or Library calls, Such calls will later be utilized to gain non-authorized enhanced privileges to access privileged resources.

Several strategies, which attempt to resolve the buffer overflow weakness, are known in the art. One such strategy is to design a compiler designed to prohibit a computer program from writing past a stack segment array. Another strategy is to detect buffer overflow vulnerable programs off line and alert the user to the possibility that the system privileges may be compromised.

Another known strategy is using a repair program. The repair program can repair or fix those vulnerable programs that can be used to exploit the buffer overflow weakness.

None of the above provide a method and apparatus for prevention of buffer overflow through controlled execution of system or other calls within a computer system.

SUBSTITUTE SHEET (RULE 26) WO 01/37094 SUMMARY OF THE PRESENT INVENTION Thus, there is a long felt need to provide a for detecting and preventing unauthorized or illégal access attempts within a computer system. More specifically, a method for detecting and preventing attempts to exploit the buffer overflow-related weakness within a computer system by validating system or other calls made within a computer system.

It is therefore the object of this invention to provide a method for preventing induced buffer overflow attack by preventing execution of high-level System calls, Library calls, Application Program Interface call and the like when such calls are illegally made.

It is therefore another object of the present invention to provide a method for preventing induced buffer overflow attack by preventing execution of high-level System calls, Library calls, Application Program Interface call and the like when such calls are made from unauthorized areas within a computer system.

It is yet a further object of the present invention to provide a method for preventing induced buffer overflow attack by preventing execution of high-level System calls, Library calls, Application Program Interface call when such calls are made from outside the user process associated with said called system or other call.

SUBSTITUTE SHEET (RULE 26) It is therefore provided in accordance with a preferred embodiment of the present invention a method of secure function execution within a computer system running an operating system platform, said operating system including a kernel space and a process space, said process space including a user application running in process space, said user application operative to intercept system calls, said method comprising the step of examining said intercepted system call validity by comparing said intercepted system call originating address with range of process valid addresses associated with said process from which said intercepted system call originated and providing notification as to the validity of said intercepted system call, or terminating said intercepted system call.

It is further provided in accordance with a preferred embodiment of the present invention a method of secure function execution within a computer system running an operating system platform, said operating system including a kemet space and a process space, said process space including a user application running in process space, said user application operative to intercept system calls, said method comprising the step of examining said intercepted system call validity by comparing said intercepted system call originating address with range of process valid addresses associated with said process from which said WO 01/37094 intercepted system call originated and responsive to process creation inserting application program interface interception module into said created process and responsive to process creation updating process vaiid addresses table or responsive to process termination updating process valid addresses table.

In accordance with yet another preferred embodiment of the present invention there is provided a method of secure function execution within a computer system running an operating system platform, said operating system including a kemel space and a process space, said process space including a user application running in process space, said user application operative to intercept library calls, said method comprising the step of examining said intercepted library call validity by comparing said intercepted library call originating address with range of process valid addresses associated with said process from which said intercepted library call originated and providing notification as to the validity of said intercepted library call or terminating said intercepted library call.

In accordance with yet another preferred embodiment of the present invention there is provided a method of secure function SUBSTITUTE SHEET (RULE 26) execution within a computer system running an operating system platform, said operating system including a kernel space and a process space, said process space including a user application running in process space, said user application operative to intercept library calls, said method comprising the step of examining said intercepted library ca41 validity by comparing said intercepted library call originating address with range of process valid addresses associated with said process from which said intercepted library call originated and responsive to system call loading dynamic link library hooking and patching library routines associated with said dynamic link library and responsive to system call unloading dynamic link library updating process valid addresses table.

In accordance with another preferred embodiment of the present invention there is provided a method of secure function execution within a computer system running an operating system platform, said operating system including a kernel space and a process space, said process space including a user application running in process space, said user application operative to system and function calls, said system or function call intercepted, said method comprising the steps of receiving caller routine return address from said process memory device, determining whether caller routine address is valid by comparing said caller address routine with process valid address table and providing notification as to the validity of said caller routine return address or performing user predetermined acts associated with said validity of caller routine address. The same method further comprising the step of determining said calier routine calling address by determining the address preceding said caller routine address.

In accordance with yet another preferred embodiment of the present invention there is provided a method of secure function execution within a computer system running an operating system platform, said operating system including a kernel space and a process space, said process space including a user application running in process space, said user application operative to system and function calls, said system or function call intercepted, said method comprising the steps of receiving caller routine return address from said process memory device, determining whether caller routine address is valid by comparing said caller address routine with associated process stack address area and providing notification as to the validity of said caller routine return address or performing user predetermined acts associated with said validity of caller routine address. The same method wherein the step of receiving caller routine return address from ests from said process memory device further comprises the step of determining term, mng said caller routine calling address by determining the address ddress preceding said caller routine address.

BRIEF DESCRIPTION OF THE DRAWINGS- The accompanying drawings, which are incorporated in and constitutes a part of the specification, illustrate preferred embodiments of the invention and, together with the description, serve to explain the principes of the invention: Fig. 1 is a block diagram of the Secure Function Execution System environment generally referenced to as system 100; Fig. 2 is a high-level flow diagram of the Secure Function Execution Server 116 operation ; Fig. 3 is a high-level flow diagram of the operation of the Secure Function Execution Server initiaiization module referred to in Fig. 2; Fig. 4 is a high-level diagram of Secure Function Execution Server or the like response to an intercepted system call referred to in Fig. 2; Fig. 5 is a high-level flow diagram of the operation of the Secure Function Execution Server and the like library call response module referred to in Fig. 2; Fig. 6 is a high-level flow diagram of the operation of the Calling Address Validation Routine module ; Fig. 7 is a high-level flow diagram of the Calling Address Validation Routine module relating to an another embodiment of the present invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS The present invention overcomes the disadvantages of the prior art by providing a novel method, which detects if an attempt to exploit the buffer overflow weakness is occurring by validating use of system or other calls within a computerized system.

Reference is now made to Fig. 1, there is provided a schematic illustration of the system environment wherein the Secure Function Execution System is operating, generally referred to as system 100, in accordance with a preferred embodiment of the present invention.

The present invention is related to Israel Patent Application Number XXXXXXX."METHOD AND SYSTEM FOR INTERCEPTING A APPLICATION PROGRAM INTERFACE"filed 14 November 1999.

As further described in detail in Israel Patent Application Number XXXXXXX, system 100 of Fig. 1 may comprise of four components of the Secure Function Execution System; a) Secure Function Execution Server 116 is an active component. Secure Function Execution Server 116 is the operational center of the Secure Function Execution System 100. Secure Function Execution Server 116 loads and controls System Call interception Component 124, loads and controls API Interception Module 134, 140, and 146, responds to diverse system and library calls and acts as an interface towards the user. The Secure Function Execution Server 116 is loaded into the user space memory device 112 of a computer system.

Secure Function Execution Server 116 incorporates the API Interception Control Server operations that were described in detail in Israel Patent Application No. xxxxxxx b) API interception Module 134, 140,146 and the like are active components. API Interception Moduie 134,140, 146 and the like operations are described in detail in Israel Patent Application Number XXXXXXX. AP) Interception Module 134,140,146 and the like consist of Dispatch Routine, Depatch Routine, Hook and Patch Routine, Pre-Entry Routine, and Post-Entry Routine. The operations of the said routines are also described in detail in Israel Patent Application No. XXXXXXX c) System Call Interception Component 124 is an active component. System Call Interception Component 124 operation is described in detail in'srael Patent Application No. XXXXXXX d) API routine 132,138,144 and the like are passive components. API routines 132,138,144 and the like are potential objects upon which Secure Function Execution System 100 might operate. API routines 132,138,144 and the like are loaded into process address space memory device 118, 120,122 and the like in user memory device 112 of system 100.

System 100 previously described in Israel Patent Application number XXXXXXX serve as model for a Secure Function Execution System in which the present invention is operative. It will be appreciated by those skilled in the art that the present invention may operate under various similar systems and that the system shown herein is an example to further ililistrate the working of the present invention.

Referring to Fig. 2 there is provided a high-level flow diagram of the Secure Function Execution Server 116 operation. Secure Function Execution Server 116 was previously'described in detail in Israel Patent Application Number XXXXXXX. The operation of said Secure Function Execution Server 116 would now be briefly expiained.

Secure Function Execution (SFE as it will be abbreviated from this point on in the text of this document) Server 116 initializes the application in step 150. Consequently SFE Server 116 commences its run-time operation in step 152 by constantly monitoring system calls made by diverse applications that run in the host operating system (step 152) and responding appropriately to the said system calls (step 154) as described in detail in Fig. 4. SFE Server is also constantly monitoring library calls made by diverse application that run in the host operating system (step 156). SFE Server responds appropriately to the said library calla (step 158) as described in detail in Fig. 5.

Referring now to Fig. 3 there is provided a high-level flow diagram of SFE Server 116 start-u. operation referenced as step 150 of Fig. 2. SFE Server 116 start-up operation was previously described in detail in Israel Patent Application Number XXXXXXX First SFE Server 116 loads System Call Interception Component 124 into kemel space memory device 114 (step 184). After establishing communication with System Call Interception Component 124 SFE Server 116 queries System Call Interception Component 124 for the list of active processes 118,120,122 and the like (step 186).

Using the said list of active processes 118,120,122 and the like SFE Server 116 creates a list of valid address ranges for each active process 118,120,122 and the like. This structure will be referenced from this point on as Process Vaiid Address Range in the text of this document.

Process Valid Address Range List holds the address range into which the process 118,120,122 and the like was loaded to.

Process Valid Address Range also holds all the address ranges into which diverse Dynamic Link Library (DLL) 130, 136,142 and the like were loaded. Dynamic Link Library is a set of callable subroutines linked as a binary image that can be dynamically loaded by applications that utilize them.

Finally, SFE Server will insert API Interception Module 134, 140, 146 and the like to all active processes 118, 120, 122 and the like (step 19) as described in detail in Israel Patent Application No. xxxxxxx SFE Server operation of monitoring system cals in step 152 of Fig. 2 is described in detail in Israel Patent Application No. xxxxxxx Turning now to Fig. 4 which is a high-level flow diagram of SFE Server or the like response to an intercepted system call, referred to as step 154 of Fig. 2. SFE Server 116 determines in step 160 whether the system call detected is an illegal call or a legal call. SFE Server determines whether said system call is valid by comparing said system call originating address with range of Process Valid Address associated with said process from which said system call originated. If illegal call was detected the SFE Server 116 may terminate the illegal function (step 164). Altematively SFE Server 116 may notify a user (typically the System Administrator) about the illegal call (step 166).

Alternatively, SFE Server 116 may perform another or other series of user predetermined actions.

If the system call detected is legal (step 160) SFE Server 116 examines the said system call to determine if it is of the type of process creation (step 162). If and when it is determined that the system call of the type of process creation SFE Server 116 inserts API interception Module 134 to the newly created process address space 118 (step 168) and updates Process Valid Address Range List (step 170) by adding said process address list to Process Valid Address Range. If the decision in step 162 is negative SFE Server optionally performs any other user predetermined or instructed action (step 166). If and when it is determined that the system call of the type of process termination SFE Server 116 updates Process Valid Address Range List (step 171) by removing said process valid addresses range from Process Valid Address Range List.

SFE Server operation of monitoring library calls in step 156 of Fig. 2 is described in detail in US Patent Application No. XXXXXXX Turning now to Fig. 5 which is a high-level flow diagram of the SFE Server and the like response to an intercepted library call referred to as step 158 of Fig. 2. SFE Server 116 determines in step 172 if the library call detected is an illegal call. SFE Server determines whether said library call is valid by comparing said library call originating address with range of Process Vaiid Address associated with said process from which said library call originated. If an illegal call is detected SFE Server 116 optionally terminates the illegal library function (step 180). Altematively, SFE Server 116 notifies a user (typically the System Administrator) (step 182). Altemativeiy, SFE Server 116 performs any other user predetermined or instructed action (step 182).

If the library call detected is legal (step 172) SFE Server 116 determines if the said library call is of the type of DLL 130 load (step 174). If the decision in step 174 is affirmative than SFE Server 116 hooks and patches the library calls (APIs) 132 existing within said loaded DLL 130. Such hooking and patching is further described in detail in Israel Patent Application. After hooking and patching said API Interception Module 134 already loaded into said associated process 118 is now operative to intercept calls made to said library calls 132. When determined that the library call is of the type DLL load SFE Server 116 updates Process Va (nid Address Range List (step 178) by adding DLL address rage into Process Valid Address Range List. If the decision in step 172 is negative SFE Server 116 determines if the intercepted library call if of the type DLL unload (step 176) by deleting DLL address range from Process Valid Address Range List. When it is determined that the library call is of the type of DLL unload SFE Server updates the Process Valid Address Range List (step 178).

Reference in now made to Fig. 6 that is a high-levet flow diagram of the operation of the Calling Address Validation Routine module. The Calling Address Validation Routine module may operate in conjunction with API interception Module Pre-Entry routine as further described in detail in Israel Patent Application No. XXXXXXX Pre-Entry routine may be activated when an API 132 or the like of Fig. 1 is intercepted. Operating under SFE System 100 Pre-Entry routine, Calling Address Validation Routine mcdule is executing a set of instructions designed to validate the API function 132 of Fig. 1 calling address (calter Routine). Caller Routine also inclues caller Application Program Interface, caller system call, caller library cali and the like.

Calling Address Validation Routine module commences its operation by reading the caller Routine return address from the Procedure Activation Record (stack frame) which is on the user stack segment (step 191). The stack frame is a dynamic area of the process stack segment used as a control area for function calls. The process stack segment is a dynamic area of memory belonging to a process. in step 192 the caller Routine calling address is calculated (step 192) and with the heip of the data in Process Valid Address Range List it is examined if the said caller Routine calling address is within valid address range limits (step 194). In step 196 it is determined whether the calling address valid or non-valid. To calculate if said caller Routine calling address is within said valid address range limit said cailer Routine calling address is matched with said valid address range limit.

If said caller Routine calling address is within said valid address range than caller Routine calting address is valid. Next, Calling Address Validation Routine module by Pre-Entry routine or the like notifies SFE Server 116 or the like about the test result (step 198 and step 200).

It will be appreciated to by persons skilled in the art that in this illustrated embodiment of the present invention any unauthorized or illegal system call or library call originating from memory areas out of active process address space memory device 118,120,122 and the like of Fig. 1 will be detected and optionally their execution will be prevented by SFE System 100.

Reference is now made to Fig. 7 which is a high-level flow diagram of the Calling Address Validation Routine module relating to an another embodiment of the present invention.

In the embodiment thereof Calling Address Validation Routine module commences its operation by reading the caller return address from the Procedure Activation Record (stack frame) on the process stack segment (step 202). It will be appreciated that reading caller Routine return address is significantly faster and more accurate. In step 204 the caller Routine calling address is calcuiated and it is examined with the help of system-levei structures to determine whether the calling address is inside the address limits of the process stack segment (step 206). Such determination is accomplished by comparing said caller Routine calling address with address limits of said process stack segment. Next, Calling Address Validation Routine module by Pre-Entry routine or the like notifies SFE Server 116 or the like about the result of the examination (step 210 and step 212).

It will be appreciated by persons skiiled in the art that in this further embodiment of the invention any unauthorized or illegal system call or library call originating from the process stack segment structure will be detected and optionally prevented by SFE System 100.

Additional advantages will readily occur to the person skilled in the art. The invention, in its broader aspects is, therefore, not limited to the specific details, representative methods, systems and examples shown and described. It will be further appreciated by persons skilled in the art that the present invention is not limited to what has been particularly shown and described hereinabove. Rather the scope of the applicant's general inventive concept and the claims which follow.