Background

Technical Field

[0001] The present disclosure relates generally to information processing systems and, more specifically, to scheduling and control of thread execution on OS-sequestered sequencers without operating system intervention.

Background Art

[0002] hi order to increase performance of information processing systems, such as those that include microprocessors, both hardware and software techniques have been employed. On the hardware side, microprocessor design approaches to improve microprocessor performance have included increased clock speeds, pipelining, branch prediction, super-scalar execution, out-of-order execution, and caches. Many such approaches have led to increased transistor count, and have even, in some instances, resulted in transistor count increasing at a rate greater than the rate of improved performance.

[0003] Rather than seek to increase performance strictly through additional transistors, other performance enhancements involve software techniques. One software approach that has been employed to improve processor performance is known as "multithreading." La software multithreading, an instruction stream may be divided

into multiple instruction streams that can be executed in parallel. Alternatively, multiple independent software streams may be executed in parallel.

[0004] In one approach, known as time-slice multithreading or time-multiplex ("TMUX") multithreading, a single processor switches between threads after a fixed period of time, hi still another approach, a single processor switches between threads upon occurrence of a trigger event, such as a long latency cache miss, hi this latter approach, known as switch-on-event multithreading ("SoEMT"), only one thread, at most, is active at a given time.

[0005] Increasingly, multithreading is supported in hardware. For instance, in one approach, processors in a multi-processor system, such as a chip multiprocessor ("CMP") system, may each act on one of the multiple software threads concurrently. In another approach, referred to as simultaneous multithreading ("SMT"), a single physical processor is made to appear as multiple logical processors to operating systems and user programs. For SMT, multiple software threads can be active and execute simultaneously on a single processor without switching. That is, each logical processor maintains a complete set of the architecture state, but many other resources of the physical processor, such as caches, execution units, branch predictors, control logic and buses are shared. For SMT, the instructions from multiple software threads thus execute concurrently on each logical processor.

[0006] For a system that supports concurrent execution of software threads, such as SMT and/or CMP systems, an operating system application may control scheduling

and execution of the software threads. Typically, however, operating system control does not scale well; the ability of an operating system application to schedule threads without negatively impacting performance is commonly limited to a relatively small number of threads.

Brief Description of the Drawings

[0007] Embodiments of the present invention may be understood with reference to the following drawings in which like elements are indicated by like numbers. These drawings are not intended to be limiting but are instead provided to illustrate selected embodiments of an apparatus, system and method to schedule OS-sequestered threads for execution.

[0008] Fig. 1 is a block diagram presenting a graphic representation of a general parallel programming approach for a multi-sequencer system.

[0009] Fig. 2 is a block diagram illustrating shared memory and state among threads and shreds for at least one embodiment of user-level multithreading.

[00010] Fig. 3 is a block diagram illustrating various embodiments of multi- sequencer systems.

[00011] Fig. 4 is a data flow diagram illustrating at least one embodiment of a scheduling mechanism for a multi-sequencer multithreading system that supports user- level thread control.

[00012] Fig. 5 is a data flow diagram illustrating at least one embodiment of a scheduling mechanism for an asymmetric multi-sequencer multithreading system.

[00013] Fig. 6 is a data flow diagram illustrating at least one embodiment of a scheduling mechanism that includes multiple work queues.

[00014] Fig. 7 is a timing diagram illustrating at least one embodiment of control flow for user-driven OS-independent scheduling of shreds.

[00015] Fig 8 is a timing diagram illustrating at least one embodiment of control flow for a specific example of user-driven OS-independent scheduling of shreds.

[00016] Fig. 9 is a block diagram illustrating at least one embodiment of a system capable of performing disclosed techniques.

Detailed Description

[00017] The following discussion describes selected embodiments of methods, systems and mechanisms to schedule and/or otherwise control threads of execution (referred to as "shreds") on one or more OS-sequestered sequencers. The OS- sequestered sequencers are sometimes referred to herein as "OS-invisible." The mechanisms described herein may be utilized with single-core or multi-core multithreading systems. In the following description, numerous specific details such as processor types, multithreading environments, system configurations, numbers and topology of sequencers in a multi-sequencer system, microarchitectural structures, and instruction nomenclature and parameters have been set forth to provide a more thorough

understanding of the present invention. It will be appreciated, however, by one skilled in the art that the invention may be practiced without such specific details. Additionally, some well known structures, circuits, and the like have not been shown in detail to avoid unnecessarily obscuring the present invention.

[00018] A shared-memory multiprocessing paradigm may be used in an approach referred to as parallel programming. According to this approach, an application programmer may split a software program, sometimes referred to as an "application" or "process," into multiple tasks to be run concurrently in order to express parallelism for a software program. All threads of the same software program ("process") share a common logical view of memory.

[00019] Fig. 1 is a block diagram illustrating a graphic representation of a parallel programming approach on a multi-sequencer multithreading system. Fig. 1 illustrates processes 100, 120 visible to an operating system 140. These processes 100, 120 may be different software application programs, such as, for example, a word processing program and an email management program. Commonly, each process operates hi a different address space.

[00020] The operating system ("OS") 140 is commonly responsible for managing the user-created tasks for a process, such as process 120 illustrated in Fig. 1. Accordingly, the operating system 140 may create a distinct thread 125, 126 for each of the user-defined tasks associated with a process 120, and may map the threads 125, 126 to thread execution resources. (Thread execution resources are not shown in Fig. 1, but

are discussed in detail below.) The OS 140 is commonly responsible for scheduling these threads 125, 126 for execution on the execution resources. The threads associated with a single process typically have the same view of memory and have visibility to each others' virtual address space.

[00021] Because the OS 140 is responsible for creating, mapping, and scheduling threads, the threads 125, 126 are "visible" to the OS 140. In addition, embodiments of the present invention comprehend additional threads 130 - 136 that are not visible to the OS 140. That is, the OS 140 does not create, manage, or otherwise acknowledge or control these additional threads 130-136. These additional threads, which are neither created nor controlled by the OS 140, are sometimes referred to herein as "shreds" 130 - 136 in order to distinguish them from OS-visible threads. The shreds are created and managed by user-level programs and are scheduled to run on sequencers that are sequestered from the operating system. The OS-sequestered sequencers share the same ring 0 state as OS-visible sequencers. Shreds thus share the same execution environment (address map) that is created for the threads associated with the same process.

[00022] As used herein, the terms "thread" and "shred" include, at least, the concept of a set of instructions to be executed concurrently with other threads and/or shreds of a process. The thread and "shred" terms both encompass the idea, therefore, of a set of software instructions. As used herein, a distinguishing factor between a thread (which is OS-controlled) and a shred (which is not visible to the operating

system and is instead user-controlled), which are both instruction streams, lies in the difference of how execution of the thread and shred instruction streams is managed. A thread is generated in response to a system call to the OS. The OS generates that thread and allocates resources to run the thread. Such resources allocated for a thread may include data structures that the operating system uses to control and schedule the threads.

[00023] In contrast, at least one embodiment of a shred is generated via a user instruction or "primitive" that invokes a software library or other OS-independent mechanism for generating a shred that the OS is not aware of. A shred may thus be generated in response to a user-level software library call.

[00024] Fig. 2 is a block diagram illustrating, in graphical form, further detail regarding the statement, made above, that all threads of the same software program or process share a common logical view of memory. For embodiments of the present invention, this statement is also true regarding shreds associated with a process 100, 120. Fig. 2 is discussed herein with reference to Fig. 1.

[00025] Fig. 2 assumes the graphical representation of a process 120, threads 125, 126 and shreds 130 - 136 illustrated in Fig. 1. However, such representation should not be taken to be limiting. Embodiments of the present invention do not necessarily impose an upper or lower bound on the number of threads or shreds associated with a process. Regarding a lower bound, Fig. 1 illustrates that every process running at a given time need not necessarily be associated with any threads or shreds at all. For

example, Process 0 100 illustrated in Fig. 1 is shown to run without threads nor shreds at the particular time illustrated in Fig. 1.

[00026] However, another process 120 may be associated with one or more threads 125, 126 as illustrated in Fig. 1. In addition, the process 120 may additionally be associated with one or more shreds 130 - 136 as well. The representation of two threads 125, 126 and four shreds 130-136 for a process 120 is illustrative only and should not be taken to be limiting. The number of OS-visible threads associated with a process may be limited by the OS program. However, the upper bound for the cumulative number shreds associated with a process is limited, for at least one embodiment, only by the number of thread execution resources available at a particular time during execution. Fig. 2 illustrates that a second thread 126 associated with a process 120 may have a different number (n) of threads associated with it than the first thread 125. (N may be 0 for either or both of the threads 125, 126.)

[00027] Fig. 2 illustrates that a particular logical view 200 of memory is shared by all threads 125, 126 associated with a particular process 120. Fig. 2 illustrates that each thread 125, 126 has its own application and system state 202a, 202b, respectively. Fig. 2 illustrates that the application and system state 202 for a thread 125, 126 is shared by all shreds (for example, shreds 130 - 136) associated with the particular thread.

[00028] Accordingly, Fig. 2 illustrates that a system for at least one embodiment of the present invention may support a 1 -to-many relationship between an OS-visible thread, such as thread 125, and the shreds 130 - 136 (which are not visible to the OS)

associated with the thread. The shreds are not "visible" to the OS (see 140, Fig. 1) in the sense that a programmer, not the OS, may employ user-level techniques to create, synchronize and otherwise manage and control operation of the shreds. While the OS 140 is aware of, and manages, a thread, the OS 140 is not aware of, and does not manage or control, shreds.

[00029] Thus, instead of relying on the operating system to manage the mapping between thread unit hardware and shreds, for at least one embodiment a user may directly control such mapping and may directly manipulate control and state transfers associated with shred execution. Accordingly, for embodiments of the methods, mechanisms and systems described herein, a user- visible feature of the architecture of the thread units is at least a canonical set of instructions that allow a user direct manipulation and control of thread unit hardware.

[00030] As used herein, a thread unit, also interchangeably referred to herein as a "sequencer", may be any physical or logical unit capable of executing a thread or shred. It may include next instruction pointer logic to determine the next instruction to be executed for the given thread or shred. For example, the OS thread 125 illustrated in Fig. 2 may execute on a sequencer, not shown, while each of the active shreds 130 — 136 may execute on other sequencers, "seq 1" - "seq 4", respectively. A sequencer may be a logical thread unit or a physical thread unit. Such distinction between logical and physical thread units is illustrated in Fig. 3.

[00031] Fig. 3 is a block diagram illustrating selected hardware features of embodiments 310, 350 of a multi-sequencer system capable of performing disclosed techniques. Fig. 3 illustrates selected hardware features of an SMT multi-sequencer multithreading environment 310. Fig. 3 also illustrates selected hardware features of a multiple-core multithreading environment 350, where each sequencer is a separate physical processor core.

[00032] In the SMT environment 310, a single physical processor 304 is made to appear as multiple logical processors (not shown), referred to herein as LP ₁ through LP _n , to operating systems and user programs. Each logical processor LP ₁ through LP _n maintains a complete set of the architecture state ASi - AS _n , respectively. The architecture state includes, for at least one embodiment, data registers, segment registers, control registers, debug registers, and most of the model specific registers. The logical processors LP ₁ - LP _n share most other resources of the physical processor 304, such as caches, execution units, branch predictors, control logic and buses. Although such features may be shared, each thread context hi the multithreading environment 310 can independently generate the next instruction address (and perform, for instance, a fetch from an instruction cache, an execution instruction cache, or trace cache). Thus, the processor 304 includes logically independent next-instruction-pointer and fetch logic 320 to fetch instructions for each thread context, even though the multiple logical sequencers may be implemented in a single physical fetch/decode unit 322. For an SMT or embodiment, the term "sequencer" encompasses at least the next-

instruction-pointer and fetch logic 320 for a thread context, along with at least some of the associated architecture state, AS, for that thread context. It should be noted that the sequencers of an SMT system 310 need not be symmetric. For example, two SMT sequencers for the same physical core may differ in the amount of architectural state information that they each maintain.

[00033] Thus, for at least one embodiment, the multi-sequencer system 310 is a single-core processor 304 that supports concurrent multithreading. For such embodiment, each sequencer is a logical processor having its own instruction next- instruction-pointer and fetch logic and its own architectural state information, although the same physical processor core 304 executes all thread instructions. For such embodiment, the logical processor maintains its own version of the architecture state, although execution resources of the single processor core may be shared among concurrently-executing threads.

[00034] Fig. 3 also illustrates at least one embodiment of a multi-core multithreading environment 350. Such an environment 350 includes two or more separate physical processors 304a - 304n that is each capable of executing a different thread/shred such that execution of at least portions of the different threads/shreds may be ongoing at the same time. Each processor 304a through 304n includes a physically independent fetch unit 322 to fetch instruction information for its respective thread or shred. In an embodiment where each processor 304a-304n executes a single thread/shred, the fetch/decode unit 322 implements a single next-instruction-pointer and

fetch logic 320. However, in an embodiment where each processor 304a-304n supports multiple thread contexts, the fetch/decode unit 322 implements distinct next-instruction- pointer and fetch logic 320 for each supported thread context. The optional nature of additional next-instruction-pointer and fetch logic 320 in a multiprocessor environment 350 is denoted by dotted lines in Fig. 3.

[00035] Thus, for at least one embodiment of the multi-core system 350 illustrated in Fig. 3, each of the sequencers may be a processor core 304, with the multiple cores 304a - 304n residing in a single chip package 360. Each core 304a - 304n may be either a single-threaded or multi-threaded processor core. The chip package 360 is denoted with a broken line in Fig. 3 to indicate that the illustrated single-chip embodiment of a multi-core system 350 is illustrative only. For other embodiments, processor cores of a multi-core system may reside on separate chips.

[00036] For ease of discussion, the following discussion focuses on embodiments of the multi-core system 350. However, this focus should not be taken to be limiting, in that the mechanisms described below may be performed in either a multi-core or single- core multi-sequencer environment.

[00037] Fig. 4 is a data flow diagram illustrating at least one embodiment of a scheduling mechanism 400 for a multi-sequencer multithreading system that supports user-level thread control. Fig. 4 illustrates that the mechanism 400 includes a work queue system 402. The mechanism also includes a scheduler routine 450, which may execute on each of multiple sequencers 403, 404.

[00038] The work queue system 402 may maintain, for at least one embodiment, descriptors for user-defined shreds that are in line for execution are therefore "pending". As is stated above, the scheduling mechanism 400 may be employed rather than an OS- provided scheduling mechanism; each work descriptor describes a shred that is to be executed, independent of OS intervention, on either an OS-sequestered or OS-visible sequencer.

[00039] Shred descriptors may be created by user-level shred creation instructions or primitives. One of skill in the art will recognize that there may be several levels of abstraction between the programmer's code and actual architectural instructions that cause a sequencer to perform actions resulting in the generation of shred descriptors and placement of the descriptors into a work queue 402. As used herein, an instruction described as being generated by a programmer or user is intended to encompass not only architectural instructions that may generated by an assembler or compiler based on user-generated code, or by a programmer working in an assembly language, but also any high-level primitive or instruction that may ultimately be assembled or compiled into architectural shred control instructions. It should also be understood that an architectural shred control instruction may be further decoded by a thread execution unit into one or micro-operations.

[00040] Fig. 4 further illustrates that the scheduler routine 450a, 450b for each of the sequencers may access the work queue system 402 in order to obtain a shred for execution on the associated sequencer 403, 404. Thus, Fig. 4 illustrates at least one

embodiment of a scheduling mechanism that allows user-level mapping and control of shreds, which may be executed concurrently with each other and/or with OS-controlled threads, without OS intervention for scheduling of the shreds.

[00041] It should be noted that the sequencers 403, 404 illustrated in Fig. 4 need not be symmetric, and their number should not be taken to be limiting. Regarding the number of sequencers, the scheduling mechanism 400 may be utilized for any number of sequencers. For example, the scheduling mechanism may be implemented for a multi-sequencer system that includes four, eight, sixteen, thirty-two or more sequencers.

[00042] Regarding symmetry, the sequencers 403, 404 may differ in any manner, including those aspects that affect quality of computation. For example, the sequencers may differ in terms of power consumption, speed of computational performance, functional features, or the like. By way of example, for one embodiment, the sequencers 403, 404 may differ in terms of functionality. For example, one sequencer may be capable of executing integer and floating point instructions, but cannot execute a single instruction multiple data ("SIMD") set of instruction extensions, such as Streaming SIMD Extensions 3 ("SSE3"). On the other hand, another sequencer may be capable of performing all the instructions that the first sequencer can execute, and can also execute SSE3 instructions.

[00043] As another example of functional asymmetry, for an embodiment of the present invention, one sequencer maybe visible to the OS (see, for example, 140 of Fig. 1) and may therefore be capable of performing "ring 0" operations such as performing

system calls, servicing a page fault, and the like. On the other hand, another sequencer may be sequestered from the OS, and therefore be incapable of performing ring 0 operations.

[00044] The sequencers of a system on which the scheduling mechanism 400 is utilized may also differ in any other manner, such as dimensions, word and/or data path size, topology, memory, power consumption, number of functional units, communication architectures (multi-drop vs. point-to-point interconnect), or any other metric related to functionality, performance, footprint, or the like.

[00045] Fig. 5 is a data flow diagram illustrating at least one embodiment of a scheduling mechanism 500 for an asymmetric multi-sequencer multithreading system. Fig. 5 illustrates just one embodiment of asymmetry, in that a first sequencer 503 is visible to an operating system 540 while a second sequencer 504 is sequestered from the operating system 540. Again, one of skill in the art will recognize that other types of asymmetry may be present in the system, and that more than two sequencers may be present in the system.

[00046] Fig. 5 illustrates that the scheduler routine 450a, 450b for each of the OS- visible sequencer 503 and the sequestered sequencer 504 probes the work queue system 502 for shreds to be executed on the sequencer 503, 504 associated with the scheduler instance 450a, 45Ob ₅ respectively. Again, such mechanism 500 provides for scheduling of shreds on asymmetric sequencers without intervention of an operating system 540.

[00047] Fig. 6 is a data flow diagram illustrating further detail for at least one embodiment of a scheduling mechanism 600 for an asymmetric multi-sequencer multithreading system. Fig. 6 illustrates that a work queue system 602 may include multiple work queues 604, 606. For at least one embodiment, each of the multiple work queues 604, 606 is associated with a sequencer type. For the sample embodiment illustrated in Fig. 6, a first queue 604 is associated with a set of one or more of a first type of sequencer 640, referred to herein as "Type A". A second queue 606 is associated with a set of one or more of a second type of sequencer 642, referred to herein as "Type B". As with the other embodiments illustrated in previous figures, the number of queues and sequencer types, as well as number of sequencers of each type as illustrated in Fig. 6 should not be taken to be limiting.

[00048] Fig. 6 illustrates, therefore, a scheduling mechanism 600 for a system that includes at least two types of asymmetric sequencers - Type A sequencers 640 and Type B sequencers 642. The queue system 602 includes a work queue 604, 606 associated with each type of sequencer. Each sequencer 640, 642 includes a portion of a distributed scheduler routine 450. The portions 450a, 450b may be identical copies of each other, but need not necessarily be so.

[00049] Fig. 6 illustrates that the first queue 604 may include shred descriptors for shreds 623 to run on the type A sequencers 640. The second queue 606 may include shred descriptors for shreds 624 to run on the type B sequencers 642. The type A shreds 623 may be software sequences written by a user to take advantage of particular

functionality provided by the type A sequencers 640. Similarly, the type B shreds 624 may be software sequences written by a user to take advantage of particular functionality provided by the type B sequencers 642.

[00050] For at least one embodiment, the functionality of type A and type B sequencers may be mutually exclusive. That is, for example, one set of sequencers 640 may support a particular functionality, such as execution of SSE3 instructions, that the other set of sequencers 642 does not support; while another set of sequencers 642 may support a particular functionality, such as ring 0 operations, that the other set of sequencers 640 does not support.

[00051] For such embodiment, the scheduling mechanism 600 may operate such that descriptors for a first type of shred 623 may be pulled from the first queue 604 by the scheduler routine 450 on a type A sequencer 640, while descriptors for a second type of shred 624 may be pulled from the first queue 604 by the scheduler routine 450 on a type A sequencer 640. The mapping of shreds to an appropriate sequencer may be directed by user-level instructions.

[00052] However, for at least one other embodiment, such as the embodiment illustrated in Fig. 6, the functionality of sequencer types A 640 and B 642 represent a superset-subset functionality relationship rather than a mutually exclusive functionality relationship. That is, a first set of sequencers (such as type A sequencers 640) provide a superset of functionality that includes all functionality of a second set of sequencers (such as type B sequencers 642), plus additional functionality that is not provided by the

second set of sequencers 642. For such embodiment, the shred descriptors from the second queue 606 may be optionally retrieved by the scheduler routine 450 for the type A sequencers 640. This optional processing may be performed, for example, if all descriptors for the type A sequencers are unavailable (such as, for example, if the first queue 604 is empty). In other words, the sequencers with superset functionality may execute shreds specifically designated for such sequencers, but may also execute shreds designated for sequencers of subset functionality.

[00053] Finally, Fig. 6 illustrates that shred descriptors may be placed 660 into the work queue system 602 responsive to shred creation instructions executed by another shred or by a shred-aware thread. For at least one embodiment, the instructions that trigger creation of shred descriptor are API-like ("Application Programmer Interface") thread control primitives such as shred_create. Software, such as that provided by a software library, may create, responsive to a shred_create primitive, a shred descriptor for the new shred and may place it into the work queue system 602.

[00054] For at least one embodiment, a shred descriptor is thus created by software responsive to a shred_create primitive and is placed into the queue system 702. The shred descriptor may be, for at least one embodiment, a record that identifies at least the following properties for a shred: a) the address at which the shed should begin execution and b) a stack descriptor. The stack descriptor identifies the memory storage area (stack) to be used by the new shred to store temporary variables, such as local variables and return addresses.

[00055] Fig. 7 is a data flow diagram illustrating a sample sequence for utilizing an embodiment of a scheduling mechanism, such as any of those illustrated in Figs. 4-6, to perform user-level scheduling of shreds for execution on a multi-sequencer system. For at least one embodiment, a method 700 is performed by a thread, T, running on a sequencer 770. For at least one embodiment, the method 700 is performed responsive to instructions generated by a user, such as a programmer. For Fig. 7, such instructions are referred as instructions of a thread, T. It will be understood, of course, that the blocks of the method 700 may be programmed by a user in a different sequence than that shown in Fig. 7.

[00056] Fig. 7 illustrates a sample embodiment of a multi-sequencer system wherein the sequencers are asymmetric with respect to a functional metric. That is, the sequencer 770 on which the thread T is executed is an OS-visible sequencer. As such, the sequencer 770 may, via interaction with the operating system, perform privileged operations, such as ring 0 operations (i.e., service page faults, perform system calls, etc.) hi contrast, the sample embodiment illustrated in Fig. 7 also includes sequestered sequencers 760 that are not visible to the OS. As such, the sequestered sequencers 760 are not able to perform the privileged operations. They may, for example, be limited to performing ring 3 operations but cannot perform ring 0 operations. Fig. 7 thus illustrates a sample embodiment of a multi-sequencer system that includes asymmetric sequencers.

[00057] For the embodiment illustrated in Fig. 7, the method 700 begins at block 701 and proceeds to block 704. At block 704, the thread, T, executes one or more instructions to perform initialization for user-level control of one or more concurrent shreds associated with thread T. For at least one embodiment, such initialization 704 may include creation and/or initialization of one or more queues (see, e.g., 604 and 606 of Fig. 6) of a work queue system 702.

[00058] In addition, Fig. 7 illustrates that such initialization 704 may include initialization of one or more sequestered sequencers 760. One of skill in the art will recognize, of course, that the queue creation and sequencer initialization illustrated at block 704 need not necessarily be performed by a single routine, method, or function, nor need they necessarily be performed in the sequential order discussed herein. For an alternative embodiment, for example, the initialization actions taken at block 704 may be performed in the opposite order, or may be performed concurrently.

[00059] The sequencer initialization of block 704 may be performed in order to prepare the sequestered sequencer(s) 760 to run shreds as indicated by user-generated shred creation instructions placed into the instruction stream for thread T by a user, such as a computer programmer. For at least one embodiment, such sequencer initialization may include the invocation of the distributed scheduler routine 450 on each of the sequestered sequencers 760. As is discussed in further detail below, invocation of the scheduler routine 450 may result in execution of a scheduling loop, wherein each

portion of the distributed scheduler begins to periodically check the work queue system 702 for work that may be performed by the associated sequestered sequencer 760.

[00060] Without such initialization 704, it is anticipated for at least one embodiment that the sequestered sequencer(s) 760 are not available to execute shreds. For at least one embodiment, the initialization 704 that prepares a sequencer 760 to execute shreds includes, at least, invocation of a scheduler routine 450. The initialization 704 may be performed for each sequencer indicated by a user in the instruction stream for thread T. For at least one embodiment, all sequencers initialized for execution by the thread T shared the same view of virtual memory that is constructed by the operating system for the process with which thread T is associated.

[00061] Processing proceeds from block 704 to block 706. At block 706, the thread, T, generates a shred descriptor for execution beginning at a user-defined main- shred function. For at least one embodiment, the user-defined main-shred function replaces a traditional OS-visible main() function. Fig. 7 illustrates that the shred descriptor for the main-shred may be placed into a work queue system 702. Accordingly, the main-shred is placed in line to be scheduled for execution by one of the distributed shred scheduler routines 450 at the next opportunity.

[00062] From block 706, processing proceeds to block 708. At block 708, the OS- visible thread T begins its own scheduling loop. Accordingly, thread T executes one or more instructions to initialize a scheduler routine 750 on sequencer 770. Fig. 7 illustrates that scheduler 750 may consult the work queue system 702 to schedule work

on the first sequencer 770. Fig. 7 thus illustrates an embodiment wherein all shred descriptors are sent to a common queue system 702 from which all scheduler routines 450a-450n, 550 can request and schedule work. Because instances of the scheduling routine 450a - 45On are distributed among multiple sequestered sequencers, the scheduling routine 450 may be referred to herein as a "distributed" scheduler.

[00063] For at least one embodiment, the OS-visible sequencer 770 (a first type) is associated with a first work queue of the queue system 702 (see, e.g., 604 of Fig. 6) while the sequestered sequencers (a second type) are associated with a second work queue of the queue system 702 (see, e.g., 606 of Fig. 6). As is explained above in connection with Fig. 6, for at least one embodiment the first sequencer 770 may pull work from any queue in the work queue system 702.

[00064] In this manner, both the OS-visible sequencer 770 and the one or more sequestered sequencers 760 have been initialized 704, 706 to be capable of executing of shreds based on user-generated instructions rather than relying on an OS to manage such scheduling. As illustrated, at least one mechanism for doing so is to invoke a scheduler routine 450 on the initialized (see block 704) sequestered sequencers 760 as well invoking a more common OS-visible scheduler routine 750 on the sequencer 770 that is to generate the shred descriptor for the main-shred routine (see block 708). From block 708, processing proceeds to block 710.

[00065] At block 710, the sequencer 770 executes instructions as scheduled on the sequencer 770 by its scheduler 750. The nature of block 710 may be iterative, in that

multiple shred or thread portions may be executed sequentially on the sequencer770. The potentially iterative nature of block 710 processing is denoted in Fig. 7 with a broken line.

[00066] The sequencer 770 may execute shred instructions as indicated by the shred descriptors in the queue of the work queue system 702 that is associated with its sequencer type. Such instructions may include those that cause additional shred descriptors to be generated and placed into the work queue system 702.

[00067] For at least one embodiment, shreds that are executed by sequencer 770 may include those that require functionality provided by the sequencer type of 770, which cannot be performed by the sequestered sequencers 760. For the example illustrated in Fig. 7, for instance, the queue associated with sequencer 770 may include shred instructions that require ring 0 operations. That is, operating system calls executed by a shred may require special handling because the OS can only service system calls issued from an OS-visible sequencer. System calls for shreds running on the OS-sequestered sequencers 760 may be executed by an OS-visible sequencer 770 via proxy mechanism.

[00068] The proxy mechanism that allows an OS-visible sequencer to impersonate a shred in order to get the attention of the operating system to make forward progress for a privileged instruction. The proxy mechanism may be implemented in any number of manners. For example, such mechanism may be invoked implicitly when a sequestered sequencer 760 attempts to perform a privileged instruction, such as a

system call, on an OS-sequestered sequencer. The attempted system call may cause an exception to be generated. The handler for the exception may save the shred state, generate a shred descriptor for the system call, and place the shred descriptor in the queue associated with the OS-visible sequencer 702. When such descriptor is scheduled for execution 710 on the OS-visible sequencer 770, the sequencer 770 may save its own state before obtaining the shred's state and executing the system call. For such proxy execution, the execution on the OS-visible sequencer 770 resumes at instruction that caused the fault on the OS-sequestered sequencer 760. The shred's state is then saved, and control is returned to the faulting OS-sequestered sequencer 760 so that it may continue execution of its shred.

[00069] For at least one other embodiment, the proxy execution for privileged instructions may be explicitly controlled by the user. For such embodiment, the user may explicitly indicate that a shred that includes a privileged instruction should be executed on an OS-visible sequencer 770. Such indication may be expressed as a parameter, or variant, of a shred creation primitive. The inclusion of a queue preference parameter, or the use of a specialized shred creation primitive, are both collectively referred to herein as scheduling hints.

[00070] A shred creation primitive (such as, for example, "shred_create()") primitive may invoke a library function. The function may cause creation of a new shred descriptor and may further cause placement of the descriptor into a queue of the work queue system 702. An optional parameter of the standard shred_create() primitive

may allow the user to specify a queue type for the descriptor. Alternatively, an optional function call, referred to herein as shred_create_affmity(), may be utilized by the user to express a queue preference for the particular shred. For example, if the shred consists of a large number of system calls, the user may specify a work queue associated with a sequencer type capable of performing ring 0 operations. One of skill in the art will recognize that similar parameter and/or alternative parameter approaches may also be utilized with a yield primitive.

[00071] The system illustrated in Fig. 7 may include additional safeguards related to the handling of system calls. For example, a mutex may be utilized to provide serialization around OS calls. The mutex may be utilized to minimize deadlock that could otherwise occur if a thread and one or more of its associated shreds attempt to acquire the same lock provided by the operating system for thread synchronization.

[00072] For any shred, or shred portion, executed at block 710, the sequencer 770 may continue execution of the shred until an exit or yield primitive is executed. If an "exit" primitive is encountered in the current shred, execution of the current shred is complete and the scheduler 750 may prod an appropriate queue of the queue system 702 to schedule another sequence of instructions for execution on the sequencer 770. For at least one embodiment, the exit instructions thus marks the shred as being complete and control returns back to the scheduler 750.

[00073] If a "yield" primitive is encountered in the current shred, a shred descriptor for the calling process may be placed back into the queue system and control returned to

the scheduler 750. Accordingly, upon execution of a "yield" primitive, a shred descriptor for the remaining shred instructions for the current shred may be placed into the work queue system 702.

[00074] For either the exit or yield case, after execution of a current shred has completed at block 710, processing may end at block 712. Alternatively, a new thread or shred may be scheduled for execution. Accordingly, the broken line looping from block 710 back to block 710 indicates that additional threads or shreds may be executed by the sequencer 770. After execution of all threads and/or shreds (or shred portions) as scheduled by the distributed scheduler 750 is complete, processing may end at block 712.

[00075] Regarding scheduling of additional shreds after execution of block 710, a new shred may be scheduled for execution, by the distributed scheduler 750, on the current sequencer 770 after the yield or end instruction has been executed. For at least one embodiment, such action occurs as a result of the scheduler 750 obtaining work from a work queue associated with a sequencer type other than its own.

[00076] Alternatively, additional OS-visible instructions may be executed after block 710. Such work may be scheduled from a work queue associated with the sequencer's 770 own sequencer type. As is described above, for at least one embodiment the sequencer's 770 scheduler 750 only schedules shreds originally designated to execute on OS-sequestered sequencers if its own queue is empty.

[00077] Fig. 8 is a timing diagram illustrating at least one embodiment of control flow for OS-independent scheduling of shreds. For purposes of illustration, the timing diagram of Fig. 8 illustrates control flow for a specific example of the method embodiment 700 illustrated in Fig. 7. Accordingly, Fig. 8 is discussed below with reference to Fig. 7. As is illustrated in Fig. 7, the method 800 illustrated in Fig. 8 may be performed by a first OS-visible sequencer (870) and a second, sequestered, sequencer (860).

[00078] Fig. 8 illustrates further detail for at least one embodiment of initialization 704 (see Fig. 7) on a first sequencer 870 of a first type. For at least one embodiment, the sequencer 870 is distinguished from the second sequencer 860 in terms of a functional metric. That is, the first sequencer 870 is OS-visible, while the second sequencer 860 is sequestered.

[00079] Fig. 8 illustrates that the initialization 704 may include execution of at least two user-generated instructions. The first instruction, referred to herein as a shred monitor ("SEMONITOR") instruction, is used to set up a channel for communication between an OS-visible sequencer and an OS-sequestered sequencer. A second initialization instruction is a control transfer instruction, referred to herein as a shred transfer ("SXFR") instruction.

[00080] One of skill in the art will recognize that the methods 700, 800 and systems described herein need not be limited to any particular nomenclature or instruction set architecture for shred control instructions. One of skill in the art will also recognize that

the SEMONITOR- and SXFR-type instructions may be generated, as a result of a compiler, assembler, or library function, for a primitive, such as shred_create(), shred_create_affmity(), shred_yield(), and shred_exit() discussed above. In this sense, the ultimate instructions are nonetheless referred to herein "user-generated", although they may have been generated responsive to a higher-level user construct. The foregoing partial list of shred primitives should not be taken to be limiting.

[00081] One of skill in the art should also recognize that the appended claims encompass embodiments wherein the ultimate shred control instructions are not necessarily hardware-supported instructions. Instead, for at least one embodiment, the "instructions" generated in response to the primitives may be, instead of architectural instructions, hardware signals. The signals, such as interrupts or other sequencer control signals, may be generated by a software or firmware layer in response to user- generated primitives in order to implement SEMONITOR- and SXFR-type functionality.

[00082] Instead, the methods 700, 800 generically assume that the sequencers 870, 860 support architectural, user-level instructions for generating and controlling shred and that traditional multithreading API's may be fully implemented using the canonical shred instruction set. For further discussion of one sample set of canonical shred instructions, one may refer to copending patent application U.S Pat. App. Ser. No., , Attorney docket number 42390.P19770, entitled "A Mechanism For Instruction Set- Based Thread Execution on a Plurality of Instruction Sequencers."

[00083] For at least one embodiment, it is assumed that the sequencers 870, 860 support at least a canonical set of shred instructions, which includes instructions along the lines of SEMONITOR and SXFR. The capability of a user to utilize this type of instruction to control the operation of multiple sequencers and, in particular, to effect inter-sequencer transfer of control, is referred to herein as "sequencer arithmetic." The shred instructions that provide for sequencer arithmetic may be utilized as building blocks to construct higher-level shredding primitives, such as fork, join, yield, exit, lock, wait, and unlock, to name a few. These primitives may be utilized to allow user- level management of concurrent shared-memory shreds that execute without OS-level scheduling. In this manner, the canonical shred instructions may be utilized to build an abstraction layer between the architectural shred instructions and application software. The canonical shred instructions may thus be utilized to build an abstraction layer that supports legacy application program interfaces.

[00084] Generally, the SXFR-type instructions may, when executed by a first sequencer, send a signal from the first sequencer to a second sequencer. The SXFR- type instruction therefore provides a mechanism to perform inter-shred signaling for service. The action of an SXFR-type instruction is synchronous, in the sense that the user can control, by judiciously placing the SXFR-type instruction into code, the timing of the execution of the SXFR instruction in relation to execution of other instructions in the shred instruction stream.

[00085] In contrast, the SEMONITOR-type instruction provides for asynchronous sequencer arithmetic. The SEMONITOR-type instructions may be used by the programmer to configure a second sequencer to monitor for a signal from the first sequencer.

[00086] Fig. 8 illustrates that the first initialization instruction, the SEMONITOR instruction, maps on the specified target sequencer a handler routine to a specific event type. That is, the embodiment of the SEMONITOR instruction illustrated in Fig. 8 maps the beginning instruction pointer address for a particular routine (here, the beginning of the distributed scheduler routine, "Scheduler",) to an event type, "init" for the specified sequencer id, 1. The result, on the target sequencer, of execution of the SEMONITOR instruction on the first sequencer is that the target sequencer monitors the indicated event indicator and begins a flow of control on the target sequencer at the designated address.

[00087] The event type may be implemented for at least one embodiment as an indicator, such as a signal line that becomes activated when the specified event is indicated by the first sequencer. As a result of execution of the SEMONITOR instruction on the first sequencer SIDO 870, the target sequencer, SIDl 860, as identified by the "1" as the sequencer identifier parameter of the SEMONITOR instruction, polls or otherwise monitors the "init" event indicator. When the init indicator is activated, SIDl 860 jumps to the specified address (that is, to the beginning of the scheduler routine) and begins execution of the scheduler code. The Scheduler

routine is merely used as an example in Fig. 8; one of skill in the art will recognize that any appropriate event-handler code may be utilized. Execution of the SEMONITOR instruction may thus set up an asynchronous control transfer to occur on the specified sequencer.

[00088] The SEMONITOR initialization illustrated in Fig. 8 is specific to the example illustrated in Fig. 8, and should not be taken to be limiting. A set of several initialization instructions, similar to the SEMONITOR instruction illustrated in Fig. 8, may be performed for alternative embodiments in order to map a series of event types to specific handler routines. After execution of the SEMONITOR instruction illustrated in Fig. 8, event type "init" is associated with the distributed scheduler.

[00089] A second initialization instruction, the SXFR instruction, is then executed by the first sequencer 870. Execution of the SXFR instruction sends an event to initialize a separate thread of execution on the OS-sequestered sequencer SIDl. At 802, when the OS-sequestered sequencer SIDl detects the signal containing the "Init" scenario, the OS-sequestered sequencer SIDl starts a thread S, and commences execution of an instruction stream corresponding to the distributed scheduler (see, e.g., 450 of Fig. 7).

[00090] Fig. 8 illustrates an embodiment of the SXFR instruction that includes the following parameters: a sequencer identifier, an event type, and an optional synchronization flag. The sequencer identifier ("SID") specifies the sequencer to which control is to be transferred as a result of execution of the SXFR instruction. For the

example illustrated in Fig. 8, the sequencer identifier for sequencer 870 is "SIDO" and the sequencer identifier for sequencer 860 is "SIDl."

[00091] The event type parameter for the SXFR instruction illustrated in Fig. 8 indicates an "init" event type. As a result of prior execution of the SEMONITOR instruction illustrated in Fig., the "init" event type has been associated with the distributed scheduler (450, Fig. 7) at the time that the SXFR instruction is executed. Accordingly, the event type parameter effectively indicates to the transferee sequencer (that is, to SIDl 860), that is should begin execution at the IP address for the distributed scheduler when the sequencer 860 begins execution.

[00092] The final parameter of the sample SXFR instruction illustrated in Fig. 8 is an optional wait/no wait flag. By specifying "no wait", the instruction essentially begins a concurrent multithreading situation - the first sequencer 870 need not wait for a synchronization indicator from the transferee sequencer 860 , but may continue with execution of its own instruction stream as soon as it has completed execution of the SXFR instruction.

[00093] The SXFR instruction is thus a shred creation and control transfer instruction. Fig. 8 illustrates that, as a result of the first sequencer's execution of the SXFR instruction, a shred S begins executing the distributed handler routine on the second sequencer 860. After execution of the SXFR instruction, therefore, thread T and shred S may execute concurrently.

[00094] Execution of thread T on SIDO 870 may continue execution after it has executed the SXFR instruction at 704, due to the "no wait" flag. Accordingly, thread T goes on to create 706 a shred descriptor for a "main-shred" program and to place such descriptor in the work queue system 702. As is discussed above in connection with Fig. 7, a main-shred descriptor may be placed into a work queue system 702 (Fig. 2), where the queue is associated with the sequestered sequencer SIDl 860.

[00095] Thread T then invokes 708 its OS-visible scheduler 750, and goes on to execute instructions 710 as directed by its scheduler 750. As is discussed above in connection with Fig. 7, the thread T may optionally look for shred descriptors 709, and may therefore execute shred instructions at 710, at least for those embodiments that a) permit the OS-visible scheduler to pull work from the OS-sequestered sequencer queue and/or b) that support proxy execution of shred instructions on the OS-visible sequencer 870. After executing one or more iterations of instruction execution 710, processing for the thread T may end at 712.

[00096] Regarding the sequestered sequencer 860, Fig. 8 illustrates at least one embodiment of a sequence of operations to be performed for a shred, S. As is discussed above, a scheduling loop (i.e., execution of the distributed scheduler 450) may be initiated on the sequencer 860 as a result of initialization processing 704 performed by the OS-visible thread, T, executing on the OS-visible sequencer SID 870.

[00097] At 804, the scheduler routine 450 running on SIDl 860 queries the work queue system 702 to determine if there are any shred descriptors available for execution

by the sequencer 860. For the sample sequence illustrated in Fig. 8, the sequencer SIDl 860 may encounter the main-shred descriptor at block 806, if block 706 (discussed above) has already been performed by SE)O 870.

[00098] If a descriptor is found, the sequencer 860 executes 806 the shred indicated by the descriptor. The shred is executed until a) completion or b) a yield instruction is encountered.

[00099] It should be noted that instructions of a shred may include additional shred creation instructions. That is, for at least one embodiment, a shred may itself perform at least two types of shred operations: shred control operations (related to creating and terminating shreds) and synchronization operations (such as a yield operation). (For at least one embodiment, such operations may be invoked by a user's use of an industry standard-compliant primitive in the shred instruction stream. One such industry standard, for example, is POSIX (Portable Operating System Interface).) A broken line from 806 to the work queue system 702 in Fig. 8 illustrates that additional shred descriptor(s) may be created during execution 806 of a shred, and that the additional descriptor(s) may be placed into the work queue system 702.

[000100] For example, a yield primitive may be placed into the shred instruction stream by the user. Such yield primitive may cause the sequencer to yield control of the shred. In so doing, the sequencer 860 may place a descriptor for the remainder of the shred instructions back into the work queue system 702 prior to terminating execution of the shred. A yield primitive may invoke a function that swaps the stack before

performing transfer of control, such that the current thread context is saved before returning control to the sequencer's scheduler.

[000101] Fig. 8 illustrates that execution of the shred, S, on SIDl 860 is terminated or suspended when a termination instruction (exit, yield, etc.) is executed. At such time, distributed scheduler 450 may optionally probe the work queue system 702 for another shred descriptor in order to identify work that may be performed by sequencer SIDl 860. Such optional iterative probe of the work queue system 702 is indicated in Fig. 8 by the broken arrow from 806 to 804.

[000102] For the example illustrated in Fig. 8, it should be noted that it is possible that the scheduler routine 450 running on SIDl 860 may likely find the shred descriptor for the "main-shred" routine before the scheduler routine 450 running on SIDO 870 probes the work queue system 702. In such case, assuming that either sequencer 860, 870 is capable of executing the "main-shred" routine, the "main-shred" routine may be executed by SIDl 860, even though its descriptor was created by SIDO 870.

[000103] It will be apparent from the foregoing discussion of Fig. 8 that, for at least one embodiment, the two illustrated sequencers 860, 870 are both capable of performing the "main-shred" routine. Accordingly, it may be assumed that, for at least one embodiment, the sequencers 860, 870 are symmetric. For at least one other embodiment, however, the sequencers 860, 870 are not fully symmetric, but each sequencer 860, 870 nonetheless has sufficient functional capability to execute the main- shred routine. Such embodiment may exist, for example, where the main-shred routine

requires only ring 3 capability, sequencer SIDl 860 has ring 3 capability, and SIDO 870 has both ring 3 and ring 0 capability. For such embodiment, as illustrated in Fig. 6, the work queue system 702 may include a first queue (for descriptors that can only be performed by SIDO 870, for example) and a second queue for descriptors that can be performed by either SIDO 870 or SIDl 860.

[000104] Regarding the distributed scheduler 450 and the work queue system 702, for at least one embodiment, the work queue system 702 may be protected by a critical section. As is discussed above, each sequestered sequencer 860 may effectively run its own copy of a portion of the distributed the scheduler routine 450 and attempt to contend access to the head of a queue of the task queue system 702 to process the next ready instruction to run its associated sequencer. Should one task on a sequencer be waiting for a synchronization variable such as mutex, a conditional variable, or a semaphore, the task may be de-scheduled and put at the tail of the a task queue after entering the corresponding critical section.

[000105] Given the example illustrated in Fig. 8, it can be seen that embodiments of the present invention are capable of scheduling threads of execution on sequencers that are sequestered from the OS, without using an OS. Thus, by virtue of the techniques disclosed herein it is possible to build a multi-sequencer system with more sequencers than an OS has the ability to support and to allow user-level scheduling of threads on sequencers of the multi-sequencer system that are not supported by the OS.

[000106] For at least some embodiments of the mechanisms, systems, and methods described above, a distributed scheduler 450 operates as an event-driven self-scheduler where shreds are created in response to queued scheduling events that are created as a result of API-like thread control (shred_create, shred_create affinity, and the like) or thread synchronization (shred_yield and the like) primitives. The scheduler and primitives may be implemented as part of a run-time library that creates an intermediate layer of abstraction between a traditional industry standard API, such as a POSIX- compliant API, and the hardware of a multi-sequencer system wherein the sequencers support at least a canonical set of shred instructions that implement user-level sequencer arithmetic. The library may include a scheduler, such as an embodiment of the distributed scheduler 450 discussed above. The run-time library may act as an intermediate level of abstraction so that a programmer may utilize a traditional thread API (such as, for instance, Pthreads API or Win32 Threads API) with hardware that supports shredding. The library may provide functions that transparently invoke the canonical shred instructions, based on user-programmed primitives.

[000107] The scheduling mechanism and techniques discussed herein may be implemented on any multi-sequencer system, including a single-core SMT system (see, e.g., 310 of Fig. 3) and a multi-core system (see, e.g., 350 of Fig. 3). Further discussion of such systems is discussed below in connection with Fig. 9.

[000108] Fig. 9 illustrates at least one sample embodiment of a computing system 900 capable of performing disclosed techniques. The computing system 900 includes at

least one processor core 904 and a memory system 940. Memory system 940 may include larger, relatively slower memory storage 902, as well as one or more smaller, relatively fast caches, such as an instruction cache 944 and/or a data cache 942. The memory storage 902 may store instructions 910 and data 912 for controlling the operation of the processor 904. The instructions 910 may include one or more copies of the distributed scheduler routine 450.

[000109] Memory system 940 is intended as a generalized representation of memory and may include a variety of forms of memory, such as a hard drive, CD-ROM, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory and related circuitry. Memory system 940 may store instructions 910 and/or data 912 represented by data signals that may be executed by processor 904. The instructions 910 and/or data 912 may include code and/or data for performing any or all of the techniques discussed herein. For example, the data 912 may include one or more queues to form a queue system 702 capable of storing shred descriptors as described above. Alternatively, the instructions 910 may include instructions to generate a queue system 702 for storing shred descriptors.

[000110] The processor 904 may include a front end 920 that supplies instruction information to an execution core 930. Fetched instruction information may be buffered in a cache 225 to await execution by the execution core 930. The front end 920 may supply the instruction information to the execution core 930 in program order. For at least one embodiment, the front end 920 includes a fetch/decode unit 322 that

determines the next instruction to be executed. For at least one embodiment of the system 900, the fetch/decode unit 322 may include a single next-instruction-pointer and fetch logic 320. However, in an embodiment where each processor 904 supports multiple thread contexts, the fetch/decode unit 322 implements distinct next-instruction- pointer and fetch logic 320 for each supported thread context. The optional nature of additional next-instruction-pointer and fetch logic 320 in a multiprocessor environment is denoted by dotted lines in Fig. 9.

[000111] Embodiments of the methods described herein may be implemented in hardware, hardware emulation software or other software, firmware, or a combination of such implementation approaches. Embodiments of the invention may be implemented for a programmable system comprising at least one processor, a data storage system (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. For purposes of this application, a processing system includes any system that has a processor, such as, for example; a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[000112] A program may be stored on a storage media or device (e.g., hard disk drive, floppy disk drive, read only memory (ROM), CD-ROM device, flash memory device, digital versatile disk (DVD), or other storage device) readable by a general or special purpose programmable processing system. The instructions, accessible to a processor in a processing system, provide for configuring and operating the processing

system when the storage media or device is read by the processing system to perform the procedures described herein. Embodiments of the invention may also be considered to be implemented as a machine-readable storage medium, configured for use with a processing system, where the storage medium so configured causes the processing system to operate in a specific and predefined manner to perform the functions described herein.

[000113] Sample system 900 is representative of processing systems based on the Pentium®, Pentium® Pro, Pentium® II, Pentium® III, Pentium® 4, and Itanium® and Itanium® 2 microprocessors available from Intel Corporation, although other systems (including personal computers (PCs) having other microprocessors, engineering workstations, personal digital assistants and other hand-held devices, set-top boxes and the like) may also be used. For one embodiment, sample system may execute a version of the Windows™ operating system available from Microsoft Corporation, although other operating systems and graphical user interfaces, for example, may also be used.

[000114] While particular embodiments of the present invention have been shown and described, it will be obvious to those skilled hi the art that changes and modifications can be made without departing from the scope of the appended claims. For example, the work queue system 702 may include a single queue that is contended by multiple sequencer types. For such embodiment, resource requirements are expressly included in each shred descriptor. Each sequencer's portion of the distributed scheduler does a check to make sure that the sequencer is capable of executing a shred

before the shred's descriptor is removed from the work queue for execution by the sequencer.

[000115] For another alternative embodiment, a hybrid approach may be implemented in the work queue system 702. For such hybrid approach, the work queue system 702 includes multiple queues. Nonetheless, one or more of the queues corresponds to multiple sequencer types. For such hybrid embodiment, an express resource indication is included in each shred descriptor for any queue that is associated with more than one sequencer type.

[000116] The distributed scheduler routine discussed above may schedule execution of shred descriptors using a round-robin scheduling approach. However, for at least one alternative embodiment, the distributed scheduler and work queue system discussed above may support a priority-based scheduling scheme. Also, for at least one alternative embodiment, queue preferences for a multi-queue queue system embodiment can be set by the user, or by the run-time library, to direct that tasks are executed by a preferred sequencer. For example, a shred that is executed on a particular sequencer before a yield operation may be sent back to that same particular sequencer when remaining instructions of the shred are executed.

[000117] Regarding dispatch of shred descriptors into the work queue system, it has been described above in connection with Fig. 7 that assignment of shred descriptors may be expressly controlled statically by a user's use of a particular parameter or

particular "affinity" primitive in order to define, at the time a shred is created, which queue is should be assigned to.

[000118] For at least one alternative embodiment, dynamic assignment may be utilized instead of, or in addition to, such static assignment approaches. Dynamic assignment may be implemented through a user-level synchronization object, such as a mutex. Such mutex may allow a shred to dynamically self-schedule a spawnee shred into a particular queue of the work queue system. Alternatively, dynamic assignment may be implemented such that the distributed scheduler observes the behavior of the queues and dynamically selects which queue should receive a descriptor. Also alternatively, dynamic assignment may be performed through the programmer's use of synchronization primitives, such as "lock" and "unlock" primitives. Such primitives may utilize OS mutex constructs. Upon execution of a lock primitive, the current sequencer may execute a yield operation and place the descriptor for the current shred into a queue corresponding to the mutex.

[000119] Accordingly, one of skill in the art will recognize that changes and modifications can be made without departing from the present invention in its broader aspects. The appended claims are to encompass within their scope all such changes and modifications that fall within the true scope of the present invention.