The dequeue operation schedules a task for execution based on the user-supplied priority. However, priority is not enforced strictly; priority is only respected on a per processor basis. The task queue may schedule a low priority task on one processor before a high priority task on another processor.
Our interpretation of priority is very different from that of a sequential priority queue, in that we use priority as a performance related ``hint'', instead of as a strict synchronization constraint. Enforcing global priority is expensive in a distributed environment, as a dequeue operation would require communication to find the top priority task. Strict global priority also lowers parallelism since low priority tasks can not start before all higher priority tasks finish. Relaxing the priority constraint, however, may lead to more work for certain applications.
For certain applications, the termination of the program depends on the fact that the highest priority task be processed in finite time. Although the task queue does not enforce global priority, it still works in such cases - since the highest priority task must have the highest priority on some processor, it will be dequeued and executed in finite time.
The setting of priority involves the following tradeoffs. In general, it is advantageous to give high priority to tasks that create more tasks for parallelism. The increase in parallelism, however, may again lead to redundant work in branch-and-bound applications. Also, the more tasks there are, the more overhead their scheduling incurs. It seems desirable for the task queue to adapt its scheduling policy in response to the load of the system. We have not addressed this problem in our current implementation.
To invoke the dequeue operation, the programmer supplies the name of the task queue, a buffer to hold the dequeue task, and a buffer to hold general information on the task (such as priority and cost estimates). The primitive returns with a code indicating the status of dequeue: success, failure, or termination. A success code means that a task has been dequeue and placed in the user supplied buffer; a failure code means that no task is found, but some task may become available in the future; a termination code means that no task is found and no task will ever be placed in the task queue. In response to a falure code, the programmer can spin on the dequeue primitive until a task is found, or switch to other computation to hide the latency of obtaining a task; in response to a termination code, the programmer enters the last phase of the program, which usually combines the results of different processors and terminates.