Introduction
StarPU's Modularized Schedulers are made of individual Scheduling Components Modularizedly assembled as a Scheduling Tree. Each Scheduling Component has an unique purpose, such as prioritizing tasks or mapping tasks over resources. A typical Scheduling Tree is shown below.
|
starpu_push_task |
|
v
Fifo_Component
| ^
| |
v |
Eager_Component
| ^
| |
v |
--------><--------------><--------
| ^ | ^
| | | |
v | v |
Fifo_Component Fifo_Component
| ^ | ^
| | | |
v | v |
Worker_Component Worker_Component
When a task is pushed by StarPU in a Modularized Scheduler, the task moves from a Scheduling Component to an other, following the hierarchy of the Scheduling Tree, and is stored in one of the Scheduling Components of the strategy. When a worker wants to pop a task from the Modularized Scheduler, the corresponding Worker Component of the Scheduling Tree tries to pull a task from its parents, following the hierarchy, and gives it to the worker if it succeded to get one.
Using Modularized Schedulers
Existing Modularized Schedulers
StarPU is currently shipped with the following pre-defined Modularized Schedulers :
- Eager-based Schedulers (with/without prefetching) :
Naive scheduler, which tries to map a task on the first available resource it finds.
- Prio-based Schedulers (with/without prefetching) :
Similar to Eager-Based Schedulers. Can handle tasks which have a defined priority and schedule them accordingly.
- Random-based Schedulers (with/without prefetching) :
Selects randomly a resource to be mapped on for each task.
- HEFT Scheduler :
Heterogeneous Earliest Finish Time Scheduler. This scheduler needs that every task submitted to StarPU have a defined performance model (Performance Model Calibration) to work efficiently, but can handle tasks without a performance model.
It is currently needed to set the environment variable STARPU_SCHED to use those Schedulers. Modularized Schedulers' naming is tree-*
An Example : The Tree-Eager-Prefetching Strategy
|
starpu_push_task |
|
v
Fifo_Component
| ^
Push | | Can_Push
v |
Eager_Component
| ^
| |
v |
--------><-------------------><---------
| ^ | ^
Push | | Can_Push Push | | Can_Push
v | v |
Fifo_Component Fifo_Component
| ^ | ^
Pull | | Can_Pull Pull | | Can_Pull
v | v |
Worker_Component Worker_Component
Interface
Each Scheduling Component must follow the following pre-defined Interface to be able to interact with other Scheduling Components.
- Push (Caller_Component, Child_Component, Task)
The calling Scheduling Component transfers a task to its Child Component. When the Push function returns, the task no longer belongs to the calling Component. The Modularized Schedulers' model relies on this function to perform prefetching.
- Pull (Caller_Component, Parent_Component) -> Task
The calling Scheduling Component requests a task from its Parent Component. When the Pull function ends, the returned task belongs to the calling Component.
- Can_Push (Caller_Component, Parent_Component)
The calling Scheduling Component notifies its Parent Component that it is ready to accept new tasks.
- Can_Pull (Caller_Component, Child_Component)
The calling Scheduling Component notifies its Child Component that it is ready to give new tasks.
Build a Modularized Scheduler
Pre-implemented Components
StarPU is currently shipped with the following four Scheduling Components :
- Flow-control Components : Fifo, Prio
Components which store tasks. They can also prioritize them if they have a defined priority. It is possible to define a threshold for those Components following two criterias : the number of tasks stored in the Component, or the sum of the expected length of all tasks stored in the Component.
- Resource-Mapping Components : Mct, Heft, Eager, Random, Work-Stealing
"Core" of the Scheduling Strategy, those Components are the ones who make scheduling choices.
- Worker Components : Worker
Each Worker Component modelize a concrete worker.
- Special-Purpose Components : Perfmodel_Select, Best_Implementation
Components dedicated to original purposes. The Perfmodel_Select Component decides which Resource-Mapping Component should be used to schedule a task. The Best_Implementation Component chooses which implementation of a task should be used on the choosen resource.
Progression And Validation Rules
Some rules must be followed to ensure the correctness of a Modularized Scheduler :
- At least one Flow-control Component without threshold per Worker Component is needed in a Modularized Scheduler, to store incoming tasks from StarPU and to give tasks to Worker Components who asks for it. It is possible to use one Flow-control Component per Worker Component, or one for all Worker Components, depending on how the Scheduling Tree is defined.
- At least one Resource-Mapping Component is needed in a Modularized Scheduler. Resource-Mapping Components are the only ones who can make scheduling choices, and so the only ones who can have several child.
Implement a Modularized Scheduler
The following code shows how the Tree-Eager-Prefetching Scheduler shown in Section An Example : The Tree-Eager-Prefetching Strategy is implemented :
#define _STARPU_SCHED_NTASKS_THRESHOLD_DEFAULT 2
#define _STARPU_SCHED_EXP_LEN_THRESHOLD_DEFAULT 1000000000.0
static void initialize_eager_prefetching_center_policy(unsigned sched_ctx_id)
{
unsigned ntasks_threshold = _STARPU_SCHED_NTASKS_THRESHOLD_DEFAULT;
double exp_len_threshold = _STARPU_SCHED_EXP_LEN_THRESHOLD_DEFAULT;
[...]
starpu_sched_component_eager_create(NULL);
(t->
root, eager_component);
eager_component->add_father
(eager_component, t->
root);
{
.ntasks_threshold = ntasks_threshold,
.exp_len_threshold = exp_len_threshold,
};
unsigned i;
for(i = 0;
i++)
{
(fifo_component, worker_component);
worker_component->add_father
(worker_component, fifo_component);
(eager_component, fifo_component);
fifo_component->add_father
(fifo_component, eager_component);
}
(sched_ctx_id, (void*)t);
}
static void deinitialize_eager_prefetching_center_policy(unsigned sched_ctx_id)
{
(sched_ctx_id);
}
{
.
init_sched = initialize_eager_prefetching_center_policy,
.deinit_sched = deinitialize_eager_prefetching_center_policy,
.pop_every_task = NULL,
.policy_name = "tree-eager-prefetching",
.policy_description = "eager with prefetching tree policy"
};
Write a Scheduling Component
Generic Scheduling Component
Each Scheduling Component is instantiated from a Generic Scheduling Component, which implements a generic version of the Interface. The generic implementation of Pull, Can_Pull and Can_Push functions are recursive calls to their parents (respectively to their children). However, as a Generic Scheduling Component do not know how much children it will have when it will be instantiated, it does not implement the Push function.
Instantiation : Redefine the Interface
A Scheduling Component must implement all the functions of the Interface. It is so necessary to implement a Push function to instantiate a Scheduling Component. The implemented Push function is the "fingerprint" of a Scheduling Component. Depending on how functionalities or properties the programmer wants to give to the Scheduling Component he is implementing, it is possible to reimplement all the functions of the Interface. For example, a Flow-control Component reimplements the Pull and the Can_Push functions of the Interface, allowing him to catch the generic recursive calls of these functions. The Pull function of a Flow-control Component can, for example, pop a task from the local storage queue of the Component, and give it to the calling Component which asks for it.
Detailed Progression and Validation Rules
- A Reservoir is a Scheduling Component which redefines a Push and a Pull function, in order to store tasks into it. A Reservoir delimit Scheduling Areas in the Scheduling Tree.
- A Pump is the engine source of the Scheduler : it pushes/pulls tasks to/from a Scheduling Component to an other. Native Pumps of a Scheduling Tree are located at the root of the Tree (incoming Push calls from StarPU), and at the leafs of the Tree (Pop calls coming from StarPU Workers). Pre-implemented Scheduling Components currently shipped with Pumps are Flow-Control Components and the Resource-Mapping Component Heft, within their defined Can_Push functions.
- A correct Scheduling Tree requires a Pump per Scheduling Area and per Execution Flow.
The Tree-Eager-Prefetching Scheduler shown in Section An Example : The Tree-Eager-Prefetching Strategy follows the previous assumptions :
starpu_push_task
Pump
|
Area 1 |
|
v
-----------------------Fifo_Component-----------------------------
Pump
| ^
Push | | Can_Push
v |
Area 2 Eager_Component
| ^
| |
v |
--------><-------------------><---------
| ^ | ^
Push | | Can_Push Push | | Can_Push
v | v |
-----Fifo_Component-----------------------Fifo_Component----------
| ^ | ^
Pull | | Can_Pull Pull | | Can_Pull
Area 3 v | v |
Pump Pump
Worker_Component Worker_Component