### Navigation

- next
- previous |
- Ganeti 2.8.0~rc3 documentation »

# Ganeti automatic instance allocation

Documents Ganeti version 2.8

Contents

- Ganeti automatic instance allocation
  - Introduction
    - User-visible changes
    - Cluster configuration
    - Command line interface changes
  - IAllocator API
    - Input message
      - Common information
      - Operation-specific input
    - Response message
  - Examples
    - Input messages to scripts
    - Response messages
    - Command line messages
    - Reference implementation

## Introduction

Currently in Ganeti the admin has to specify the exact locations for an instance’s node(s). This prevents a completely automatic node evacuation, and is in general a nuisance.

The iallocator framework will enable automatic placement via external scripts, which allows customization of the cluster layout per the site’s requirements.

### User-visible changes

There are two parts of the ganeti operation that are impacted by the auto-allocation: how the cluster knows what the allocator algorithms are and how the admin uses these in creating instances.

An allocation algorithm is just the filename of a program installed in a defined list of directories.

### Cluster configuration

At configure time, the list of the directories can be selected via the `--with-iallocator-search-path=LIST` option, where LIST is a comma-separated list of directories. If not given, this defaults to `$libdir/ganeti/iallocators`, i.e. for an installation under `/usr`, this will be `/usr/lib/ganeti/iallocators`.

Ganeti will then search for allocator script in the configured list, using the first one whose filename matches the one given by the user.

### Command line interface changes

The node selection options in instance add and instance replace disks can be replace by the new `--iallocator=NAME` option (shortened to `-I`), which will cause the auto-assignement of nodes with the passed iallocator. The selected node(s) will be show as part of the command output.

## IAllocator API

The protocol for communication between Ganeti and an allocator script will be the following:

1. ganeti launches the program with a single argument, a filename that contains a JSON-encoded structure (the input message)
2. if the script finishes with exit code different from zero, it is considered a general failure and the full output will be reported to the users; this can be the case when the allocator can’t parse the input message
3. if the allocator finishes with exit code zero, it is expected to output (on its stdout) a JSON-encoded structure (the response)

### Input message

The input message will be the JSON encoding of a dictionary containing all the required information to perform the operation. We explain the contents of this dictionary in two parts: common information that every type of operation requires, and operation-specific information.

#### Common information

All input dictionaries to the IAllocator must carry the following keys:

version
  the version of the protocol; this document specifies version 2
cluster_name
  the cluster name
cluster_tags
  the list of cluster tags
enabled_hypervisors
  the list of enabled hypervisors
ipolicy
  the cluster-wide instance policy (for information; the per-node group values take precedence and should be used instead)
request
  a dictionary containing the details of the request; the keys vary depending on the type of operation that’s being requested, as explained in Operation-specific input below.
nodegroups

  a dictionary with the data for the cluster’s node groups; it is keyed on the group UUID, and the values are a dictionary with the following keys:

  name
    the node group name
  alloc_policy
    the allocation policy of the node group (consult the semantics of this attribute in the gnt-group(8) manpage)
  networks
    the list of network UUID’s this node group is connected to
  ipolicy
    the instance policy of the node group
  tags
    the list of node group tags
instances

  a dictionary with the data for the current existing instance on the cluster, indexed by instance name; the contents are similar to the instance definitions for the allocate mode, with the addition of:

  admin_state
    if this instance is set to run (but not the actual status of the instance)
  nodes
    list of nodes on which this instance is placed; the primary node of the instance is always the first one
nodes

  dictionary with the data for the nodes in the cluster, indexed by the node name; the dict contains [*] :

  total_disk
    the total disk size of this node (mebibytes)
  free_disk
    the free disk space on the node
  total_memory
    the total memory size
  free_memory
    free memory on the node; note that currently this does not take into account the instances which are down on the node
  total_cpus
    the physical number of CPUs present on the machine; depending on the hypervisor, this might or might not be equal to how many CPUs the node operating system sees;
  primary_ip
    the primary IP address of the node
  secondary_ip
    the secondary IP address of the node (the one used for the DRBD replication); note that this can be the same as the primary one
  tags
    list with the tags of the node
  master_candidate:
    a boolean flag denoting whether this node is a master candidate
  drained:
    a boolean flag denoting whether this node is being drained
  offline:
    a boolean flag denoting whether this node is offline
  i_pri_memory:
    total memory required by primary instances
  i_pri_up_memory:
    total memory required by running primary instances
  group:
    the node group that this node belongs to

  No allocations should be made on nodes having either the `drained` or `offline` flags set. More details about these of node status flags is available in the manpage ganeti(7).

[*] | Note that no run-time data is present for offline, drained or non-vm_capable nodes; this means the tags total_memory, reserved_memory, free_memory, total_disk, free_disk, total_cpus, i_pri_memory and i_pri_up memory will be absent

#### Operation-specific input

All input dictionaries to the IAllocator carry, in the `request` dictionary, detailed information about the operation that’s being requested. The required keys vary depending on the type of operation, as follows.

In all cases, it includes:

> type

>   the request type; this can be either `allocate`, `relocate`, `change-group` or `node-evacuate`. The `allocate` request is used when a new instance needs to be placed on the cluster. The `relocate` request is used when an existing instance needs to be moved within its node group.

>   The `multi-evacuate` protocol used to request that the script computes the optimal relocate solution for all secondary instances of the given nodes. It is now deprecated and needs only be implemented if backwards compatibility with Ganeti 2.4 and lower is needed.

>   The `change-group` request is used to relocate multiple instances across multiple node groups. `node-evacuate` evacuates instances off their node(s). These are described in a separate design document.

>   The `multi-allocate` request is used to allocate multiple instances on the cluster. The request is beside of that very similiar to the `allocate` one. For more details look at Ganeti bulk create.

For both allocate and relocate mode, the following extra keys are needed in the `request` dictionary:

> name
>   the name of the instance; if the request is a realocation, then this name will be found in the list of instances (see below), otherwise is the FQDN of the new instance; type string
> required_nodes
>   how many nodes should the algorithm return; while this information can be deduced from the instace’s disk template, it’s better if this computation is left to Ganeti as then allocator scripts are less sensitive to changes to the disk templates; type integer
> disk_space_total
>   the total disk space that will be used by this instance on the (new) nodes; again, this information can be computed from the list of instance disks and its template type, but Ganeti is better suited to compute it; type integer

Allocation needs, in addition:

> disks

>   list of dictionaries holding the disk definitions for this instance (in the order they are exported to the hypervisor):

>   mode
>     either `ro` or `rw` denoting if the disk is read-only or writable
>   size
>     the size of this disk in mebibytes
> nics

>   a list of dictionaries holding the network interfaces for this instance, containing:

>   ip
>     the IP address that Ganeti know for this instance, or null
>   mac
>     the MAC address for this interface
>   bridge
>     the bridge to which this interface will be connected
> vcpus
>   the number of VCPUs for the instance
> disk_template
>   the disk template for the instance
> memory
>   the memory size for the instance
> os
>   the OS type for the instance
> tags
>   the list of the instance’s tags
> hypervisor
>   the hypervisor of this instance

Relocation:

> relocate_from
>   a list of nodes to move the instance away from; for DRBD-based instances, this will contain a single node, the current secondary of the instance, whereas for shared-storage instance, this will contain also a single node, the current primary of the instance; type list of strings

As for `node-evacuate`, it needs the following request arguments:

> instances
>   a list of instance names to evacuate; type list of strings
> evac_mode
>   specify which instances to evacuate; one of `primary-only`, `secondary-only`, `all`, type string

`change-group` needs the following request arguments:

> instances
>   a list of instance names whose group to change; type list of strings
> target_groups
>   must either be the empty list, or contain a list of group UUIDs that should be considered for relocating instances to; type list of strings

`multi-allocate` needs the following request arguments:

> instances
>   a list of request dicts

### Response message

The response message is much more simple than the input one. It is also a dict having three keys:

success
  a boolean value denoting if the allocation was successful or not
info
  a string with information from the scripts; if the allocation fails, this will be shown to the user
result

  the output of the algorithm; even if the algorithm failed (i.e. success is false), this must be returned as an empty list

  for allocate/relocate, this is the list of node(s) for the instance; note that the length of this list must equal the `requested_nodes` entry in the input message, otherwise Ganeti will consider the result as failed

  for the `node-evacuate` and `change-group` modes, this is a dictionary containing, among other information, a list of lists of serialized opcodes; see the design document for a detailed description

  for the `multi-allocate` mode this is a tuple of 2 lists, the first being element of the tuple is a list of succeeded allocation, with the instance name as first element of each entry and the node placement in the second. The second element of the tuple is the instance list of failed allocations.

Note

Current Ganeti version accepts either `result` or `nodes` as a backwards-compatibility measure (older versions only supported `nodes`)

## Examples

### Input messages to scripts

Input message, new instance allocation (common elements are listed this time, but not included in further examples below):

    {
      "version": 2,
      "cluster_name": "cluster1.example.com",
      "cluster_tags": [],
      "enabled_hypervisors": [
        "xen-pvm"
      ],
      "nodegroups": {
        "f4e06e0d-528a-4963-a5ad-10f3e114232d": {
          "name": "default",
          "alloc_policy": "preferred",
          "networks": ["net-uuid-1", "net-uuid-2"],
          "ipolicy": {
            "disk-templates": ["drbd", "plain"],
            "minmax": [
              {
                "max": {
                  "cpu-count": 2,
                  "disk-count": 8,
                  "disk-size": 2048,
                  "memory-size": 12800,
                  "nic-count": 8,
                  "spindle-use": 8
                },
                "min": {
                  "cpu-count": 1,
                  "disk-count": 1,
                  "disk-size": 1024,
                  "memory-size": 128,
                  "nic-count": 1,
                  "spindle-use": 1
                }
              }
            ],
            "spindle-ratio": 32.0,
            "std": {
              "cpu-count": 1,
              "disk-count": 1,
              "disk-size": 1024,
              "memory-size": 128,
              "nic-count": 1,
              "spindle-use": 1
            },
            "vcpu-ratio": 4.0
          },
          "tags": ["ng-tag-1", "ng-tag-2"]
        }
      },
      "instances": {
        "instance1.example.com": {
          "tags": [],
          "should_run": false,
          "disks": [
            {
              "mode": "w",
              "size": 64
            },
            {
              "mode": "w",
              "size": 512
            }
          ],
          "nics": [
            {
              "ip": null,
              "mac": "aa:00:00:00:60:bf",
              "bridge": "xen-br0"
            }
          ],
          "vcpus": 1,
          "disk_template": "plain",
          "memory": 128,
          "nodes": [
            "nodee1.com"
          ],
          "os": "debootstrap+default"
        },
        "instance2.example.com": {
          "tags": [],
          "should_run": false,
          "disks": [
            {
              "mode": "w",
              "size": 512
            },
            {
              "mode": "w",
              "size": 256
            }
          ],
          "nics": [
            {
              "ip": null,
              "mac": "aa:00:00:55:f8:38",
              "bridge": "xen-br0"
            }
          ],
          "vcpus": 1,
          "disk_template": "drbd",
          "memory": 512,
          "nodes": [
            "node2.example.com",
            "node3.example.com"
          ],
          "os": "debootstrap+default"
        }
      },
      "nodes": {
        "node1.example.com": {
          "total_disk": 858276,
          "primary_ip": "198.51.100.1",
          "secondary_ip": "192.0.2.1",
          "tags": [],
          "group": "f4e06e0d-528a-4963-a5ad-10f3e114232d",
          "free_memory": 3505,
          "free_disk": 856740,
          "total_memory": 4095
        },
        "node2.example.com": {
          "total_disk": 858240,
          "primary_ip": "198.51.100.2",
          "secondary_ip": "192.0.2.2",
          "tags": ["test"],
          "group": "f4e06e0d-528a-4963-a5ad-10f3e114232d",
          "free_memory": 3505,
          "free_disk": 848320,
          "total_memory": 4095
        },
        "node3.example.com.com": {
          "total_disk": 572184,
          "primary_ip": "198.51.100.3",
          "secondary_ip": "192.0.2.3",
          "tags": [],
          "group": "f4e06e0d-528a-4963-a5ad-10f3e114232d",
          "free_memory": 3505,
          "free_disk": 570648,
          "total_memory": 4095
        }
      },
      "request": {
        "type": "allocate",
        "name": "instance3.example.com",
        "required_nodes": 2,
        "disk_space_total": 3328,
        "disks": [
          {
            "mode": "w",
            "size": 1024
          },
          {
            "mode": "w",
            "size": 2048
          }
        ],
        "nics": [
          {
            "ip": null,
            "mac": "00:11:22:33:44:55",
            "bridge": null
          }
        ],
        "vcpus": 1,
        "disk_template": "drbd",
        "memory": 2048,
        "os": "debootstrap+default",
        "tags": [
          "type:test",
          "owner:foo"
        ],
        hypervisor: "xen-pvm"
      }
    }

Input message, reallocation:

    {
      "version": 2,
      ...
      "request": {
        "type": "relocate",
        "name": "instance2.example.com",
        "required_nodes": 1,
        "disk_space_total": 832,
        "relocate_from": [
          "node3.example.com"
        ]
      }
    }

### Response messages

Successful response message:

    {
      "success": true,
      "info": "Allocation successful",
      "result": [
        "node2.example.com",
        "node1.example.com"
      ]
    }

Failed response message:

    {
      "success": false,
      "info": "Can't find a suitable node for position 2 (already selected: node2.example.com)",
      "result": []
    }

Successful node evacuation message:

    {
      "success": true,
      "info": "Request successful",
      "result": [
        [
          "instance1",
          "node3"
        ],
        [
          "instance2",
          "node1"
        ]
      ]
    }

### Command line messages

    # gnt-instance add -t plain -m 2g --os-size 1g --swap-size 512m --iallocator hail -o debootstrap+default instance3
    Selected nodes for the instance: node1.example.com
    * creating instance disks...
    [...]

    # gnt-instance add -t plain -m 3400m --os-size 1g --swap-size 512m --iallocator hail -o debootstrap+default instance4
    Failure: prerequisites not met for this operation:
    Can't compute nodes using iallocator 'hail': Can't find a suitable node for position 1 (already selected: )

    # gnt-instance add -t drbd -m 1400m --os-size 1g --swap-size 512m --iallocator hail -o debootstrap+default instance5
    Failure: prerequisites not met for this operation:
    Can't compute nodes using iallocator 'hail': Can't find a suitable node for position 2 (already selected: node1.example.com)

### Reference implementation

Ganeti’s default iallocator is “hail” which is available when “htools” components have been enabled at build time (see Ganeti quick installation guide for more details).

### Navigation

- next
- previous |
- Ganeti 2.8.0~rc3 documentation »