NOTE: the fix balance command referred to here for dynamic load balancing has not yet been released.
Syntax:
balance keyword args ...
x args = uniform or Px-1 numbers between 0 and 1 uniform = evenly spaced cuts between processors in x dimension numbers = Px-1 ascending values between 0 and 1, Px - # of processors in x dimension y args = uniform or Py-1 numbers between 0 and 1 uniform = evenly spaced cuts between processors in y dimension numbers = Py-1 ascending values between 0 and 1, Py - # of processors in y dimension z args = uniform or Pz-1 numbers between 0 and 1 uniform = evenly spaced cuts between processors in z dimension numbers = Pz-1 ascending values between 0 and 1, Pz - # of processors in z dimension dynamic args = Nrepeat Niter dimstr thresh Nrepeat = # of times to repeat dimstr sequence Niter = # of times to iterate within each dimension of dimstr sequence dimstr = sequence of letters containing "x" or "y" or "z" thresh = stop balancing when this imbalance threshhold is reached
Examples:
balance x uniform y 0.4 0.5 0.6 balance dynamic 1 5 xzx 1.1 balance dynamic 5 10 x 1.0
Description:
This command adjusts the size of processor sub-domains within the simulation box, to attempt to balance the number of particles and thus the computational cost (load) evenly across processors. The load balancing is "static" in the sense that this command performs the balancing once, before or between simulations. The processor sub-domains will then remain static during the subsequent run. To perform "dynamic" balancing, see the fix balance command, which can adjust processor sub-domain sizes on-the-fly during a run.
Load-balancing is only useful if the particles in the simulation box have a spatially-varying density distribution. E.g. a model of a vapor/liquid interface, or a solid with an irregular-shaped geometry containing void regions. In this case, the LAMMPS default of dividing the simulation box volume into a regular-spaced grid of processor sub-domain, with one equal-volume sub-domain per procesor, may assign very different numbers of particles per processor. This can lead to poor performance in a scalability sense, when the simulation is run in parallel.
Note that the processors command gives you some control over how the box volume is split across processors. Specifically, for a Px by Py by Pz grid of processors, it lets you choose Px, Py, and Pz, subject to the constraint that Px * Py * Pz = P, the total number of processors. This can be sufficient to achieve good load-balance for some models on some processor counts. However, all the processor sub-domains will still be the same shape and have the same volume.
This command does not alter the topology of the Px by Py by Pz grid or processors. But it shifts the cutting planes between processors (in 3d, or lines in 2d), which adjusts the volume (area in 2d) assigned to each processor, as in the following 2d diagram. The left diagram is the default partitioning of the simulation box across processors (one sub-box for each of 16 processors); the right diagram is after balancing.
When the balance command completes, it prints out the final positions of all cutting planes in each of the 3 dimensions (as fractions of the box length). It also prints statistics about its results, including the change in "imbalance factor". This factor is defined as the maximum number of particles owned by any processor, divided by the average number of particles per processor. Thus an imbalance factor of 1.0 is perfect balance. For 10000 particles running on 10 processors, if the most heavily loaded processor has 1200 particles, then the factor is 1.2, meaning there is a 20% imbalance. The change in the maximum number of particles (on any processor) is also printed.
IMPORTANT NOTE: This command attempts to minimize the imbalance factor, as defined above. But because of the topology constraint that only the cutting planes (lines) between processors are moved, there are many irregular distributions of particles, where this factor cannot be shrunk to 1.0, particuarly in 3d. Also, computational cost is not strictly proportional to particle count, and changing the relative size and shape of processor sub-domains may lead to additional computational and communication overheads, e.g. in the PPPM solver used via the kspace_style command. Thus you should benchmark the run times of your simulation before and after balancing.
The x, y, and z keywords adjust the position of cutting planes between processor sub-domains in a specific dimension. The uniform argument spaces the planes evenly, as in the left diagram above. The numeric argument requires you to list Ps-1 numbers that specify the position of the cutting planes. This requires that you know Ps = Px or Py or Pz = the number of processors assigned by LAMMPS to the relevant dimension. This assignment is made (and the Px, Py, Pz values printed out) when the simulation box is created by the "create_box" or "read_data" or "read_restart" command and is influenced by the settings of the "processors" command.
Each of the numeric values must be between 0 and 1, and they must be listed in ascending order. They represent the fractional position of the cutting place. The left (or lower) edge of the box is 0.0, and the right (or upper) edge is 1.0. Neither of these values is specified. Only the interior Ps-1 positions are specified. Thus is there are 2 procesors in the x dimension, you specify a single value such as 0.75, which would make the left processor's sub-domain 3x larger than the right processor's sub-domain.
The dynamic keyword changes the cutting planes between processors in an iterative fashion, seeking to reduce the imbalance factor, similar to how the fix balance command operates. Note that this keyword begins its operation from the current processor partitioning, which could be uniform or the result of a previous balance command.
The dimstr argument is a string of characters, each of which must be an "x" or "y" or "z". The characters can appear in any order, and can be repeated as many times as desired. These are all valid dimstr arguments: "x" or "xyzyx" or "yyyzzz".
Balancing proceeds by adjusting the cutting planes in each of the dimensions listed in dimstr, one dimension at a time. The entire sequence of dimensions is repeated Nrepeat times. For a single dimension, the balancing operation (described below) is iterated on Niter times. After each dimension finishes, the imbalance factor is re-computed, and the balancing operation halts if the thresh criterion is met.
The interplay between Nrepeat, Niter, and dimstr means that these commands do essentially the same thing, the only difference being how often the imbalance factor is computed and checked against the threshhold:
balance y dynamic 5 10 x 1.2 balance y dynamic 1 10 xxxxx 1.2 balance y dynamic 50 1 x 1.2
A rebalance operation in a single dimension is performed using an iterative "diffusive" load-balancing algorithm (Cybenko). One iteration on a dimension (which is repeated Niter times), works as follows. Assume there are Px processors in the x dimension. This defines Px slices of the simulation, each of which contains Py*Pz processors. The task is to adjust the position of the Px-1 cuts between slices, leaving the end cuts unchanged (left and right edges of the simulation box).
The iteration beings by calculating the number of atoms within each of the Px slices. Then for each slice, its atom count is compared to its neighbors. If a slice has more atoms than its left (or right) neighbor, the cut is moved towards the center of the slice, effectively shrinking the width of the slice and migrating atoms to the other slice. The distance to move the cut is a function of the "density" of atoms in the donor slice and the difference in counts between the 2 slices. A damping factor is also applied to avoid oscillations in the position of the cutting plane as iterations proceed. Hence the "diffusive" nature of the algorithm as work (atoms) effectively diffuses from highly loaded processors to less-loaded processors.
Restrictions:
The dynamic keyword cannot be used with the x, y, or z arguments.
For 2d simulations, the z keyword cannot be used. Nor can a "z" appear in dimstr for the dynamic keyword.
Related commands:
Default: none
(Cybenko) Cybenko, J Par Dist Comp, 7, 279-301 (1989).