This is a package of tools in use in Skype for replication and failover. Also it includes a generic queuing mechanism PgQ and utility library for Python scripts.

Overview

It contains following tools:

PgQ

This is the queue mechanism we use. It consists of PL/pgsql, and C code in database, with Python framework on top of it. It is based on snapshot based event handling ideas from Slony-I, written for general usage.

Features:

  • There can be several queues in database.

  • There can be several producers than can insert into any queue.

  • There can be several consumers on one queue and all consumers see all events.

Documentation:

Londiste

Replication tool written in Python, using PgQ as event transport.

Features:

  • Tables can be added one-by-one into set.

  • Initial COPY for one table does not block event replay for other tables.

  • Can compare tables on both sides.

Documentation:

walmgr

This script will setup WAL archiving, does initial backup and runtime WAL archive and restore.

Source tree contents

doc/
  Documentation in asciidoc format.  Source for both html
  and man pages.
python/
  Python modules and primary executables - walmgr, londiste, qadmin, pgqadm.
python/pgq/
  Python framework for PgQ.
python/londiste/
  Londiste replication.
python/skytools/
  Low-level utilities for writing database scripts in Python.
sql/
  Database modules.
sql/pgq/
  Table definitions and functions for PgQ queueing.
sql/pgq_coop/
  Functions for cooperative consuming.
sql/londiste/
  Table definitions and functions for Londiste replication.
sql/ticker/
  PgQ ticker written in C.
scripts/
  Python scripts with lesser priority.
lib/
  libusual C libary, for pgqd.
debian/
  Debian packaging.  This is for creating private packages,
  official Debian packages uses it's own packagin code.
misc/
  Random scripts used for building.