Package CedarBackup2 :: Package extend :: Module mysql
[hide private]
[frames] | no frames]

Source Code for Module CedarBackup2.extend.mysql

  1  # -*- coding: iso-8859-1 -*- 
  2  # vim: set ft=python ts=3 sw=3 expandtab: 
  3  # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # 
  4  # 
  5  #              C E D A R 
  6  #          S O L U T I O N S       "Software done right." 
  7  #           S O F T W A R E 
  8  # 
  9  # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # 
 10  # 
 11  # Copyright (c) 2005,2010 Kenneth J. Pronovici. 
 12  # All rights reserved. 
 13  # 
 14  # This program is free software; you can redistribute it and/or 
 15  # modify it under the terms of the GNU General Public License, 
 16  # Version 2, as published by the Free Software Foundation. 
 17  # 
 18  # This program is distributed in the hope that it will be useful, 
 19  # but WITHOUT ANY WARRANTY; without even the implied warranty of 
 20  # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. 
 21  # 
 22  # Copies of the GNU General Public License are available from 
 23  # the Free Software Foundation website, http://www.gnu.org/. 
 24  # 
 25  # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # 
 26  # 
 27  # Author   : Kenneth J. Pronovici <pronovic@ieee.org> 
 28  # Language : Python (>= 2.5) 
 29  # Project  : Official Cedar Backup Extensions 
 30  # Purpose  : Provides an extension to back up MySQL databases. 
 31  # 
 32  # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # 
 33   
 34  ######################################################################## 
 35  # Module documentation 
 36  ######################################################################## 
 37   
 38  """ 
 39  Provides an extension to back up MySQL databases. 
 40   
 41  This is a Cedar Backup extension used to back up MySQL databases via the Cedar 
 42  Backup command line.  It requires a new configuration section <mysql> and is 
 43  intended to be run either immediately before or immediately after the standard 
 44  collect action.  Aside from its own configuration, it requires the options and 
 45  collect configuration sections in the standard Cedar Backup configuration file. 
 46   
 47  The backup is done via the C{mysqldump} command included with the MySQL 
 48  product.  Output can be compressed using C{gzip} or C{bzip2}.  Administrators 
 49  can configure the extension either to back up all databases or to back up only 
 50  specific databases.  Note that this code always produces a full backup.  There 
 51  is currently no facility for making incremental backups.  If/when someone has a 
 52  need for this and can describe how to do it, I'll update this extension or 
 53  provide another. 
 54   
 55  The extension assumes that all configured databases can be backed up by a 
 56  single user.  Often, the "root" database user will be used.  An alternative is 
 57  to create a separate MySQL "backup" user and grant that user rights to read 
 58  (but not write) various databases as needed.  This second option is probably 
 59  the best choice. 
 60   
 61  The extension accepts a username and password in configuration.  However, you 
 62  probably do not want to provide those values in Cedar Backup configuration. 
 63  This is because Cedar Backup will provide these values to C{mysqldump} via the 
 64  command-line C{--user} and C{--password} switches, which will be visible to 
 65  other users in the process listing. 
 66   
 67  Instead, you should configure the username and password in one of MySQL's 
 68  configuration files.  Typically, that would be done by putting a stanza like 
 69  this in C{/root/.my.cnf}:: 
 70   
 71     [mysqldump] 
 72     user     = root 
 73     password = <secret> 
 74   
 75  Regardless of whether you are using C{~/.my.cnf} or C{/etc/cback.conf} to store 
 76  database login and password information, you should be careful about who is 
 77  allowed to view that information.  Typically, this means locking down 
 78  permissions so that only the file owner can read the file contents (i.e. use 
 79  mode C{0600}). 
 80   
 81  @author: Kenneth J. Pronovici <pronovic@ieee.org> 
 82  """ 
 83   
 84  ######################################################################## 
 85  # Imported modules 
 86  ######################################################################## 
 87   
 88  # System modules 
 89  import os 
 90  import logging 
 91  from gzip import GzipFile 
 92  from bz2 import BZ2File 
 93   
 94  # Cedar Backup modules 
 95  from CedarBackup2.xmlutil import createInputDom, addContainerNode, addStringNode, addBooleanNode 
 96  from CedarBackup2.xmlutil import readFirstChild, readString, readStringList, readBoolean 
 97  from CedarBackup2.config import VALID_COMPRESS_MODES 
 98  from CedarBackup2.util import resolveCommand, executeCommand 
 99  from CedarBackup2.util import ObjectTypeList, changeOwnership 
100   
101   
102  ######################################################################## 
103  # Module-wide constants and variables 
104  ######################################################################## 
105   
106  logger = logging.getLogger("CedarBackup2.log.extend.mysql") 
107  MYSQLDUMP_COMMAND = [ "mysqldump", ] 
108 109 110 ######################################################################## 111 # MysqlConfig class definition 112 ######################################################################## 113 114 -class MysqlConfig(object):
115 116 """ 117 Class representing MySQL configuration. 118 119 The MySQL configuration information is used for backing up MySQL databases. 120 121 The following restrictions exist on data in this class: 122 123 - The compress mode must be one of the values in L{VALID_COMPRESS_MODES}. 124 - The 'all' flag must be 'Y' if no databases are defined. 125 - The 'all' flag must be 'N' if any databases are defined. 126 - Any values in the databases list must be strings. 127 128 @sort: __init__, __repr__, __str__, __cmp__, user, password, all, databases 129 """ 130
131 - def __init__(self, user=None, password=None, compressMode=None, all=None, databases=None): # pylint: disable=W0622
132 """ 133 Constructor for the C{MysqlConfig} class. 134 135 @param user: User to execute backup as. 136 @param password: Password associated with user. 137 @param compressMode: Compress mode for backed-up files. 138 @param all: Indicates whether to back up all databases. 139 @param databases: List of databases to back up. 140 """ 141 self._user = None 142 self._password = None 143 self._compressMode = None 144 self._all = None 145 self._databases = None 146 self.user = user 147 self.password = password 148 self.compressMode = compressMode 149 self.all = all 150 self.databases = databases
151
152 - def __repr__(self):
153 """ 154 Official string representation for class instance. 155 """ 156 return "MysqlConfig(%s, %s, %s, %s)" % (self.user, self.password, self.all, self.databases)
157
158 - def __str__(self):
159 """ 160 Informal string representation for class instance. 161 """ 162 return self.__repr__()
163
164 - def __cmp__(self, other):
165 """ 166 Definition of equals operator for this class. 167 @param other: Other object to compare to. 168 @return: -1/0/1 depending on whether self is C{<}, C{=} or C{>} other. 169 """ 170 if other is None: 171 return 1 172 if self.user != other.user: 173 if self.user < other.user: 174 return -1 175 else: 176 return 1 177 if self.password != other.password: 178 if self.password < other.password: 179 return -1 180 else: 181 return 1 182 if self.compressMode != other.compressMode: 183 if self.compressMode < other.compressMode: 184 return -1 185 else: 186 return 1 187 if self.all != other.all: 188 if self.all < other.all: 189 return -1 190 else: 191 return 1 192 if self.databases != other.databases: 193 if self.databases < other.databases: 194 return -1 195 else: 196 return 1 197 return 0
198
199 - def _setUser(self, value):
200 """ 201 Property target used to set the user value. 202 """ 203 if value is not None: 204 if len(value) < 1: 205 raise ValueError("User must be non-empty string.") 206 self._user = value
207
208 - def _getUser(self):
209 """ 210 Property target used to get the user value. 211 """ 212 return self._user
213
214 - def _setPassword(self, value):
215 """ 216 Property target used to set the password value. 217 """ 218 if value is not None: 219 if len(value) < 1: 220 raise ValueError("Password must be non-empty string.") 221 self._password = value
222
223 - def _getPassword(self):
224 """ 225 Property target used to get the password value. 226 """ 227 return self._password
228
229 - def _setCompressMode(self, value):
230 """ 231 Property target used to set the compress mode. 232 If not C{None}, the mode must be one of the values in L{VALID_COMPRESS_MODES}. 233 @raise ValueError: If the value is not valid. 234 """ 235 if value is not None: 236 if value not in VALID_COMPRESS_MODES: 237 raise ValueError("Compress mode must be one of %s." % VALID_COMPRESS_MODES) 238 self._compressMode = value
239
240 - def _getCompressMode(self):
241 """ 242 Property target used to get the compress mode. 243 """ 244 return self._compressMode
245
246 - def _setAll(self, value):
247 """ 248 Property target used to set the 'all' flag. 249 No validations, but we normalize the value to C{True} or C{False}. 250 """ 251 if value: 252 self._all = True 253 else: 254 self._all = False
255
256 - def _getAll(self):
257 """ 258 Property target used to get the 'all' flag. 259 """ 260 return self._all
261
262 - def _setDatabases(self, value):
263 """ 264 Property target used to set the databases list. 265 Either the value must be C{None} or each element must be a string. 266 @raise ValueError: If the value is not a string. 267 """ 268 if value is None: 269 self._databases = None 270 else: 271 for database in value: 272 if len(database) < 1: 273 raise ValueError("Each database must be a non-empty string.") 274 try: 275 saved = self._databases 276 self._databases = ObjectTypeList(basestring, "string") 277 self._databases.extend(value) 278 except Exception, e: 279 self._databases = saved 280 raise e
281
282 - def _getDatabases(self):
283 """ 284 Property target used to get the databases list. 285 """ 286 return self._databases
287 288 user = property(_getUser, _setUser, None, "User to execute backup as.") 289 password = property(_getPassword, _setPassword, None, "Password associated with user.") 290 compressMode = property(_getCompressMode, _setCompressMode, None, "Compress mode to be used for backed-up files.") 291 all = property(_getAll, _setAll, None, "Indicates whether to back up all databases.") 292 databases = property(_getDatabases, _setDatabases, None, "List of databases to back up.") 293
294 295 ######################################################################## 296 # LocalConfig class definition 297 ######################################################################## 298 299 -class LocalConfig(object):
300 301 """ 302 Class representing this extension's configuration document. 303 304 This is not a general-purpose configuration object like the main Cedar 305 Backup configuration object. Instead, it just knows how to parse and emit 306 MySQL-specific configuration values. Third parties who need to read and 307 write configuration related to this extension should access it through the 308 constructor, C{validate} and C{addConfig} methods. 309 310 @note: Lists within this class are "unordered" for equality comparisons. 311 312 @sort: __init__, __repr__, __str__, __cmp__, mysql, validate, addConfig 313 """ 314
315 - def __init__(self, xmlData=None, xmlPath=None, validate=True):
316 """ 317 Initializes a configuration object. 318 319 If you initialize the object without passing either C{xmlData} or 320 C{xmlPath} then configuration will be empty and will be invalid until it 321 is filled in properly. 322 323 No reference to the original XML data or original path is saved off by 324 this class. Once the data has been parsed (successfully or not) this 325 original information is discarded. 326 327 Unless the C{validate} argument is C{False}, the L{LocalConfig.validate} 328 method will be called (with its default arguments) against configuration 329 after successfully parsing any passed-in XML. Keep in mind that even if 330 C{validate} is C{False}, it might not be possible to parse the passed-in 331 XML document if lower-level validations fail. 332 333 @note: It is strongly suggested that the C{validate} option always be set 334 to C{True} (the default) unless there is a specific need to read in 335 invalid configuration from disk. 336 337 @param xmlData: XML data representing configuration. 338 @type xmlData: String data. 339 340 @param xmlPath: Path to an XML file on disk. 341 @type xmlPath: Absolute path to a file on disk. 342 343 @param validate: Validate the document after parsing it. 344 @type validate: Boolean true/false. 345 346 @raise ValueError: If both C{xmlData} and C{xmlPath} are passed-in. 347 @raise ValueError: If the XML data in C{xmlData} or C{xmlPath} cannot be parsed. 348 @raise ValueError: If the parsed configuration document is not valid. 349 """ 350 self._mysql = None 351 self.mysql = None 352 if xmlData is not None and xmlPath is not None: 353 raise ValueError("Use either xmlData or xmlPath, but not both.") 354 if xmlData is not None: 355 self._parseXmlData(xmlData) 356 if validate: 357 self.validate() 358 elif xmlPath is not None: 359 xmlData = open(xmlPath).read() 360 self._parseXmlData(xmlData) 361 if validate: 362 self.validate()
363
364 - def __repr__(self):
365 """ 366 Official string representation for class instance. 367 """ 368 return "LocalConfig(%s)" % (self.mysql)
369
370 - def __str__(self):
371 """ 372 Informal string representation for class instance. 373 """ 374 return self.__repr__()
375
376 - def __cmp__(self, other):
377 """ 378 Definition of equals operator for this class. 379 Lists within this class are "unordered" for equality comparisons. 380 @param other: Other object to compare to. 381 @return: -1/0/1 depending on whether self is C{<}, C{=} or C{>} other. 382 """ 383 if other is None: 384 return 1 385 if self.mysql != other.mysql: 386 if self.mysql < other.mysql: 387 return -1 388 else: 389 return 1 390 return 0
391
392 - def _setMysql(self, value):
393 """ 394 Property target used to set the mysql configuration value. 395 If not C{None}, the value must be a C{MysqlConfig} object. 396 @raise ValueError: If the value is not a C{MysqlConfig} 397 """ 398 if value is None: 399 self._mysql = None 400 else: 401 if not isinstance(value, MysqlConfig): 402 raise ValueError("Value must be a C{MysqlConfig} object.") 403 self._mysql = value
404
405 - def _getMysql(self):
406 """ 407 Property target used to get the mysql configuration value. 408 """ 409 return self._mysql
410 411 mysql = property(_getMysql, _setMysql, None, "Mysql configuration in terms of a C{MysqlConfig} object.") 412
413 - def validate(self):
414 """ 415 Validates configuration represented by the object. 416 417 The compress mode must be filled in. Then, if the 'all' flag I{is} set, 418 no databases are allowed, and if the 'all' flag is I{not} set, at least 419 one database is required. 420 421 @raise ValueError: If one of the validations fails. 422 """ 423 if self.mysql is None: 424 raise ValueError("Mysql section is required.") 425 if self.mysql.compressMode is None: 426 raise ValueError("Compress mode value is required.") 427 if self.mysql.all: 428 if self.mysql.databases is not None and self.mysql.databases != []: 429 raise ValueError("Databases cannot be specified if 'all' flag is set.") 430 else: 431 if self.mysql.databases is None or len(self.mysql.databases) < 1: 432 raise ValueError("At least one MySQL database must be indicated if 'all' flag is not set.")
433
434 - def addConfig(self, xmlDom, parentNode):
435 """ 436 Adds a <mysql> configuration section as the next child of a parent. 437 438 Third parties should use this function to write configuration related to 439 this extension. 440 441 We add the following fields to the document:: 442 443 user //cb_config/mysql/user 444 password //cb_config/mysql/password 445 compressMode //cb_config/mysql/compress_mode 446 all //cb_config/mysql/all 447 448 We also add groups of the following items, one list element per 449 item:: 450 451 database //cb_config/mysql/database 452 453 @param xmlDom: DOM tree as from C{impl.createDocument()}. 454 @param parentNode: Parent that the section should be appended to. 455 """ 456 if self.mysql is not None: 457 sectionNode = addContainerNode(xmlDom, parentNode, "mysql") 458 addStringNode(xmlDom, sectionNode, "user", self.mysql.user) 459 addStringNode(xmlDom, sectionNode, "password", self.mysql.password) 460 addStringNode(xmlDom, sectionNode, "compress_mode", self.mysql.compressMode) 461 addBooleanNode(xmlDom, sectionNode, "all", self.mysql.all) 462 if self.mysql.databases is not None: 463 for database in self.mysql.databases: 464 addStringNode(xmlDom, sectionNode, "database", database)
465
466 - def _parseXmlData(self, xmlData):
467 """ 468 Internal method to parse an XML string into the object. 469 470 This method parses the XML document into a DOM tree (C{xmlDom}) and then 471 calls a static method to parse the mysql configuration section. 472 473 @param xmlData: XML data to be parsed 474 @type xmlData: String data 475 476 @raise ValueError: If the XML cannot be successfully parsed. 477 """ 478 (xmlDom, parentNode) = createInputDom(xmlData) 479 self._mysql = LocalConfig._parseMysql(parentNode)
480 481 @staticmethod
482 - def _parseMysql(parentNode):
483 """ 484 Parses a mysql configuration section. 485 486 We read the following fields:: 487 488 user //cb_config/mysql/user 489 password //cb_config/mysql/password 490 compressMode //cb_config/mysql/compress_mode 491 all //cb_config/mysql/all 492 493 We also read groups of the following item, one list element per 494 item:: 495 496 databases //cb_config/mysql/database 497 498 @param parentNode: Parent node to search beneath. 499 500 @return: C{MysqlConfig} object or C{None} if the section does not exist. 501 @raise ValueError: If some filled-in value is invalid. 502 """ 503 mysql = None 504 section = readFirstChild(parentNode, "mysql") 505 if section is not None: 506 mysql = MysqlConfig() 507 mysql.user = readString(section, "user") 508 mysql.password = readString(section, "password") 509 mysql.compressMode = readString(section, "compress_mode") 510 mysql.all = readBoolean(section, "all") 511 mysql.databases = readStringList(section, "database") 512 return mysql
513
514 515 ######################################################################## 516 # Public functions 517 ######################################################################## 518 519 ########################### 520 # executeAction() function 521 ########################### 522 523 -def executeAction(configPath, options, config):
524 """ 525 Executes the MySQL backup action. 526 527 @param configPath: Path to configuration file on disk. 528 @type configPath: String representing a path on disk. 529 530 @param options: Program command-line options. 531 @type options: Options object. 532 533 @param config: Program configuration. 534 @type config: Config object. 535 536 @raise ValueError: Under many generic error conditions 537 @raise IOError: If a backup could not be written for some reason. 538 """ 539 logger.debug("Executing MySQL extended action.") 540 if config.options is None or config.collect is None: 541 raise ValueError("Cedar Backup configuration is not properly filled in.") 542 local = LocalConfig(xmlPath=configPath) 543 if local.mysql.all: 544 logger.info("Backing up all databases.") 545 _backupDatabase(config.collect.targetDir, local.mysql.compressMode, local.mysql.user, local.mysql.password, 546 config.options.backupUser, config.options.backupGroup, None) 547 else: 548 logger.debug("Backing up %d individual databases." % len(local.mysql.databases)) 549 for database in local.mysql.databases: 550 logger.info("Backing up database [%s]." % database) 551 _backupDatabase(config.collect.targetDir, local.mysql.compressMode, local.mysql.user, local.mysql.password, 552 config.options.backupUser, config.options.backupGroup, database) 553 logger.info("Executed the MySQL extended action successfully.")
554
555 -def _backupDatabase(targetDir, compressMode, user, password, backupUser, backupGroup, database=None):
556 """ 557 Backs up an individual MySQL database, or all databases. 558 559 This internal method wraps the public method and adds some functionality, 560 like figuring out a filename, etc. 561 562 @param targetDir: Directory into which backups should be written. 563 @param compressMode: Compress mode to be used for backed-up files. 564 @param user: User to use for connecting to the database (if any). 565 @param password: Password associated with user (if any). 566 @param backupUser: User to own resulting file. 567 @param backupGroup: Group to own resulting file. 568 @param database: Name of database, or C{None} for all databases. 569 570 @return: Name of the generated backup file. 571 572 @raise ValueError: If some value is missing or invalid. 573 @raise IOError: If there is a problem executing the MySQL dump. 574 """ 575 (outputFile, filename) = _getOutputFile(targetDir, database, compressMode) 576 try: 577 backupDatabase(user, password, outputFile, database) 578 finally: 579 outputFile.close() 580 if not os.path.exists(filename): 581 raise IOError("Dump file [%s] does not seem to exist after backup completed." % filename) 582 changeOwnership(filename, backupUser, backupGroup)
583
584 -def _getOutputFile(targetDir, database, compressMode):
585 """ 586 Opens the output file used for saving the MySQL dump. 587 588 The filename is either C{"mysqldump.txt"} or C{"mysqldump-<database>.txt"}. The 589 C{".bz2"} extension is added if C{compress} is C{True}. 590 591 @param targetDir: Target directory to write file in. 592 @param database: Name of the database (if any) 593 @param compressMode: Compress mode to be used for backed-up files. 594 595 @return: Tuple of (Output file object, filename) 596 """ 597 if database is None: 598 filename = os.path.join(targetDir, "mysqldump.txt") 599 else: 600 filename = os.path.join(targetDir, "mysqldump-%s.txt" % database) 601 if compressMode == "gzip": 602 filename = "%s.gz" % filename 603 outputFile = GzipFile(filename, "w") 604 elif compressMode == "bzip2": 605 filename = "%s.bz2" % filename 606 outputFile = BZ2File(filename, "w") 607 else: 608 outputFile = open(filename, "w") 609 logger.debug("MySQL dump file will be [%s]." % filename) 610 return (outputFile, filename)
611
612 613 ############################ 614 # backupDatabase() function 615 ############################ 616 617 -def backupDatabase(user, password, backupFile, database=None):
618 """ 619 Backs up an individual MySQL database, or all databases. 620 621 This function backs up either a named local MySQL database or all local 622 MySQL databases, using the passed-in user and password (if provided) for 623 connectivity. This function call I{always} results a full backup. There is 624 no facility for incremental backups. 625 626 The backup data will be written into the passed-in backup file. Normally, 627 this would be an object as returned from C{open()}, but it is possible to 628 use something like a C{GzipFile} to write compressed output. The caller is 629 responsible for closing the passed-in backup file. 630 631 Often, the "root" database user will be used when backing up all databases. 632 An alternative is to create a separate MySQL "backup" user and grant that 633 user rights to read (but not write) all of the databases that will be backed 634 up. 635 636 This function accepts a username and password. However, you probably do not 637 want to pass those values in. This is because they will be provided to 638 C{mysqldump} via the command-line C{--user} and C{--password} switches, 639 which will be visible to other users in the process listing. 640 641 Instead, you should configure the username and password in one of MySQL's 642 configuration files. Typically, this would be done by putting a stanza like 643 this in C{/root/.my.cnf}, to provide C{mysqldump} with the root database 644 username and its password:: 645 646 [mysqldump] 647 user = root 648 password = <secret> 649 650 If you are executing this function as some system user other than root, then 651 the C{.my.cnf} file would be placed in the home directory of that user. In 652 either case, make sure to set restrictive permissions (typically, mode 653 C{0600}) on C{.my.cnf} to make sure that other users cannot read the file. 654 655 @param user: User to use for connecting to the database (if any) 656 @type user: String representing MySQL username, or C{None} 657 658 @param password: Password associated with user (if any) 659 @type password: String representing MySQL password, or C{None} 660 661 @param backupFile: File use for writing backup. 662 @type backupFile: Python file object as from C{open()} or C{file()}. 663 664 @param database: Name of the database to be backed up. 665 @type database: String representing database name, or C{None} for all databases. 666 667 @raise ValueError: If some value is missing or invalid. 668 @raise IOError: If there is a problem executing the MySQL dump. 669 """ 670 args = [ "-all", "--flush-logs", "--opt", ] 671 if user is not None: 672 logger.warn("Warning: MySQL username will be visible in process listing (consider using ~/.my.cnf).") 673 args.append("--user=%s" % user) 674 if password is not None: 675 logger.warn("Warning: MySQL password will be visible in process listing (consider using ~/.my.cnf).") 676 args.append("--password=%s" % password) 677 if database is None: 678 args.insert(0, "--all-databases") 679 else: 680 args.insert(0, "--databases") 681 args.append(database) 682 command = resolveCommand(MYSQLDUMP_COMMAND) 683 result = executeCommand(command, args, returnOutput=False, ignoreStderr=True, doNotLog=True, outputFile=backupFile)[0] 684 if result != 0: 685 if database is None: 686 raise IOError("Error [%d] executing MySQL database dump for all databases." % result) 687 else: 688 raise IOError("Error [%d] executing MySQL database dump for database [%s]." % (result, database))
689