Package CedarBackup2 :: Module filesystem :: Class FilesystemList
[hide private]
[frames] | no frames]

Class FilesystemList

source code

object --+    
         |    
      list --+
             |
            FilesystemList
Known Subclasses:

Represents a list of filesystem items.

This is a generic class that represents a list of filesystem items. Callers can add individual files or directories to the list, or can recursively add the contents of a directory. The class also allows for up-front exclusions in several forms (all files, all directories, all items matching a pattern, all items whose basename matches a pattern, or all directories containing a specific "ignore file"). Symbolic links are typically backed up non-recursively, i.e. the link to a directory is backed up, but not the contents of that link (we don't want to deal with recursive loops, etc.).

The custom methods such as addFile will only add items if they exist on the filesystem and do not match any exclusions that are already in place. However, since a FilesystemList is a subclass of Python's standard list class, callers can also add items to the list in the usual way, using methods like append() or insert(). No validations apply to items added to the list in this way; however, many list-manipulation methods deal "gracefully" with items that don't exist in the filesystem, often by ignoring them.

Once a list has been created, callers can remove individual items from the list using standard methods like pop() or remove() or they can use custom methods to remove specific types of entries or entries which match a particular pattern.


Notes:
Instance Methods [hide private]
new empty list
__init__(self)
Initializes a list with no configured exclusions.
source code
 
addFile(self, path)
Adds a file to the list.
source code
 
addDir(self, path)
Adds a directory to the list.
source code
 
addDirContents(self, path, recursive=True, addSelf=True, linkDepth=0, dereference=False)
Adds the contents of a directory to the list.
source code
 
removeFiles(self, pattern=None)
Removes file entries from the list.
source code
 
removeDirs(self, pattern=None)
Removes directory entries from the list.
source code
 
removeLinks(self, pattern=None)
Removes soft link entries from the list.
source code
 
removeMatch(self, pattern)
Removes from the list all entries matching a pattern.
source code
 
removeInvalid(self)
Removes from the list all entries that do not exist on disk.
source code
 
normalize(self)
Normalizes the list, ensuring that each entry is unique.
source code
 
_setExcludeFiles(self, value)
Property target used to set the exclude files flag.
source code
 
_getExcludeFiles(self)
Property target used to get the exclude files flag.
source code
 
_setExcludeDirs(self, value)
Property target used to set the exclude directories flag.
source code
 
_getExcludeDirs(self)
Property target used to get the exclude directories flag.
source code
 
_setExcludeLinks(self, value)
Property target used to set the exclude soft links flag.
source code
 
_getExcludeLinks(self)
Property target used to get the exclude soft links flag.
source code
 
_setExcludePaths(self, value)
Property target used to set the exclude paths list.
source code
 
_getExcludePaths(self)
Property target used to get the absolute exclude paths list.
source code
 
_setExcludePatterns(self, value)
Property target used to set the exclude patterns list.
source code
 
_getExcludePatterns(self)
Property target used to get the exclude patterns list.
source code
 
_setExcludeBasenamePatterns(self, value)
Property target used to set the exclude basename patterns list.
source code
 
_getExcludeBasenamePatterns(self)
Property target used to get the exclude basename patterns list.
source code
 
_setIgnoreFile(self, value)
Property target used to set the ignore file.
source code
 
_getIgnoreFile(self)
Property target used to get the ignore file.
source code
 
_addDirContentsInternal(self, path, includePath=True, recursive=True, linkDepth=0, dereference=False)
Internal implementation of addDirContents.
source code
 
verify(self)
Verifies that all entries in the list exist on disk.
source code

Inherited from list: __add__, __contains__, __delitem__, __delslice__, __eq__, __ge__, __getattribute__, __getitem__, __getslice__, __gt__, __iadd__, __imul__, __iter__, __le__, __len__, __lt__, __mul__, __ne__, __new__, __repr__, __reversed__, __rmul__, __setitem__, __setslice__, __sizeof__, append, count, extend, index, insert, pop, remove, reverse, sort

Inherited from object: __delattr__, __format__, __reduce__, __reduce_ex__, __setattr__, __str__, __subclasshook__

Class Variables [hide private]

Inherited from list: __hash__

Properties [hide private]
  excludeFiles
Boolean indicating whether files should be excluded.
  excludeDirs
Boolean indicating whether directories should be excluded.
  excludeLinks
Boolean indicating whether soft links should be excluded.
  excludePaths
List of absolute paths to be excluded.
  excludePatterns
List of regular expression patterns (matching complete path) to be excluded.
  excludeBasenamePatterns
List of regular expression patterns (matching basename) to be excluded.
  ignoreFile
Name of file which will cause directory contents to be ignored.

Inherited from object: __class__

Method Details [hide private]

__init__(self)
(Constructor)

source code 

Initializes a list with no configured exclusions.

Returns: new empty list
Overrides: object.__init__

addFile(self, path)

source code 

Adds a file to the list.

The path must exist and must be a file or a link to an existing file. It will be added to the list subject to any exclusions that are in place.

Parameters:
  • path (String representing a path on disk) - File path to be added to the list
Returns:
Number of items added to the list.
Raises:
  • ValueError - If path is not a file or does not exist.
  • ValueError - If the path could not be encoded properly.

addDir(self, path)

source code 

Adds a directory to the list.

The path must exist and must be a directory or a link to an existing directory. It will be added to the list subject to any exclusions that are in place. The ignoreFile does not apply to this method, only to addDirContents.

Parameters:
  • path (String representing a path on disk) - Directory path to be added to the list
Returns:
Number of items added to the list.
Raises:
  • ValueError - If path is not a directory or does not exist.
  • ValueError - If the path could not be encoded properly.

addDirContents(self, path, recursive=True, addSelf=True, linkDepth=0, dereference=False)

source code 

Adds the contents of a directory to the list.

The path must exist and must be a directory or a link to a directory. The contents of the directory (as well as the directory path itself) will be recursively added to the list, subject to any exclusions that are in place. If you only want the directory and its immediate contents to be added, then pass in recursive=False.

Parameters:
  • path (String representing a path on disk) - Directory path whose contents should be added to the list
  • recursive (Boolean value) - Indicates whether directory contents should be added recursively.
  • addSelf (Boolean value) - Indicates whether the directory itself should be added to the list.
  • linkDepth (Integer value, where zero means not to follow any soft links) - Maximum depth of the tree at which soft links should be followed
  • dereference (Boolean value) - Indicates whether soft links, if followed, should be dereferenced
Returns:
Number of items recursively added to the list
Raises:
  • ValueError - If path is not a directory or does not exist.
  • ValueError - If the path could not be encoded properly.
Notes:
  • If a directory's absolute path matches an exclude pattern or path, or if the directory contains the configured ignore file, then the directory and all of its contents will be recursively excluded from the list.
  • If the passed-in directory happens to be a soft link, it will be recursed. However, the linkDepth parameter controls whether any soft links within the directory will be recursed. The link depth is maximum depth of the tree at which soft links should be followed. So, a depth of 0 does not follow any soft links, a depth of 1 follows only links within the passed-in directory, a depth of 2 follows the links at the next level down, etc.
  • Any invalid soft links (i.e. soft links that point to non-existent items) will be silently ignored.
  • The excludeDirs flag only controls whether any given directory path itself is added to the list once it has been discovered. It does not modify any behavior related to directory recursion.
  • If you call this method on a link to a directory that link will never be dereferenced (it may, however, be followed).

removeFiles(self, pattern=None)

source code 

Removes file entries from the list.

If pattern is not passed in or is None, then all file entries will be removed from the list. Otherwise, only those file entries matching the pattern will be removed. Any entry which does not exist on disk will be ignored (use removeInvalid to purge those entries).

This method might be fairly slow for large lists, since it must check the type of each item in the list. If you know ahead of time that you want to exclude all files, then you will be better off setting excludeFiles to True before adding items to the list.

Parameters:
  • pattern - Regular expression pattern representing entries to remove
Returns:
Number of entries removed
Raises:
  • ValueError - If the passed-in pattern is not a valid regular expression.

removeDirs(self, pattern=None)

source code 

Removes directory entries from the list.

If pattern is not passed in or is None, then all directory entries will be removed from the list. Otherwise, only those directory entries matching the pattern will be removed. Any entry which does not exist on disk will be ignored (use removeInvalid to purge those entries).

This method might be fairly slow for large lists, since it must check the type of each item in the list. If you know ahead of time that you want to exclude all directories, then you will be better off setting excludeDirs to True before adding items to the list (note that this will not prevent you from recursively adding the contents of directories).

Parameters:
  • pattern - Regular expression pattern representing entries to remove
Returns:
Number of entries removed
Raises:
  • ValueError - If the passed-in pattern is not a valid regular expression.

removeLinks(self, pattern=None)

source code 

Removes soft link entries from the list.

If pattern is not passed in or is None, then all soft link entries will be removed from the list. Otherwise, only those soft link entries matching the pattern will be removed. Any entry which does not exist on disk will be ignored (use removeInvalid to purge those entries).

This method might be fairly slow for large lists, since it must check the type of each item in the list. If you know ahead of time that you want to exclude all soft links, then you will be better off setting excludeLinks to True before adding items to the list.

Parameters:
  • pattern - Regular expression pattern representing entries to remove
Returns:
Number of entries removed
Raises:
  • ValueError - If the passed-in pattern is not a valid regular expression.

removeMatch(self, pattern)

source code 

Removes from the list all entries matching a pattern.

This method removes from the list all entries which match the passed in pattern. Since there is no need to check the type of each entry, it is faster to call this method than to call the removeFiles, removeDirs or removeLinks methods individually. If you know which patterns you will want to remove ahead of time, you may be better off setting excludePatterns or excludeBasenamePatterns before adding items to the list.

Parameters:
  • pattern - Regular expression pattern representing entries to remove
Returns:
Number of entries removed.
Raises:
  • ValueError - If the passed-in pattern is not a valid regular expression.

Note: Unlike when using the exclude lists, the pattern here is not bounded at the front and the back of the string. You can use any pattern you want.

removeInvalid(self)

source code 

Removes from the list all entries that do not exist on disk.

This method removes from the list all entries which do not currently exist on disk in some form. No attention is paid to whether the entries are files or directories.

Returns:
Number of entries removed.

_setExcludeFiles(self, value)

source code 

Property target used to set the exclude files flag. No validations, but we normalize the value to True or False.

_setExcludeDirs(self, value)

source code 

Property target used to set the exclude directories flag. No validations, but we normalize the value to True or False.

_setExcludeLinks(self, value)

source code 

Property target used to set the exclude soft links flag. No validations, but we normalize the value to True or False.

_setExcludePaths(self, value)

source code 

Property target used to set the exclude paths list. A None value is converted to an empty list. Elements do not have to exist on disk at the time of assignment.

Raises:
  • ValueError - If any list element is not an absolute path.

_setExcludePatterns(self, value)

source code 

Property target used to set the exclude patterns list. A None value is converted to an empty list.

_setExcludeBasenamePatterns(self, value)

source code 

Property target used to set the exclude basename patterns list. A None value is converted to an empty list.

_setIgnoreFile(self, value)

source code 

Property target used to set the ignore file. The value must be a non-empty string if it is not None.

Raises:
  • ValueError - If the value is an empty string.

_addDirContentsInternal(self, path, includePath=True, recursive=True, linkDepth=0, dereference=False)

source code 

Internal implementation of addDirContents.

This internal implementation exists due to some refactoring. Basically, some subclasses have a need to add the contents of a directory, but not the directory itself. This is different than the standard FilesystemList behavior and actually ends up making a special case out of the first call in the recursive chain. Since I don't want to expose the modified interface, addDirContents ends up being wholly implemented in terms of this method.

The linkDepth parameter controls whether soft links are followed when we are adding the contents recursively. Any recursive calls reduce the value by one. If the value zero or less, then soft links will just be added as directories, but will not be followed. This means that links are followed to a constant depth starting from the top-most directory.

There is one difference between soft links and directories: soft links that are added recursively are not placed into the list explicitly. This is because if we do add the links recursively, the resulting tar file gets a little confused (it has a link and a directory with the same name).

Parameters:
  • path - Directory path whose contents should be added to the list.
  • includePath - Indicates whether to include the path as well as contents.
  • recursive - Indicates whether directory contents should be added recursively.
  • linkDepth - Depth of soft links that should be followed
  • dereference - Indicates whether soft links, if followed, should be dereferenced
Returns:
Number of items recursively added to the list
Raises:
  • ValueError - If path is not a directory or does not exist.

Note: If you call this method on a link to a directory that link will never be dereferenced (it may, however, be followed).

verify(self)

source code 

Verifies that all entries in the list exist on disk.

Returns:
True if all entries exist, False otherwise.

Property Details [hide private]

excludeFiles

Boolean indicating whether files should be excluded.

Get Method:
_getExcludeFiles(self) - Property target used to get the exclude files flag.
Set Method:
_setExcludeFiles(self, value) - Property target used to set the exclude files flag.

excludeDirs

Boolean indicating whether directories should be excluded.

Get Method:
_getExcludeDirs(self) - Property target used to get the exclude directories flag.
Set Method:
_setExcludeDirs(self, value) - Property target used to set the exclude directories flag.

excludeLinks

Boolean indicating whether soft links should be excluded.

Get Method:
_getExcludeLinks(self) - Property target used to get the exclude soft links flag.
Set Method:
_setExcludeLinks(self, value) - Property target used to set the exclude soft links flag.

excludePaths

List of absolute paths to be excluded.

Get Method:
_getExcludePaths(self) - Property target used to get the absolute exclude paths list.
Set Method:
_setExcludePaths(self, value) - Property target used to set the exclude paths list.

excludePatterns

List of regular expression patterns (matching complete path) to be excluded.

Get Method:
_getExcludePatterns(self) - Property target used to get the exclude patterns list.
Set Method:
_setExcludePatterns(self, value) - Property target used to set the exclude patterns list.

excludeBasenamePatterns

List of regular expression patterns (matching basename) to be excluded.

Get Method:
_getExcludeBasenamePatterns(self) - Property target used to get the exclude basename patterns list.
Set Method:
_setExcludeBasenamePatterns(self, value) - Property target used to set the exclude basename patterns list.

ignoreFile

Name of file which will cause directory contents to be ignored.

Get Method:
_getIgnoreFile(self) - Property target used to get the ignore file.
Set Method:
_setIgnoreFile(self, value) - Property target used to set the ignore file.