disco.ddfs – Client interface for Disco Distributed Filesystem

See also: Disco Distributed Filesystem.

Note

Parameters below which are indicated as tags can be specified as a tag:// URL, or the name of the tag.

class disco.ddfs.DDFS(master=None, proxy=None, settings=None)

Opens and encapsulates a connection to a DDFS master.

Parameters:master – address of the master, for instance disco://localhost.
attrs(tag, token=None)

Get a list of the attributes of the tag tag and their values.

blobs(tag, ignore_missing=True, token=None)

Walks the tag graph starting at tag.

Yields only the terminal nodes of the graph (blobs).

Parameters:ignore_missing (bool) – Whether or not missing tags will raise a disco.error.CommError.
chunk(tag, urls, replicas=None, forceon=[], retries=10, delayed=False, update=False, token=None, chunk_size=67108864, max_record_size=1048576, **kwargs)

Chunks the contents of urls, pushes the chunks to ddfs and tags them with tag.

delattr(tag, attr, token=None)

Delete the attribute attr of the tag tag.

delete(tag, token=None)

Delete tag.

exists(tag)

Returns whether or not tag exists.

get(tag, token=None)

Return the tag object stored at tag.

getattr(tag, attr, token=None)

Return the value of the attribute attr of the tag tag.

list(prefix='')

Return a list of all tags starting with prefix.

pull(tag, blobfilter=<function <lambda>>, token=None)

Iterate over the blobs in a tag after optionally applying a blobfilter over the blob names.

push(tag, files, replicas=None, forceon=[], retries=10, delayed=False, update=False, token=None)

Pushes a bunch of files to ddfs and tags them with tag.

Parameters:files (a list of paths, (path, name)-tuples, or (fileobject, name)-tuples.) – the files to push as blobs to DDFS. If names are provided, they will be used as prefixes by DDFS for the blobnames. Names may only contain chars in r'[^A-Za-z0-9_\-@:]'.
put(tag, urls, token=None)

Put the list of urls to the tag tag.

Warning

Generally speaking, concurrent applications should use DDFS.tag() instead.

setattr(tag, attr, val, token=None)

Set the value of the attribute attr of the tag tag.

tag(tag, urls, token=None, **kwargs)

Append the list of urls to the tag.

urls(tag, token=None)

Return the urls in the tag.

walk(tag, ignore_missing=True, tagpath=(), token=None)

Walks the tag graph starting at tag.

Yields a 3-tuple (tagpath, tags, blobs).

Parameters:ignore_missing (bool) – Whether or not missing tags will raise a disco.error.CommError.