anaconda-cloud-storage

FSSpec driver and anaconda CLI plugin for Anaconda Cloud storage service. This doc will cover special cases for the Anaconda Cloud Storage driver and is not general FSSpec manual. Refer to the FSSpec documentation for more information about how to use FSSpec drivers for filesystem operations.

Additionally, a Universal Pathlib implementation for the FSSpec driver is added here.

Authentication

To use the CLI or FSSpec driver you must either login using the interactive login commands provided by anaconda-cloud-auth or set the API key environment variable. All anaconda-cloud-auth environment variables are supported by both the CLI and the FSSpec driver. Additionally, the FSSpec driver allows you to set api_key, domain, extra_headers, and api_version when instantiating the driver, which take precedence over environment variables.

Anaconda Storage URI

This package defines the Anaconda Storage URI used to reference projects and paths to files and folders in those projects. This URI format is used both in the FSSpec Python driver and in the CLI.

anaconda://<owner>/<project>[#<checkpoint>]/path/to/file[#<revision>]

where checkpoint and revision are optional. Note that checkpoint and revision cannot be specified together.

File metadata

You can set file metadata using .open(), .write_text() or .write_bytes() by passing optional keyword-arguments. At this time two fields are supported

description=<string> a string description of the file
metadata=<dict[str, Any]> a dictionary or user-supplied key-value pairs.

Example:

import fsspect
fs = fsspec.filesystem('anaconda')

with open('my/project/file', description='My new file', metadata={"key": "value"}) as f:
    f.write(...)

fs.write_text('my/project/another_file', description="Another file", metadata={"key": ['v1', 'v2']})

File revisions

Projects on the Anaconda Storage API save every change (including deletions) to all files in the project. Each revision is accessible using the #<revision> flag in the URI.

The following values for revision are supported

Integer decrement -1, -2, etc. to reference older versions in descending chronological order
File version UUID. UUIDs can be obtained from the stat CLI command or the fs.info(<path-to-file>) method

Project checkpoints

Projects on the Anaconda Storage API have checkpoints that capture the state of the project at a point in time. Two types of checkpoints are supported, time-point and ad-hoc. Time-point access is available whether or not you have created an ad-hoc checkpoint.

The following values for checkpoint are supported

Time-point in YYYY-MM-DDTHH:MM:SS format to access the state of the project as of the requested point in time
Integer decrement -1, -2, etc. to reference older ad-hoc checkpoints in descending chronological order
Ad-hoc Checkpoint UUID. UUIDs can be obtained from the stat CLI command or the fs.info(<owner>/<project>) method

Modify storage permissions (chmod)

The .chmod() method and chmod CLI command are used to grant and revoke permissions or transfer ownership of a project to users or organizations on Anaconda.cloud.

The input is a list of permission strings in the format

<u or o>/<name or UUID><+ or -><o, w, or r>

The prefix specifies whether the name or UUID is an organization or user

u/ means the following name or UUID refers to a user
o/ means the following name or UUID refers to an organization

To add a specific permission for a user or org use the + to add or - to remove.

The available permissions are

o to transfer ownership (A project can only be transferred to being owned by an organization and only if you are a member)
r to grant or revoke read permissions to files and metadata about your project
w to grant or revoke write permissions to files and metadata about your project

Here are some examples and descriptions of action

u/49aa1e19-0527-4bc0-b8bd-31c6ea82710a+r: grant read access for a single user
o/d8c4f686-aff9-45f4-b6ec-8ed3976bca27-w: revoke write access for all members of the organization
- o/collective-w: Alternate format using the Organization name collective

Modify storage attributes (chattr)

The .chattr() method and chattr CLI command are used to modify attributes and metadata of projects and files.

The following attributes can be changed for Projects:

name: String of 120 ascii lower-case characters according to this regex [a-z0-9-_]
title: String of 120 unicode characters
description: String of 1024 unicode characters
tags: A list of strings. Accepted values are data and notebook

The following attributes can be changed for Files. Note that this will create a new file revision.

description: String of 1024 unicode characters
metadata: Dictionary (JSON) mapping of key-value pairs

Directories and trailing /

The Anaconda Cloud Storage API does not officially support empty directories. Virtual directories are handled by filenames containing one or more /. You'll see below that the mkdir('<owner>/<project>/path/to/dir') method creates an empty file called .keep in the desired directory. Not all virtual directories will have a .keep file.

For cp, mv, upload, and download operations a trailing /, like rsync, will refer to the contents of the directory rather than the directory itself.

CLI subcommands

The storage subcommands provides a Unix-like interface to the storage service.

The following subcommands are provided

cat             Print the contents of a file to the console
chattr          Modify attributes and metadata of projects and files
checkpoint      Create a new checkpoint the current contents of a project
chmod           Modify permissions granted to users and organizations for a project
cp              Copy files and folders within and between projects
download        Download whole projects, files, or directories
ls              List usernames, projects, versions, directories and files
mkdir           Create a new project or empty directory within a project
mv              Move/rename files and folders within and between projects
rm              Remove files or directories within a project
rmdir           Delete a project or empty directories within a project
stat            Information about projects (including checkpoints and permissions), directories, and files (including revisions)
touch           Create a new empty file. Use --truncate to overwrite an existing file.
upload          Upload files or directories

File operations require the full path of the file to be specified using the URI definition above, The anaconda:// protocol prefix in the URI is optional.

FSSpec driver

This package defines the anaconda FSSpec protocol with the URI format above.

Initialization

By default the FSSpec driver uses the account you logged in with using anaconda login on the terminal or with Anaconda Navigator.

Additionally, the following keyword arguments can be specified when instantiating a FileSystem object or reading data by URI

api_key: The API key or access_token.
- If not present defaults to, in order, ANACONDA_CLOUD_API_KEY env var, token stored in keyring using anaconda login CLI command
domain: The domain to send requests
- defaults to anaconda.cloud otherwise the value from the ANACONDA_CLOUD_API_DOMAIN env var
api_version: The minimum API version to request
- defaults to "2023.11.28"
extra_headers: Pass extra headers to the Client session
- defaults to {}. The value can either be a dict or JSON string of key-value pairs.
- Only new keys will be passed on to the client. If you attempt to overwrite, for example, Api-Version in extra_headers it will be ignored. For this case use api_version=.

These arguments can be provided in the 4 filesystem invocation variants

by protocol

import fsspec
fs = fsspec.filesystem('anaconda', api_key=..., api_domain=..., api_version=..., extra_headers=...)

by driver class

from anaconda_cloud_storage import AnacondaFileSystem
fs = AnacondaFileSystem(api_key=..., api_domain=..., api_version=..., extra_headers=...)

as storage_options

import fsspec

with fsspec.open('anaconda://<owner>/<project>[#checkpoint]/path/to/file[#revision]',
    storage_options={'api_key': ..., 'domain': ..., 'api_version': ..., 'extra_headers':...}) as f:
    # read or write data

import pandas as pd

df = pd.read_csv('anaconda://<owner>/<project>[#checkpoint]/path/to/file[#revision]',  storage_options={'api_key': ..., 'domain': ..., 'api_version': ..., 'extra_headers':...})

(the same works for to_csv and Dask)

uri parameters

df = pd.read_csv('anaconda://<owner>/<project>[#checkpoint]/path/to/file[#revision]?api_key=...&domain=...&api_version=...&extra_headers=...')

(the same works for to_csv and Dask)

Special methods

In addition to the standard FSSpec methods the anaconda protocol provides the following specialized methods when instantiating a file system object.

.mkdir('<owner>/<project>[/path/to/dir]')
- Create a new project as <owner>/<project>
- or an empty directory within a project as <owner>/<project>/path/to/dir by writing an empty file called <owner>/<project>/path/to/dir/.keep
- You cannot have a file and a directory of the same name within a project
- .mkdir() supports kwargs to set metadata for projects and folders
  - If creating a project you can set description=<str> and title=<str>
  - If creating a directory in a project you can set description=<str> and metadata=<dict>
.rmdir('<owner>/<project>[/path/to/dir]')
- Delete a project as <owner>/<project>
- or delete an "empty" directory as <owner>/<project>/path/to/dir, only if the directory only contains a file called .keep
.chmod('<owner>/<project>', [<list-of-permissions]) to modify permissions of the project (see above for permission format)
.checkpoint('<owner>/<project>) to create an ad-hoc checkpoint
.stat('<anaconda uri>') and .info('<anaconda uri>') return detailed metadata, including
- ad-hoc checkpoints (for projects only)
- permissions (for projects only)
- revisions (for files only)
.chattr('<owner>/<project>[/path/to/file]', name=<str>, title=<str>, tags=<List[str]>, metadata=<Dict[str, str]>, description=<str>, hidden=<bool>)
- For projects <owner>/<project> you can change name, title, tags, hidden flag, or description
- For files <owner>/<project>/path/to/file you can change description or metadata

UPath implementation

This package defines the anaconda UPath protocol with the URI format above.

Setup for development

Ensure you have conda installed. Then run:

make setup

Run the unit tests

make test

Run the unit tests across isolated environments with tox

make tox