Client for Anaconda Cloud Storage service
pip install -i https://pypi.anaconda.org/anaconda-cloud/simple anaconda-cloud-storage
pip install -i https://pypi.anaconda.org/anaconda-cloud/label/dev/simple anaconda-cloud-storage
conda install anaconda-cloud::anaconda-cloud-storage
conda install anaconda-cloud/label/dev::anaconda-cloud-storage
FSSpec driver and anaconda CLI plugin for Anaconda Cloud storage service. This doc will cover special cases for the Anaconda Cloud Storage driver and is not general FSSpec manual. Refer to the FSSpec documentation for more information about how to use FSSpec drivers for filesystem operations.
Additionally, a Universal Pathlib implementation for the FSSpec driver is added here.
To use the CLI or FSSpec driver you must either login using the interactive login commands provided by
anaconda-cloud-auth or set the API key environment variable.
All anaconda-cloud-auth environment variables are supported by both the CLI and the FSSpec driver.
Additionally, the FSSpec driver allows you to set api_key
, domain
, extra_headers
, and api_version
when instantiating the driver, which take precedence over environment variables.
This package defines the Anaconda Storage URI used to reference projects and paths to files and folders in those projects. This URI format is used both in the FSSpec Python driver and in the CLI.
anaconda://<owner>/<project>[#<checkpoint>]/path/to/file[#<revision>]
where checkpoint
and revision
are optional. Note that checkpoint
and revision
cannot be specified together.
Projects on the Anaconda Storage API save every change (including deletions) to all files in the project.
Each revision is accessible using the #<revision>
flag in the URI.
The following values for revision
are supported
-1
, -2
, etc. to reference older versions in descending chronological orderstat
CLI command or the fs.info(<path-to-file>)
methodProjects on the Anaconda Storage API have checkpoints that capture the state of the project at a point in time. Two types of checkpoints are supported, time-point and ad-hoc. Time-point access is available whether or not you have created an ad-hoc checkpoint.
The following values for checkpoint
are supported
YYYY-MM-DDTHH:MM:SS
format to access the state of the project as of the requested point in time-1
, -2
, etc. to reference older ad-hoc checkpoints in descending chronological orderstat
CLI command or the fs.info(<owner>/<project>)
methodThe .chmod()
method and chmod
CLI command are used to grant and revoke permissions
or transfer ownership of a project to users or organizations on Anaconda.cloud.
The input is a list of permission strings in the format
<u or o>/<name or UUID><+ or -><o, w, or r>
The prefix specifies whether the name
or UUID
is an organization or user
u/
means the following name or UUID refers to a usero/
means the following name or UUID refers to an organizationTo add a specific permission for a user or org use the +
to add or -
to remove.
The available permissions are
o
to transfer ownership (A project can only be transferred to being owned by an organization and only if you are a member)r
to grant or revoke read permissions to files and metadata about your projectw
to grant or revoke write permissions to files and metadata about your projectHere are some examples and descriptions of action
u/49aa1e19-0527-4bc0-b8bd-31c6ea82710a+r
: grant read access for a single usero/d8c4f686-aff9-45f4-b6ec-8ed3976bca27-w
: revoke write access for all members of the organization
o/collective-w
: Alternate format using the Organization name collectiveThe .chattr()
method and chattr
CLI command are used to modify attributes and
metadata of projects and files.
The following attributes can be changed for Projects:
[a-z0-9-_]
data
and notebook
The following attributes can be changed for Files. Note that this will create a new file revision.
The Anaconda Cloud Storage API does not officially support empty directories. Virtual directories are handled by
filenames containing one or more /
. You'll see below that the mkdir('<owner>/<project>/path/to/dir')
method creates
an empty file called .keep
in the desired directory. Not all virtual directories will have a .keep
file.
For cp
, mv
, upload
, and download
operations a trailing /
, like
rsync, will
refer to the contents of the directory rather than the directory itself.
The storage
subcommands provides a Unix-like interface to the storage service.
The following subcommands are provided
cat Print the contents of a file to the console
chattr Modify attributes and metadata of projects and files
checkpoint Create a new checkpoint the current contents of a project
chmod Modify permissions granted to users and organizations for a project
cp Copy files and folders within and between projects
download Download whole projects, files, or directories
ls List usernames, projects, versions, directories and files
mkdir Create a new project or empty directory within a project
mv Move/rename files and folders within and between projects
rm Remove files or directories within a project
rmdir Delete a project or empty directories within a project
stat Information about projects (including checkpoints and permissions), directories, and files (including revisions)
touch Create a new empty file. Use --truncate to overwrite an existing file.
upload Upload files or directories
File operations require the full path of the file to be specified using the URI definition above, The anaconda://
protocol prefix in the URI is optional.
This package defines the anaconda
FSSpec protocol
with the URI format above.
By default the FSSpec driver uses the account you logged in with using anaconda login
on the terminal or with
Anaconda Navigator.
Additionally, the following keyword arguments can be specified when instantiating a FileSystem object or reading data by URI
api_key
: The API key or access_token.
ANACONDA_CLOUD_API_KEY
env var, token stored in keyring using anaconda login
CLI commanddomain
: The domain to send requests
anaconda.cloud
otherwise the value from the ANACONDA_CLOUD_API_DOMAIN
env varapi_version
: The minimum API version to request
"2023.11.28"
extra_headers
: Pass extra headers to the Client session
{}
. The value can either be a dict or JSON string of key-value pairs.Api-Version
in extra_headers it will be ignored. For this case use api_version=
.These arguments can be provided in the 4 filesystem invocation variants
import fsspec
fs = fsspec.filesystem('anaconda', api_key=..., api_domain=..., api_version=..., extra_headers=...)
from anaconda_cloud_storage import AnacondaFileSystem
fs = AnacondaFileSystem(api_key=..., api_domain=..., api_version=..., extra_headers=...)
import fsspec
with fsspec.open('anaconda://<owner>/<project>[#checkpoint]/path/to/file[#revision]',
storage_options={'api_key': ..., 'domain': ..., 'api_version': ..., 'extra_headers':...}) as f:
# read or write data
import pandas as pd
df = pd.read_csv('anaconda://<owner>/<project>[#checkpoint]/path/to/file[#revision]', storage_options={'api_key': ..., 'domain': ..., 'api_version': ..., 'extra_headers':...})
(the same works for to_csv
and Dask)
df = pd.read_csv('anaconda://<owner>/<project>[#checkpoint]/path/to/file[#revision]?api_key=...&domain=...&api_version=...&extra_headers=...')
(the same works for to_csv
and Dask)
In addition to the standard FSSpec methods the
anaconda
protocol provides the following specialized methods when instantiating a file system object.
.mkdir('<owner>/<project>[/path/to/dir]')
<owner>/<project>
<owner>/<project>/path/to/dir
by writing an empty file called <owner>/<project>/path/to/dir/.keep
.rmdir('<owner>/<project>[/path/to/dir]')
<owner>/<project>
<owner>/<project>/path/to/dir
, only if the directory only contains a file called .keep
.chmod('<owner>/<project>', [<list-of-permissions])
to modify permissions of the project (see above for permission format).checkpoint('<owner>/<project>)
to create an ad-hoc checkpoint.stat('<anaconda uri>')
and .info('<anaconda uri>')
return detailed metadata, including
.chattr('<owner>/<project>[/path/to/file]', name=<str>, title=<str>, tags=<List[str]>, metadata=<Dict[str, str]>, description=<str>, hidden=<bool>)
<owner>/<project>
you can change name
, title
, tags
, hidden
flag, or description
<owner>/<project>/path/to/file
you can change description
or metadata
This package defines the anaconda
UPath protocol
with the URI format above.
Ensure you have conda
installed.
Then run:
make setup
make test
make tox