Command line interface

Faculty has a command line client, faculty. This lets you:

• create, list, and terminate servers;

• access your servers via SSH;

• transfer large files using appropriate protocols like scp and rsync.

faculty is designed to be used from a terminal on your local machine. It will work on macOS, Linux and Windows.

Note

The command line client was formerly called sml. The API will remain the same. The sml client will no longer receive updates.

Installation on MacOS and Linux

You can install it using pip:

$ pip install -U faculty-cli

If you don’t have pip, follow these instructions. If you get an error along the lines of Permission denied on macOS or Linux, you may need to re-run this command with sudo.

Once you have done this, continue to the Initialising faculty section.

Installation on Windows

The recommended way to use faculty on Windows is via Cygwin. This has only been tested on Windows 10.

Make sure you have an installation of Cygwin with the following packages. Installing new packages normally involves restarting the Cygwin installer.

• python3 (in the Interpreters, Python category)

• openssh (in the Net category)

• rsync (in the Net category)

In the Cygwin terminal, run:

$ python3 -m ensurepip
$ pip3 install -U faculty-cli

Once you have done this, continue to the Initialising faculty section.

Initialising faculty

The first time you use faculty, you will need to configure it with a client ID and secret. This is used to authenticate access to your resources in Faculty, but is different from your usual username and password. You can create client credentials through the My account page.

Once you’ve created your credentials, run the following and enter the credentials when prompted:

$ faculty login

Once faculty is configured, you can use it to create a new server on a project, sync data up to a project, and connect to a server from a terminal on your local machine.

To see the full list of commands run:

$ faculty --help

While most of the Faculty functionality is accessible through the UI, there are some things that you can only do through the command line (for instance, bulk uploading code to your workspace).

List projects and servers

You can list your Faculty projects with:

$ faculty project list

and you can list a project’s running servers with:

$ faculty server list <project>

Note

faculty server list, like many other faculty subcommands, takes a project as an argument. For projects with a unique name, you can simply give the project name, however for projects with non-unique names, you must pass the project ID. Project IDs are discoverable using faculty project list -v.

SSH into a server

Having successfully listed your servers, you can get SSH access using:

$ faculty shell <project> <server>

You can also provide faculty shell with arbitrary extra arguments, which will be passed through to ssh. For example, to set up an SSH tunnel to an arbitrary network port on the server:

$ faculty shell <project> <server> -L 9000:localhost:8888

When you SSH into a server, you will be in the /home/faculty directory. You probably want to access your project files first. These are located in /project. You can jump to that directory with:

$ cd /project

Note

As with projects, faculty shell and other commands that take a server as an argument can be passed either the server name or server ID. If you have servers with non-unique names, use faculty server list <project> -v to discover their IDs for use in faculty shell.

Create a new server

You can create new servers for a project using:

$ faculty server new <options> <project>

For example:

$ faculty server new --cores=8 --memory=16 --name=new-server user-playground

Full options can be listed with faculty server new --help.

Note

If the name of the project is not unique among projects you have access to, you must pass the project ID rather than the project name. Project IDs are discoverable using faculty project list -v.

Open the web interface of a server in your browser

You can open the web interface of a server in your browser with:

$ faculty server open <project>

faculty will open the web interface of one of the running servers in the project. If you wish to open the interface of a specific server, pass the server’s name or ID with --server <server>.

Copying files to and from your workspace

You can also use faculty to upload and download files to and from your project on the command line. To upload a file, use faculty file put:

$ faculty file put <project> <local-source> <remote-destination>

Similarly, to download a file, use faculty file get:

$ faculty file get <project> <remote-source> <local-destination>

These commands will copy files from the source path provided to the destination path.

You need to have a server running on the project to upload files. If you have multiple servers, you can specify which server the files are routed through with the --server option.

Note

The remote path in faculty file put and faculty file get will be relative to the Unix home directory (/home/faculty) on the Faculty server. To put files directly in the project workspace, you will want to use a remote path under /project.

faculty file put and faculty file get use the scp command internally, and any additional command line arguments provided on the command line will be passed through to scp. Advanced users may find this useful to customise the behaviour of faculty file put and faculty file get.

Uploading large files to Faculty

When uploading large files, we recommend using faculty file sync-up, as this supports resuming the upload process if your connection drops. To start a new upload with resume support, use the following command:

$ faculty file sync-up <project> path/on/local/machine path/on/faculty --partial --progress

To resume an unfinished upload:

$ faculty file sync-up <project> path/on/local/machine path/on/faculty --append --progress

Uploading and downloading a large number of files to and from Faculty

If you have an entire directory of files to upload to Faculty, you can do so with faculty file sync-up. To upload a directory:

$ faculty file sync-up <project> path/on/localmachine/ path/on/faculty

To download an entire directory from Faculty to your local machine use faculty file sync-down:

$ faculty file sync-down <project> path/on/faculty/ path/on/localmachine

You can also provide faculty file sync-down and faculty file sync-up with arbitrary extra arguments, which will be passed through to the rsync program. For example, to upload a directory with compression and delete files in the directory on Faculty not present on your local machine, use:

$ faculty file sync-up <project> path/on/localmachine/ path/on/faculty --compress --delete

Note

Users unfamiliar with the rsync program, and especially the meaning of trailing slashes on the local and remote arguments, and the meaning of the --delete option, are strongly recommended to read its documentation before using faculty file sync-down or faculty file sync-up. Incorrect use of these features may result in data loss.

Server environments

Server Environments can be created in Faculty and applied through the web interface or through faculty. The faculty environment subcommands provide the ability to list and apply environments in a project, and to view the results of applying environments.

To list the environments in a project, run:

$ faculty environment list <project>

This will list all environments in the specified project.

You can apply one of these environments to an existing server using the faculty environment apply command:

$ faculty environment apply <project> <server> <environment>

As with other commands, each of the project, server and environment can be a name or an ID. To get the ID of server environments in a project, you can use the -v option with faculty environment list.

The status of the last environment applied to a server can be checked with the faculty environment status command:

$ faculty environment status <project> <server>

This will summarise the steps of the environment being applied, along with the overall status of the execution. To stream the logs of the environment application as it’s being applied, use the faculty environment logs command:

$ faculty environment logs <project> <server>

You can also specify a particular step to stream the logs of using the --step command line option. This accepts the index of the step as given by faculty environment status.

Jobs

Jobs can be created in Faculty and run through the web interface or through faculty. The faculty job subcommands provide the ability to list and run jobs in a project, and to view the logs for a run.

To list the jobs in a project, run:

$ faculty job list <project>

This will list all jobs in the specified project.

You can run a job using the faculty job run command:

$ faculty job run <project> <job>

The project and job can be a name or ID. To get the ID of job in a project, you can use the -v option with faculty job list.

To run a job with parameters, run:

$ faculty job run <project> <job> "foo=bar,eggs=spam"

This will start a single job with the parameters “foo” and “eggs” set to “bar” and “spam” respectively.

To run a job multiple times with different parameters, run:

$ faculty job run <project> <job> "foo=bar1,eggs=spam1" "foo=bar2,eggs=spam2"

This will start two jobs, the first with parameters “bar1” and “spam1” and the second with parameters “bar2” and “spam2”.

You can also run a job multiple times with no parameters using the --num-subruns command line option.

To list the runs of a job, use the faculty job list-runs command:

$ faculty job list-runs <project> <job>

You can view the logs for a run using:

$ faculty job logs <project> <job> <run number>

For example, to view the logs for the second run of a job, use:

$ faculty job logs <project> <job> 2

To view the logs for the fifth sub-run of the third run of a job, use:

$ faculty job logs <project> <job> 3.5

Managing the credentials for multiple deployments

By default, when you have completed the Initialising faculty section, there will be a single default set of credentials saved on your machine.

If you are working on multiple Platform deployments you have to manually edit the credentials file at initialization and add the credentials for the additional deployments (each will generate you a different set of credentials) The credentials are found at ~/.config/faculty/credentials on MacOS/Linux/Cygwin), and you need to add another section, following the existing pattern as:

[profile-name]
domain = ...
client_id = ...
client_secret = ...

Here the value of profile-name can be anything, just have to be unique in the file. Afterwards you can switch which Platform deployment your CLI is interacting with by setting the FACULTY_PROFILE environment variable to the relevant profile-name value. For example, to set this value for the current shell on MacOS/Linux:

$ export FACULTY_PROFILE=profile-name

after which the faculty will interact with that profile for its operation.

Using the Faculty CLI from a Faculty server with multiple deployments

It is possible to use the Faculty CLI from a Faculty server to interact with a second Faculty deployment. For example, a usecase could be to transfer data between Faculty deployments.

• First you will need to setup the Faculty CLI to use multiple deployments from within a Faculty server.

• Ensure that ~/.config/faculty/credentials contains a default profile for the current source deployment, and a named profile for the destination deployment.

• Clear the following environment variables from the Faculty server environment:

  unset FACULTY_CLIENT_ID
  unset FACULTY_CLIENT_SECRET
  unset FACULTY_PROTOCOL
  unset FACULTY_DOMAIN
  unset SHERLOCKML_CLIENT_ID
  unset SHERLOCKML_CLIENT_SECRET
  unset SHERLOCKML_PROTOCOL
  unset SHERLOCKML_DOMAIN
  

• Override the Faculty CLI credentials to use the destination deployment profile on a single line.

The following command will list projects from the destination deployment:

FACULTY_PROFILE=<Destination Profile> faculty project list

• To transfer a local workspace file from the source Faculty server, to a project dataset on a destination Faculty deployment:

  FACULTY_PROFILE=<Destination Profile> faculty datasets put <Destination Project> <Local File> <Destination Dataset Path>