CLI usage
The SSF command line interface consists of the following main commands
and a --config path.
These commands can be used with various options.
The format is:
Multiple commands can be used within the same line:(Note: If --config is not defined, SSF will look for an ssf_config.yaml file in the current directory.)
All SSF commands will return a result code. RESULT_OK (0) is expected for succesful completion.
If multiple commands are issued, each command is actioned in turn while they each return RESULT_OK. If a command fails to return RESULT_OK, then SSF will return immediately and subsequents steps will be skipped.
For a complete list of result codes, see result codes.
By default, SSF will print INFO level log to the terminal (stdout) and DEBUG level log to its log file ssf.log. The log file, ssf.log, can contain useful additional information if SSF returns with an error or throws an exception. The log levels can be modified; see --file-log-level and --stdout-log-level in options for more details.
SSF is tolerant to unknown arguments and commands. SSF will log these as warnings.
SSF init
init is designed to set up a clean environment before you build.
This will:
- Remove endpoint artifacts generated by a previous
build - Remove custom
artifacts(if any) specified in ssf_config.yaml - Remove the virtual environment generated by a previous
buildorrun - Clone a repo if using a remote model repository
For example:
# Example 1: remove all generated endpoint files related to ssf_config.yaml
gc-ssf --config examples/simple/ssf_config.yaml init
# Example 2: remove all generated endpoint files related to ssf_config.yaml
# also clone a fresh copy of the remote repo git@github.com:graphcore/simple-server-framework.git
gc-ssf --config 'git@github.com:graphcore/simple-server-framework.git|examples/simple/ssf_config.yaml' init
SSF build
This shows how to issue build for an example.
This step will generate the API endpoint files.
It will instantiate the application in its own virtual environment and run the build method.
You can use this to write custom preparation steps to run offline, for example compilation, weights download,
cache generation, and so on.
$ gc-ssf --config examples/simple/ssf_config.yaml build
2023-03-28 16:40:20,893 644580 INFO ==== Build ==== (build.py:15)
2023-03-28 16:40:20,893 644580 INFO > Generate_endpoints (build.py:17)
2023-03-28 16:40:20,898 644580 INFO > Load application (build.py:20)
2023-03-28 16:40:20,898 644580 INFO Creating application (application.py:95)
2023-03-28 16:40:20,898 644580 INFO Checking application dependencies (application.py:97)
2023-03-28 16:40:20,898 644580 INFO Loading simple_test application from /home/examples/simple/my_application.py with module id simple_test (application.py:103)
2023-03-28 16:40:20,898 644580 INFO loading module /home/examples/simple/my_application.py with module name simple_test (utils.py:110)
2023-03-28 16:40:20,898 644580 INFO Create MyApplication instance (my_application.py:34)
2023-03-28 16:40:20,898 644580 INFO > Build application (build.py:23)
2023-03-28 16:40:20,898 644580 INFO Changed directory to /home/examples/simple (utils.py:123)
2023-03-28 16:40:20,898 644580 INFO MyApp build (my_application.py:14)
2023-03-28 16:40:20,898 644580 INFO Returned directory to /home (utils.py:127)
The endpoint files are auto-generated in the current working directory:
NOTE:
Use init to remove build artifacts.
SSF run
Start running the application (server).
It is assumed the endpoints already exist (=> issue build first.)
The application will run in its own virtual environment.
This shows how to issue run for an example.
Example output:
$ gc-ssf --config examples/simple/ssf_config.yaml run
2023-03-28 16:41:29,813 644749 INFO ==== Run ==== (run.py:17)
2023-03-28 16:41:29,813 644749 INFO > Starting fastapi runtime with examples/simple/ssf_config.yaml (run.py:28)
2023-03-28 16:41:29,813 644749 INFO > Running Uvicorn (ssf_run.py:24)
2023-03-28 16:41:29,989 644749 INFO > Running FastAPI server (server.py:27)
2023-03-28 16:41:29,995 644749 INFO > Test API : A very simple test API (server.py:38)
2023-03-28 16:41:29,996 644749 INFO > Creating FastAPI applications (server.py:43)
2023-03-28 16:41:29,996 644749 INFO > Creating FastAPI instance (server.py:50)
2023-03-28 16:41:29,997 644749 INFO > Loading endpoints for simple_test (server.py:79)
2023-03-28 16:41:29,997 644749 INFO > Loading simple_test endpoint from /home/ssf_simple_test_endpoint_0_fastapi.py with module id simple_test_endpoint_0 (server.py:85)
2023-03-28 16:41:29,997 644749 INFO loading module /home/ssf_simple_test_endpoint_0_fastapi.py with module name simple_test_endpoint_0 (utils.py:110)
2023-03-28 16:41:29,998 644749 INFO Loaded /home/ssf_simple_test_endpoint_0_fastapi.py for simple_test endpoint (ssf_simple_test_endpoint_0_fastapi.py:33)
2023-03-28 16:41:30,000 644749 INFO > Loading simple_test endpoint from /home/ssf_simple_test_endpoint_1_fastapi.py with module id simple_test_endpoint_1 (server.py:85)
2023-03-28 16:41:30,000 644749 INFO loading module /home/ssf_simple_test_endpoint_1_fastapi.py with module name simple_test_endpoint_1 (utils.py:110)
2023-03-28 16:41:30,002 644749 INFO Loaded /home/ssf_simple_test_endpoint_1_fastapi.py for simple_test endpoint (ssf_simple_test_endpoint_1_fastapi.py:33)
2023-03-28 16:41:30,004 644749 WARNING API key has not been specified, endpoints are not secured. (server.py:94)
2023-03-28 16:41:30,005 644749 INFO Started server process [644749] (server.py:74)
2023-03-28 16:41:30,005 644749 INFO Waiting for application startup. (on.py:48)
2023-03-28 16:41:30,005 644749 INFO > Startup (server.py:106)
2023-03-28 16:41:30,011 644815 INFO Dispatcher started for simple_test [644749->644815] (dispatcher.py:59)
2023-03-28 16:41:30,011 644815 INFO > Getting user application instance (dispatcher.py:64)
2023-03-28 16:41:30,011 644815 INFO Creating application (application.py:95)
2023-03-28 16:41:30,011 644815 INFO Checking application dependencies (application.py:97)
2023-03-28 16:41:30,011 644815 INFO Loading simple_test application from /home/examples/simple/my_application.py with module id simple_test (application.py:103)
2023-03-28 16:41:30,011 644815 INFO loading module /home/examples/simple/my_application.py with module name simple_test (utils.py:110)
2023-03-28 16:41:30,012 644815 INFO Create MyApplication instance (my_application.py:34)
2023-03-28 16:41:30,012 644815 INFO instance=<simple_test.MyApplication object at 0x7fc0503ab250> (dispatcher.py:66)
2023-03-28 16:41:30,012 644815 INFO > Initialising user application instance (dispatcher.py:69)
2023-03-28 16:41:30,012 644815 INFO MyApp initialise (my_application.py:18)
2023-03-28 16:41:31,012 644749 INFO Application startup complete. (on.py:62)
2023-03-28 16:41:31,013 644749 INFO Uvicorn running on http://0.0.0.0:8100 (Press CTRL+C to quit) (server.py:217)
It is valid to combine build and run in one step:
SSF package
This shows how to issue package for an example.
It is assumed the endpoints already exist (=> issue build first.)
This will pull resources into a local directory (.package) and build a docker image with name <application_id>:latest
Example output:
$ gc-ssf --config examples/simple/ssf_config.yaml package
2023-03-28 21:03:52,071 1307287 INFO ==== Package ==== (package.py:15)
2023-03-28 21:03:52,071 1307287 INFO > Packaging simple_test to /home/.package (package.py:30)
2023-03-28 21:03:52,072 1307287 INFO > Package SSF from /home/ssf (package.py:70)
2023-03-28 21:03:52,077 1307287 INFO > Package Application from /home/examples/simple (package.py:86)
2023-03-28 21:03:52,077 1307287 INFO > Package Endpoint files (package.py:90)
2023-03-28 21:03:52,078 1307287 INFO > Generate docker image (package.py:126)
2023-03-28 21:03:52,114 1307287 INFO [Docker image build] Sending build context to Docker daemon 103.9kB (utils.py:73)
2023-03-28 21:03:52,198 1307287 INFO [Docker image build] Step 1/14 : FROM graphcore/pytorch:3.1.0-ubuntu-20.
[...]
2023-03-28 21:03:52,226 1307287 INFO [Docker image build] ---> Using cache (utils.py:73)
2023-03-28 21:03:52,226 1307287 INFO [Docker image build] ---> 1a39476a9bb4 (utils.py:73)
2023-03-28 21:03:52,226 1307287 INFO [Docker image build] Step 14/14 : CMD cd src && ./run.sh (utils.py:73)
2023-03-28 21:03:52,227 1307287 INFO [Docker image build] ---> Using cache (utils.py:73)
2023-03-28 21:03:52,227 1307287 INFO [Docker image build] ---> ed9945bd6e3a (utils.py:73)
2023-03-28 21:03:52,228 1307287 INFO [Docker image build] Successfully built ed9945bd6e3a (utils.py:73)
2023-03-28 21:03:52,230 1307287 INFO [Docker image build] Successfully tagged simple_test:latest (utils.py:73)
2023-03-28 21:03:52,237 1307287 INFO > Run: 'docker run --rm -d --network host --name simple_test simple_test:latest' (package.py:137)
2023-03-28 21:03:52,237 1307287 INFO > Logs: 'docker logs simple_test' (package.py:138)
2023-03-28 21:03:52,237 1307287 INFO > Stop: 'docker stop simple_test' (package.py:139)
It is valid to combine build and package in one step:
SSF publish
This shows how to issue 'publish' for an example.
This will push the most recent packaged docker image to a docker server (default Docker Hub).
This assumes the docker server is already logged in, but if a docker username and password are provided, then these will be used to login before pushing.
A specific server may also be specified if required using --container-server.
gc-ssf --config examples/simple/ssf_config.yaml --docker-username <username> --docker-password <password> publish
Example output:
gc-ssf --config examples/simple/ssf_config.yaml publish
2023-04-11 17:41:36,382 1273566 INFO > Config examples/simple/ssf_config.yaml (cli.py:334)
2023-04-11 17:41:36,387 1273566 INFO > ==== Publish ==== (publish.py:15)
2023-04-11 17:41:37,212 1273566 INFO > Docker push user/myapp:simple_test-1.0-latest to DockerHub (publish.py:43)
2023-04-11 17:41:37,252 1273566 INFO [Push user/myapp:simple_test-1.0-latest] The push refers to repository [docker.io/user/myapp] (utils.py:193)
2023-04-11 17:41:37,509 1273566 INFO [Push user/myapp:simple_test-1.0-latest] 0f277cce0197: Preparing (utils.py:193)
[...]
2023-04-11 17:41:40,249 1273566 INFO [Push user/myapp:simple_test-1.0-latest] simple_test-1.0-latest: digest:
sha256:cb...0d6 size: 5996 (utils.py:193)
It is assumed the package already exist (=> issue 'package' first.)
It is valid to combine package and publish in one step:
SSF deploy
This shows how to issue 'deploy' for an example.
It is assumed the image already exists on Docker hub, two possibilities:
- You have packaged it yourself and used
publishfirst, then use the option--deploy-package(see example 1) - You use the public pre-build SSF image (see example 2)
It is valid to combine publish and deploy in one step:
Deployment to Gcore
For Gcore the deployment commands are passed using SSH. So the required options will be:
--deploy-platform(Gcore)--deploy-gcore-target-address: The Gcore VM IP address to SSH--deploy-gcore-target-username: The Gcore VM username to SSH (by default VMs useubuntu)
gc-ssf --config examples/simple/ssf_config.yaml --deploy-platform Gcore --deploy-gcore-target-address <ipaddr> --deploy-gcore-target-username <username> deploy
For example:
gc-ssf deploy --config ssf_config.yaml --deploy-platform Gcore --port 8100 --deploy-gcore-target-address 123.456.789.0 --deploy-gcore-target-username ubuntu --docker-username dockerUser --docker-password my-docker-token --add-ssh-key KEY_ENV_VARIABLE --deploy-package
This will deploy the most recently published image to the target platform (Gcore).
The config application ID is used for the deployment name unless --deploy-name is used to override it.
This will create a new deployment or update an existing deployment with the same deployment name if one already exists.
The target server must have already been created.
Use --add-ssh-key to add an SSH key via environment variable if required.
Deployment to Paperspace
For Paperspace the deployment is made using the Gradient API. So the required options will be:
--deploy-platform(Paperspace)--deployment-paperspace-api-key: The Paperspace API key--deploy-paperspace-project-id: The Paperspace project ID--deployment-paperspace-cluster-id: The Paperspace cluster ID--deploy-paperspace-registry: The Paperspace container registry
gc-ssf --config examples/simple/ssf_config.yaml --deploy-platform Paperspace --deploy-paperspace-project-id <projectId> --deployment-paperspace-cluster-id <clusterId> --deploy-paperspace-registry <registry> --deployment-paperspace-api-key <apiKey> deploy
For example:
gc-ssf deploy --config examples/simple/ssf_config.yaml --deploy-platform Paperspace --deploy-paperspace-project-id p01234567890 --deployment-paperspace-cluster-id c01234567 --deploy-paperspace-registry "Graphcore Cloud Solutions Dev R-O" --deploy-paperspace-api-key PAPERSPACE_API_KEY --package-tag graphcore/cloudsolutions-dev:mnist_api-1.0 --deploy-package
This will deploy the most recently published image, graphcore/cloudsolutions-dev:mnist_api-1.0, as a Paperspace deployment where the project ID is p01234567890, the cluster ID is c01234567, and the API key has been pre-set in environment variable PAPERSPACE_API_KEY.
The config application ID is used for the deployment name unless --deploy-name is used to override it.
This will create a new deployment or update an existing deployment if one already exists.
The project and API key must have already been created and a container registry must have already been configured (see --deploy-paperspace-registry).
Deployment arguments
See gc-ssf --help for other --deploy- arguments.
Some SSF CLI arguments, for example --port, --key, --replication-dispatcher and --fastapi-replicate-server, are passed to the remote deployment using the SSF_OPTIONS environment variable. The ssf.log can be used to check arguments passed to the remote deployment.
The SSF CLI argument --deploy-custom-args can be used to append additional arguments to the SSF_OPTIONS environment variable. This might be useful to pass additional arguments or override existing arguments.
SSF test
This shows how to issue 'test' for an example.
This will start and test the current packaged Docker image (run gc-ssf package first.)\
2 types of test are issued:
- Built-in SSF tests (always run): Testing that the server runs normally (for instance accessing root endpoint)
- Custom tests (optional): Users can define their own tests to run with this command, see the section "Create a test interface for your application".
If the current system can not satisfy the application dependency (for example the required Poplar version), then the test will be skipped and a warning will be logged.
If IPUs are required then remember to set 'ipus' in the SSF config.
Example output:
$ gc-ssf --config examples/simple/ssf_config.yaml test
2023-05-04 10:33:04,800 1578140 INFO > Config examples/simple/ssf_config.yaml (cli.py:404)
2023-05-04 10:33:04,805 1578140 INFO > ==== Test ==== (test.py:12)
2023-05-04 10:33:04,805 1578140 INFO > Start simple-test user/repo:simple-test-1.0-latest (test.py:263)
2023-05-04 10:33:04,871 1578140 INFO SSF_OPTIONS=-p 8100 --key test_key (test.py:65)
2023-05-04 10:33:05,118 1578140 INFO > Wait simple-test (test.py:276)
2023-05-04 10:33:10,202 1578140 INFO > Subtest Check root endpoint (test.py:279)
2023-05-04 10:33:10,223 1578140 INFO > Subtest Check security logout (test.py:286)
2023-05-04 10:33:10,243 1578140 INFO > Subtest Check security forbidden (test.py:293)
2023-05-04 10:33:10,260 1578140 INFO > Subtest Check security accepted (test.py:300)
2023-05-04 10:33:10,277 1578140 INFO OK 4 KO 0 (test.py:307)
2023-05-04 10:33:10,324 1578140 INFO > Stop simple-test (test.py:314)
2023-05-04 10:33:20,559 1578140 INFO > Remove simple-test (test.py:316)
It is assumed the package already exist (=> issue 'package' first.)
It is valid to combine package and test in one step:
If the test passes, then gc-ssf test will return RESULT_OK (0).
If the test is skipped (for example, due to configuration) then gc-ssf test will return RESULT_UNMET_REQUIREMENT (255).
If the test fails, then a SSFExceptionApplicationTestError exception will be thrown.
Example: