Skip to content

Setup with sc_pack


To manage the edge servers running ShimmerCat Accelerator, we install a "pack" of programs called sc_pack. The installation consists of three steps: first you need to install sc_pack itself, second you need to ask sc_pack to prepare the hierarchy of directories, and third you need to create deployment sites and domains to serve.

Preliminary requirements
  • Make sure you have created an authentication token.
  • Ubuntu 16.04 or 18.04 is recommended.
  • We strongly suggest not to use root when executing the sc_pack commands during setup.

1. Installation


The PC on which sc_pack will be deployed is required to have python3. python3-dev, virtualenv, and gcc. You can install them by executing:

$ sudo apt-get install python3 python3-dev virtualenv gcc

We highly recommend that you use a Python3 virtualenv, and that you create the virtualenv as a subdirectory <venv> in the hierarchy we mentioned before. This can be done by executing:

$ virtualenv /path/to/install_dir/venv -p python3

Don't forget to replace /path/to/install_dir/ with the actual path you use. Then you can activate the virtualenv:

$ source /path/to/install_dir/venv/bin/activate

Download sc_pack

You can download the sc_pack distributable by clicking the below link (for other alternatives, see here):

After you have downloaded sc_pack to your server, with the virtualenv activated, do:

$ pip install /path/to/<sc_pack-version>.whl

Now sc_pack should be installed. To verify that it is properly installed run:

$ sc_pack -h

2. Directory hierarchy and configuration

The programs need a directory as the top of the hierarchy, and some information for configuration. This information contains details about your setup, and secrets needed for the programs to send and receive data from our cloud service.

The configuration file

All configuration is structured in a YAML file, called sc_pack.conf.yaml, which has nesting levels. There is an example file inside the sc_pack redistributable, that can be obtained by executing:

$ sc_pack extract example.sc_pack.conf.yaml

Copy the content that appears in the terminal and create a file with extension .yaml, called sc_pack.conf.yaml. You can find more details about all the fields here.

We have created an example file below - remember to edit the fields according to your details, mainly make sure that the install_dir is the one you have been using so far in the commands you have already run before. Also change the user and user_group to shimmercat.

# This is a YAML document with the configuration items
# we need to set up the server with sc_pack.
# For the format, see example below.
# All the directories defined are relative to install_dir
    root_dir: www
    devlove_file_path: devlove.yaml
    scratch_dir_name: shimmercat-scratch-folder
    listen_port: "4043"
    http2https_port: "8084"
    bots_views_dir: views-dir
api_access_token: <your_authentication_token> # Received in the first step of the getting started tutorial
user: shimmercat
user_group: shimmercat
install_dir: /path/to/sc_pack_files # Update this with the path you want the sc_pack files to be at
logs_host_name: localhost
deployment_site_long_name: <your_name16> # Update this value with the value of your deployment site's `name16` you will create on the next step
deployment_site_long_secret: <your_secret16> # Update this value with the value of your deployment site's `secret16` you will create on the next step
amqp_vhost: skyloft
enable_bots_blocking: False
enable_images_optimization: False
improve_images_quality: False
humanity_validator_host: localhost
humanity_validator_port: 8060 # The port where the service that send the Google 
google_recaptcha_site_key: <your_google_recaptcha_site_key> # Only needed if enable_bots_blocking is True. If needed you should get it from
google_recaptcha_site_secret: <your_google_recaptcha_site_secret> # Only needed if enable_bots_blocking is True. If needed you should get it from
transit_encryption_key: <your_transit_encryption_key> # User-defined variable. This will be the encryption key to encrypt and decrypt data sent from the edge servers to the Accelerator Platform and vice versa. Among other things, it is used to synchronize the certificates and their private keys. Make sure to set the same key to all the deployments you would like to synchronize. You choose the key yourself.
deployment_group: test

Directory hierarchy

If you don't have a user and user_group called shimmercat we recommend that you create it:

$ sudo adduser shimmercat

and provide it with permission to the installation dir:

$ sudo chown -R shimmercat:shimmercat <install_dir>

Now change to user shimmercat

$ su shimmercat

And activate the virtualenv:

$ source /path/to/install_dir/venv/bin/activate

Once the configuration file is created, it's time to setup the directory hierarchy. If you have already a file called sc_pack.conf.yaml in the directory from where you are running that command, run:

$ sc_pack create

Otherwise you can let sc_pack create the deployment environment by typing:

$ sc_pack create -f configuration/file/path/sc_pack.conf.yaml

Subject to the same rule, it's possible to omit the -f in many sc_pack sub-commands. E.g, if the current directory is /home/shimmercat/test/shimmercat-accelerator and the file /home/shimmercat/test/shimmercat-accelerator/sc_pack.conf.yaml exists and you run sc_pack create, the configuration file taken is /home/shimmercat/test/shimmercat-accelerator/sc_pack.conf.yaml.

The sc_pack create command also creates an example devlove.yaml file, which is ShimmerCat's configuration file with domains and such. You will edit this file in the coming steps, and for more info regarding the file you can check here.

You can double check that the directories are properly created:

$ ls
$ celery redis shimmercat   devlove.yaml sc_logs_agent shimmercat-scratch-folder views-dir  # And some others

And now you can start the accelerator:

$ sc_pack supervisord

You will see some logs similar to:

2019-05-18 14:52:00,385 INFO success: redis entered RUNNING state,
process has stayed up for > than 10 seconds (startsecs)

2019-05-18 14:52:00,385 INFO success: shimmercat entered RUNNING state,
process has stayed up for > than 10 seconds (startsecs)

2019-05-18 14:52:00,385 INFO success: kv_store entered RUNNING state,
process has stayed up for > than 10 seconds (startsecs)

2019-05-18 14:52:00,385 INFO success: humanity_validator entered RUNNING state,
process has stayed up for > than 10 seconds (startsecs)

To start the accelerator at boot time, see here.

Inside every service directory there is a directory named data, where the logs for each service are stored. If there are errors, please check the logs for more details. For example, if you see the error celery (exit status 1; not expected), you can check more details in the log file .../celery/data/celery.log.


Note that it's expected that the celery and the sc_logs_agent don't start with the current configuration. The reason is that we are still missing the info for deployment_site_long_name and deployment_site_long_secret that are needed to let Celery authenticate with Celery broker.

3. Create deployment sites and domains

A DeploymentSite represents a site where sc_pack has been deployed, and it is the root of the tree to configure everything else. In the commands below, make sure to replace <your_authentication_token> with your authentication token received in the first step of the getting started tutorial.

Create a deployment site

To create a DeploymentSite use the below command:

$ curl -X POST -H 'Authorization: Token <your_authentication_token>' -H 'Content-type: application/json' -d '{"description": "Deployment site on server test1"}'

You will get the response data:

  "customer": {
      "id": <your_customer_id>,
      "name":"ShimmerCat Accelerator Test"
  "description":"Deployment site on server test1",

Now edit the file sc_pack.conf.yaml and add the responses for <your_deployment_site_long_name> and <your_deployment_site_long_secret>. To update the configuration files on the file system with the name16, and secret16 execute:

$ sc_pack create

Whenever you change sc_pack.conf.yaml you should run sc_pack create - otherwise the changes won't be synchronized to all the configuration files used by the services.

After sc_pack create we need to start all the services again by executing:

$ sc_pack ctl start all

Note that with the command sc_pack ctl you can control the services. For example, to see if a service is running you can use sc_pack ctl status shimmercat, sc_pack ctl status redis, and sc_pack ctl status sc_logs_agent. To restart a sevice you can use the command sc_pack ctl restart.

Create a domain for the deployment site

To create a domain for the deployment site, use the command below:

$ curl -X POST -H 'Authorization: Token <your_authentication_token>' -H 'Content-type: application/json' -d '{"name": "", "deployment_site_id": <your_deployment_id>}'

Note that you need to change <your_deployment_id> to make sure that the domain you are creating will be linked to the deployment site with this id. You will get the response:


If you would like to add more deployment sites to your domain, see more information on Adding a new deployment for an existing domain. You are now ready to serve a website with ShimmerCat.

Serve the example website

In your install_dir (as an example we use /home/shimmercat/test/shimmercat-accelerator) there is a server config file named devlove.yaml, which was automatically created in the previous sections. If you want to try our static website example, edit the devlove.yaml file according to:

              root-dir: www

Then download our static website example and decompress it in the directory /home/shimmercat/test/shimmercat-accelerator/www. Be sure to check that you have the file /home/shimmercat/test/shimmercat-accelerator/www/index.html.

Now edit your /etc/hosts to have the line

Last thing is to run the command:

$ sc_pack supervisord

Then open a browser and visit:


Congratulations, you have now completed the tutorial and ShimmerCat should be running for the domain!

To learn more about sc_pack, check out the info here. For the next recommended tutorials, see:

Thanks a lot for your time, and keep reading!