Skip to content

Setup sc_pack in three steps

Preliminaries

To manage the edge servers running ShimmerCat Accelerator, we install a "pack" of programs called sc_pack. The installation consists of two steps: first you need to install sc_pack itself, and then you need to ask sc_pack to prepare the hierarchy of directories.

1. Installation

First, you will need to install python3, python3-dev, virtualenv and git on the server. You can install them by executing:

$ sudo apt-get install python3 python3-dev virtualenv git

We highly recommend that you use a Python3 virtualenv. This virtualenv will contain an isolated copy of Python3, sc_pack, supervisord and all the libraries needed for the tools and for the Celery worker. Since it's so tied to sc_pack, we recommend that you create the virtualenv as a subdirectory in the hierarchy we mentioned before, the so called <venv> directory. This can be done by executing:

$ virtualenv /path/to/install_dir/venv -p python3

Don't forget to replace /path/to/install_dir/ with the actual path you use. Then you can activate it:

$ source /path/to/install_dir/venv/bin/activate

You will need the sc_pack distributable to install. You can either find the download link by visiting the API page, or by directly using the below download command. It (requires jq), which you can install with:

$ sudo apt-get install jq

You can find the download_url at this API page. Now you can download the latest sc_pack with the below download command:

$ curl -O $(curl -s  https://accelerator.shimmercat.com/presentation-api/v1/download/sc_pack/?format=json | jq -r .download_url)

After you have downloaded sc_pack to your server, with the venv activated, do:

$ pip install /path/to/<sc_pack-version>.whl

Now sc_pack should be installed. To verify that it is properly installed run:

$ sc_pack -h

2. Directory hierarchy and configuration

The programs need a directory as the top of the hierarchy, and some information for configuration. This information contains details about your setup, and secrets needed for the programs to send and receive data from our cloud service.

The configuration file

All configuration is structured in a YAML file, called sc_pack.conf.yaml, which has nesting levels. There is an example file inside the sc_pack redistributable, that can be obtained by executing:

$ sc_pack extract example.sc_pack.conf.yaml

Copy the content that appears in the terminal and create a file with extension .yaml, called sc_pack.conf.yaml. You can find more details about all the fieldshere.

We have created an example file below - remember to edit the fields according to your details, mainly make sure the install_dir is the one you have been using so far in the commands you have already run before.

---
# This is a YAML document with the configuration items
# we need to set up the server with sc_pack.
# For the format, see example below.
#
# All the directories defined are relative to install_dir
shimmercat:
    root_dir: www
    devlove_file_path: devlove.yaml
    scratch_dir_name: shimmercat-scratch-folder
    listen_port: "4043"
    http2https_port: "8084"
    bots_views_dir: views-dir
api_access_token: <your_authentication_token> # Received in the first step of the getting started tutorial
user: shimmercat
user_group: shimmercat
install_dir: /path/to/sc_pack_files # Update this with the path you want the sc_pack files to be at
logs_host_name: localhost
deployment_site_long_name: <your_name16> # Update this value with the value of your deployment site's `name16` you will create on the next step
deployment_site_long_secret: <your_secret16> # Update this value with the value of your deployment site's `secret16` you will create on the next step
amqp_server_url: amqp.staging.c.shimmercat.com
amqp_vhost: skyloft
enable_bots_blocking: False
enable_images_optimization: False
improve_images_quality: False
humanity_validator_host: localhost
humanity_validator_port: 8060 # The port where the service that send the Google 
google_recaptcha_site_key: <your_google_recaptcha_site_key> # Only needed if enable_bots_blocking is True. If needed you should get it from https://www.google.com/recaptcha/admin/.
google_recaptcha_site_secret: <your_google_recaptcha_site_secret> # Only needed if enable_bots_blocking is True. If needed you should get it from https://www.google.com/recaptcha/admin/.
transit_encryption_key: <your_transit_encryption_key> # User-defined variable. This will be the encryption key to encrypt and decrypt data sent from the edge servers to the Accelerator Platform and vice versa. Among other things, it is used to synchronize the certificates and their private keys. Make sure to set the same key to all the deployments you would like to synchronize. You choose the key yourself.
deployment_group: test

Directory hierarchy

If you don't have a user and user_group called shimmercat you should create it:

$ chown -R shimmercat:shimmercat <install_dir>

Now change to user shimmercat

$ su shimmercat

Once the configuration file is created, it's time to setup the directory hierarchy. If you have already a file called sc_pack.conf.yaml in the directory from where you are running that command, run:

$ sc_pack create

Otherwise you can let sc_pack create the deployment environment by typing:

$ sc_pack create -f configuration/file/path/sc_pack.conf.yaml

Subject to the same rule, it's possible to omit the -f in many sc_pack sub-commands. E.g, if the current directory is /home/shimmercat/test/shimmercat-accelerator and the file /home/shimmercat/test/shimmercat-accelerator/sc_pack.conf.yaml exists and you run sc_pack create, the configuration file taken is /home/shimmercat/test/shimmercat-accelerator/sc_pack.conf.yaml.

The sc_pack create command also creates an example devlove.yaml file, which is ShimmerCat's configuration file with domains and such. You will edit this file in the coming steps, and for more info regarding the file you can check here.

You can double check that the directories are properly created:

$ ls
$ celery redis shimmercat   devlove.yaml sc_logs_agent shimmercat-scratch-folder views-dir  # And some others

And now you can start the accelerator:

$ sc_pack supervisord

You will see some logs similar to:

2019-05-18 14:52:00,385 INFO success: redis entered RUNNING state,
process has stayed up for > than 10 seconds (startsecs)

2019-05-18 14:52:00,385 INFO success: shimmercat entered RUNNING state,
process has stayed up for > than 10 seconds (startsecs)

2019-05-18 14:52:00,385 INFO success: kv_store entered RUNNING state,
process has stayed up for > than 10 seconds (startsecs)

2019-05-18 14:52:00,385 INFO success: humanity_validator entered RUNNING state,
process has stayed up for > than 10 seconds (startsecs)

To start the accelerator at boot time, see here.

Inside every service directory there is a directory named data, where the logs for each service are stored. If there are errors, please check the logs for more details. For example, if you see the error celery (exit status 1; not expected), you can check more details in the log file .../celery/data/celery.log.

Logging

Note that it's expected that the celery and the sc_logs_agent don't start with the current configuration. The reason is that we are still missing the info for deployment_site_long_name and deployment_site_long_secret that are needed to let Celery authenticate with Celery broker.


3. Create deployment sites and domains

A DeploymentSite represents a site where sc_pack has been deployed, and it is the root of the tree to configure everything else. In the commands below, make sure to replace <your_authentication_token> with your authentication token received in the first step of the getting started tutorial.

Create a deployment site

To create a DeploymentSite use the below command:

$ curl -X POST https://accelerator.shimmercat.com/presentation-api/v1/deployment-site/ -H 'Authorization: Token <your_authentication_token>' -H 'Content-type: application/json' -d '{"description": "Deployment site on server test1"}'

You will get the response data:

{
  "id":<your_deployment_id>,
  "customer": {
      "id": <your_customer_id>,
      "name":"ShimmerCat Accelerator Test"
  },
  "description":"Deployment site on server test1",
  "name16":"<your_deployment_site_long_name>",
  "secret16":"<your_deployment_site_long_secret>"
}

Now edit the file sc_pack.conf.yaml and add the responses for <your_deployment_site_long_name> and <your_deployment_site_long_secret>. To update the configuration files on the file system with the name16, and secret16 execute:

$ sc_pack create

Whenever you change sc_pack.conf.yaml you should run sc_pack create - otherwise the changes won't be synchronized to all the configuration files used by the services.

Create a domain for the deployment site

To create a domain for the deployment site, use the command below:

$ curl -X POST https://accelerator.shimmercat.com/presentation-api/v1/domain/ -H 'Authorization: Token <your_authentication_token>' -H 'Content-type: application/json' -d '{"name": "example-accelerator.shimmercat.com", "deployment_site_id": <your_deployment_id>}'

Note that you need to change <your_deployment_id> to make sure that the domain you are creating will be linked to the deployment site with this id. You will get the response:

{"id":<your_domain_id>,"name":"example-accelerator.shimmercat.com","sub_domains":[]}

If you would like to add more deployment sites to your domain, see more information on Adding a new deployment for an existing domain. You are now ready to serve a website with ShimmerCat.


Serve the example website

In your install_dir (as an example we use /home/shimmercat/test/shimmercat-accelerator) there is a server config file named devlove.yaml, which was automatically created in the previous sections. If you want to try our static website example, edit the devlove.yaml file according to:

---
shimmercat-devlove:
      domains:
          elec example-accelerator.shimmercat.com:
              root-dir: www

Then download our static website example and decompress it in the directory /home/shimmercat/test/shimmercat-accelerator/www. Be sure to check that you have the file /home/shimmercat/test/shimmercat-accelerator/www/index.html.

Now edit your /etc/hosts to have the line 127.0.0.1 example-accelerator.shimmercat.com.

Last thing is to run the command:

$ sc_pack supervisord

Then open a browser and visit: https://example-accelerator.shimmercat.com:4043/.

Summary

Congratulations, you have now completed the tutorial and ShimmerCat should be running for the domain example-accelerator.shimmercat.com!

To learn more about sc_pack, check out the info here. For the next recommended tutorials, see:

Thanks a lot for your time, and keep reading!