3. Setup with sc_pack

3.1. Preliminaries

To manage the edge servers running ShimmerCat Accelerator, we install a “pack” of programs called sc_pack. The installation consists of three steps: first you need to install sc_pack itself, second you need to create deployment sites and domains to serve, and third you need to ask sc_pack to prepare the hierarchy of directories.

sc_pack comes with a self-sufficient installer that should hopefully work in any x64 Linux.

All the sc_pack services (shimmercat, sc_logsagent, redis, bots-blocking, usher(s), among others) run in a rkt pod. The advantage of this approach is that we can set individual resources quota to each service.

• One host (VPS or physical) contains several deployments.

• Each deployment runs in its own rkt pod.

• Each deployment/rkt pod is managed by a single host service, e.g. Systemd service.

The rkt pod uses its own filesystem, except for the directory /srv/deployment that is mapped to the outer host’s deployment directory. This is exactly how Docker+Kubernetes does it, but without having to run Docker nor Kubernetes. Since the pod comes with its own filesystem, Python is installed directly on the image and not in a virtual environment. The rkt pod uses the same network namespace than the host. This means that we still need to assign ports in such a way that they don’t collide with other ports in the host or other deployments.

3.2. 1. Installation

The installer can be found at https://storage.googleapis.com/downloads-shimmercat-com-561059/sc_pack/rkt/sc_pack-linux-amd64-<sc_pack_version>.bootstrap, versions will be something like: 2020.1 and we will increase the integer after the dot. For instance the first version can be found at: https://storage.googleapis.com/downloads-shimmercat-com-561059/sc_pack/rkt/sc_pack-linux-amd64-2020.1.bootstrap. You can download the installer and run it in a VPS to get sc_pack installed.

3.2.1. Download sc_pack

wget https://storage.googleapis.com/downloads-shimmercat-com-561059/sc_pack/rkt/sc_pack-linux-amd64-2020.1.bootstrap

3.2.2. How to run the installer?

The .bootstrap files are like shell scripts, so just give execution permissions to the downloaded file:

$ chmod +x sc_pack-linux-amd64-2020.1.bootstrap

and, run it:

$ ./sc_pack-linux-amd64-2020.1.bootstrap

Now sc_pack should be installed. To verify that it is properly installed run:

$ sc_pack -h

Note that after installation, the sc_pack command is available anywhere in the host but it still needs to be executed from a valid deployment directory most of the times.

---

3.3. 2. Create deployment sites and domains

A DeploymentSite represents a site where sc_pack has been deployed, and it is the root of the tree to configure everything else. In the commands below, make sure to replace <your_authentication_token> with your authentication token received in the first step of the getting started tutorial.

Create a deployment site

To create a DeploymentSite use the below command:

$ curl -X POST https://accelerator.shimmercat.com/presentation-api/v1/deployment-site/ -H 'Authorization: Token <your_authentication_token>' -H 'Content-type: application/json' -d '{"description": "Deployment site on server test1"}'

You will get the response data:

{
  "id":<your_deployment_id>,
  "customer": {
      "id": <your_customer_id>,
      "name":"ShimmerCat Accelerator Test"
  },
  "description":"Deployment site on server test1",
  "name16":"<your_deployment_site_long_name>",
  "secret16":"<your_deployment_site_long_secret>"
}

The returned values must be kept to be used later.

Create a domain for the deployment site

To create a domain for the deployment site, use the command below:

$ curl -X POST https://accelerator.shimmercat.com/presentation-api/v1/domain/ -H 'Authorization: Token <your_authentication_token>' -H 'Content-type: application/json' -d '{"name": "example-accelerator.shimmercat.com", "deployment_site_id": <your_deployment_id>}'

Note that you need to change <your_deployment_id> to make sure that the domain you are creating will be linked to the deployment site with this id. You will get the response:

{"id":<your_domain_id>,"name":"example-accelerator.shimmercat.com","sub_domains":[]}

If you would like to add more deployment sites to your domain, see more information on Adding a new deployment for an existing domain. You are now ready to serve a website with ShimmerCat.

---

3.4. 3. Directory hierarchy and configuration

The programs need a directory as the top of the hierarchy, and some information for configuration. This information contains details about your setup, and secrets needed for the programs to send and receive data from our cloud service.

3.4.1. The configuration file

All configuration is structured in a YAML file, called sc_pack.conf.yaml, which has nesting levels. Below is an example of the configuration file, copy the content and create a file with extension .yaml, called sc_pack.conf.yaml. You can find more details about all the fields here.

• We recommend creating a user especially to work with sc_pack, let’s call it shimmercat. So run $ sudo adduser shimmercat . The user that runs the sc_pack should be on the sudo group, to do that just run: $ sudo usermod -aG sudo shimmercat

Remember to edit the fields according to your details:

amqp_server_url: amqp.staging.c.shimmercat.com
amqp_vhost: skyloft
api_access_token: <your_authentication_token> # Received in the first step of the getting started tutorial
customer_id: <your_customer_id> # Received in the first step of the getting started tutorial
backup_config_secret: <your_backup_config_secret> # Received in the first step of the getting started tutorial
deployment_site_long_name: <your_name16> # Update this value with the value of your deployment site's `name16` you will create on the next step
deployment_site_long_secret: <your_secret16> # Update this value with the value of your deployment site's `secret16` you will create on the next step
enable_bots_blocking: false # true or false.
enable_images_optimization: false # true or false.
enable_sc_logs_agent: false # true or false.
usher3_monitoring_listen_on: 4080
usher3_disable_monitoring: false # true or false.
usher3_toilmore_api_prefix: "https://accelerator.shimmercat.com" # or for demo "https://canary.shimmercat.com"
enable_usher3: false # true or false.
usher3_max_overhead: 1.0
usher3_toilmore_subservice: light # light or lux
usher3_toilmore_api_version: 2020.1 # if light 2020.1, if lux 2020.4
usher3_lux_adjustments:
  master: {}
  shifter: {}
  encoder:
    max_overhead: 1.0
google_recaptcha_site_key:  <your_google_recaptcha_site_key> # Only needed if enable_bots_blocking is true. If needed you should get it from https://www.google.com/recaptcha/admin/.
google_recaptcha_site_secret: <your_google_recaptcha_site_secret> # Only needed if enable_bots_blocking is True. If needed you should get it from https://www.google.com/recaptcha/admin/.
humanity_validator_host: localhost
humanity_validator_port: 8060 # The port where the service that send the Google
images_optimization_with_aws: false
improve_images_quality: false
install_dir: /path/to/sc_pack_files # Update this with the path you want the sc_pack files to be at
logs_host_name: localhost
transit_encryption_key: <your_transit_encryption_key> # User-defined variable. This will be the encryption key to encrypt and decrypt data sent from the edge servers to the Accelerator Platform and vice versa. Among other things, it is used to synchronize the certificates and their private keys. Make sure to set the same key to all the deployments you would like to synchronize. You choose the key yourself.
user: shimmercat
user_group: shimmercat
shimmercat: {bots_views_dir: views-dir, devlove_file_path: devlove.yaml, http2https_port: '8010',
  listen_port: '4010', root_dir: www, scratch_dir_name: shimmercat-scratch-folder}

3.4.2. Directory hierarchy

Once the configuration file is created, it’s time to setup the directory hierarchy, run:

$ sc_pack create -f /path/to/sc_pack.conf.yaml or if you are already on the path where the sc_pack.conf.yaml you can miss the /path/to/sc_pack.conf.yaml and just run the $ sc_pack create.

You will see now that the directories and files were created for the new deployment, take a look inside the install_dir. Each deployment has a hidden folder .sc_pack-rkt with some files. The files are:

• deployment-images.json: the images that were used on this deployment. That’s probably the most important file for you because in case you need a new version of sc_pack, celery, sc_logs-agent, or shimmercat this is the one you need to change.

• detailed-versions.dhall: the detailed versions and sha of the images that were installed on this deployment.

• pod-manifest.json: the manifest from where all the services were created for the deployment.

• sc-<install_dir>.service: a systemd service template ready to be installed at /etc/systemd/system.

All those files need a sudo user to be edited.

$ sudo chown -R shimmercat:shimmercat <install_dir>

Now change to user shimmercat

$ su shimmercat

$ cd <install_dir>

The sc_pack create command also creates an example devlove.yaml file, which is ShimmerCat’s configuration file with domains and such. You will edit this file in the coming steps, and for more info regarding the file you can check here.

You can double check that the directories are properly created:

$ ls
$ celery redis shimmercat   devlove.yaml sc_logs_agent shimmercat-scratch-folder views-dir  # And some others

3.4.3. Run the services

Do a symlink (as root) of <install_dir>/.sc_pack-rkt/sc-<install_dir>.service to /etc/systemd/system

$ sudo ln -s <install_dir>.sc_pack-rkt/sc-<install_dir>.service /etc/systemd/system

Reload the systemctl daemon:

$ sudo systemctl daemon-reload

Start the services:

$ sudo systemctl start sc-<install_dir>.service

Check the services statuses:

$ sudo systemctl status sc-<install_dir>.service

Inside every service directory there is a directory named data, where the logs for each service are stored. If there are errors, please check the logs for more details. For example, if you see the error celery (exit status 1; not expected), you can check more details in the log file <install_dir>/celery/data/logs/current.log.

---

3.5. Serve the example website

In your install_dir (as an example we use /home/shimmercat/test/shimmercat-accelerator) there is a server config file named devlove.yaml, which was automatically created in the previous sections. If you want to try our static website example, edit the devlove.yaml file according to:

---
shimmercat-devlove:
      domains:
          elec example-accelerator.shimmercat.com:
              root-dir: www

Then download our static website example and decompress it in the directory /home/shimmercat/test/shimmercat-accelerator/www. Be sure to check that you have the file /home/shimmercat/test/shimmercat-accelerator/www/index.html.

Now edit your /etc/hosts to have the line 127.0.0.1 example-accelerator.shimmercat.com.

Last thing is to run the command:

Restart the services:

$ sudo systemctl start sc-<install_dir>.service

Then open a browser and visit: https://example-accelerator.shimmercat.com:4010/.

• Whenever you change devlove.yaml you should run $ sc_pack ctl reload shimmercat.

3.6. Summary

Congratulations, you have now completed the tutorial and ShimmerCat should be running for the domain example-accelerator.shimmercat.com!

To learn more about sc_pack, check out the info here. For the next recommended tutorials, see:

• Enable and configure HTTP/2 Push.

• Enable and configure bot blocking.

Thanks a lot for your time, and keep reading!