3. Install sc_pack

To manage the edge servers running ShimmerCat Accelerator, we install a “pack” of programs called sc_pack. The installation consists of two steps: first you need to install sc_pack itself, and then you need to ask sc_pack to prepare the hierarchy of directories.

3.1. 1. Installation

First, you will need to install python3, python3-dev, virtualenv and git on the server. You can install them by executing:

$ sudo apt-get install python3 python3-dev virtualenv git

We highly recommend that you use a Python 3 virtualenv. This virtualenv will contain an isolated copy of Python 3, sc_pack, supervisord and all the libraries needed for the tools and for the Celery worker. Since it’s so tied to sc_pack, we recommend that you create the virtualenv as a subdirectory in the hierarchy we mentioned before, the so called <venv> directory. This can be done by executing:

$ virtualenv /path/to/install_dir/venv -p python3

Don’t forget to replace /path/to/install_dir/ with the actual path you use. Then you can activate it:

$ source /path/to/install_dir/venv/bin/activate

You will need the sc_pack distributable to install. You can either find the download link by visiting the API page, or by directly using the below download command (requires jq):

curl -O $(curl -s | jq -r .download_url)

After you have downloaded sc_pack to your server, with the venv activated, do:

$ pip install /path/to/<sc_pack-version>.whl

Now sc_pack should be installed. To verify that it is properly installed run:

$ sc_pack -h

3.2. 2. Prepare the configuration file and the directory hierarchy

The programs need a directory as the top of the hierarchy, and some information for configuration. This information contains details about your setup, and secrets needed for the programs to send and receive data from our cloud service.

3.2.1. The configuration file

All configuration is structured in a YAML file, which has nesting levels. In the rest of this documentation, we sometimes refer to a nested field by using a /, for example shimmercat/root-dir refers to the field root-dir under the object shimmercat.

There is an example file inside the sc_pack redistributable, that can be obtained by executing:

$ sc_pack extract example.sc_pack.conf.yaml

Copy the content that appears in the terminal and create a file with extension .yaml, eg. my_sc_pack_conf.yaml.

In the configuration file, you need to edit the following:

  • install_dir: the top directory for the hierarchy of files.

  • user: the user that will be used to run all the programs in sc_pack. This can be root, although is not needed nor recommended. Instead, we recommend you create a special user for this. Said user needs all access to the hierarchy of files, but to nothing outside of it. If you want, you can use the command adduser --system --disabled-login --no-create-home --group sc_pack to create a user called sc_pack and an associated group.

  • shimmercat/port: the port where ShimmerCat will listen for web requests.

  • shimmercat/root-dir: this setting is used to put an initial folder in the devlove.yaml file for where to serve files from. By default it’s inside the hierarchy and called www, but you can also write an absolute directory here. Just be sure the user you provided before has read access to this directory.

  • shimmercat/bots-views-dir: this directory contains ShimmerCat views for the blocking bots feature. It is only required when enable_bots_blocking is True, and there is not a views-dir defined in devlove.yaml. If there is a views-dir in devlove.yaml, it is used by default.

  • logs_host_name: a hostname to identify your logs. Any hostname will do, so you can either write the output of the hostname program, or some other label that you see fit. This hostname will be used later in the dashboard with our service to identify your deployment.

  • enable_bots_blocking: <True/False> will enable the bots blocking mechanism.

  • humanity_validator_host: host where the service to enable the humanity validator runs. We need it because when an IP address has been automatically detected as a prospector bot, we ensure that it is actually a bot with a mechanism using Google reCAPTCHA. That way a user can still access the website after defeating the CAPTCHA challenge, if an IP address has been wrongly classified as bot. Note that this host should be set in the devlove.yaml consultants too.

  • humanity_validator_port: port that will use the service to enable the humanity validator. Note that this host should also be set in the devlove.yaml.

  • google_recaptcha_site_key: The Google reCAPTCHA mechanism is used to validate human visitors vs bots. You can create the key through Google reCAPTCHA admin.

  • google_recaptcha_site_secret: As mentioned before we use the Google reCAPTCHA mechanism to validate human visitors vs bots. You can create the secret through Google reCAPTCHA admin.

  • transit_encryption_key: a free-form password that sc_pack uses to encrypt files to synchronized across deployments of the same deployment group. Among other things, we use it to synchronize the certificates and their private keys. Make sure to set the same transit_encryption_key to all the deployments you would like to synchronize. Also, keep this key safe, as access to it could allow a third party to gain access to your certificates and their private keys. The optimization cloud service does not store this key.

  • deployment_site_long_name: the name16 provided by ShimmerCat cloud service

  • deployment_site_long_secret: the secret16 provided by ShimmerCat cloud service

  • deployment_group: an arbitrary name for a deployment group. A deployment group is a set of deployments that share the same set of domains, and thus are expected to be in sync regarding the devlove.yaml, the certificates and the views. We suggest you strive to keep this name globally unique and hard to guess, as in .

3.2.2. Directory hierarchy

Once the configuration file is created, it’s time to setup the directory hierarchy. If you have already a file called sc_pack.conf.yaml in the directory from where you are running that command, run:

    $ sudo sc_pack create

Otherwise you should run:

    $ sudo sc_pack create -f configuration/file/path/sc_pack.conf.yaml

You can get a copy of the config template content that sc_pack has by executing:

    $ sc_pack extract example.sc_pack.conf.yaml

You can then copy and paste that content on a file called sc_pack.conf.yaml and then run the create command without arguments. Subject to the same rule, it’s possible to omit the -f in many sc_pack sub-commands. E.g, if the current dir is /srv/ and the file /srv/, and you run sc_pack create, the configuration file taken is /srv/

We recommend running the command above with root privileges so that shimmercat can be authorized to bind to port 443, without running ShimmerCat itself as root.

The sc_pack ... create command also creates an example devlove.yaml, which is ShimmerCat’s configuration file with domains and such. The format of that file is documented elsewhere, but this is a good moment to either edit it by hand or to request our cloud service to send you a version.

3.3. 3. Deploy and setup sc_pack

You are now ready to setup and deploy your sc_pack configuration, see using sc_pack