To manage the edge servers running ShimmerCat Accelerator, we install a "pack" of programs called
sc_pack. The installation consists of two steps: first you need to install
sc_pack itself, and then you need to ask
sc_pack to prepare the hierarchy of directories.
You will need to have installed: python3, python3-dev, virtualenv and git on the server. You can install them by executing:
$ sudo apt-get install python3 python3-dev virtualenv git
We highly recommend that you use a Python 3 virtualenv for all of this. This virtualenv will contain an isolated copy of Python 3,
sc_pack, supervisord and the libraries we need with these tools and with the celery worker. Because it's so tied to
sc_pack, we recommend you create the virtualenv as a subdirectory of the hierarchy we mentioned before, the so called
<venv> directory. You can have all of this by doing:
$ virtualenv /path/to/install_dir/venv -p python3
(don't forget to replace
/path/to/install_dir/ with the actual thing)
and then to activate it:
$ source /path/to/install_dir/venv/bin/activate
You will need the
sc_pack distributable to install, and you a version here: https://accelerator.shimmercat.com/presentation-api/v1/download/sc_pack/. Download it to your server, and with the venv activated do:
$ pip install /path/to/<sc_pack-version>.whl
Now it should be installed. To verify that it is properly installed run:
$ sc_pack -h
2. Prepare the directory hierarchy
The programs need a directory as the top of the hierarchy, and some configuration information. Of that information, some of them are secrets needed for the programs to send and receive data from our cloud service, and other are configuration details that you should edit to your needs.
All of it is structured as a YAML file, which has nesting levels. In the rest of this documentation, we sometimes refer to a nested field by using a
/, for example
shimmercat/root-dir refers to the field
root-dir under the object
There is an example file inside the sc_pack redistributable, that can be obtained by executing:
$ sc_pack extract example.sc_pack.conf.yaml
Copy the content that appears in the terminal and create a file with extension yaml, eg. my_sc_pack_conf.yaml.
In the configuration file, you need to edit the following:
install_dir: the top directory for the hierarchy of files.
user: the user that will be used to run all the programs in
sc_pack. This can be
root, although is not needed nor recommended. Instead, we recommend you create a special user for this. Said user needs all access to the hierarchy of files, but to nothing outside of it. If you want, you can use the command
adduser --system --disabled-login --no-create-home --group sc_packto create a user called
sc_packand an associated group.
shimmercat/port: the port where shimmercat will listen for web requests.
shimmercat/root-dir: this setting is used to put an initial folder in the
devlove.yamlfile for where to serve files from. By default is inside the hierarchy and called
www, but you can also write an absolute directory here. Just be sure the user you provided before has read access to this directory.
shimmercat/bots-views-dir: this directory contains ShimmerCat views for the blocking bots feature. It is only required when enable_bots_blocking is True, and there is not a views-dir defined in devlove.yaml. If there is a views-dir in devlove.yaml, it is used by default.
logs_host_name: a hostname to identify your logs. Any hostname will do, so you can either write the output of the
hostnameprogram, or some other label that you see fit. This hostname will be used later in the dashboard with our service to identify your deployment.
will enable the bots blocking mechanism.
humanity_validator_host: host where the service to enable the humanity validator runs. We need it because when an IP address has been automatically detected as a prospector bot, we ensure that it is actually a bot with a mechanism using Google reCAPTCHA. That way a user can still access the website after defeating the CAPTCHA challenge, if an IP address has been wrongly classified as bot. Note that this host should be set in the devlove.yaml consultants too.
humanity_validator_port: port that will use the service to enable the humanity validator. Note that this host should be set in the devlove.yaml consultants too.
google_recaptcha_site_key: As mentioned before we use the Google reCAPTCHA mechanism to validate human visitors vs bots. You can create the key through Google reCAPTCHA admin.
google_recaptcha_site_secret: As mentioned before we use the Google reCAPTCHA mechanism to validate human visitors vs bots. You can create the secret through Google reCAPTCHA admin.
transit_encryption_key: a free-form password that sc_pack uses to encrypt files to synchronized across deployments of the same deployment group. Among other things, we use it to synchronize the certificates and their private keys. Make sure to set the same
transit_encryption_keyto all the deployments you would like to synchronize. Also, keep this key safe, as access to it could allow a third party to gain access to your certificates and their private keys. The optimization cloud service does not store this key.
deployment_site_long_name: the name16 provided by ShimmerCat cloud service
deployment_site_long_secret: the secret16 provided by ShimmerCat cloud service
deployment_group: an arbitrary name for a deployment group. A deployment group is a set of deployments that share the same set of domains, and thus are expected to be in sync regarding the
devlove.yaml, the certificates and the views. We suggest you strive to keep this name globally unique and hard to guess, as in
dht_bootstrap_node: Sc_pack uses and runs a distributed key-value store to quickly synchronize configuration across a deployment group, and the value of
dht_bootstrap_nodepoints to a bootstrap node. The default value should work find; we run
dht.shimmercat.comjust for this case.
dht_bind_port: a port for sc_pack to bind and run its node of the distributed hash table.
Once the configuration file is ready, it's time to setup the directory hierarchy:
$ sudo sc_pack create -f configuration/file/path/sc_pack.conf.yaml
$ sudo sc_pack create
...if you have already a file called
sc_pack.conf.yaml on the directory from where you are running that command.
You can get a copy of the config template content the sc_pack has by typing:
$ sc_pack extract example.sc_pack.conf.yaml
You can then copy and paste that content on a file called
sc_pack.conf.yaml and then run the
create command without arguments. Subject to the same rule, it's possible to omit the
-f in many
E.g, if the current dir is
/srv/www.example.com and the file
/srv/www.example.com/sc_pack.conf.yaml and you run
sc_pack create, the configuration file taken is
We recommend running the command above with root privileges so that
shimmercat can be authorized to bind to port 443, without running ShimmerCat itself as root.
sc_pack ... create command also creates an example
devlove.yaml, which is ShimmerCat's configuration
file with domains and such. The format of that file is documented elsewhere, but this is a good moment
to either edit it by hand or to request our cloud service to send you a version.
3. Deploy and setup
To setup and deploy your
sc_pack configuration, see Setup sc_pack