26. Views and devlove¶
A single domain can be served by any number of backends, which in ShimmerCat lingo we call “consultants”. To organize how HTTP requests are forwarded to these, we use the term “view”. In ShimmerCat, views function as access paths to dynamic content generated by an HTTP or FastCGI application.
As a first approximation, one can think of a view as what other web servers and caches would call a “cache hole”.
Views serve as configuration last-stops for HTTP requests traveling to the application backend, and as configuration first entry points for HTTP responses coming back. If a request is traveling forward to a backend through ShimmerCat, there is one and only one view processing that request.
Physically speaking, views are files with special fixed names,
The directory where they live influences how they are selected, such that
with consideration to which URL paths they handle, views form a single-parent inheritance
However settings in a view are always standalone with regard to other views; they
do not mingle in any manner.
26.1. General structure of a view¶
View files have one of two names: either
Because they can only have those two names, we can only fit at most two in a single directory.
All views are inside a special directory called the
which we can be set independently for each domain.
This way, the relative path of a view file, taken from the
gives us a unique view path.
The first kind of view, the one at
index.html, is used when a requested URL path matches exactly the
containing relative directory.
For example, if the user requests
/supershoes/brown/, ShimmerCat will the file at
if it exists.
The second file,
__index.html, is used when a request path matches a directory inside the directory of the view,
if no other more specific view would match.
That is, if one requests
/supershoes/brown/high-heels/, ShimmerCat would use
if none of the following files exist:
26.1.1. What do view files contain?¶
If you don’t have a backend application, and just static HTML, then you could use the views to write that HTML. If you have a backend application, then the linking aspect of the view is needed.
HTML code to send to the browser under certain settings, and
Instructions on how to forward and recover a request from the application backend
Sometimes you need only one of those two aspects, sometimes you need both.
How they are mixed is controlled by something called
HTML code is the default content in the view file. For the other contents, the instructions inside a view, we use a special HTML comment syntax:
<!-- shimmercat: … -->
where the three dots represent entries in a YAML dictionary. For example, the content-disposition is written like this:
<!-- shimmercat: content-disposition: …. -->
26.2. Instructions in a view¶
26.2.1. Changing again URL paths in views¶
Views are located by URL path, but the URL path of a request can be changed just before looking up a view.
For that, we use a special section in the
devlove.yaml file, called
change-url, containing rules to
18.104.22.168. Example 1¶
For example, we can use the
change-url section at the
devlove.yaml file to make
it so that when a visitor comes to the URL path
/super-shoes/bright-pink/, we send them to
a view at
<views-dir>/shoes/__index.html and form there a request is made to the
/shoes-product.php in the backend:
# Section at devlove.yaml … change-url: - /super-shoes/bright-pink/ -> /shoes/color/pink/ … # Special HTML comment (surrounded by '<!--' and '-->') # in file <views-dir>/shoes/__index.html: shimmercat: content-disposition: replace change-url: - '/shoes/color/<color>/ -> /shoes-product.php?product_type=shoes&color=<color>'
22.214.171.124. Example 2¶
Suppose that we are using a PHP application, and that a request will be handled by a file called
/admin/login.php … In that case, PHP expects a request to that precise URL path, but views always expect a request to something that looks like a directory.
So, we re-write the URL path just before passing it down to the application engine.
Say that we have placed the view file under
Then we just need to write a
change-url section like this one:
<!-- shimmercat: content-disposition: … change-url: - /admin/login/ -> /admin/login.php -->
The view above will work if your users expect to login by using the URL
/admin/login/, but what happens if we invested a lot of time training our users and our search engines to use
/admin/login.php instead? Then we use the
shimmercat-devlove: domains: elec www.supershoes.com: ... change-url: - /admin/login.php -> /admin/login/
26.2.2. Deciding for a web backend¶
As stated before, a domain’s configuration admits multiple backends.
A view can set which backend it will send a request to via the
# In the `devlove.yaml`: shimmercat-devlove: domains: ... elec admin.newacme.com: consultants: backstore-app: connect-to: ... # In a particular view <!-- shimmercat: content-disposition: replace use-backend: backstore-app -->
When not specified via the
use-backend directive, a view selects a special consultant
26.2.3. Processing the request/response body¶
A view decides, via the
content-disposition directive, how the body (and in some case even the method)
of an HTTP request and response are passed to the backend.
More details at the content disposition page.
26.2.4. Inserting the metrics snippet¶
This is functionality for inserting a performance-reporting snippet, mostly useful with our data-collection pipelines. See here.
26.2.5. Stacked settings¶
Stacked settings are settings that can be set on a view, an entire backend, or an entire domain in the
When they are set in multiple places, ShimmerCat combines them — using a semigroup — and uses the
combined result instead.
The way the settings at different levels are combined is specific for each setting.
This section details some of those settings.
126.96.36.199. Modifying request and response headers¶
change-headers-in allows modifying HTTP request headers just before passing them to the backend,
and the setting
change-headers-out allows modifying HTTP response headers just after they come from the backend
and before passing them to the browser.
In what follows, we will describe
change-headers-in, but bear in mind that the same applies to
Here is an example of a view that uses
<!-- shimmercat: content-disposition: replace change-headers-in: | headers['X-Forward-Proto'] = 'HTTPS' return headers -->
headers: a Lua table with the input headers
The code should return a table with the modified headers. If it doesn’t, ShimmerCat QS won’t change the headers before passing them to the application.
Being a stacked setting,
change-headers-in can also be defined in the backend stanza at the
shimmercat-devlove: domains: elec www.example.com: consultants: our-iis-server: connect-to: ... change-headers-in: headers['x-real-ip']=headers['x-forwarded-for']
ShimmerCat regards the value of
change-headers-in as a function from headers to headers, and combines
them using the endomorphism monoid.
In practical terms, when
change-headers-in is set both at a view and at the consultant used by
that view, the headers are first passed to the function at the consultant, and its result is then passed
to the function at the view.
The composition order is reversed for
change-headers-out, so that when the setting is at two places
the view processes the headers first and then the consultant goes second.
188.8.131.52. Accessing GeoIP functionality via IPRecords¶
Shimmercat QS can read
GeoIP2 files and make their information accessible to the
A stacked boolean setting is used for this:
enable-ip-records, which can be used both at a view
level and in the top object of a consultant.
When used in both places, the value set at the view takes precedence.
When set, a table
ip_records is added to the Lua context of both
Accessing an attribute of this table triggers a lookup of a corresponding entry for the current request’s IP
address, which is either the IP address of the connecting session or, if configured, the
address provided via proxy-v2.
Databases are copied or linked inside the scratch folder, at
Take note of the filename with which you copy or link your database inside that folder, as it needs to
be used to access that database’s entry from inside the file.
For example, if the GeoIP2 database file is a symbolic link with the name
ip2city”, then you can use
something like this to access the corresponding IP record:
... headers['Geo-IP-Country'] = ip_records.ip2city.en.country_code return headers
ip_records.ip2city will hold whatever object in the database matches the current
IP address, or
nil if none matches.
Because of how GeoIP2 is defined, the contents of the
record vary from database to database, and you must find out which one applies in your particular case.
We recommend you to use the tool
sc-iprecord-lookup in the ShimmerCat QS distro, perhaps in combination with jq,
to do some sample queries in your database and find out how objects inside it look like.
It’s possible to install and use multiple GeoIP2 databases simultaneously, and there is
an auxiliary program in the distribution to enable creating your own GeoIP2 files:
Run it passing via standard input a sequence of lines, each containing a one-line JSON document.
Each document should be a map with two entries:
The subnet is a string denoting an IPv4 subnet (we havent gotten to implement IPv6 support yet) as in
record is any valid JSON value.
The output of
sc-iprecords-meld will be a valid GeoIP2 file.
184.108.40.206. Forwarding backend responses for error cases¶
forward-error-pages instructs ShimmerCat QS to send error pages from the backend
to the browser.
This setting combines exactly as
enable-ip-records, and its details are described in a
220.127.116.11. Blowing HTML up¶
This setting enabling writting a log message with HTML received from a backend, from URLs which have been whitelisted for this. This is a non-destructive side effect, in particular it doesn’t imply changes to the transmitted HTML. That is, here “blow up” is used in its literal sense, not the idiomatic one.
This functionality is necessary to report data for analysis with acceleration rules that rely on understandig HTML structure.
The setting, a boolean value for a
blowup field, can be used in the definition of the application
port and the definition at the view.
A definition at the view takes precedence over any definition at the application port.
In addition to these stacked settings, a particular URL path should be listed for reporting.
The list must be in a sqlite3 database
Said datbase should include the URLs whose contents we are interested in in a table defined with the
CREATE TABLE urls_to_report_t( url_path VARCHAR PRIMARY KEY, weight INT NOT NULL )
In the table, the
url_path should be the URL path, without query string — this is a temporary restriction —
and weight should be a number that tells ShimmerCat QS how important this URL path is for reporting.
Matching is done against the original URL path coming from the browser, before any rewrites in ShimmerCat.
If a URL path for a request is not in that table, its HTML is not reported.
When a Shimmercat worker starts, it copies the database file to a temporary file, opens it in read-only mode,
and unlinks the temporary file.
This way, it is safe to overwrite the original file at
at any moment without crashing SQLite.
However, for ShimmerCat to use the updated file, a server reload is needed — e.g. via
kill -SIGHUP $(cat scratch-folder/master.pid).
The log messages report the HTML as a XZ stream which has been text-encoded with Ascii85. The log message codes assigned for this are M60005 to M60007. An additional log message, M44005, will be issued when ShimmerCat finds and activates a SQLite3 database for a particular domain.