26. Views and devlove¶

A single domain can be served by any number of backends, which in ShimmerCat lingo we call “consultants”. To organize how HTTP requests are forwarded to these, we use the term “view”. In ShimmerCat, views function as access paths to dynamic content generated by an HTTP or FastCGI application.

As a first approximation, one can think of a view as what other web servers and caches would call a “cache hole”.

Views serve as configuration last-stops for HTTP requests traveling to the application backend, and as configuration first entry points for HTTP responses coming back. If a request is traveling forward to a backend through ShimmerCat, there is one and only one view processing that request.

Physically speaking, views are files with special fixed names, index.html and __index.html. The directory where they live influences how they are selected, such that with consideration to which URL paths they handle, views form a single-parent inheritance hierarchy. However settings in a view are always standalone with regard to other views; they do not mingle in any manner.

26.1. General structure of a view¶

View files have one of two names: either index.html, or __index.html. Because they can only have those two names, we can only fit at most two in a single directory. All views are inside a special directory called the views-dir, which we can be set independently for each domain.

This way, the relative path of a view file, taken from the views-dir, gives us a unique view path.

The first kind of view, the one at index.html, is used when a requested URL path matches exactly the containing relative directory. For example, if the user requests /supershoes/brown/, ShimmerCat will the file at <views-dir>/supershoes/brown/index.html if it exists. The second file, __index.html, is used when a request path matches a directory inside the directory of the view, if no other more specific view would match. That is, if one requests /supershoes/brown/high-heels/, ShimmerCat would use /supershoes/__index.html if none of the following files exist:

/supershoes/brown/high-heels/index.html,
/supershoes/brown/__index.html

26.1.1. What do view files contain?¶

If you don’t have a backend application, and just static HTML, then you could use the views to write that HTML. If you have a backend application, then the linking aspect of the view is needed.

Views contain:

HTML code to send to the browser under certain settings, and
Instructions on how to forward and recover a request from the application backend

Sometimes you need only one of those two aspects, sometimes you need both. How they are mixed is controlled by something called content-disposition.

HTML code is the default content in the view file. For the other contents, the instructions inside a view, we use a special HTML comment syntax:

<!--
shimmercat:
…
-->

where the three dots represent entries in a YAML dictionary. For example, the content-disposition is written like this:

<!--
shimmercat:
content-disposition: ….
-->

26.2. Instructions in a view¶

26.2.1. Changing again URL paths in views¶

Views are located by URL path, but the URL path of a request can be changed just before looking up a view. For that, we use a special section in the devlove.yaml file, called change-url, containing rules to re-write URLs.

26.2.1.1. Example 1¶

For example, we can use the change-url section at the devlove.yaml file to make it so that when a visitor comes to the URL path /super-shoes/bright-pink/, we send them to a view at <views-dir>/shoes/__index.html and form there a request is made to the URL /shoes-product.php in the backend:

# Section at devlove.yaml
…
change-url:

  - /super-shoes/bright-pink/ -> /shoes/color/pink/
…

# Special HTML comment (surrounded by '<!--' and '-->') 
# in file <views-dir>/shoes/__index.html: 
shimmercat:
   content-disposition: replace
   change-url:
     - '/shoes/color/<color>/ -> 
        /shoes-product.php?product_type=shoes&color=<color>'

26.2.1.2. Example 2¶

Suppose that we are using a PHP application, and that a request will be handled by a file called /admin/login.php … In that case, PHP expects a request to that precise URL path, but views always expect a request to something that looks like a directory. So, we re-write the URL path just before passing it down to the application engine.

Say that we have placed the view file under /admin/login/, as /admin/login/index.html. Then we just need to write a change-url section like this one:

<!--
shimmercat:
content-disposition: …
change-url:
  - /admin/login/ -> /admin/login.php
-->

The view above will work if your users expect to login by using the URL /admin/login/, but what happens if we invested a lot of time training our users and our search engines to use /admin/login.php instead? Then we use the change-url section at the devlove.yaml file!

shimmercat-devlove:
  domains:
    elec www.supershoes.com:
       ...  
       change-url:  
          - /admin/login.php -> /admin/login/

26.2.2. Deciding for a web backend¶

As stated before, a domain’s configuration admits multiple backends. A view can set which backend it will send a request to via the use-consultant directive:

# In the `devlove.yaml`:
shimmercat-devlove:
  domains:
    ... 
    elec admin.newacme.com:
       consultants:
         backstore-app:
            connect-to: ...

# In a particular view
<!-- 
shimmercat:
   content-disposition: replace
   use-backend: backstore-app
-->

When not specified via the use-backend directive, a view selects a special consultant default.

26.2.3. Processing the request/response body¶

A view decides, via the content-disposition directive, how the body (and in some case even the method) of an HTTP request and response are passed to the backend. More details at the content disposition page.

26.2.4. Inserting the metrics snippet¶

This is functionality for inserting a performance-reporting snippet, mostly useful with our data-collection pipelines. See here.

26.2.5. Stacked settings¶

Stacked settings are settings that can be set on a view, an entire backend, or an entire domain in the devlove file. When they are set in multiple places, ShimmerCat combines them — using a semigroup — and uses the combined result instead. The way the settings at different levels are combined is specific for each setting.

This section details some of those settings.

26.2.5.1. Modifying request and response headers¶

The setting change-headers-in allows modifying HTTP request headers just before passing them to the backend, and the setting change-headers-out allows modifying HTTP response headers just after they come from the backend and before passing them to the browser. In what follows, we will describe change-headers-in, but bear in mind that the same applies to change-headers-out.

Here is an example of a view that uses change-headers-in:

<!--
shimmercat:
  content-disposition: replace
  change-headers-in: |
    headers['X-Forward-Proto'] = 'HTTPS'
    return headers
-->

The value of the setting is a string which should be valid Lua code. ShimmerCat evaluates the snippet in a Lua environment with the following additional parameter:

headers: a Lua table with the input headers

The code should return a table with the modified headers. If it doesn’t, ShimmerCat QS won’t change the headers before passing them to the application.

Being a stacked setting, change-headers-in can also be defined in the backend stanza at the devlove:

shimmercat-devlove:
  domains:
    elec www.example.com:
       consultants:
         our-iis-server:
            connect-to: ...
            change-headers-in:
               headers['x-real-ip']=headers['x-forwarded-for']

ShimmerCat regards the value of change-headers-in as a function from headers to headers, and combines them using the endomorphism monoid. In practical terms, when change-headers-in is set both at a view and at the consultant used by that view, the headers are first passed to the function at the consultant, and its result is then passed to the function at the view. The composition order is reversed for change-headers-out, so that when the setting is at two places the view processes the headers first and then the consultant goes second.

26.2.5.2. Accessing GeoIP functionality via IPRecords¶

Shimmercat QS can read GeoIP2 files and make their information accessible to the change-headers-in and change-headers-out function.

A stacked boolean setting is used for this: enable-ip-records, which can be used both at a view level and in the top object of a consultant. When used in both places, the value set at the view takes precedence.

When set, a table ip_records is added to the Lua context of both change-headers-in and change-headers-out. Accessing an attribute of this table triggers a lookup of a corresponding entry for the current request’s IP address, which is either the IP address of the connecting session or, if configured, the address provided via proxy-v2.

Databases are copied or linked inside the scratch folder, at rdonly-dbs/ip-records/. Take note of the filename with which you copy or link your database inside that folder, as it needs to be used to access that database’s entry from inside the file. For example, if the GeoIP2 database file is a symbolic link with the name ip2city”, then you can use something like this to access the corresponding IP record:

...
headers['Geo-IP-Country'] = ip_records.ip2city.en.country_code
return headers

The fragment ip_records.ip2city will hold whatever object in the database matches the current IP address, or nil if none matches. Because of how GeoIP2 is defined, the contents of the record vary from database to database, and you must find out which one applies in your particular case. We recommend you to use the tool sc-iprecord-lookup in the ShimmerCat QS distro, perhaps in combination with jq, to do some sample queries in your database and find out how objects inside it look like.

It’s possible to install and use multiple GeoIP2 databases simultaneously, and there is an auxiliary program in the distribution to enable creating your own GeoIP2 files: sc-iprecords-meld. Run it passing via standard input a sequence of lines, each containing a one-line JSON document. Each document should be a map with two entries: subnet and record. The subnet is a string denoting an IPv4 subnet (we havent gotten to implement IPv6 support yet) as in 183.128.0.0/9, and record is any valid JSON value. The output of sc-iprecords-meld will be a valid GeoIP2 file.

26.2.5.3. Forwarding backend responses for error cases¶

The setting forward-error-pages instructs ShimmerCat QS to send error pages from the backend to the browser. This setting combines exactly as enable-ip-records, and its details are described in a separate page.

26.2.5.4. Blowing HTML up¶

This setting enabling writting a log message with HTML received from a backend, from URLs which have been whitelisted for this. This is a non-destructive side effect, in particular it doesn’t imply changes to the transmitted HTML. That is, here “blow up” is used in its literal sense, not the idiomatic one.

This functionality is necessary to report data for analysis with acceleration rules that rely on understandig HTML structure.

The setting, a boolean value for a blowup field, can be used in the definition of the application port and the definition at the view. A definition at the view takes precedence over any definition at the application port.

In addition to these stacked settings, a particular URL path should be listed for reporting. The list must be in a sqlite3 database file at <scratch-folder>/rdonly-dbs/blowuphtml/<domain_name>.sq3. Said datbase should include the URLs whose contents we are interested in in a table defined with the following DDL:

CREATE TABLE urls_to_report_t(
  url_path VARCHAR PRIMARY KEY,
  weight INT NOT NULL
  )

In the table, the url_path should be the URL path, without query string — this is a temporary restriction — and weight should be a number that tells ShimmerCat QS how important this URL path is for reporting. Matching is done against the original URL path coming from the browser, before any rewrites in ShimmerCat. If a URL path for a request is not in that table, its HTML is not reported.

When a Shimmercat worker starts, it copies the database file to a temporary file, opens it in read-only mode, and unlinks the temporary file. This way, it is safe to overwrite the original file at <scratch-folder>/rdonly-dbs/blowuphtml/<domain_name>.sq3 at any moment without crashing SQLite. However, for ShimmerCat to use the updated file, a server reload is needed — e.g. via kill -SIGHUP $(cat scratch-folder/master.pid).

The log messages report the HTML as a XZ stream which has been text-encoded with Ascii85. The log message codes assigned for this are M60005 to M60007. An additional log message, M44005, will be issued when ShimmerCat finds and activates a SQLite3 database for a particular domain.