Table of Contents
  • Docs >
  • 31. Structuring your applications
Shortcuts

31. Structuring your applications

31.1. Two types of domains

ShimmerCat uses two types of domains, each with a different function inside the application: electric domains, and api domains.

For example, you can map “www.mysite.com” to an electric domain, and “api1.mysite.com” to an API domain. Both types of domains are configured in the devlove.yaml file.

Electric domains can be recognized automatically by the subkeys root-dir or consultant. Like this:

shimmercat-devlove:
    domains:
        www.mysite.com:
            root-dir: www

They can also be distinguished by prefixing the word elec to the key with the domain name.

API domains can be recognized automatically by specifying the subkey port with the web application port to which ShimmerCat should proxy requests, like this:

shimmercat-devlove:
    domains:
        api1.mysite.com:
            port: "8080"

Important: This is not the port where ShimmerCat listens, but rather the port it will use to connect to an internal HTTP/1.1 or FastCGI web application. To indicate a port for ShimmerCat to listen on, use the --listen option in the command line.

Domains can also be recognized by prefixing the word api to the key with the domain name.

Of course, you can mix and match domains of different types in a single devlove.yaml file. Like this:

shimmercat-devlove:
    domains:
        api1.mysite.com:
            port: "8080"
        www.mysite.com:
            root-dir: www

Notice that from version 1.5 of ShimmerCat electric domains can both serve static assets and rely requests to an inner application. When electric domains are used to proxy requests to an inner application, powerful logic can be used to simplify your development workflows or to make ShimmerCat serve traditional applications like Wordpress.

API domains should be used when all you want is to proxy HTTP/2 requests to a HTTP/1.1 or FastCGI application, without serving any static files on the same domain.

For details on the differences between each type of domain, keep reading.

31.1.1. Electric domains serve views

Electric domains can serve static resource and gate some requests to your inner application. The static files are your .html files, JavaScript files, CSS files, images and other files that your application needs to load in the user’s browser. Gating requests to an inner application is achieved through something we call SC-views, which in short are bundles of information that influence how an HTML document is delivered. We have an entire article dedicated to SC-views.

At electric domains, ShimmerCat can use HTTP/2 Push with static assets. ShimmerCat can even assist you on creating descriptions of what to push, see the article about learning and pushing.

To distinguish electric domains from other domains, use the root-dir attribute, or prefix the domain name with the keyword elec, as in the following example:

shimmercat-devlove:
    domains:
        elec www.mysite.com:
            root-dir: www

31.1.2. API domains

These are domains in which requests are forwarded to a backend application with little to no processing. We support two forwarding mechanisms to inner applications: HTTP/1.1 and FastCGI. In both cases, the port attribute is used to distinguish an API domain, and ShimmerCat probes the port to determine if the application is using HTTP/1.1 or FastCGI. The probing operation shows in ShimmerCat’s logs, and may also show in the error log of the application. We recommend using HTTP/1.1 whenever possible.

FastCGI may require setting some variables in the request in order for the application to function. For example, PHP-FPM requires setting DOCUMENT_ROOT and SCRIPT_FILENAME. Getting that to work is easy with ShimmerCat, just use the document-root field for the API domain, which is exclusive for the FastCGI transport. The example below shows how that would go with a site divided in two folders, one for the assets delivered to the browser and another for php files executed by a backend application, e.g. PHP-FPM.

---
shimmercat-devlove:
    domains:
        www.test.com:
            root-dir: my-site/frontend-files
        api.test.com:
            port:
                connect-to: "9101"
                document-root: my-site/backend-php-scripts

Important: Do not use a raw number for the value of port. You can either quote the number as shown in the example, or prefix it with a colon :. ShimmerCat interprets the number inside quotes or after a colon as a port number in the loopback network address 127.0.0.1. You can also specify a full IPv4 address, as in 192.168.5.100:8080, or a Unix domain socket, as in unix://app.sock.

Notice that it is possible to use relative paths in the document-root, ShimmerCat will make them absolute by using the directory where the devlove file is located, before passing them to the backend application. Also take note that now port is an object with a connect-to attribute and document-root is where the document root is.

The forwarding mechanism also includes a “Forwarded” header that can be used by the inner application to gather information about the client, for example, its IP address, the latency, and if it supports HTTP/2 Push.

31.2. How electric domains map URLs to filesystem paths

For electric domains we use a fixed, common convention to map URL paths to filesystem paths. Simply put, this convention presents HTML files behind URL paths ending in a slash.

Following this convention alone allows to write rich frontend applications with pretty URLs, but if you would prefer to follow a different convention, there is a powerfull URL path re-write engine that can help. However, because the re-write engine just routes URL paths in and out of core, it is still worth knowing the convention presented here.

31.2.1. How the mapping works in an example

We describe the rules in more detail below, but an example goes a long way, so let’s start with one.

31.2.1.1. Basic-url-mapping:

Basic example demonstrating how ShimmerCat maps URL paths to filesystem paths.

31.2.1.2. Files:

  • devlove.yaml

---
shimmercat-devlove:
    domains:
        elec www.mystaticsite.com:
            root-dir: www-files

  • www-files/index.html

<!--
Served when the user types https://www.mystaticsite.com/
in the browser.
-->
<html>
 <head>
     <title>This is the homepage</title>
     <script src="/js/app.js"></script>
 </head>
 <body>
     This is the index.html file right under root-dir.
 </body>
</html>

  • www-files/places/__404.html

<!--
 Served when the user types any address that
 can not be resolved to a HTML page.
 E.g. https://www.mystaticsite.com/not-existent
-->
<html>
  <head>
      <title>404: The page you were looking for was not found</title>
      <script src="/js/app.js"></script>
  </head>
  <body>
      404 Not found:
      You were trying to access this location:
      <span id="location"></span>
  </body>
</html>

  • www-files/places/index.html

<!--
 Served when the user tries to access
 E.g. https://www.mystaticsite.com/places/
-->
<html>
  <head>
      <title>Hello, this is a non-generic place</title>
      <script src="/js/app.js"></script>
  </head>
  <body>
      You are accessing this location: <span id="location"></span>,
      and this file is located at /places/index.html on the filesystem.
  </body>
</html>

  • www-files/places/__index.html

<!--
 Served when the user tries to access a location that looks
 like a page under the `places` directory, but no actual `index.html`
 file is present right below the corresponding sub-directory.
 E.g. https://www.mystaticsite.com/places/my-latest-blog-article/
-->
<html>
  <head>
      <title>Hello, this is one generic place</title>
      <script src="/js/app.js"></script>
  </head>
  <body>
      You are accessing this location: <span id="location"></span>,
      and this file is located at /places/__index.html on the filesystem.
      By looking to window.location in the browser, I can
      complete this page with location-specific contents.
  </body>
</html>

  • www-files/js/app.js

// Reads the URL from the browser's address bar
// and writes it as the contents of an element
// named "location"
window.onload=function(){
  var loc_span = document.getElementById("location");
  loc_span.innerHTML = window.location;
}

Run ShimmerCat in devlove mode:

$ /place/where/you/uncompressed/shimmercat/program/bin/shimmercat devlove

Then try the following URLs with a browser or with cURL:

  • https://www.mystaticsite.com is served with the file index.html.

  • https://www.mystaticsite.com/where-are-you/ . This returns the contents of the __404.html file.

  • https://www.mystaticsite.com/places . This redirects to https://www.mystaticsite.com/places/ , then returns the contents of places/index.html.

  • https://www.mystaticsite.com/places/a-place-on-earth. This redirects to https://www.mystaticsite.com/places/a-place-on-earth/ , then returns the contents of places/__index.html.

To get a better idea of how mapping works, use a browser with SOCKS5 configured to connect to the SOCKS5 port where ShimmerCat listens. You can do that painlessly by running sc-tool chrome after you have installed sc-tool.

31.2.2. The rules of the convention

First, a root directory is indicated in the devlove file, using the root-dir key. Then URL paths are used as relative paths inside that root-dir. This is the usual way in which web servers providing static resources work, so you should be able to get used to it in no time.

ShimmerCat also increases the set of URLs that map to a single set of files through something called template lookup.

Here is the convention:

  1. If the last part of a URL doesn’t contain a dot, and it doesn’t end with a slash, then a redirect is returned to the client asking it to access instead the path ending with a slash, e.g. /places → /places/. But /js/app.js contains a dot, and therefore the server just tries to return a file at that relative path.

  2. If the last part of a path ends with a slash, then it is treated like a relative directory inside the root-dir, and a file index.html is sought in that relative directory. If found, that file is returned, otherwise a template lookup will be started. More details about template lookups are provided below, but in the example above they allow to return the file at places/__index.html when the user tries to access https://www.mystaticsite.com/places/some-new-place/

  3. If a resource can’t be found, a template lookup for a 404 page will be initiated. If found, that page will be rendered. Otherwise, a very simple generic 404 page will be returned.

A template lookup looks for a file named __index.html in successive parent directories to URL path requested. For example, if a request to the relative URL path /places/my-place-group/a-place/ is done, and there is no file in the filesystem at path places/my-place-group/a-place/index.html, then places/my-place-group/__index.html is tried, and then places/__index.html, and finally __index.html just below root-dir. For the example above, the search ends when places/__index.html is found.

31.3. Consulting

When using template lookups in electric domains, ShimmerCat may be provided with a way to discover if a URL path refers to a valid entity in the site or not. We call web applications in that role a “consultant”.

The consultant is an ordinary HTTP/1.1 or FastCGI application. In its simplest form, “pure consulting”, the application is queried with a HEAD request. The request is the original request received by ShimmerCat from the browser, but changing the method from GET to HEAD.

If the consultant returns a 404, then ShimmerCat returns a 404 to the client (possibly doing a template lookup for a __404.html file, which is not consulted). If the consultant returns a 200 HTTP status code, ShimmerCat returns the contents of the __index.html template.

Here is a tiny project illustrating the setup:

31.3.1. Simple-consult:

A basic consulting example. For running this example you will need, in addition to ShimmerCat and a browser, a Python interpreter and Bottle’s web framework.

31.3.2. Files:

  • devlove.yaml

---
shimmercat-delove:
    domains:
       elec www.myexample.com:
          consultant: :8080
          root-dir: site-files

  • site-files/docs/__index.html

<html>
  <head>
      <title>This is a web application page</html>
      <!--
      Include anything needed to make a rich client web application,
      the consulting mechanism ensures that this page is only
      returned when there are actual contents for the URL
      that the user requested.
      -->
      <script src="/js/app.js"></script>
  </head>
  <body>
      This is a non-empty page, the URL used was <span id="location"></span>
  </body>
</html>

  • app.py

from bottle import route, run

@route('/docs/<doc_name>/')
def index(doc_name):
    # In a real world example, this would include code
    # to query the database or any other content backend to
    # find out if doc_name refers to a valid document.
    #
    # Instead of doing that, and for the sake of simplicity,
    # we are just going to return a 200 response if doc_name
    # refers to any of our favorite web frameworks
    if doc_name in LIKED_FRAMEWORKS:
        # For a pure consulting backend, the contents don't matter, only
        # the status code and some headers are used.
        return 'ok'
    else:
        return HTTPError(status=404)

LIKED_FRAMEWORKS = [
    'django',
    'yesod',
    'express',
    'ruby-on-rails',
    'larabel',
    'snap',
    'bottle',
]

run(host='localhost', port=8080)

  • site-files/js/app.js

// Reads the URL from the browser's address bar
// and writes it as the contents of an element
// named "location"
window.onload=function(){
  var loc_span = document.getElementById("location");
  loc_span.innerHTML = window.location;
}

For the HTTP status codes which represent redirects, ShimmerCat will forward the intention from the application to the client. That is, if the consultant returns a 301, 302, 303 or 307 code, ShimmerCat will take a look to any Location header returned by the consultant and build a response with the same code and same Location header.

This simple form of consulting can be used to create web applications which load very fast, because the consulting request only needs to establish if there are contents at the requested URL, while the contents themselves can be fetched with ulterior AJAX requests, presumably when the bulk of static assets making up the application are already in their way to the client.

However the consulting mechanism has been vastly improved from version 1.5, and is no longer limited to changing HEAD to GET requests. Now it can also relay all common HTTP verbs (POST, PUT, DELETE, OPTIONS) and it can use the response from the application to build the HTML contents sent to the browser. For more information, check the Handling Application Contents article.