31. Structuring your applications¶
31.1. Two types of domains¶
ShimmerCat uses two types of domains, each with a different function inside the application: electric domains, and api domains.
For example, you can map “www.mysite.com” to an electric domain, and “api1.mysite.com”
to an API domain.
Both types of domains are configured in the devlove.yaml
file.
Electric domains can be recognized automatically by the subkeys root-dir
or consultant
.
Like this:
shimmercat-devlove:
domains:
www.mysite.com:
root-dir: www
They can also be distinguished by prefixing the word elec
to the key with the domain
name.
API domains can be recognized automatically by specifying the subkey port
with the web
application port to which ShimmerCat should proxy requests, like this:
shimmercat-devlove:
domains:
api1.mysite.com:
port: "8080"
Important: This is not the port where ShimmerCat listens, but rather the port it will
use to connect to an internal HTTP/1.1 or FastCGI web application.
To indicate a port for ShimmerCat to listen on, use the --listen
option in the command line.
Domains can also be recognized by prefixing the word api
to the key with the domain name.
Of course, you can mix and match domains of different types in a single devlove.yaml
file.
Like this:
shimmercat-devlove:
domains:
api1.mysite.com:
port: "8080"
www.mysite.com:
root-dir: www
Notice that from version 1.5 of ShimmerCat electric domains can both serve static assets and rely requests to an inner application. When electric domains are used to proxy requests to an inner application, powerful logic can be used to simplify your development workflows or to make ShimmerCat serve traditional applications like Wordpress.
API domains should be used when all you want is to proxy HTTP/2 requests to a HTTP/1.1 or FastCGI application, without serving any static files on the same domain.
For details on the differences between each type of domain, keep reading.
31.1.1. Electric domains serve views¶
Electric domains can serve static resource and gate some requests to your inner application. The static files are your .html files, JavaScript files, CSS files, images and other files that your application needs to load in the user’s browser. Gating requests to an inner application is achieved through something we call SC-views, which in short are bundles of information that influence how an HTML document is delivered. We have an entire article dedicated to SC-views.
At electric domains, ShimmerCat can use HTTP/2 Push with static assets. ShimmerCat can even assist you on creating descriptions of what to push, see the article about learning and pushing.
To distinguish electric domains from other domains, use the root-dir
attribute,
or prefix the domain name with the keyword elec
, as in the following example:
shimmercat-devlove:
domains:
elec www.mysite.com:
root-dir: www
31.1.2. API domains¶
These are domains in which requests are forwarded to a backend application with
little to no processing.
We support two forwarding mechanisms to inner applications: HTTP/1.1 and FastCGI.
In both cases, the port
attribute is used to distinguish an
API domain, and ShimmerCat probes the port to determine if the application is using
HTTP/1.1 or FastCGI.
The probing operation shows in ShimmerCat’s logs, and may also show in
the error log of the application.
We recommend using HTTP/1.1 whenever possible.
FastCGI may require setting some variables in the request in order for the application
to function.
For example, PHP-FPM requires setting DOCUMENT_ROOT
and SCRIPT_FILENAME
.
Getting that to work is easy with ShimmerCat, just use the document-root
field
for the API domain, which is exclusive for the FastCGI transport.
The example below shows how that would go with a site divided in two folders,
one for the assets delivered to the browser and another for php files
executed by a backend application, e.g. PHP-FPM.
---
shimmercat-devlove:
domains:
www.test.com:
root-dir: my-site/frontend-files
api.test.com:
port:
connect-to: "9101"
document-root: my-site/backend-php-scripts
Important: Do not use a raw number for the value of port
.
You can either quote the number as shown in the example, or prefix it with a colon :
.
ShimmerCat interprets the number inside quotes or after a colon as a port number in the
loopback network address 127.0.0.1.
You can also specify a full IPv4 address, as in 192.168.5.100:8080
, or a Unix domain
socket, as in unix://app.sock
.
Notice that it is possible to use relative paths in the document-root
, ShimmerCat
will make them absolute by using the directory where the devlove file is located,
before passing them to the backend application.
Also take note that now port
is an object with a connect-to
attribute and
document-root
is where the document root is.
The forwarding mechanism also includes a “Forwarded” header that can be used by the inner application to gather information about the client, for example, its IP address, the latency, and if it supports HTTP/2 Push.
31.2. How electric domains map URLs to filesystem paths¶
For electric domains we use a fixed, common convention to map URL paths to filesystem paths. Simply put, this convention presents HTML files behind URL paths ending in a slash.
Following this convention alone allows to write rich frontend applications with pretty URLs, but if you would prefer to follow a different convention, there is a powerfull URL path re-write engine that can help. However, because the re-write engine just routes URL paths in and out of core, it is still worth knowing the convention presented here.
31.2.1. How the mapping works in an example¶
We describe the rules in more detail below, but an example goes a long way, so let’s start with one.
31.2.1.1. Basic-url-mapping:¶
Basic example demonstrating how ShimmerCat maps URL paths to filesystem paths.
31.2.1.2. Files:¶
devlove.yaml
---
shimmercat-devlove:
domains:
elec www.mystaticsite.com:
root-dir: www-files
www-files/index.html
<!--
Served when the user types https://www.mystaticsite.com/
in the browser.
-->
<html>
<head>
<title>This is the homepage</title>
<script src="/js/app.js"></script>
</head>
<body>
This is the index.html file right under root-dir.
</body>
</html>
www-files/places/__404.html
<!--
Served when the user types any address that
can not be resolved to a HTML page.
E.g. https://www.mystaticsite.com/not-existent
-->
<html>
<head>
<title>404: The page you were looking for was not found</title>
<script src="/js/app.js"></script>
</head>
<body>
404 Not found:
You were trying to access this location:
<span id="location"></span>
</body>
</html>
www-files/places/index.html
<!--
Served when the user tries to access
E.g. https://www.mystaticsite.com/places/
-->
<html>
<head>
<title>Hello, this is a non-generic place</title>
<script src="/js/app.js"></script>
</head>
<body>
You are accessing this location: <span id="location"></span>,
and this file is located at /places/index.html on the filesystem.
</body>
</html>
www-files/places/__index.html
<!--
Served when the user tries to access a location that looks
like a page under the `places` directory, but no actual `index.html`
file is present right below the corresponding sub-directory.
E.g. https://www.mystaticsite.com/places/my-latest-blog-article/
-->
<html>
<head>
<title>Hello, this is one generic place</title>
<script src="/js/app.js"></script>
</head>
<body>
You are accessing this location: <span id="location"></span>,
and this file is located at /places/__index.html on the filesystem.
By looking to window.location in the browser, I can
complete this page with location-specific contents.
</body>
</html>
www-files/js/app.js
// Reads the URL from the browser's address bar
// and writes it as the contents of an element
// named "location"
window.onload=function(){
var loc_span = document.getElementById("location");
loc_span.innerHTML = window.location;
}
Run ShimmerCat in devlove mode:
$ /place/where/you/uncompressed/shimmercat/program/bin/shimmercat devlove
Then try the following URLs with a browser or with cURL:
https://www.mystaticsite.com
is served with the fileindex.html
.https://www.mystaticsite.com/where-are-you/
. This returns the contents of the__404.html
file.https://www.mystaticsite.com/places
. This redirects tohttps://www.mystaticsite.com/places/
, then returns the contents ofplaces/index.html
.https://www.mystaticsite.com/places/a-place-on-earth
. This redirects tohttps://www.mystaticsite.com/places/a-place-on-earth/
, then returns the contents ofplaces/__index.html
.
To get a better idea of how mapping works, use a browser with SOCKS5
configured to connect to the SOCKS5 port where ShimmerCat listens.
You can do that painlessly by running sc-tool chrome
after you
have installed sc-tool.
31.2.2. The rules of the convention¶
First, a root directory is indicated in the devlove file, using the root-dir
key.
Then URL paths are used as relative paths inside that root-dir
.
This is the usual way in which web servers providing static resources work,
so you should be able to get used to it in no time.
ShimmerCat also increases the set of URLs that map to a single set of files through something called template lookup.
Here is the convention:
If the last part of a URL doesn’t contain a dot, and it doesn’t end with a slash, then a redirect is returned to the client asking it to access instead the path ending with a slash, e.g.
/places
→/places/
. But/js/app.js
contains a dot, and therefore the server just tries to return a file at that relative path.If the last part of a path ends with a slash, then it is treated like a relative directory inside the
root-dir
, and a fileindex.html
is sought in that relative directory. If found, that file is returned, otherwise a template lookup will be started. More details about template lookups are provided below, but in the example above they allow to return the file atplaces/__index.html
when the user tries to accesshttps://www.mystaticsite.com/places/some-new-place/
If a resource can’t be found, a template lookup for a 404 page will be initiated. If found, that page will be rendered. Otherwise, a very simple generic 404 page will be returned.
A template lookup looks for a file named __index.html
in successive parent directories to
URL path requested.
For example, if a request to the relative URL path /places/my-place-group/a-place/
is done,
and there is no file in the filesystem at path places/my-place-group/a-place/index.html
,
then places/my-place-group/__index.html
is tried, and then places/__index.html
, and
finally __index.html
just below root-dir.
For the example above, the search ends when places/__index.html
is found.
31.3. Consulting¶
When using template lookups in electric domains, ShimmerCat may be provided with a way to discover if a URL path refers to a valid entity in the site or not. We call web applications in that role a “consultant”.
The consultant is an ordinary HTTP/1.1 or FastCGI application. In its simplest form, “pure consulting”, the application is queried with a HEAD request. The request is the original request received by ShimmerCat from the browser, but changing the method from GET to HEAD.
If the consultant returns a 404, then ShimmerCat returns a 404 to the client (possibly doing
a template lookup for a __404.html
file, which is not consulted).
If the consultant returns a 200 HTTP status code, ShimmerCat returns the
contents of the __index.html
template.
Here is a tiny project illustrating the setup:
31.3.1. Simple-consult:¶
A basic consulting example. For running this example you will need, in addition to ShimmerCat and a browser, a Python interpreter and Bottle’s web framework.
31.3.2. Files:¶
devlove.yaml
---
shimmercat-delove:
domains:
elec www.myexample.com:
consultant: :8080
root-dir: site-files
site-files/docs/__index.html
<html>
<head>
<title>This is a web application page</html>
<!--
Include anything needed to make a rich client web application,
the consulting mechanism ensures that this page is only
returned when there are actual contents for the URL
that the user requested.
-->
<script src="/js/app.js"></script>
</head>
<body>
This is a non-empty page, the URL used was <span id="location"></span>
</body>
</html>
app.py
from bottle import route, run
@route('/docs/<doc_name>/')
def index(doc_name):
# In a real world example, this would include code
# to query the database or any other content backend to
# find out if doc_name refers to a valid document.
#
# Instead of doing that, and for the sake of simplicity,
# we are just going to return a 200 response if doc_name
# refers to any of our favorite web frameworks
if doc_name in LIKED_FRAMEWORKS:
# For a pure consulting backend, the contents don't matter, only
# the status code and some headers are used.
return 'ok'
else:
return HTTPError(status=404)
LIKED_FRAMEWORKS = [
'django',
'yesod',
'express',
'ruby-on-rails',
'larabel',
'snap',
'bottle',
]
run(host='localhost', port=8080)
site-files/js/app.js
// Reads the URL from the browser's address bar
// and writes it as the contents of an element
// named "location"
window.onload=function(){
var loc_span = document.getElementById("location");
loc_span.innerHTML = window.location;
}
For the HTTP status codes which represent redirects,
ShimmerCat will forward the intention from the application to the
client.
That is, if the consultant returns a 301, 302, 303 or 307 code, ShimmerCat will take a look
to any Location
header returned by the consultant and build a response with the same code
and same Location
header.
This simple form of consulting can be used to create web applications which load very fast, because the consulting request only needs to establish if there are contents at the requested URL, while the contents themselves can be fetched with ulterior AJAX requests, presumably when the bulk of static assets making up the application are already in their way to the client.
However the consulting mechanism has been vastly improved from version 1.5, and is no longer limited to changing HEAD to GET requests. Now it can also relay all common HTTP verbs (POST, PUT, DELETE, OPTIONS) and it can use the response from the application to build the HTML contents sent to the browser. For more information, check the Handling Application Contents article.