32. Handling application contents¶
32.1. Brokering between two peers¶
Here we explain how ShimmerCat’s electric domains do reverse-proxying between the Internet and one or more web applications.
ShimmerCat can use the reponse returned from an application in different ways,
through a directive called content-disposition
.
This directive must be present as a special HTML comment (see below),
it is not an HTTP header that your application must set.
The directive affects what ShimmerCat does with the contents
returned in the application response, represented by the big
pinkish rectangle in our header figure.
The content-disposition
directive can be used to instruct ShimmerCat to do things the conventional way (value replace
).
But it can also be used to offset the delays introduced by the backend and to
simplify deployment of rich client applications (values consult
and use-json
),
with no need for backend templates.
32.2. Sending contents verbatim from the backend¶
32.2.1. Example 1: A very simple NodeJS application¶
This is a very simple “hello-world” example, where the application backend
is a JavaScript file for NodeJS.
Here we show the content-disposition: replace
directive, which simply sends
the contents from the backend unchanged to the browser.
For illustration, we have also added a styles.css
file which should be served directly
by ShimmerCat.
How to serve contents from a simple NodeJS backend.
Get following files we mention below in a folder, cd
to it and execute
your NodeJS backend with:
$ nodejs nodejs_backend.js
Then execute ShimmerCat:
$ shimmercat devlove -5 2016
and then test that it works using curl:
curl -k –socks5-hostname 127.0.0.1:2016 https://www.example.com/nodejsapp/
32.2.1.1. Files:¶
nodejs_backend.js
var http = require('http');
const PORT=8095;
const contents =
'<html><head><link rel="stylesheet" href="/styles.css"></head>\n' +
'<body>Made inside NodeJS</body></html>';
function handleRequest(request, response){
response.end(contents);
}
var server = http.createServer(handleRequest);
server.listen(PORT, function(){
console.log("Server listening on: http://localhost:%s", PORT);
});
devlove.yaml
shimmercat-devlove:
domains:
elec www.example.com:
consultant: "8095"
root-dir: site
site/styles.css
body {
color: green;
}
site/nodejsapp/index.html
<!--
shimmercat:
content-disposition: replace
-->
The NodeJS application should listen on port 8095 and ShimmerCat will connect to that port to fetch contents.
To test that this setup is working, you can point your
development browser configured with SOCKS5
to the url https://www.example.com/nodejsapp/
.
Alternatively, you can use one of the following curl command lines:
# If you let ShimmerCat listen for SOCKS5 at port 2006
$ curl -k --socks5-hostname 127.0.0.1:2006 https://www.example.com/nodejsapp/
...
# Or if you prefer to set the Host header:
$ curl -HHost:www.example.com https://127.0.0.1:4043/nodejsapp/
<html><head><link rel="stylesheet" href="/styles.css"></head>
<body>Made inside NodeJS</body></html>
We are requesting the URL path nodejsapp/
in our curl command,
and for that to work, notice the presence of the index.html
file just below
that directory in in the root-dir
location.
This file is the one containing the content-disposition
directive as part
of a special HTML comment.
To know more in general about our mapping from URL paths to filesystem paths,
check the section on URL path mapping
in our article about application structure.
32.2.2. Streaming and buffering¶
With content-disposition: replace
ShimmerCat will exceptionally buffer contents from the application,
but as a general rule, it
will relay bytes as soon as they are received.
This is because most application frameworks already take good care of
doing buffering when it makes sense, and additional buffering from ShimmerCat is more likely
to get in the way than to help.
The only cases when ShimmerCat will do buffering is when the application backend sets
a Content-Length
HTTP header but sends plain, uncompressed contents.
If ShimmerCat knows that the client supports compression, it will buffer the contents,
compress them, and set a new Content-Length
with the adjusted length for the
compressed contents.
If the application does not set a Content-Length
header but sends plain, uncompressed
contents, and ShimmerCat knows that the client supports compressed contents, it
will compress the contents on-the-fly.
In any other case, ShimmerCat will just relay the contents unchanged from the backend
as they arrive.
In all cases, if ShimmerCat can’t provide a Content-Length
header to the client,
HTTP/1.1 clients will receive chunked-encoded contents.
32.3. Using JSON data from the backend¶
32.3.1. Example 2: Making your site SEO-friendly¶
Many web applications can be built as a set of static files that once loaded in the browser will fetch any necessary data via AJAX, websockets or even use local storage.
It turns out this is not good for sites whose contents you want to have indexed or whose URLs you want to share in social networks. Some services like Google Bot are just fine with contents and metadata set on the page via Javascript, but we have seen Twitter, LinkedIn and Facebook fail misserably at the same task.
Solving this problem and also saving round-trips is what motivated us to introduce
content-disposition: use-json
in version 1.5.
The directive enables a very simple and efficient form of HTML templates using
JSON data from the backend.
It helps in simple cases where one needs to splice a few strings in the
HTML and to embed JSON data.
It should fit well frontend frameworks like AngularJS and EmberJS which take
care of rendering contents in the client, by enabling the backend to just
speak JSON.
Notice however that it is not a replacement for a full-fledged template engine.
With the use-json
value, the backend response is interpreted as JSON,
and the index.html
or __index.html
pages are used as templates.
These templates are compiled and stored
in ShimmerCat’s transparent cache.
Here again we use directives embedded in HTML comments with a special
form.
Namely, contents that form a valid YAML for an object holding a single shimmercat
key.
This allows to use the same HTML files with other web servers (which of course
we don’t recommend ;-) ).
Our tiny template language can do two things: intersperse string values in
the HTML, and express JSON literal objects directly in the document, presumably
inside a <script>
tag.
In the first case, the string is inserted literally in the document, and in the second,
ShimmerCat escapes as necessary to create contents which are allowed inside <script>
tags,
as specified in section 4.11.1.2
of the HTML5 script recommendation.
Both facilities are shown in action in the example below:
This example is very similar to the the previous one.
Get the files we mention below in a folder, cd
to it and execute
your NodeJS backend with:
$ nodejs nodejs_backend.js
Then execute ShimmerCat:
$ shimmercat devlove -5 2016 -w
and then test that it works using curl:
$ curl -k --socks5-hostname 127.0.0.1:2016 https://www.example.com/article/my-dynamic-article
Of course, you can also connect with a browser and examine the results.
If everything goes alright, the HTML that you get should be similar to this one:
<html>
<head>
<title>
This is the article for /articles/my-dynamic-article/
</title>
</head>
<body>
<div id="content"></div>
<script src="https://crash.shimmercat.com/js/marked.min.js?vh=1e56f4bc6"></script>
<script>
var contents=
{"markdown":"## Title: an article at the URL Path /articles/my-dynamic-article/ ...\n\nLorem ipsum....litle HTML comment comes: \u003c!-- html comment --\u003e."};null;
if ( contents != null )
{
document.getElementById('content').innerHTML =
marked(contents.markdown);
} else {
// If you want your application to also work without ShimmerCat,
// use this branch to fetch the contents using XHR.
//
// ... use promises and such ...
//
//
}
</script>
</body>
</html>
32.3.1.1. Files:¶
site/articles/__index.html
<html>
<head>
<title>
<!--shimmercat:
insert: $.metadata.title
-->
</title>
</head>
<body>
<div id="content"></div>
<script
src="https://crash.shimmercat.com/js/marked.min.js?vh=1e56f4bc6">
</script>
<script>
var contents=
<!--shimmercat:
insert: $.contents;
-->null;
if ( contents != null )
{
document.getElementById('content').innerHTML =
marked(contents.markdown);
} else {
// If you want your application to also work without ShimmerCat,
// use this branch to fetch the contents using XHR.
//
// ... use promises and such ...
//
//
}
</script>
</body>
</html>
<!--
shimmercat:
content-disposition: use-json
-->
nodejs_backend.js
var http = require('http')
, url = require('url');
const PORT=8095;
function handleRequest(request, response){
var url_parts = url.parse(request.url);
var contents = JSON.stringify({
metadata: {
title: "This is the article for " + url_parts.pathname
},
contents: {
markdown: "## Title: an article at the URL Path " + url_parts.pathname + " ...\n\n" +
"Lorem ipsum...." +
"litle HTML comment comes: <!-- html comment -->."
}
});
response.end(contents);
}
var server = http.createServer(handleRequest);
server.listen(PORT, function(){
console.log("Server listening on: http://localhost:%s", PORT);
});
devlove.yaml
shimmercat-devlove:
domains:
elec www.example.com:
consultant: "8095"
root-dir: site
32.3.2. Micro-template syntax for use-json
¶
The syntax for the value at the insert
key inside the YAML is very simple:
Use a dollar sign
$
to refer to the root JSON object.Use a dot to access a member of a JSON object. For example,
$.mykey
will get you the value1
when applied to the object{"mykey":1}
. Keys are valid if they are a sequence of latin alphanumeric characters, digits, underscore (_
) and dash (-
). You can use more dots and keys to access deeper into the JSON object passed from the backend.If at some point you try to access a key that doesn’t exist , the access expression yields an empty string.
If you want to access elements of an array, use a key which is a valid integer, as in
$.0
. We use zero-based indexing.An expression to access a value in the JSON object is parsed as further to the right as possible. Parsing ends at the first space or at any other character which is not a dot or a valid part of a key.
Because it is possible to combine object access with literal contents in the same place, as in
<!--
shimmercat:
insert: There are $.page_count pages
-->
we also have a few additional facilities:
Use two dollar signs in the template
$$
to get a literal$
in the output.If your literal contents include characters which are valid parts of a key and you want to write them immediately after an access expression with no intervening space, use two dots to terminate the access expression.
Use a dollar sign followed by an ampersand to query for a piece of runtime information implicitly provided by ShimmerCat. Right now, we have only one:
$&backend-time
, which outputs how much time the backend took to produce an answer. The contents of this variable are exactly the same that appear on message log code 31002.
Here is an example from our test suite that exercises the rules above. Provided that the backend returns the following JSON:
{
'salute': '()',
'nested': {
'alpha': 15,
'beta': 'karaoke',
'arr': [
0,
101,
303,
{
'inner': 10
}
]
}
}
the following template (expressed without the HTML comment marks around for clarity):
shimmercat:
content-disposition: use-json
insert: |
$.salute$.salute
and$.salute
but$$
and$;thelast
It's also possible $.nested.alpha..see
And using numbers is ok: $.nested.arr.1
Things can be very nested, as in $.nested.arr.3.inner..pixels
But some expand to empty NOT$.not.exists..EXISTS
would produce this output:
()()
and()
but$
and{"nested":{"arr":[0,101,303,{"inner":10}],"alpha":15,"beta":"karaoke"},"salute":"()"};thelast
It's also possible 15see
And using numbers is ok: 101
Things can be very nested, as in 10pixels
But some expand to empty NOTEXISTS
32.4. What else?¶
We are always looking for ways to make the web faster an easier to code, and we really need your feedback. If you would like to help us improve our content handling functionality, please contact us at ops@shimmercat.com or through our ticket system.