Table of Contents
  • Docs >
  • 32. Handling application contents
Shortcuts

32. Handling application contents

Content structure

32.1. Brokering between two peers

Here we explain how ShimmerCat’s electric domains do reverse-proxying between the Internet and one or more web applications.

ShimmerCat can use the reponse returned from an application in different ways, through a directive called content-disposition. This directive must be present as a special HTML comment (see below), it is not an HTTP header that your application must set.

The directive affects what ShimmerCat does with the contents returned in the application response, represented by the big pinkish rectangle in our header figure. The content-disposition directive can be used to instruct ShimmerCat to do things the conventional way (value replace). But it can also be used to offset the delays introduced by the backend and to simplify deployment of rich client applications (values consult and use-json), with no need for backend templates.

32.2. Sending contents verbatim from the backend

32.2.1. Example 1: A very simple NodeJS application

This is a very simple “hello-world” example, where the application backend is a JavaScript file for NodeJS. Here we show the content-disposition: replace directive, which simply sends the contents from the backend unchanged to the browser.

For illustration, we have also added a styles.css file which should be served directly by ShimmerCat.

How to serve contents from a simple NodeJS backend. Get following files we mention below in a folder, cd to it and execute your NodeJS backend with:

$ nodejs nodejs_backend.js

Then execute ShimmerCat:

$ shimmercat devlove -5 2016

and then test that it works using curl:

curl -k –socks5-hostname 127.0.0.1:2016 https://www.example.com/nodejsapp/

32.2.1.1. Files:

  • nodejs_backend.js

var http = require('http');
const PORT=8095;

const contents =
   '<html><head><link rel="stylesheet" href="/styles.css"></head>\n' +
   '<body>Made inside NodeJS</body></html>';

function handleRequest(request, response){
   response.end(contents);
}

var server = http.createServer(handleRequest);

server.listen(PORT, function(){
   console.log("Server listening on: http://localhost:%s", PORT);
});

  • devlove.yaml

shimmercat-devlove:
   domains:
       elec www.example.com:
           consultant: "8095"
           root-dir: site

  • site/styles.css

body {
   color: green;
}

  • site/nodejsapp/index.html

<!--
shimmercat:
   content-disposition: replace
-->

The NodeJS application should listen on port 8095 and ShimmerCat will connect to that port to fetch contents.

To test that this setup is working, you can point your development browser configured with SOCKS5 to the url https://www.example.com/nodejsapp/. Alternatively, you can use one of the following curl command lines:

# If you let ShimmerCat listen for SOCKS5 at port 2006
$ curl -k --socks5-hostname 127.0.0.1:2006 https://www.example.com/nodejsapp/
...
# Or if you prefer to set the Host header:
$ curl -HHost:www.example.com https://127.0.0.1:4043/nodejsapp/
<html><head><link rel="stylesheet" href="/styles.css"></head>
<body>Made inside NodeJS</body></html>

We are requesting the URL path nodejsapp/ in our curl command, and for that to work, notice the presence of the index.html file just below that directory in in the root-dir location. This file is the one containing the content-disposition directive as part of a special HTML comment. To know more in general about our mapping from URL paths to filesystem paths, check the section on URL path mapping in our article about application structure.

32.2.2. Streaming and buffering

With content-disposition: replace ShimmerCat will exceptionally buffer contents from the application, but as a general rule, it will relay bytes as soon as they are received. This is because most application frameworks already take good care of doing buffering when it makes sense, and additional buffering from ShimmerCat is more likely to get in the way than to help.

The only cases when ShimmerCat will do buffering is when the application backend sets a Content-Length HTTP header but sends plain, uncompressed contents. If ShimmerCat knows that the client supports compression, it will buffer the contents, compress them, and set a new Content-Length with the adjusted length for the compressed contents.

If the application does not set a Content-Length header but sends plain, uncompressed contents, and ShimmerCat knows that the client supports compressed contents, it will compress the contents on-the-fly. In any other case, ShimmerCat will just relay the contents unchanged from the backend as they arrive. In all cases, if ShimmerCat can’t provide a Content-Length header to the client, HTTP/1.1 clients will receive chunked-encoded contents.

32.3. Using JSON data from the backend

32.3.1. Example 2: Making your site SEO-friendly

Many web applications can be built as a set of static files that once loaded in the browser will fetch any necessary data via AJAX, websockets or even use local storage.

It turns out this is not good for sites whose contents you want to have indexed or whose URLs you want to share in social networks. Some services like Google Bot are just fine with contents and metadata set on the page via Javascript, but we have seen Twitter, LinkedIn and Facebook fail misserably at the same task.

Solving this problem and also saving round-trips is what motivated us to introduce content-disposition: use-json in version 1.5. The directive enables a very simple and efficient form of HTML templates using JSON data from the backend. It helps in simple cases where one needs to splice a few strings in the HTML and to embed JSON data. It should fit well frontend frameworks like AngularJS and EmberJS which take care of rendering contents in the client, by enabling the backend to just speak JSON. Notice however that it is not a replacement for a full-fledged template engine.

With the use-json value, the backend response is interpreted as JSON, and the index.html or __index.html pages are used as templates. These templates are compiled and stored in ShimmerCat’s transparent cache.

Here again we use directives embedded in HTML comments with a special form. Namely, contents that form a valid YAML for an object holding a single shimmercat key. This allows to use the same HTML files with other web servers (which of course we don’t recommend ;-) ).

Our tiny template language can do two things: intersperse string values in the HTML, and express JSON literal objects directly in the document, presumably inside a <script> tag. In the first case, the string is inserted literally in the document, and in the second, ShimmerCat escapes as necessary to create contents which are allowed inside <script> tags, as specified in section 4.11.1.2 of the HTML5 script recommendation.

Both facilities are shown in action in the example below:

This example is very similar to the the previous one. Get the files we mention below in a folder, cd to it and execute your NodeJS backend with:

$ nodejs nodejs_backend.js

Then execute ShimmerCat:

$ shimmercat devlove -5 2016 -w

and then test that it works using curl:

$ curl  -k --socks5-hostname 127.0.0.1:2016 https://www.example.com/article/my-dynamic-article

Of course, you can also connect with a browser and examine the results.

If everything goes alright, the HTML that you get should be similar to this one:

<html>
   <head>
       <title>
This is the article for /articles/my-dynamic-article/
        </title>

   </head>
   <body>
       <div id="content"></div>
   <script src="https://crash.shimmercat.com/js/marked.min.js?vh=1e56f4bc6"></script>
   <script>
       var contents=
{"markdown":"## Title: an article at the URL Path /articles/my-dynamic-article/ ...\n\nLorem ipsum....litle HTML comment comes: \u003c!-- html comment --\u003e."};null;
       if ( contents != null )
       {
           document.getElementById('content').innerHTML =
               marked(contents.markdown);
       } else {
           // If you want your application to also work without ShimmerCat,
           // use this branch to fetch the contents using XHR.
           //
           // ... use promises and such ...
           //
           //
       }
   </script>
   </body>
</html>

32.3.1.1. Files:

  • site/articles/__index.html

<html>
 <head>
     <title>
<!--shimmercat:
 insert: $.metadata.title
-->
      </title>

 </head>
 <body>
     <div id="content"></div>
 <script
     src="https://crash.shimmercat.com/js/marked.min.js?vh=1e56f4bc6">
 </script>
 <script>
     var contents=
<!--shimmercat:
 insert: $.contents;
-->null;
     if ( contents != null )
     {
         document.getElementById('content').innerHTML =
             marked(contents.markdown);
     } else {
         // If you want your application to also work without ShimmerCat,
         // use this branch to fetch the contents using XHR.
         //
         // ... use promises and such ...
         //
         //
     }
 </script>
 </body>
</html>

<!--
shimmercat:
 content-disposition: use-json
-->

  • nodejs_backend.js

var http = require('http')
  , url = require('url');

const PORT=8095;

function handleRequest(request, response){
   var url_parts = url.parse(request.url);
   var contents = JSON.stringify({
       metadata: {
           title: "This is the article for " + url_parts.pathname
       },
       contents: {
           markdown: "## Title: an article at the URL Path " + url_parts.pathname + " ...\n\n" +
                     "Lorem ipsum...." +
                     "litle HTML comment comes: <!-- html comment -->."
       }
   });

   response.end(contents);
}

var server = http.createServer(handleRequest);

server.listen(PORT, function(){
   console.log("Server listening on: http://localhost:%s", PORT);
});

  • devlove.yaml

shimmercat-devlove:
   domains:
       elec www.example.com:
           consultant: "8095"
           root-dir: site

32.3.2. Micro-template syntax for use-json

The syntax for the value at the insert key inside the YAML is very simple:

  • Use a dollar sign $ to refer to the root JSON object.

  • Use a dot to access a member of a JSON object. For example, $.mykey will get you the value 1 when applied to the object {"mykey":1}. Keys are valid if they are a sequence of latin alphanumeric characters, digits, underscore (_) and dash (-). You can use more dots and keys to access deeper into the JSON object passed from the backend.

  • If at some point you try to access a key that doesn’t exist , the access expression yields an empty string.

  • If you want to access elements of an array, use a key which is a valid integer, as in $.0. We use zero-based indexing.

  • An expression to access a value in the JSON object is parsed as further to the right as possible. Parsing ends at the first space or at any other character which is not a dot or a valid part of a key.

Because it is possible to combine object access with literal contents in the same place, as in

<!--
shimmercat:
 insert: There are $.page_count pages
-->

we also have a few additional facilities:

  • Use two dollar signs in the template $$ to get a literal $ in the output.

  • If your literal contents include characters which are valid parts of a key and you want to write them immediately after an access expression with no intervening space, use two dots to terminate the access expression.

  • Use a dollar sign followed by an ampersand to query for a piece of runtime information implicitly provided by ShimmerCat. Right now, we have only one: $&backend-time, which outputs how much time the backend took to produce an answer. The contents of this variable are exactly the same that appear on message log code 31002.

Here is an example from our test suite that exercises the rules above. Provided that the backend returns the following JSON:

{
    'salute': '()',
    'nested': {
        'alpha': 15,
        'beta': 'karaoke',
        'arr': [
            0,
            101,
            303,
            {
                'inner': 10
            }
        ]
    }
}

the following template (expressed without the HTML comment marks around for clarity):

shimmercat:
content-disposition: use-json
insert: |
    $.salute$.salute
    and$.salute
    but$$
    and$;thelast
    It's also possible $.nested.alpha..see
    And using numbers is ok: $.nested.arr.1
    Things can be very nested, as in $.nested.arr.3.inner..pixels
    But some expand to empty NOT$.not.exists..EXISTS

would produce this output:

()()
and()
but$
and{"nested":{"arr":[0,101,303,{"inner":10}],"alpha":15,"beta":"karaoke"},"salute":"()"};thelast
It's also possible 15see
And using numbers is ok: 101
Things can be very nested, as in 10pixels
But some expand to empty NOTEXISTS

32.4. What else?

We are always looking for ways to make the web faster an easier to code, and we really need your feedback. If you would like to help us improve our content handling functionality, please contact us at ops@shimmercat.com or through our ticket system.