ZeroSharp

Robert Anderson's ones and zeros

Serverless Framework - Part 3: The Guts of a Serverless Service

| Comments

This is part of an ongoing series about the Serverless framework. For those following along, part 1 and part 2 have been updated for the current latest version of Serverless 0.3.1.

In this post, we’ll discuss how a Serverless function actually works.

The guts of a serverless function

When we visited the deployed endpoint at the end of part 1, it correctly returned some JSON content.

1
2
3
{
    message: "Your Serverless function ran successfully!"
}

Where does this message come from? Look at index.js in the component’s lib folder.

nodejscomponent/lib/index.js
1
2
3
4
5
6
7
8
9
10
11
12
/**
 * Lib
 */

module.exports.respond = function(event, cb) {

  var response = {
    message: "Your Serverless function ran successfully!"
  };

  return cb(null, response);
};

And it’s the handler.js file in the function’s subfolder which calls it.

nodejscomponent/potd/check/handler.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
'use strict';

/**
 * Serverless Module: Lambda Handler
 * - Your lambda functions should be a thin wrapper around your own separate
 * modules, to keep your code testable, reusable and AWS independent
 * - 'serverless-helpers-js' module is required for Serverless ENV var support.  Hopefully, AWS will add ENV support to Lambda soon :)
 */

// Require Serverless ENV vars
var ServerlessHelpers = require('serverless-helpers-js').loadEnv();

// Require Logic
var lib = require('../../lib');

// Lambda Handler
module.exports.handler = function(event, context) {

  lib.respond(event, function(error, response) {
    return context.done(error, response);
  });
};

In our case we’re coding a password checking function. The URI will look something like this:

http://something.amazonaws.com/development/potd/check?password=P455w0rd

We’ll modify lib/index.js to retrieve the value from the query parameter password and return true if the password is correct and false otherwise. But first we need to set up the parameter.

Configuring the function parameter

In each function’s directory, there is a file named s-function.json which allows you to specify the details of the function call. Add a section to the requestTemplates as follows:

nodejscomponent/potd/check/s-function.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
{
  "name": "check",
  "handler": "potd/check/handler.handler",
  "timeout": 6,
  "memorySize": 1024,
  "custom": {
    "excludePatterns": [],
    "envVars": []
  },
  "endpoints": [
    {
      "path": "potd/check",
      "method": "GET",
      "authorizationType": "none",
      "apiKeyRequired": false,
      "requestParameters": {},
      "requestTemplates": {
+       "application/json": {
+           "password": "$input.params('password')"
+       }
      },
      "responses": {
        "400": {
          "statusCode": "400"
        },
        "default": {
          "statusCode": "200",
          "responseParameters": {},
          "responseModels": {},
          "responseTemplates": {
            "application/json": ""
          }
        }
      }
    }
  ]
}

Note that the s-function.json file is also where you can configure if the service accepts POST or PUT or DELETE requests. Here we are only interested in GET.

You can easily tailor the requestTemplates in this file to extract whatever parameters you need in your lambda function.

Retrieving the parameter value

Now back in index.js you will find that the function’s event parameter has a property password which is set to the value of the querystring parameter.

nodejscomponent/lib/index.js
1
2
3
4
5
6
7
8
9
module.exports.respond = function(event, cb) {

  var parameterValue = event.password; // the querystring parameter
  var response = {
    message: parameterValue
  };

  return cb(null, response);
};

Redeploy.

 $ serverless dash deploy

Visit the URI.

http://something.amazonaws.com/development/potd/check?password=P455w0rd

The response is:

1
2
3
{
    message: "P455w0rd"
}

Ready for implementation

We now have all the pieces we need. Serverless, Typescript, Mocha and AWS. In the next post I’ll show how to wire up everything get it working.

Serverless Framework - Part 2: TypeScript and Mocha

| Comments

Happy New Year everyone!

This is the second part of a series about the Serverless framework. Read the first part to get up and running.

First I’ll describe the webservice I’m building. Then we’ll configure our environment for Typescript and Mocha testing.

Poor man’s dual factor authentication via a password of the day

I’m the technical lead for an enterprise application which is in use by about 100 large multinational corporates. As part of the installation process, we ask for a registration code which is based on the date. The customer has to call us to get the password of the day. This gives us an opportunity to engage with the customer and also gives us little more control. It’s a simple form of dual factor authorization where one of the factors requires a phone call.

In the old days, the routine for checking the validity of the password was part of the source code, but we’ve since moved the checking function to a web service.

It’s my goal to replace this ‘password of the day’ check function with a Serverless module. The service will take a password as input and check that it matches the password of the day.

It’s a tiny, simple, rarely-used web service but AWS lambda is still a great fit for it. Although lambda can scale if necessary, in this case it’s about not having the hassle of administering a server.

Mocha and TypeScript

New version 0.3.1

Edit: since the original version of this post, a new version 0.3.1 of Serverless was released. I have updated the tutorial below to reflect the newer version. Also, TSD has been deprecated in favour of Typings so I’ve updated to use Typings instead.

Let’s do things properly and set up a testing framework.

Make sure you’re in the component folder.

$ cd nodejscomponent

Then we’ll install Mocha.

$ npm install mocha --save-dev
mocha@2.4.5 node_modules/mocha
├── escape-string-regexp@1.0.2
├── commander@2.3.0
├── diff@1.4.0
├── supports-color@1.2.0
├── growl@1.8.1
├── debug@2.2.0 (ms@0.7.1)
├── mkdirp@0.5.1 (minimist@0.0.8)
├── jade@0.26.3 (commander@0.6.1, mkdirp@0.3.0)
└── glob@3.2.3 (inherits@2.0.1, graceful-fs@2.0.3, minimatch@0.2.14)

Next we’ll install TypeScript. Of course you can use plain javascript if you prefer. My background is C#: I make fewer dumb mistakes with TypeScript. It looks like there is a typescript plugin in the pipeline which will make typescript integration even easier in the future, but for now:

$ npm install typescript --save
typescript@1.7.5 node_modules/typescript

$ npm install typings -g
/usr/local/bin/typings -> /usr/local/lib/node_modules/typings/dist/bin/typings.js
typings@0.6.6 /usr/local/lib/node_modules/typings
├── array-uniq@1.0.2
├── elegant-spinner@1.0.1
├── thenify@3.2.0
├── popsicle-status@1.0.1
... etc ...

Initialise typings.

$ typings init
-> written typings.json

Now we add the type definitions for Mocha.

$ typings install mocha --ambient --save
? Found mocha typings for DefinitelyTyped. Continue? Yes
Installing mocha@~2.2.5 (DefinitelyTyped)...

mocha
└── (No dependencies)

We need a config file for the TypeScript compiler. This goes in the same module directory (back/modules/potd in my case).

tsconfig.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{
    "compilerOptions": {
        "module": "commonjs",
        "target": "es5",
        "noImplicitAny": true,
        "sourceMap": false,
        "declaration": false,
        "outDir": "lib"
    },
    "exclude": [
        "node_modules",
        "typings/browser",
        "typings/browser.d.ts"
    ]
}

Now make a subdirectory for our TypeScript source files.

$ mkdir src
$ mkdir src/test

Next up we need to modify the package.json file to add a scripts section. Only the "scripts" section needs changing.

package.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
{
  "name": "potd",
  "version": "0.0.1",
  "description": "Dependencies for a Password of the day Serverless Module",
  "author": "Robert Anderson",
  "license": "MIT",
  "private": true,
  "repository": {
    "type": "git",
    "url": "git://github.com/"
  },
  "keywords": [],
  "devDependencies": {
    "mocha": "^2.3.4"
  },
  "dependencies": {
    "serverless-helpers-js": "~0.0.3",
    "typescript": "^1.7.5"
  },
  "scripts": {
    "prepublish": "tsc",
    "pretest": "tsc",
    "test": "mocha ./lib/test"
  }
}

We’ve finished setting up everything for TypeScript and Mocha. Whenever you run tsc, any TypeScript files in /src will get compiled to javascript in /lib. And running npm test will compile and then run any Mocha tests in /lib/test.

The source code so far is on GitHub. Note the default .gitignore file skips the admin.env file which contains the (sensitive) AWS keys in it so don’t forget to add your own.

In the next post we’ll create a TypeScript class for the guts of the lambda function which checks the password of the day, along with some corresponding Mocha tests, also written in TypeScript.

Serverless Framework - Part 1: Up and Running

| Comments

New version 0.3.1

Edit: since the original version of this post, a new version 0.3.1 of Serverless was released. I have updated the tutorial below to reflect the newer version.

I was in the middle of a blog post about the JAWS framework and before I had finished it changed its name to the Serverless framework. It is a very clever way to build apps without worrying about provisioning server or whether it will scale. This is because it uses Amazon Web Services and in particular the Amazon lambda compute service. It’s currently in beta.

Follow the instructions for setting up an administrative IAM user for use with the framework.

Make sure you have node and npm installed. You need node 4.0 or greater.

$ node -v
v4.2.3
$ npm -v
2.14.7

Install the Serverless framework.

$ npm install serverless -g 

Create a new project

$ serverless project create
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
 _______                             __
|   _   .-----.----.--.--.-----.----|  .-----.-----.-----.
|   |___|  -__|   _|  |  |  -__|   _|  |  -__|__ --|__ --|
|____   |_____|__|  \___/|_____|__| |__|_____|_____|_____|
|   |   |             The Serverless Application Framework
|       |                           serverless.com, v0.3.1
`-------'

Serverless: Initializing Serverless Project...  
Serverless: Enter a name for this project:  (serverless-vyedql) serverlessPotd
Serverless: Enter a universally unique project bucket name:  (serverless-vyedql-4jboce.com) potd-zerosharp.com
Serverless: Enter an email to use for AWS alarms:  (me@serverless-vyedql.com) potd@nosredna.com
Serverless: Select a region for your project: 
    us-east-1
    us-west-2
  > eu-west-1
    ap-northeast-1
Serverless: Select an AWS profile for your project: 
  > default
Serverless: Creating stage "dev"...  
Serverless: Creating region "eu-west-1" in stage "dev"...  
Serverless: Creating your project bucket on S3: serverless.eu-west-1.potd-zerosharp.com...  
Serverless: Deploying resources to stage "dev" in region "eu-west-1" via Cloudformation (~3 minutes)...  
Serverless: Successfully deployed "dev" resources to "eu-west-1"  
Serverless: Successfully created region "eu-west-1" within stage "dev"  
Serverless: Successfully created stage "dev"  
Serverless: Successfully initialized project "serverlessPotd"  

It takes about 3 minutes to setup the necessary CloudFormation stack for your project. Change directory to the newly created project.

$ cd serverlessPotd

Create a new component.

$ serverless component create
1
2
3
4
5
6
7
8
9
10
11
Serverless: Enter a name for your new component:  (nodejscomponent) 
Serverless: Enter a name for your component's first module:  (resource) potd
Serverless: Enter a name for your module's first function:  (show) check
Serverless: Successfully created function: "check"  
Serverless: Successfully created new serverless module "potd" inside the component "nodejscomponent"  
Serverless: Installing "serverless-helpers" for this component via NPM...  
Serverless: -----------------  
serverless-helpers-js@0.0.3 node_modules/serverless-helpers-js
└── dotenv@1.2.0
Serverless: -----------------  
Serverless: Successfully created new serverless component: nodejscomponent  

This has created the javascript code for a basic lambda function which we can immediately deploy.

$ serverless dash deploy

At the prompt select both the function and the endpoint and then select Deploy.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
 _______                             __
|   _   .-----.----.--.--.-----.----|  .-----.-----.-----.
|   |___|  -__|   _|  |  |  -__|   _|  |  -__|__ --|__ --|
|____   |_____|__|  \___/|_____|__| |__|_____|_____|_____|
|   |   |             The Serverless Application Framework
|       |                           serverless.com, v0.3.1
`-------'

Use the <up>, <down>, <pageup>, <pagedown>, <home>, and <end> keys to navigate.
Press <enter> to select/deselect, or <space> to select/deselect and move down.
Press <ctrl> + <enter> to immediately deploy selected.


Serverless: Select the assets you wish to deploy:
    nodejscomponent - potd - check
      function - nodejscomponent/potd/check
      endpoint - nodejscomponent/potd/check@potd/check~GET
    - - - - -
  > Deploy

Serverless: Deploying functions in "dev" to the following regions: eu-west-1  
Serverless: ------------------------  
Serverless: Successfully deployed functions in "dev" to the following regions:   
Serverless: eu-west-1 ------------------------  
Serverless:   nodejscomponent/potd/check: arn:aws:lambda:eu-west-1:962613113552:function:serverlessPotd-nodejscomponent-potd-check:dev  

Serverless: Deploying endpoints in "dev" to the following regions: eu-west-1  
Serverless: Successfully deployed endpoints in "dev" to the following regions:  
Serverless: eu-west-1 ------------------------  
Serverless:   GET - potd/check - https://rhnjv4ms2b.execute-api.eu-west-1.amazonaws.com/dev/potd/check  

Now open a browser and navigate to the URL in the last line. You should see the following JSON response.

1
2
3
{
    message: "Your Serverless function ran successfully!"
}

Now that’s already fantastic. With a handful of commands we have deployed an arbitrary javascript function to a URL endpoint very cheaply and with automatic scaling. We never had to consider instance size or memory or operating system.

And it’s extremely extensible too - thanks to the other AWS services, we can easily make it secure (with Amazon Cognito) kick off emails (SES), store files (S3), add persistence (DynamoDB), etc.

In the next post, I’ll be applying it to a real life scenario to replace an existing web service.

DevExpress 2015.2 Review Part 2

| Comments

This is the second and final dive into some of the new DevExpress XAF 2015.2 features. The first part covers the Report Designer and the new XML serialisation.

Batch editing

Another feature I’m excited about is the support for batch editing within the web application grids.

Let’s see what happens when combined with the validation rules. What happens if I edit two rows but only one has a validation problem - does the whole batch get rejected? Or just the row with the problem?

First I modified the edit mode of the Tasks grid to Batch via the model.

Then I added a new RuleRequiredField validation rule via the model.

I started up the MainDemo application and navigated to the Tasks view and tried to delete the subject of multiple rows at the same time.

I never even got to press Save because of another new 15.2 feature: inplace validation! The rules are being validated without a round trip to the server!

So let’s try another way. Inplace Validation does not work with all rule types, so I deleted my RuleRequiredField from the model and instead added a new RuleFromBoolProperty to the DemoTask object as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
[MemberDesignTimeVisibility(false)]
[RuleFromBoolProperty("SubjectIsRequired", 
    DefaultContexts.Save, 
    "Subject is required.", 
    UsedProperties = "Subject", 
    SkipNullOrEmptyValues = false)]
public bool IsSubjectRequired
{
    get
    {
        return !String.IsNullOrWhiteSpace(Subject);
    }
}

Now we go back to the list view for Task and try to set multiple subjects to empty.

Fantastic! This time the broken rules appear for each row and it is quite clear which message belongs to which object.

So the batch editing makes use of inplace validation when it can, but handles more complex validation rules well too. This is an excellent combination because the inpalce validation will help to make the client seem very quick and responsive.

Conclusion

This concludes my review of DevExpress 2015.2. We’ve looked in some detail at two of the most impressive new features - the way reports are serialised and the improvements to client-side validation. These changes help with performance and ease of maintenance and I’m very happy to see that the DevExpress team has focused on these areas. Looking forward to the 2016 releases!

DevExpress 2015.2 Review Part 1

| Comments

Last week, DevExpress released 2015.2.3, their second major version of the year.

There are already some good blog posts about the changes:

Rather than repeat general overviews provided in these, this two-part blog post is more of a ‘deep dive’. In particular I’ll be looking at two of the new features in the expressApp Framework (XAF). Today I’ll cover the new XML serialisation in the report designer. Tomorrow’s post will examine the new batch editing features.

Reports

Reporting is definitely one of the areas where XAF has progressed the most in recent versions. There is now an in-browser report designer (since 14.2) and an alternative implementation of the reports (reports v2, since 13.2).

In this release, I was a little worried that the support for Reports v1 would be deprecated, since our production system has over 100 custom reports and we have not yet looked for an easy way to migrate these to v2. I’m happy to report that v1 reports are still very much present.

However there are also a number of new features in v2 Reports, not least of which is the ability to store the report’s layout in XML. There’s currently not a lot of documentation about the XML serialization, so let’s dig in and see what we can discover.

XML serialization deep dive

First I ran the demo and ran the Copy Predefined Report on the Contacts Report.

Then I ran the report designer and added a dummy OnBeforePrint() script to one of the table’s cells.

1
2
3
private void xrTableCell8_BeforePrint(object sender, System.Drawing.Printing.PrintEventArgs e) {
  var x = 23;
}

Then, in the MainDemo’s Updater.cs file, I placed the following code.

1
2
3
4
5
6
7
var reportData = ObjectSpace.FindObject<ReportDataV2>(new BinaryOperator("DisplayName", "Contacts Report") & new BinaryOperator("IsPredefined", new OperandValue(false)));
if (reportData != null)
{
    var report = ReportDataProvider.ReportsStorage.LoadReport(reportData);
    report.SaveLayoutToXml(@"C:\Temp\ContactsReport.xml");
    report.SaveLayout(@"C:\Temp\ContactsReport.repx");
}

This creates two output files. One of them contains the familiar .repx format. The other contains the newer .xml serialization.

The contents of the XML file is displayed below and is 77 lines long.

By comparison, the ContactReport.repx is 408 lines long and much harder to read.

As you can see the XML file is much smaller and simpler than the .repx file. At first I didn’t believe it contained all the necessary information, so I started up the MainDemo WinForms application, created a blank new report and imported the layout and the layout looks correct and the preview loads with data as expected.

Scripts

What about scripts? Are they serialized in the xml version? You bet. In the XML export you can see this has been serialised near the top of the file.

1
2
3
ScriptsSource=
   "&#xD;&#xA;private void xrTableCell8_BeforePrint(object sender, System.Drawing.Printing.PrintEventArgs e)
   {&#xD;&#xA;  var x = 23;&#xD;&#xA;}&#xD;&#xA;"

Setup

In order to use this new XML serialization of report layouts, you must set the ReportsV2 module to use it.

1
reportsModuleV2.ReportStoreMode = ReportStoreModes.XML

This will most likely invalidate any reports which have already been serialized to the database. There are some notes on how to fix this here.

Default for new projects

What about new projects? I created a new solution and chose DevExpress 15.2 XAF Solution Wizard as the type of solution and added the reports module. Now when I navigate to the WinApplication.Designer.cs file, I find:

1
2
3
4
5
//
// reportsModuleV2
//
this.reportsModuleV2.ReportDataType = typeof(DevExpress.Persistent.BaseImpl.ReportDataV2);
this.reportsModuleV2.ReportStoreMode = DevExpress.ExpressApp.ReportsV2.ReportStoreModes.XML;

So the default storage method for new projects is now XML.

Conclusion

The XML serialisation looks like a considerable upgrade to the mechanism for storing, loading and saving reports. Now I just need to find a good way of converting my existing v1 reports… Perhaps a future blog post.

Coming up

Tomorrow I’ll be looking at the new batch editing in grids in more detail.

Migrating a Large Web Application From XAF 12.1 to 15.1 - Part 3

| Comments

This is the third part of a series about migrating a large application from XAF 12.1 to XAF 15.1. In this part I will compare the results of a simple stress test between the versions.

I have described in previous posts how to stress test XAF applications. One of our most basic tests is to simulate 25 users cycling through all the navigation tabs for an hour. I’m happy to report there is a considerable improvement under load in version 15.1.

(Note that we purposefully stress test against a single web application so that we can compare apples with apples. In production we have multiple instances load-balanced.)

Here is an interactive summary of the 15.1 results:

Here is version 15.1. There were zero errors and 382 completed scripts.

By comparison, the same test against DevExpress 12.1 yielded only 258 completions. So 15.1 shows a 48% performance improvement over 12.1.