ZeroSharp

Robert Anderson's ones and zeros

A Concrete PHP Serverless Example - Export Chess Games in PDF

| Comments

In the last post I built a PHP capable sample project for the Serverless Framework. In this post, I’ll show a concrete use of it.

The service I’m building connects runs a PHP function for pretty-printing chess games from the lichess online chess server. James Clarke has written a PHP function to do this using fpdf17.

The lichess exporter takes the game id of any game that has been played on the lichess server and produced a PDF output. Take for example, Game 8 of the current World Championship which is here. When I open the resulting file, I see this:

In this blog post I’ll describe how I turned this into a serverless service. The goal is to create:

  • Add an endpoint which takes the game id as a parameter
  • Run the PHP function via an AWS lambda function
  • Return the result as a stream

Prerequisites

First check everything we need is installed.

$ serverless --version
1.2.1
$ node --version
v7.1.0

Initial setup

$ mkdir serverless-lichess-to-pdf
$ cd serverless-lichess-to-pdf
$ sls install --url https://github.com/ZeroSharp/serverless-php

Next copy in the source from https://github.com/clarkerubber/lichessPDFExporter.

You can check it works by running the following.

$ php main.php COQChpzH > COQChpzH.pdf

What’s going on here? The php binary (from the serverless-php project) is running main.php (from the lichess-pdf-exporter project) with argument COQChpzH (which corresponds to a chess game on the lichess server. The main.php function downloads the game from the lichess API and passes it through the fpdf17 library to create a pdf stream which is written out to the COQChpzH.pdf file.

Lessons learned

I learned a few things while trying to get this project working. The basic plan is to modify handler.js so that it return the output of the call described above. Turns out there are quite a few gotchas along the way.

Lesson 1 - Defining a path parameter

I want my API to look like this:

http://.../serverless-lichess-to-pdf/export/{gameid}

I could not find an example in the serverless docs for getting a parameter that is passed in the URL.

Turns out your serverless.yml file should look like this:

serverless.yml
1
2
3
4
5
6
7
functions:
  exportToPdf:
    handler: handler.exportToPdf
    events:
      - http:
          path: export/{gameid}
          method: get

Then, in your handler.js you can retrieve the parameter with:

1
2
3
4
module.exports.exportToPdf = (event, context, callback) => {
  var gameid = event.pathParameters.gameid;
  // etc...
}

Lesson 2 - API Gateway does not support binary data

I was hoping I could just do something like this:

handler.js
1
2
3
4
5
6
7
8
// this does NOT work
const response = {
    statusCode: 200,
    body: outputFromPhpCall,
    content-type: "application/pdf"
};

return callback(null, response);

At present, you cannot return a binary file. Amazon have just (November 2016) released support for binary types in API Gateway but it’s currently an open issue in the Serverless Framework.

Lesson 3 - You can redirect the response to an S3 bucket

So instead of returning the binary output, I can write the output to an S3 bucket and return a 302 redirection to the S3 resource. Like this:

handler.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
// body contains the output from the PHP call
const params = {
    Bucket: bucket,
    Key: key,
    ACL: 'public-read-write',
    Body: body,
    ContentType: 'application/pdf'
};

// Save the pdf file to S3    
s3.putObject(params, function(err, data) {
if (err)
{
    return callback(new Error(`Failed to put s3 object: ${err}`));
}

// respond with a 302 redirect to the PDF file
const response = {
    statusCode: 302,
    headers: {
        location : `https://s3-eu-west-1.amazonaws.com/${bucket}/${key}`
    }
};

return callback(null, response);

Lesson 4 - You can automatically delete S3 objects after a number of days

Each S3 bucket has optional lifecycle rules where you can specify that files are automatically removed after a time period. I wanted to set this up within the serverless.yml resources section, but the syntax for the lifecycle rules were not very obvious and I could not find any examples online. The following seems to work:

serverless.yml
1
2
3
4
5
6
7
8
9
10
11
resources:
  Resources:
    PackageStorage:
      Type: AWS::S3::Bucket
      Properties:
        AccessControl: PublicRead
        BucketName: ${self:custom.exportToPdfBucket}
        LifecycleConfiguration:
          Rules:
            - ExpirationInDays: 1
              Status: Enabled

It’s all working now

You can check it out by visiting this link.

The source code is on Github.

I also wrote a Chrome extension which injects the link into the lichess page.

The Serverless Framework and PHP

| Comments

The goal of this post is to explain how to call a PHP function from within an AWS lambda using the Serverless Framework.

Prerequisites

First check everything we need is installed.

$ serverless --version
1.1.0
$ node --version
v7.1.0

Install the sample PHP function

Install my sample Hello function from my github repository.

$ sls install --url https://github.com/ZeroSharp/serverless-php
1
2
Serverless: Downloading and installing "serverless-php"Serverless: Successfully installed "serverless-php".

The code

$ cd serverless-php

Let’s have a look at the serverless.yml file.

serverless.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
service: serverless-php

provider:
  name: aws
  runtime: nodejs4.3
  # region: eu-west-1

functions:
  hello:
    handler: handler.hello
    events:
      - http:
          path: hello
          method: get

Now look at the php function index.php that we’d like our lambda to call.

index.php
1
2
3
4
5
6
<?php

# $argv will contain the event object. You can output its contents like this if you like
#var_export($argv, true);

printf('Go Serverless v1.0! Your PHP function executed successfully!');

And the handler.js for the hello function looks as follows. It defines a simple lambda which calls the PHP binary, logs any errors and returns the result.

handler.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
'use strict';

var child_process = require('child_process');

module.exports.hello = (event, context, callback) => {

  var strToReturn = '';

  var php = './php';

  // workaround to get 'sls invoke local' to work
  if (typeof process.env.PWD !== "undefined") {
    php = 'php';
  }

  var proc = child_process.spawn(php, [ "index.php", JSON.stringify(event), { stdio: 'inherit' } ]);

  proc.stdout.on('data', function (data) {
    var dataStr = data.toString()
    // console.log('stdout: ' + dataStr);
    strToReturn += dataStr
  });

  // this ensures any error messages raised by the PHP function end up in the logs
  proc.stderr.on('data', function (data) {
    console.log(`stderr: ${data}`);
  });

  proc.on('close', function(code) {
    if(code !== 0) {
      return callback(new Error(`Process exited with non-zero status code ${code}`));
    }

    const response = {
      statusCode: 200,
      body: JSON.stringify({
        message: strToReturn,
        //input: event,
      }),
    };

    callback(null, response);
  });
};

Included is the PHP binary to bundle with our serverless function.

(You may need to compile it yourself with different options. See below for help on how to do this.)

Check it works from your shell.

$ php index.php
1
Go Serverless v1.0! Your PHP function executed successfully!

Run it locally through the Serverless Framework.

$ sls invoke local --function hello
1
2
3
4
5
6
Serverless: Your function ran successfully.

{
    "statusCode": 200,
    "body": "{\"message\":\"Go Serverless v1.0! Your PHP function executed successfully!\"}"
}

Looks good. Let’s deploy.

$ sls deploy
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Serverless: Packaging service…
Serverless: Uploading CloudFormation file to S3…
Serverless: Uploading service .zip file to S3…
Serverless: Updating Stack…
Serverless: Checking Stack update progress…
..........
Serverless: Stack update finished…

Service Information
service: serverless-php
stage: dev
region: eu-west-1
api keys:
  None
endpoints:
  GET - https://c1w0hct166.execute-api.eu-west-1.amazonaws.com/dev/hello
functions:
  serverless-php-dev-hello: arn:aws:lambda:eu-west-1:962613113552:function:serverless-php-dev-hello

Run the remote function via Serverless.

$ sls invoke --function hello
1
2
3
4
{
    "statusCode": 200,
    "body": "{\"message\":\"Go Serverless v1.0! Your PHP function executed successfully!\",\"input\":{}}"
}

Visit the endpoint in your browser.

1
2
3
{
    "message": "Go Serverless v1.0! Your PHP function executed successfully!"
}

Nice. It’s all working.

Rebuilding the PHP binary

Depending on the PHP function you need to run, it may be necessary to rebuild the php binary with different flags and dependencies. You can do this best with docker.

$ docker --version
Docker version 1.12.3, build 6b644ec

Modify dockerfile.buildphp as necessary.

Then run:

$ sh buildphp.sh

This will build a new PHP binary and copy it to the project root. You can immediately deploy for testing with:

$ sls deploy

Thanks

Shout out to Danny Linden whose code got me started on this.

Smart Hiding of the Selection Boxes in XAF Web Applications

| Comments

When an XAF list view has no selection-based actions available, the selection box still appears in the grid. Users get confused. In this post, we’ll look at a workaround.

The problem

In the XAF MainDemo, lets make Departments read-only for the User role.

Updater.cs
1
2
3
userRole.AddTypePermissionsRecursively<Department>(SecurityOperations.Create, SecurityPermissionState.Deny);
userRole.AddTypePermissionsRecursively<Department>(SecurityOperations.Write, SecurityPermissionState.Deny);
userRole.AddTypePermissionsRecursively<Department>(SecurityOperations.Delete, SecurityPermissionState.Deny);

Then start the web application, login as John and navigate to the Departments list view. There is a column selection box, but it serves no purpose. There are no actions that depend on a grid selection.

Without the SelectionColumnVisibilityController

The fix

Here is a controller which calculates whether there are any available actions which require one or more rows to be selected. If there are none, the selection box will not appear.

Add the following controller to the MainDemo.Module.Web project. It hides the selection box if there are no actions which depend on a grid selection.

SelectionColumnVisibilityController.cs
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
using System;
using DevExpress.ExpressApp;
using DevExpress.ExpressApp.Actions;
using DevExpress.ExpressApp.Editors;
using DevExpress.ExpressApp.SystemModule;
using DevExpress.Web;
using System.Linq;

namespace MainDemo.Module.Web.Controllers
{
    public class SelectionColumnVisibilityController : ViewController
    {
        public SelectionColumnVisibilityController()
        {
            TargetViewType = ViewType.ListView;
        }

        private bool IsSelectionColumnVisible()
        {
            bool isSelectionColumnRequired = false;
            // remove checkbox if there are no available actions
            foreach (Controller controller in Frame.Controllers)
            {
                if (!controller.Active)
                    continue;

                if (controller.Actions.Count == 0)
                    continue;

                bool allowEdit = true;
                if ((Frame is NestedFrame) && (((NestedFrame)Frame).ViewItem is PropertyEditor))
                    allowEdit = (bool)((PropertyEditor)((NestedFrame)Frame).ViewItem).AllowEdit;

                foreach (ActionBase action in controller.Actions)
                {
                    if (action.SelectionDependencyType == SelectionDependencyType.RequireMultipleObjects)
                    {
                        if (action.Active || IsActionInactiveBySelectionContext(action))
                        {
                            if (action.Enabled || IsActionDisabledBySelectionContext(action))
                            {
                                isSelectionColumnRequired = true;
                                break;
                            }
                        }
                    }
                }
                if (isSelectionColumnRequired)
                    break;
            }
            return isSelectionColumnRequired;
        }

        private bool IsActionInactiveBySelectionContext(ActionBase action)
        {
            if (action.Active)
                return true;
            else
            {
                foreach (string item in action.Active.GetKeys())
                {
                    if (item == ActionBase.RequireMultipleObjectsContext || item == ActionBase.RequireSingleObjectContext)
                        continue;
                    if (!action.Active[item])
                        return false;
                }
                return true;
            }
        }

        private bool IsActionDisabledBySelectionContext(ActionBase action)
        {
            if (action.Enabled)
                return true;
            else
            {
                foreach (string item in action.Enabled.GetKeys())
                {
                    if (item == ActionBase.RequireMultipleObjectsContext ||
                        item == ActionBase.RequireSingleObjectContext ||
                        item == ActionsCriteriaViewController.EnabledByCriteriaKey)
                        continue;
                    if (!action.Enabled[item])
                        return false;
                }
                return true;
            }
        }

        protected override void OnViewControlsCreated()
        {
            base.OnViewControlsCreated();
            ASPxGridView grid = ((ListView)this.View).Editor.Control as ASPxGridView;
            if (grid != null)
            {
                grid.Load += grid_Load;
                grid.DataBound += grid_DataBound;
            }
        }

        protected override void OnDeactivated()
        {
            base.OnDeactivated();
            ASPxGridView grid = ((ListView)this.View).Editor.Control as ASPxGridView;
            if (grid != null)
            {
                grid.DataBound -= grid_DataBound;
                grid.Load -= grid_Load;
            }
        }

        void grid_Load(object sender, EventArgs e)
        {
            SetSelectionColumnVisibility(sender, e);
        }

        void grid_DataBound(object sender, EventArgs e)
        {
            SetSelectionColumnVisibility(sender, e);
        }

        private void SetSelectionColumnVisibility(object sender, EventArgs e)
        {
            bool isSelectionColumnVisible = IsSelectionColumnVisible();
            if (!isSelectionColumnVisible)
            {
                var grid = (ASPxGridView)sender;
                var selectionBoxColumn =
                    grid.Columns
                        .OfType<GridViewCommandColumn>()
                        .Where(x => x.ShowSelectCheckbox)
                        .FirstOrDefault();

                if (selectionBoxColumn != null)
                {
                    selectionBoxColumn.Visible = false;
                }
            }
        }
    }
}

Run the application again and see the difference. Now the grid looks like this. Notice, there is no longer a selection box on the row.

By the way, this is how it looks with old-style XAF web apps.

Without the SelectionColumnVisibilityController

With the SelectionColumnVisibilityController

Sometimes You’ve Just Got to Deploy

| Comments

Sometimes the deadline has arrived and you still have some failing tests. After a discussion with the dev team, you decide to deploy anyway and fix the bugs for the next release. You need to get the build server to ignore the tests.

One way is just to mark the test with the [Ignore] attribute.

1
2
3
4
5
6
[Test]
[Ignore] // TODO: Fix this test before the next release!
public void Test()
{
    // Some failing test code...
}

After the weekend, everyone forgets about the ignored tests and they never get fixed.

Instead, I like to do this.

1
2
3
4
5
6
7
[Test]
public void Test()
{
    if (DateTime.Now < new DateTime(2016, 10, 17))
        Assert.Ignore("Temporarily ignored until October 17.");
    // Some failing test code...
}

This is a fairly rare occurrence for my team, so the above approach is sufficient and works with all test frameworks. But if you want to go further Richard Slater shows how to create an NUnit attribute.

Boxstarter and Checksums

| Comments

I’ve blogged before about using BoxStarter to efficiently provision a new development machine.

This is working very well for our developers. Maintaining the installation script takes a bit of effort but the benefits are worth it.

Recently, the chocolatey developers have been making things more secure and recent versions now require a checksum with any downloaded package. For now, there seems to be some difficulty using Install-ChocolateyVsixPackage within BoxStarter scripts.

If I try to run a script with

1
2
3
Install-ChocolateyVsixPackage
  StopOnFirstBuildError
  https://visualstudiogallery.msdn.microsoft.com/91aaa139-5d3c-43a7-b39f-369196a84fa5/file/44205/7/StopOnFirstBuildError.vsix

I get the following error message:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
 WARNING: Missing package checksums are not allowed (by default for HTTP/FTP,
  HTTPS when feature 'allowEmptyChecksumsSecure' is disabled) for
  safety and security reasons. Although we strongly advise against it,
  if you need this functionality, please set the feature
  'allowEmptyChecksums' ('choco feature enable -n
  allowEmptyChecksums')
 There were errors attempting to retrieve the vsix from https://visualstudiogallery.msdn.microsoft.com/91aaa139-5d3c-43a
  or pass in the option '--allow-empty-checksums'.
 7-b39f-369196a84fa5/file/44205/7/StopOnFirstBuildError.vsix. The error message was 'Empty checksums are no longer
 allowed by default for non-secure sources. Please ask the maintainer to add checksums to this package. In the meantime 
 if you need this package to work correctly, please enable the feature allowEmptyChecksums or provide the runtime
 switch '--allowEmptyChecksums'. We strongly advise against allowing empty checksums for HTTP/FTP sources.'.
 At C:\ProgramData\chocolatey\helpers\functions\Install-ChocolateyVsixPackage.ps1:173 char:13

etc...

I tried the suggested allowEmptyChecksums but it was still raising an error. After some experimentation it seems that the combination of (temporarily) disabling the checksumFiles feature and enabling allowEmptyChecksums works. Here is the relevant section of my boxstarter script.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# temporarily enable/disable features to bypass checksums
choco feature disable -n=checksumFiles
choco feature enable -n=allowEmptyChecksums
try {
    # VS extensions for all versions of Visual Studio
    Install-ChocolateyVsixPackage StopOnFirstBuildError https://visualstudiogallery.msdn.microsoft.com/91aaa139-5d3c-43a7-b39f-369196a84fa5/file/44205/7/StopOnFirstBuildError.vsix
    Install-ChocolateyVsixPackage VisualHG https://visualhg.codeplex.com/downloads/get/1475782

    # VS extensions for VS2015
    Install-ChocolateyVsixPackage WebEssentials2015 https://visualstudiogallery.msdn.microsoft.com/ee6e6d8c-c837-41fb-886a-6b50ae2d06a2/file/146119/19/WebEssentials2015.vsix
    Install-ChocolateyVsixPackage WebExtensionPack https://visualstudiogallery.msdn.microsoft.com/f3b504c6-0095-42f1-a989-51d5fc2a8459/file/186606/23/Web%20Extension%20Pack%20v1.5.50.vsix
    Install-ChocolateyVsixPackage T4Toolbox2015 https://visualstudiogallery.msdn.microsoft.com/34b6d489-afbc-4d7b-82c3-dded2b726dbc/file/165481/2/T4Toolbox.14.0.0.71.vsix

    # CodeRush for Roslyn Preview
    Install-ChocolateyVsixPackage CodeRushForRoslyn https://visualstudiogallery.msdn.microsoft.com/8a8390ae-1f71-4659-9d8d-5dd56fd8a72e/file/163212/15/DevExpress.CodeRush.Roslyn-16.1.6.vsix
}
finally {
    choco feature enable -n=checksumFiles
    choco feature disable -n=allowEmptyChecksums
}

And then everything works as expected.

An XAF Workaround for ‘Prevent This Page From Creating Additional Dialogs.’

| Comments

Both Chrome and Firefox have a ‘feature’ which allows a user to ignore future calls to confirm(). Once this has been checked, any subsequent calls to confirm() return false immediately without showing the window.

In XAF, this behaviour prevents the application from working correctly. For instance, it becomes impossible to confirm a deletion.

The following controller provides a work around. It injects some javascript into the page wrapping the call to confirm(). The new code detects when the Prevent this page from creating additional dialogs checkbox has been checked and returns true instead. The confirmation window still does not appear, but the XAF application works as if the user had pressed confirm instead of cancel.

# HandleDisabledConfirmationsController.cs
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
using DevExpress.ExpressApp;
using DevExpress.ExpressApp.Web;
using DevExpress.Web.Internal;

namespace NetModule.Web.Controllers
{
    public class HandleDisabledConfirmationsController : Controller
    {
        public static bool IsChrome
        {
            get
            {
                return RenderUtils.Browser.IsChrome;
            }
        }

        public static bool IsFirefox
        {
            get
            {
                return RenderUtils.Browser.IsFirefox;
            }
        }

        protected override void OnFrameAssigned()
        {
            base.OnFrameAssigned();
            if (IsChrome || IsFirefox)
            {
                WebWindow window = Frame as WebWindow;
                if (window != null)
                {
                    window.CustomRegisterTemplateDependentScripts += window_CustomRegisterTemplateDependentScripts;
                }
            }
        }

        private void window_CustomRegisterTemplateDependentScripts(object sender, CustomRegisterTemplateDependentScriptsEventArgs e)
        {
            // wrapper for 'confirm'
            WebWindow window = (WebWindow)sender;

            // Detect the user has checked the 'prevent this page from creating additional dialogs'.
            // In which case assume all confirmations are accepted, rather than the default rejected
            var confirmWrapper = @"
<script>
window.nativeConfirm = window.confirm;
window.confirm = function(message)
{
    var timeBefore = new Date();
    var confirmBool = nativeConfirm(message);
    var timeAfter = new Date();
    if ((timeAfter - timeBefore) < 350)
    {
        confirmBool = true;
    }
    return confirmBool;
}
</script>";

            e.Page.ClientScript.RegisterClientScriptBlock(GetType(), "WrapConfirmations", confirmWrapper);
        }

        protected override void OnDeactivated()
        {
            if (IsChrome || IsFirefox)
            {
                WebWindow window = Frame as WebWindow;
                if (window != null)
                {
                    window.CustomRegisterTemplateDependentScripts -= window_CustomRegisterTemplateDependentScripts;
                }
                base.OnDeactivated();
            }
        }
    }
}

The controller works by timing the milliseconds to close the confirmation window. If it is less than 350 milliseconds we can assume the confirmation window never opened owing to the checkbox. In this scenario, it returns true (confirm) rather than false (cancel) in order for XAF to function correctly.

A Basic Chrome Extension - Analyze Your chess.com Games on lichess.org

| Comments

I’ve been trying to get better at chess by playing on chess.com. I often analyse my games with the help of a computer to see where I made mistakes. My favourite way of doing this is to make use of Lichess.org’s import functionality. The latest version of chess.com has greatly improved its own post game analysis, but I still much prefer Lichess’s. So I would often download the finished game in pgn and upload it to Lichess.

Somewhere along the way I came across a Chrome extension written by Allan Rempel which would do this in one step. Unfortunately it stopped working when chess.com upgraded its site and Allan does not seem to be maintaining it any longer. I decided to jump in and try to fix it and maybe add a couple of features myself.

The extension

When a chess.com game is finished, you can click on a little knight icon and jump straight to the analysis of the game in Lichess.

It takes a few seconds to process but then you can see all your mistakes and blunders.

You can play through the game and experiment with different variations. Chess.com’s new v3 has greatly improved its own post-game analysis but it still lacks many of the features of the Lichess one.

Try it out! The extension is available here.

How to modify an existing extension

First thing was to find the code of the existing broken extension. A bit of googling lead me to:

$ cd ~/Library/Application\ Support/Google/Chrome/Default/Extensions/
$ ls

total 0
drwx------+ 30 ra  staff  1020 29 Feb 20:39 .
drwx------+ 66 ra  staff  2244 29 Feb 20:56 ..
drwx------+  2 ra  staff    68 29 Feb 20:39 Temp
drwx------+  3 ra  staff   102 14 Sep  2012 allibancfngcpjmjnnboebggipbaalhn
drwx------+  3 ra  staff   102 28 Mar  2013 anfemphbgknehemmbbajjcmakfijiohp
drwx------+  3 ra  staff   102 25 Oct 08:15 apdfllckaahabafndbhieahigkjlhalf
drwx------+  3 ra  staff   102  5 Mar  2015 bepbmhgboaologfdajaanbcjmnhjmhfn
drwx------+  3 ra  staff   102  5 Jun  2014 bfbameneiokkgbdmiekhjnmfkcnldhhm
drwx------+  3 ra  staff   102 22 Feb 20:09 bhjlkimpkkgkmfjlcfngmakenalgleap
drwx------+  3 ra  staff   102 24 Sep 20:33 blpcfgokakmgnkcojhhkbfbldkacnbeo
drwx------+  3 ra  staff   102 29 Jul  2013 bmihblnpomgpjkfddepdpdafhhepdbek
drwx------+  3 ra  staff   102 13 Sep  2012 boeajhmfdjldchidhphikilcgdacljfm
... etc.

Each extension is in a not very helpfully named folder. If you open chrome://extensions/, you can find the id of any installed extension and work out which directory corresponds to which extension.

So then I dug around in the folder and found ext/background.js where the bulk of the code lies. There is also a manifest.json file which is specifies various parameters such as what permissions are required and which websites will be affected by the extension. I copied all the important files into a development folder.

Loading the extension from the development folder.

  • Navigate to chrome://extensions
  • Expand the developer dropdown menu and click “Load Unpacked Extension”
  • Navigate to development folder

Assuming there are no errors, the extension should load into your browser. Don’t forget to disable any conflicting versions you’ve installed from the Chrome Web Store.

I used the Chrome’s built-in Developer Tools and peppered the code with console.log() to explore how it worked. There are plenty of good resources on the Chrome developer site. I managed to fix the broken parts and added support for analysis from chess.com’s game archive. Then I posted the link on reddit/r/chess.

The source code for the extension is on GitHub.

Serverless Framework Part 5: Pulling in a Node Package

| Comments

This is the final part of an ongoing series about the Serverless framework.

In the previous posts, the PasswordGenerator always returned ‘Password’. Instead each date should corresponds to a new unique password. We’ll make use of the Crypto-js node package and we’ll see that the AWS lambda copes just fine.

Installing a node package

Pull in the crypto-js package into the serverless component.

$ cd nodejscomponent/
$ npm install crypto-js --save
crypto-js@3.1.6 node_modules/crypto-js

Now we need the typescript definitions. Watch out there are two different TypeScript typings called cryptojs and crypto-js. The first one is more complete.

$ typings install cryptojs --ambient --save
? Found cryptojs typings for DefinitelyTyped. Continue? Yes
Installing cryptojs@~3.1.2 (DefinitelyTyped)...

cryptojs
└── (No dependencies)

I’m not sure why, but there’s no export in the typings file for cryptojs. Add the following to the bottom of the cryptojs.d.ts file.

nodejscomponent/typings/main/ambient/cryptojs.d.ts
1
2
3
4
  ...
declare module "crypto-js" {
    export = CryptoJS;
}

Improved PasswordGenerator

nodejscomponent/src/passwordOfTheDay.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
/// <reference path="../typings/main.d.ts" />
import CryptoJS = require("crypto-js");

export function checkPotd(password : string) : boolean
{
    return new PasswordGenerator().check(password);
}

export class PasswordGenerator
{
  generate(date: Date) : string
  {
        // Get the current date as a YYYYMMDD string
        var yyyy = date.getFullYear().toString();
        var mm = (date.getMonth()+1).toString(); // getMonth() is zero-based
        var dd  = date.getDate().toString();
        var plain = `${yyyy}${mm}${dd}`;

        // Using AES CTR with 32 byte key and iv ensures the encrypted string is not too long
        // See http://stackoverflow.com/a/13298019/1077279

        var key = CryptoJS.enc.Hex.parse('108c786594543687891374723e809ec5e475a8361f7ad82df04e91ba2c139321');
        // Use a different initialization vector each time by using the date as part of the vector
        var iv  = CryptoJS.enc.Hex.parse(plain + '3a8fe4440be1e113a271574f379d70a76c3477aaff036d1e83fcd4b9');
        var options = { mode: CryptoJS.mode.CTR, padding: CryptoJS.pad.NoPadding, iv: iv };

        var encrypted = CryptoJS.AES.encrypt(plain, key, options);

      return encrypted.ciphertext.toString();
  }    
  
  check(password : string) : boolean
  {
        // check the value matches today's password of the day
      return password == this.generate(new Date());
  }
}

Run the tests

We expect the tests to fail now since we are no longer returning the same password.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
$ npm test
> @0.0.1 pretest /Users/ra/Projects/Coprocess/serverlessPotd/nodejscomponent
> tsc

> @0.0.1 test /Users/ra/Projects/Coprocess/serverlessPotd/nodejscomponent
> mocha ./lib/test

  Generator
    #generate
      1) should generate the password
    #check
      ✓ should return false when the password is incorrect
    #check
      2) should return true when the password is correct

  1 passing (16ms)
  2 failing

  1) Generator #generate should generate the password:
     Error: Expected 'Password' but was bb4bde4d76b055
      at Context.<anonymous> (lib/test/passwordOfTheDayTest.js:12:23)

  2) Generator #check should return true when the password is correct:
     Error: Expected 'true' but was false
      at Context.<anonymous> (lib/test/passwordOfTheDayTest.js:28:23)

npm ERR! Test failed.  See above for more details.

Update and improve the tests

It’s getting a little more complicated so let’s pull in chai which is a pretty assertions library.

1
2
3
4
5
$ npm install chai --save-dev
chai@3.5.0 node_modules/chai
├── assertion-error@1.0.1
├── type-detect@1.0.0
└── deep-eql@0.1.3 (type-detect@0.1.1)

And the Typescript definitions for chai.

1
2
3
4
5
6
$ typings install chai --save --ambient
? Found chai typings for DefinitelyTyped. Continue? Yes
Installing chai@~3.4.0 (DefinitelyTyped)...

chai
└── (No dependencies)

Now let’s flesh out the tests for the PasswordGenerator class.

nodejscomponent/src/test/passwordOfTheDayTest.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
/// <reference path="../../typings/main.d.ts" />
import PasswordOfTheDay = require("../passwordOfTheDay");
import Chai = require("chai");

// Tell chai that we'll be using the "should" style assertions.
Chai.should();

describe("Generator", () => {
    var subject : PasswordOfTheDay.PasswordGenerator;

    beforeEach(function () {
        subject = new PasswordOfTheDay.PasswordGenerator();
    });

    describe("#generate", () => {
        it("should generate the password when the date is 24th July 2010", () => {
            var date : Date = new Date(2010, 6, 24);
            var password : string = subject.generate(date);

            password.should.equal("92ab1ff89bf9af");
        });
    });

    describe("#generate", () => {
        it("should generate a different password when the date is 25th July 2010", () => {
            var date : Date = new Date(2010, 6, 25);
            var password : string = subject.generate(date);

            password.should.equal("26a394b21800f1");
        });
    });
    describe("#check", () => {
        it("should return false when the password is incorrect", () => {
            var password : string = "garbage";
            var result : boolean = subject.check(password);

            result.should.be.false;
        });
    });

    describe("#check", () => {
        it("should return false when the password is null", () => {
            var password : string = null;
            var result : boolean = subject.check(password);

            result.should.be.false;
        });
    });

    describe("#check", () => {
        it("should return true when the password is correct", () => {
            var password : string = subject.generate(new Date());
            var result : boolean = subject.check(password);

            result.should.be.true;
        });
    });
});

Run our tests:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
$ npm test
> @0.0.1 pretest /Users/ra/Projects/Coprocess/serverlessPotd/nodejscomponent
> tsc

> @0.0.1 test /Users/ra/Projects/Coprocess/serverlessPotd/nodejscomponent
> mocha ./lib/test

  Generator
    #generate
      ✓ should generate the password when the date is 24th July 2010
    #generate
      ✓ should generate a different password when the date is 25th July 2010
    #check
      ✓ should return false when the password is incorrect
    #check
      ✓ should return false when the password is null
    #check
      ✓ should return true when the password is correct


  5 passing (20ms)

Deploy

$ serverless dash deploy

Now when we visit the endpoint with the correct password for today’s date which happens to be 89366e6199f3.

https://...amazonaws.com/dev/potd/check?password=89366e6199f3
1
2
3
{
    message: true
}

Mission accomplished! Notice that using a node package did not require any special steps on the AWS side. In fact we have not had to login to AWS since the very beginning when we created an IAM user for the project. And yet we’ve managed to build and deploy a cheap and scalable cloud-based service.

The source code is on GitHub. Note the default .gitignore file skips the admin.env file which contains the (sensitive) AWS keys in it so don’t forget to add your own.

That wraps up my series on building a small, but real-world Serverless application. In a future post, I’d like to look at providing a secured ‘generate’ service to allow authorized users to get today’s password. Stay tuned.

Serverless Framework - Part 4: Connecting the Parts

| Comments

This is part of an ongoing series about the Serverless framework: Part 1, part 2, part 3.

New version 0.3.1

Edit: since the original version of this post, a new version 0.3.1 of Serverless was released. I have updated the tutorial below to reflect the newer version. Also, TSD has been deprecated in favour of Typings so I’ve updated to use Typings instead. All parts have been updated for the latest version of the framework 0.3.1.

The Password of the Day Generator class

First up we need a class to generate and check the password of the day. For the moment, let’s pretend the password of the day is always the string “Password”. Put the following typescript class in nodejscomponent/src.

nodejscomponent/src/passwordOfTheDay.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
export function checkPotd(password : string) : boolean
{
    return new PasswordGenerator().check(password);
}

export class PasswordGenerator
{
  generate(date: Date) : string
  {
        // generate today's password
      return "Password";
  }    
  
  check(password : string) : boolean
  {
        // check the value matches today's password of the day
      return password == this.generate(new Date());
  }
}

Now add a mocha test for it.

nodejscomponent/src/test/passwordOfTheDayTest.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
/// <reference path="../../typings/main.d.ts" />
import PasswordOfTheDay = require("../passwordOfTheDay");

describe("Generator", () => {
    var subject : PasswordOfTheDay.PasswordGenerator;

    beforeEach(function () {
        subject = new PasswordOfTheDay.PasswordGenerator();
    });

    describe("#generate", () => {
        it("should generate the password", () => {
            var result : string = subject.generate(new Date(2010, 6, 24));
            if (result !== "Password") {
                throw new Error("Expected 'Password' but was " + result);
            }
        });
    });

    describe("#check", () => {
        it("should return false when the password is incorrect", () => {
            var result : boolean = subject.check("garbage");
            if (result !== false) {
                throw new Error("Expected 'false' but was " + result);
            }
        });
    });

    describe("#check", () => {
        it("should return true when the password is correct", () => {
            var result : boolean = subject.check("Password");
            if (result !== true) {
                throw new Error("Expected 'true' but was " + result);
            }
        });
    });

});

Now compile everything.

$ cd nodejscomponent
$ tsc

You will now find that there is a corresponding javascript file in the lib folder

nodejscomponent/src/passwordOfTheDay.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
function checkPotd(password) {
    return new PasswordGenerator().check(password);
}
exports.checkPotd = checkPotd;
var PasswordGenerator = (function () {
    function PasswordGenerator() {
    }
    PasswordGenerator.prototype.generate = function (date) {
        // generate today's password
        return "Password";
    };
    PasswordGenerator.prototype.check = function (password) {
        // check the value matches today's password of the day
        return password == this.generate(new Date());
    };
    return PasswordGenerator;
})();
exports.PasswordGenerator = PasswordGenerator;

And likewise for the mocha test in lib/test. Now to run those tests:

$ npm test

> @0.0.1 pretest /Users/ra/Projects/Coprocess/serverlessPotd/nodejscomponent
> tsc

> @0.0.1 test /Users/ra/Projects/Coprocess/serverlessPotd/nodejscomponent
> mocha ./lib/test

Generator
    #generate
    ✓ should generate the password
    #check
    ✓ should return false when the password is incorrect
    #check
    ✓ should return true when the password is correct

3 passing (10ms)

Nice. Next, modify the main entry point of the component index.js.

nodejscomponent/lib/index.js
1
2
3
4
5
6
7
8
9
10
11
12
// Dependencies
var PasswordOfTheDay = require('./passwordOfTheDay');

module.exports.respond = function(event, cb) {

  var result = PasswordOfTheDay.checkPotd(event.password);
  var response = {
    message: result
  };

  return cb(null, response);
};

Notice how we make use of event.password which is the parameter we configured in part 3 in the s_function.json file.

Let’s deploy!

$ serverless dash deploy
_______                             __
|   _   .-----.----.--.--.-----.----|  .-----.-----.-----.
|   |___|  -__|   _|  |  |  -__|   _|  |  -__|__ --|__ --|
|____   |_____|__|  \___/|_____|__| |__|_____|_____|_____|
|   |   |             The Serverless Application Framework
|       |                           serverless.com, v0.3.1
`-------'

Use the <up>, <down>, <pageup>, <pagedown>, <home>, and <end> keys to navigate.
Press <enter> to select/deselect, or <space> to select/deselect and move down.
Press <ctrl> + <enter> to immediately deploy selected.


Serverless: Select the assets you wish to deploy:
    nodejscomponent - potd - check
    function - nodejscomponent/potd/check
    endpoint - nodejscomponent/potd/check@potd/check~GET
    - - - - -
> Deploy

Serverless: Deploying functions in "dev" to the following regions: eu-west-1  
Serverless: ------------------------  
Serverless: Successfully deployed functions in "dev" to the following regions:   
Serverless: eu-west-1 ------------------------  
Serverless:   nodejscomponent/potd/check: arn:aws:lambda:eu-west-1:962613113552:function:serverlessPotd-nodejscomponent-potd-check:dev  

And lets visit that URI

https://rhnjv4ms2b.execute-api.eu-west-1.amazonaws.com/development/potd/check?password=nonsense
1
2
3
{
    message: false
}
https://rhnjv4ms2b.execute-api.eu-west-1.amazonaws.com/development/potd/check?password=Password
1
2
3
{
    message: true
}

Rock and roll. A working password checker running on Lambda in the Amazon cloud.

Next up - we’ll extend the PasswordGenerator class to pull in a node package and generate a better password.

The source code so far is on GitHub. Note the default .gitignore file skips the admin.env file which contains the (sensitive) AWS keys in it so don’t forget to add your own.

Serverless Framework - Part 3: The Guts of a Serverless Service

| Comments

This is part of an ongoing series about the Serverless framework. For those following along, part 1 and part 2 have been updated for the current latest version of Serverless 0.3.1.

In this post, we’ll discuss how a Serverless function actually works.

The guts of a serverless function

When we visited the deployed endpoint at the end of part 1, it correctly returned some JSON content.

1
2
3
{
    message: "Your Serverless function ran successfully!"
}

Where does this message come from? Look at index.js in the component’s lib folder.

nodejscomponent/lib/index.js
1
2
3
4
5
6
7
8
9
10
11
12
/**
 * Lib
 */

module.exports.respond = function(event, cb) {

  var response = {
    message: "Your Serverless function ran successfully!"
  };

  return cb(null, response);
};

And it’s the handler.js file in the function’s subfolder which calls it.

nodejscomponent/potd/check/handler.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
'use strict';

/**
 * Serverless Module: Lambda Handler
 * - Your lambda functions should be a thin wrapper around your own separate
 * modules, to keep your code testable, reusable and AWS independent
 * - 'serverless-helpers-js' module is required for Serverless ENV var support.  Hopefully, AWS will add ENV support to Lambda soon :)
 */

// Require Serverless ENV vars
var ServerlessHelpers = require('serverless-helpers-js').loadEnv();

// Require Logic
var lib = require('../../lib');

// Lambda Handler
module.exports.handler = function(event, context) {

  lib.respond(event, function(error, response) {
    return context.done(error, response);
  });
};

In our case we’re coding a password checking function. The URI will look something like this:

http://something.amazonaws.com/development/potd/check?password=P455w0rd

We’ll modify lib/index.js to retrieve the value from the query parameter password and return true if the password is correct and false otherwise. But first we need to set up the parameter.

Configuring the function parameter

In each function’s directory, there is a file named s-function.json which allows you to specify the details of the function call. Add a section to the requestTemplates as follows:

nodejscomponent/potd/check/s-function.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
{
  "name": "check",
  "handler": "potd/check/handler.handler",
  "timeout": 6,
  "memorySize": 1024,
  "custom": {
    "excludePatterns": [],
    "envVars": []
  },
  "endpoints": [
    {
      "path": "potd/check",
      "method": "GET",
      "authorizationType": "none",
      "apiKeyRequired": false,
      "requestParameters": {},
      "requestTemplates": {
+       "application/json": {
+           "password": "$input.params('password')"
+       }
      },
      "responses": {
        "400": {
          "statusCode": "400"
        },
        "default": {
          "statusCode": "200",
          "responseParameters": {},
          "responseModels": {},
          "responseTemplates": {
            "application/json": ""
          }
        }
      }
    }
  ]
}

Note that the s-function.json file is also where you can configure if the service accepts POST or PUT or DELETE requests. Here we are only interested in GET.

You can easily tailor the requestTemplates in this file to extract whatever parameters you need in your lambda function.

Retrieving the parameter value

Now back in index.js you will find that the function’s event parameter has a property password which is set to the value of the querystring parameter.

nodejscomponent/lib/index.js
1
2
3
4
5
6
7
8
9
module.exports.respond = function(event, cb) {

  var parameterValue = event.password; // the querystring parameter
  var response = {
    message: parameterValue
  };

  return cb(null, response);
};

Redeploy.

 $ serverless dash deploy

Visit the URI.

http://something.amazonaws.com/development/potd/check?password=P455w0rd

The response is:

1
2
3
{
    message: "P455w0rd"
}

Ready for implementation

We now have all the pieces we need. Serverless, Typescript, Mocha and AWS. In the next post I’ll show how to wire up everything get it working.