How to Build a Serverless API With AWS DynamoDB, Lambda, and API Gateway

Lambda symbol

Imagine running your entire IT department or SaaS without servers. The age of serverless architecture is here now. So, what’s serverless architecture? According to Mike Roberts:

Serverless can also mean applications where some amount of server-side logic is still written by the application developer but unlike traditional architectures is run in stateless compute containers that are event-triggered, ephemeral (may only last for one invocation), and fully managed by a 3rd party. One way to think of this is Functions as a service (FaaS). AWS Lambda is one of the most popular implementations of FaaS at present, but there are others. I’ll be using ‘FaaS’ as a shorthand for this meaning of Serverless throughout the rest of this article.

Source: Roberts, M. (2016, August 4). Serverless Architectures.

At Rodax Software, we’re thinking about scrapping our AWS EC2s running our Skedi services and migrating them into serverless microservices running on AWS Lambda. That said, it’s important to have some healthy skepticism about making this transition. There are some potentially significant drawbacks to consider such as vendor lock-in; however, I’m not going address these in this post. For more information about the drawbacks click here.

Given the scope of transiting our API and potential risks, I wanted to get a better idea of the level effort and overall feasibility. The primary goal of this article is to walkthrough how to build a serverless RESTful API that’s integrated with Lambda and uses a NoSQL database as its data store. So, I’ve put together this sample FaaS that exposes an AWS API Gateway method, which invokes a Lambda function, which then queries a DynamoDB table. Sounds like fun, right? Anyway, I purposely minimized wizard use because I wanted to have a clear understanding of the all steps involved. Moreover, I think it’s easier to learn by stepping through AWS Management Console.

In a nutshell, here’s what we’re going to do in this tutorial:

  • Create a table in DynamoDB and populate it with sample data
  • Create a Lambda function that queries the DynamoDB table
  • Create an API Gateway method that invokes the Lambda function

The prerequisites include the following:

  1. An AWS Account and AWS CLI configured. Learn more here.
  2. Experience with the AWS Management Console. Learn more here.
  3. AWS SDK for Java is setup and configured properly (optional). For more information click here.
  4. Git is installed and configured, click here for instructions.
  5. Maven is installed and configured, click here for instructions.
  6. cURL or equivalent to download files using the command line (optional).

Populate DynamoDB

DynamoDB is Amazon’s premier fully managed proprietary NoSQL database available on AWS. SimpleDB is another NoSQL database offered by Amazon. It’s designed for smaller workloads and I intended to use it; however, it’s unsupported in the Lambda execution environment. If you’re interested, I have written code to populate my sample data in SimpleDB in Java and C#, here and here, respectively.

In this section, we’re going to create a DynamoDB table and populate it in two different ways. Choose whichever option you prefer or try both.

Option 1: Create DynamoDB Table using Java

  1. Clone the project:
    $ git clone https://github.com/johnboyer/aws-dynamodb-demo-java.git
  2. Review the code that creates the table and populates it with sample data:
    private static void createTable() throws InterruptedException {
        AttributeDefinition[] defs = {
                                    new AttributeDefinition(EMAIL, S)
                                    };
    
        ProvisionedThroughput throughput = new ProvisionedThroughput()
                                                .withReadCapacityUnits(1L)
                                                .withWriteCapacityUnits(1L);
    
        //Email address is the key
        KeySchemaElement emailKey = new KeySchemaElement(EMAIL, KeyType.HASH);
        CreateTableRequest createTableRequest = new CreateTableRequest()
                                .withTableName(TABLE)
                                .withKeySchema(emailKey)
                                .withAttributeDefinitions(defs)
                                .withProvisionedThroughput(throughput);
    
        // Create table if it does not exist yet
        TableUtils.createTableIfNotExists(sDynamoDB, createTableRequest);
        // wait for the table to move into ACTIVE state
        TableUtils.waitUntilActive(sDynamoDB, TABLE);
    }
    
    private static void addSampleItems() {
        // Add an item
        Map item = createItem("john@example.com", "John", "Doe");
        PutItemRequest putItemRequest = new PutItemRequest(TABLE, item);
        PutItemResult putItemResult = sDynamoDB.putItem(putItemRequest);
        //...
    }
    

For information about programming in DynamoDB click here.

  1. From the aws-dynamodb-demo-java/dynamo-db directory, package the project:
    $ mvn package
  2. Then run the app:
    $ mvn exec:java

Option 2: Create DynamoDB Table using AWS CLI

  1. In the terminal window, create a directory for the project.
  2. To create the table, we’ll use the following JSON:
    {
        "AttributeDefinitions": [{
            "AttributeName": "email",
            "AttributeType": "S"
        }],
        "TableName": "customer",
        "KeySchema": [{
            "AttributeName": "email",
            "KeyType": "HASH"
        }],
        "ProvisionedThroughput": {
            "ReadCapacityUnits": 1,
            "WriteCapacityUnits": 1
        }
    }
    
  3. Download the table.json file at the prompt:
    $ curl https://johnboyer.me/files/blog/table.json >table.json or use your web browser.
  4. Create the customer table using the AWS CLI:
    $ aws dynamodb create-table --table-name customer --cli-input-json file://table.json
  5. To populate the table, we’ll use the following JSON format:
    {
    "customer": [{
        "PutRequest": {
            "Item": {
            "email": {
                "S": "john@example.com"
            },
            "first_name": {
                "S": "John"
            },
            "last_name": {
                "S": "Doe"
            }
           }
        }
       },
        ...
      ]
    }
    
  6. Download the data.json file:
    $ curl https://johnboyer.me/files/blog/data.json >data.json
  7. Populate the customer table:
    $ aws dynamodb batch-write-item --request-items file://data.json

If you tried both of these methods, which one did you prefer?

Create a Lambda Function

Now that we’ve populated the database, we’re ready to create and configure our Lambda function. In the following steps we’ll:

  • Build and package a Java Lambda function
  • Create, configure, and deploy the function
  • Create and configure the function’s AWS IAM role
  • Verify and test the function
  1. Ensure that Maven is installed and configured on your computer.
  2. Clone the project:
    $ git clone https://github.com/johnboyer/aws-lambda-demo-java.git
  3. From the aws-lambda-demo-java/lambda-demo directory, package the project:
    $ mvn package
  4. In the AWS Management Console, go to the Lambda Console and click Create a Lambda Function > Blank Function > Next.
  5. For the Name, enter getCustomers and Java 8 for the runtime.
  6. Click Upload and navigate to aws-lambda-demo-java/lambda-demo/target/lambda-demo-1.0.0.jar
  7. In the Lambda function handler and role section, set the Handler to me.johnboyer.aws.samples.lambda.DynamoDBFunctionHandler::handleRequest, in the dropdown click Create new role from templates, set the Role name to customerLambdaRole and set the Policy templates to Simple Microservice permissions, click Next and Create Function.

    Note the Java 8 requires the handler value to have the following format: packageName::methodName.

  8. Then in the IAM Management Console, click Roles > customerLambdaRole > Permissions > Attach Policy > AWSLambdaDynamoDBExecutionRole > Attach Policy.
  9. In the Lambda Console, click Functions > getCustomers > Test and verify if it works.
  10. Bummer, the test fails because of a pesky permissions error.
  11. In the IAM Management Console, click Policies > Create policy > Copy an AWS Managed Policy > Select.
  12. Search for dynamodbread and select AmazonDynamoDBReadOnlyAccess.
  13. In the Policy Document, set the Resource to your DyanomoDB’s customer table ARN, e.g., arn:aws:dynamodb:us-west-2:123456789:table/customer, click Validate Policy and Create Policy. Note the generated name of the policy.
  14. Click Roles > customerLambdaRole > Permissions > AWSLambdaDynamoDBExecutionRole > Detach Policy > Detach.
  15. Then click Attach Policy > Policy Type > Customer Managed > policy from step 13 > Attach Policy.
  16. In the Lambda Console, click Functions > getCustomers > Test and verify if it works. It should and the output will look like the following:
    "{Items: [{last_name={S: Smith,}, created_date={S: 2017-07-11T00:44:40.209Z,}, first_name={S: Mary,}, email={S: mary@example.com,}}, {last_name={S: Smith,}, created_date={S: 2017-07-11T00:44:40.225Z,}, first_name={S: Bob,}, email={S: bob@example.com,}}, {last_name={S: Doe,}, created_date={S: 2017-07-11T00:44:40.191Z,}, first_name={S: Jane,}, email={S: jane@example.com,}}, {last_name={S: Boyer,}, created_date={S: 2017-07-11T00:44:40.144Z,}, first_name={S: John,}, email={S: john@example.com,}}],Count: 4,ScannedCount: 4,}"
    

Although I’m not going to cover it here in detail, the AWS CLI for a creating our Lambda function would look something like the following:

    $ aws lambda create-function \
        --region us-west-2 \
        --function-name getCustomers \
        --zip-file fileb:///aws-lambda-demo-java/lambda-demo/target/lambda-demo-1.0.0.jar \
        --role arn:aws:iam::123456789:role/service-role/customerLambdaRole \
        --handler me.johnboyer.aws.samples.lambda.DynamoDBFunctionHandler::handleRequest \
        --runtime java8 \
        --profile <adminuser>
 

Note the path for the ZIP file can also be an AWS S3 bucket. For more information about the AWS Lambda CLI click here.

Anyway, there’s a lot more to learn about Lambda such as versioning, supported event triggers, automated deployment and so on. I hope to cover some of these topics in future posts. In the meantime, I recommend checking out Lambda’s documentation here.

Create an API Gateway Method

  1. In Amazon API Gateway Console, click Create API > New API, enter CustomerSample for the name and click Create API.
  2. Then click Actions > Create Method, click GET in the dropdown menu and the OK checkmark.
  3. For the Integration type, click Lambda Function, your region (e.g., us-west-2), getCustomers for the Lambda Function and click Save.
  4. Review the overview of the API’s invocation sequence.
  5. Then click TEST > Test, the output should be the same as the Lambda function.
  6. To invoke the API via a web browser, you’ll need to deploy it. Click Actions, create a new deployment stage, e.g., test, and click Deploy.
  7. Copy and paste the Invoke URL into your web browser, e.g., https://l093tb3xa9.execute-api.us-west-2.amazonaws.com/test

As with Lambda, there’s a lot more to learn about API Gateway an a good place to start is the developer guide here.

Conclusions

After developing this exercise and writing the post. I’ve come to the realization that there are lots of moving parts between DynamoDB, Lambda, and API Gateway, especially with respect to setup and configuration. Creating the Lambda function in the console required sixteen steps. Clearly, we would need to automate these steps to ease the pain of setup and deployment. That will require further learnings through prototyping and code katas.

Additionally, I observed that after I deployed my function and waited a day between executions, the latency was very high. This is what’s known as a cold start:

A cold start occurs when an AWS Lambda function is invoked after not being used for an extended period of time resulting in increased invocation latency…
From the data, it’s clear that AWS Lambda shuts down idle functions around the hour mark.

Source: Cui, Y. (2017, July 3). How long does AWS Lambda keep your idle functions around before a cold start?

From an architectural perspective, it will be important to consider the cold start scenario. For example, asynchronous background tasks maybe better suited for Lambda than infrequent client API invocations that usually require low latency. Consequently, when we decide to migrate our first Lambda function into production, we’ll choose a background task that isn’t dependent on low latency.

In the meantime, please let me know if you have any additional thoughts or comments on this tutorial.

Advertisements
How to Build a Serverless API With AWS DynamoDB, Lambda, and API Gateway