Clojure in AWS Serverless: DynamoDB
Welcome back! Did you miss me? No? Uh… Brushing that aside, it’s time to continue evolving our Clojure Lambda with DynamoDB!In the last tutorial we set up a basic Lambda that generated prime numbers from a test request. While generating primes is a good exercise, generating every single one per request is silly.
First though, I have to admit this will be shorter than my other introductions because Numergent has a much larger tutorial specifically on DynamoDB with Faraday . It seems silly to cover ground that is already well trod. I’ll fill the gap in with a sweet segway instead.
As always, the code is in Github .
But first, the segue…
Follow me to yourdreams! Docker
I can’t keep this a secret: I Love Docker! Ok, maybe I should qualify that. I love Docker for local development.
One of the hardest parts of software development is setting up the local environment. It’s the source of countless bugs because it works on my machine! Even if the process is straightforward, it can still take up a large amount of a developer’s time. Over the years I’ve worked with virtualized environments, stub databases, and Vagrant. In the end Docker won. If this were a car commercial, I would call it best-in-class.
I included a Docker Compose configuration file that sets up the external dependencies with a simple command line, letting you spin up Dynamo Local in an isolated container. More information can be found in the README on requirements and how to run.Boot tasks
If you peeked below the Docker section of the README, you might have noticed the local boot task. In boot you can define your own commands , called tasks. Tasks act a lot like Ring middleware where they return a function. Let’s take a look at few I’ve defined.
The first helper, with-dev , ropes in the development directory. This task performs a similar function as Leiningen’s :profiles map would accomplish. It takes the existing values in :source-paths and conj es the development directory into it. I use this as a scratchpad for development tools.
The fun part is it’s just Clojure functions. You could perform all sorts of operations on your configuration in these tasks.
The second, local , takes the with-dev task and injects some environment variables with the boot-environ feature of environ library. In this case, I’m injecting :development "true" into my configuration.
With these two tasks, we can now launch a development repl complete with our configuration environment and development namespace folder.
boot local repl
The nice thing about isolating all of these steps in the local task is we control when to insert the development namespace and environment variables.
The third task you might recognize from the last tutorial. build performs the compilation of our Clojure code into an actual file. Each smaller function is a boot built-in task for each compilation step. The task-options! let us set some task arguments globally rather than every time we call it. If you’re a Leiningen fan, you’ll notice similar configuration for :main , the project name and version, as well as :uberjar-name . Even the version is a function!
Boot has many built-in tasks. If you want to know more or see a list, call boot -h . Even the tasks we defined above are listed!
Ok, the segue is over. Now for the fun part.
I’m ready for the fun part!DynamoDB withFaraday
The first trick in this pony show is to set up our database. If you are following along from the last tutorial, make sure you include Faraday in your configuration.DynamoDB interface
Because we will be doing this locally, we need to programmatically set up the database.
Here we set up some configuration. We use an environment variable :development to configure whether or not we are using development credentials. Faraday is intelligent enough to grab the production keys from the Lambda environment (like IAM roles) if we pass an incomplete configuration. It is important your :endpoint matches the region you are targeting, otherwise it’ll throw errors about US-East-1 by default. In my case I’m using US-West-2.
Notice we also define a table-name with :primes . This is important later when we set up our DynamoDB table in AWS.
For our local development, we use Faraday’s create-table to create the table, but we would avoid this call in production. I’ll talk more about that later.
Here’s our API for our prime store. Pretty simple. We can list, put and get primes from our store. However, there are some important gotchas about DynamoDB that should be brought up:scan operations are discouraged by AWS on Dynamo because they can potentially eat up many read units. For large data sets, this can get expensive (as in $$$) really quickly. Any read operation is, by default, eventually consistent. This requires only half of the read units than a Consistent Read operation. Sieve changes
Now that we are storing our primes, our sieve needs some adjustment. Why calculate primes that we’ve already found?
The key parts that changed:We now use what’s in our store first, calculating only new primes When we find a new prime, we add it to the store along with it’s location in the prime list.
Now we can pack everything up with boot build !DynamoDB in production
We are almost ready to push our code. The last two steps are to procure a DynamoDB table in AWS and set the IAM permissions for our Lambda to talk to it.
Creating a Dynamo table is pretty simple. Since this is a tutorial, we will delete it at the end. If you don’t, it could cost you money!Open the DynamoDB dashboard in AWS Click Create Table Name your table “primes” (remember from above?) Set the Primary Key to “index” with a Number type. Keep the default settings for now. Click Create.
Once the table is created, you will see the table dashboard. Near the bottom of Table Details is the ARN for our new table. We’ll need that for the IAM role.IAM Dynamo permissions forLambda You might be wondering why we chose to manually create the table instead of using our Lambda. The reason is simple: We don’t want our Lambda to have any more permissions than it needs. This is Amazon’s suggestion of Least Privilege . We will restrict our Lambda to only perform actions on o