How to work from AWS Lambda with Elasticache and DynamoDB

Since February 2016, AWS Lambda has been able to access resources inside the Virtual Private Cloud , but by default all lambdas work outside of the VPC. Since this opportunity appeared relatively recently and there are not so many articles on how to implement this, we would like to share our experience with you.

We have developed a mobile application built on AWS. The Lambda service was great for writing small server logic, which is also easily scalable.

The lambda task was set quite trivial: execute a request to DynamoDB and cache the most popular requests using Elasticache. The problem we encountered is related to setting access from VPC to Amazon's internal and external resources.

In our case, it was necessary to ensure the availability of the Elasticache service (in VPC) and DynamoDB (outside VPC). To achieve this goal, a structure was developed in VPC, which is presented in the diagram:



The following is instructions on how to create such a VPC working pattern from the AWS Console.

Configuring VPC
First you need to create a VPC, if you have not already done so. Create two subnets (Subnets) public and private.

The second step is to create routing tables (Route Tables). By default, you have already created one main table, it will serve as a private table. It remains only to create a public table. When creating, you must specify our VPC.

For each subnet (public and private), select the appropriate routing table.

Next, we create a gateway (internet gateway), which will provide access to all services outside the VPC and to external addresses in general. After attachment created a gateway to our VPC. We also create a NAT gateway, which we attach to the public subnet.

And after all, it remains only to configure the routing tables:

• We create a route in the public table for the created Internet gateway. Example:

• Create a route in a private table for NAT. Example:

Configuring Elasticache

In our example, we will consider connecting Elasticache to a private subnet. To do this, in the Elasticache panel, create a group for a private subnet (Cache Subnet Group). And then, when creating the cluster, we indicate the created group.

After all, it remains only in the VPC panel to find the group in the Security Groups for our VPC (it is created automatically) and add a rule for the port of our cluster (redis by default - 6379; memcached - 11211). Example:



Let's check how it all flies with us. In the lambda settings, select VPC and a private network. Run the code:

from __future__ import print_function
import json
import boto3
import decimal
import logging
import elasticache_auto_discovery
from pymemcache.client.hash import HashClient
CACHE_ENDPOINTS = 'exaplecache.wjbxgg.cfg.euw1.cache.amazonaws.com:11211'
def decimal_default(obj):
   if isinstance(obj, decimal.Decimal):
       return float(obj)
   raise TypeError
def lambda_handler(event, context):
   nodes_endpoints = elasticache_auto_discovery.discover(CACHE_ENDPOINTS)
   nodes = map(lambda x: (x[1], int(x[2])), nodes_endpoints)
   memcache_client = HashClient(nodes, timeout=60, connect_timeout=2)
   table = boto3.resource('dynamodb').Table('Photo')
   response = table.scan(Limit=20)
   memcache_client.set('example', json.dumps(response['Items'], default=decimal_default))
   cache_elements = memcache_client.get('example')
   if json.loads(cache_elements) == response['Items']:
       return "It works!"
   else:
       return "Doesn't work"

And after the launch, we will see the cherished result:


Conclusion
Working with Amazon services outside and inside the VPC opens up many possibilities for using AWS Lambda. The configuration scheme for VPC presented in this article can be used as a universal solution to provide access to any other services inside and outside the VPC.

Useful Links:
Lambda in VPC
Your VPC and Subnets
Getting Started with Amazon ElastiCache