We love to use AWS CloudFormation to deploy our environments. Its like configuration management for our AWS infrastructure in the sense that we write a desired state as code and apply it to our environment. But sometimes, there are tasks that we want to complete that aren’t part of CloudFormation. For instance, what if we wanted to use CloudFormation to deploy a new account which needs to be done through the CLI, or if we need to return some information to our CloudFormation template before deploying it? Luckily for us we can use a Custom Resource to achieve our goals. This post shows how you can use CloudFormation with a Custom Resource to execute a very basic Lambda function as part of a deployment.

Solution Overview

To demonstrate our Custom Resource, we’ll need a Lambda function that we can call. CloudFormation will deploy this function from a Zip file and after deployed, will execute this function and return the outputs to our CloudFormation template. The diagram below demonstrates the process of retrieving this zip file form an existing S3 bucket, deploying it, executing it and having the Lambda function return data to CloudFormation.

The CloudFormation Template

First, lets take a look at the CloudFormation template that we’ll be using to deploy our resources.

---
AWSTemplateFormatVersion: '2010-09-09'
Description: Example of a Lambda Custom Resource that returns a message
Parameters:
  ModuleName: #Name of the Lambda Module
    Description: The name of the Python file
    Type: String
    Default: helloworld
  S3Bucket: #S3 bucket in which to retrieve the python script with the Lambda handler
    Description: The name of the bucket that contains your packaged source
    Type: String
    Default: hollow-lambda1
  S3Key: #Name of the zip file
    Description: The name of the ZIP package
    Type: String
    Default: helloworld.zip
  Message: #Message input for you to enter
    Description: The message to display
    Type: String
    Default: Test
Resources:
  HelloWorld: #Custom Resource
    Type: Custom::HelloWorld
    Properties:
      ServiceToken:
        Fn::GetAtt:
        - TestFunction #Reference to Function to be run
        - Arn #ARN of the function to be run
      Input1:
        Ref: Message
  TestFunction: #Lambda Function
    Type: AWS::Lambda::Function
    Properties:
      Code:
        S3Bucket:
          Ref: S3Bucket
        S3Key:
          Ref: S3Key
      Handler:
        Fn::Join:
        - ''
        - - Ref: ModuleName
          - ".lambda_handler"
      Role:
        Fn::GetAtt:
        - LambdaExecutionRole
        - Arn
      Runtime: python2.7
      Timeout: '30'
  LambdaExecutionRole: #IAM Role for Custom Resource
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - lambda.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: "/"
      Policies:
      - PolicyName: root
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action:
            - logs:CreateLogGroup
            - logs:CreateLogStream
            - logs:PutLogEvents
            Resource: arn:aws:logs:*:*:*
Outputs: #Return output from the Lambda Function Run
  Message:
    Description: Message returned from Lambda
    Value:
      Fn::GetAtt:
      - HelloWorld #Output from the HelloWorld Custom Resource
      - Message #Return property

In the parameters section you can see we’re looking for the S3 bucket with our module in it, the name of the module and a generic input for the CloudFormation template to pass to Lambda as a string.

Parameters:
  ModuleName: #Name of the Lambda Module
    Description: The name of the Python file
    Type: String
    Default: helloworld
  S3Bucket: #S3 bucket in which to retrieve the python script with the Lambda handler
    Description: The name of the bucket that contains your packaged source
    Type: String
    Default: hollow-lambda1
  S3Key: #Name of the zip file
    Description: The name of the ZIP package
    Type: String
    Default: helloworld.zip
  Message: #Message input for you to enter
    Description: The message to display
    Type: String
    Default: Test

In the resources section we have a HelloWorld object which is our custom resource of type Custom::DESCRIPTIONHERE. We need to pass a ServiceToken along, which tells the stack which Custom Resource to be executed. We’re also adding an input which will be passed to Lambda named “Input1” and we’ll reference the parameter seen earlier.

HelloWorld: #Custom Resource
    Type: Custom::HelloWorld
    Properties:
      ServiceToken:
        Fn::GetAtt:
        - TestFunction #Reference to Function to be run
        - Arn #ARN of the function to be run
      Input1:
        Ref: Message

Below this, is the Lambda function deployment. This piece of the resources section of the template shows where the Lambda module comes from, the runtime, timeout and which role will have permissions be used for it.

TestFunction: #Lambda Function
    Type: AWS::Lambda::Function
    Properties:
      Code:
        S3Bucket:
          Ref: S3Bucket
        S3Key:
          Ref: S3Key
      Handler:
        Fn::Join:
        - ''
        - - Ref: ModuleName
          - ".lambda_handler"
      Role:
        Fn::GetAtt:
        - LambdaExecutionRole
        - Arn
      Runtime: python2.7
      Timeout: '30'

Next, there is a section for setting up permissions for the Lambda function to write to CloudWatch. Depending on your environment, you may need to provide access to other resources. For example if your function reads EC2 data, then you’d need to ensure it had the appropriate permissions to read those properties.

  LambdaExecutionRole: #IAM Role for Custom Resource
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - lambda.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: "/"
      Policies:
      - PolicyName: root
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action:
            - logs:CreateLogGroup
            - logs:CreateLogStream
            - logs:PutLogEvents
            Resource: arn:aws:logs:*:*:*

We won’t look at the next piece until our Lambda function finishes running, but we’re going to get some return information from the function and print it as a CloudFormation output.

Outputs: #Return output from the Lambda Function Run
  Message:
    Description: Message returned from Lambda
    Value:
      Fn::GetAtt:
      - HelloWorld #Output from the HelloWorld Custom Resource
      - Message #Return property

Lambda Function

Let’s look at the Lambda Function we’ll be using for this example. This is a python 2.7 function that takes a basic string input from CloudFormation, concatenates it with another string and returns it. Nothing too crazy here for the example, but there are some important pieces that must be in your Lambda function so that CloudFormation knows that the function is done running, and if it executed correctly.

#Import modules
import json, boto3, logging
from botocore.vendored import requests

#Define logging properties
log = logging.getLogger()
log.setLevel(logging.INFO)



#Main Lambda function to be excecuted
def lambda_handler(event, context):
    #Initialize the status of the function
    status="SUCCESS"
    responseData = {}


    #Read and log the input value named "Input1"
    inputValue = event['ResourceProperties']['Input1']
    log.info("Input value is:" + inputValue)

    #transform the input into a new value as an exmaple operation
    data = inputValue + "Thanks to AWS Lambda"
    responseData = {"Message" : data} #If you need to return data use this json object

    #return the response back to the S3 URL to notify CloudFormation about the code being run
    response=respond(event,context,status,responseData,None)

    #Function returns the response from the S3 URL
    return {
        "Response" :response
    }

def respond(event, context, responseStatus, responseData, physicalResourceId):
    #Build response payload required by CloudFormation
    responseBody = {}
    responseBody['Status'] = responseStatus
    responseBody['Reason'] = 'Details in: ' + context.log_stream_name
    responseBody['PhysicalResourceId'] = context.log_stream_name
    responseBody['StackId'] = event['StackId']
    responseBody['RequestId'] = event['RequestId']
    responseBody['LogicalResourceId'] = event['LogicalResourceId']
    responseBody['Data'] = responseData

    #Convert json object to string and log it
    json_responseBody = json.dumps(responseBody)
    log.info("Response body: " + str(json_responseBody))

    #Set response URL
    responseUrl = event['ResponseURL']

    #Set headers for preparation for a PUT
    headers = {
    'content-type' : '',
    'content-length' : str(len(json_responseBody))
    }

    #Return the response to the signed S3 URL
    try:
        response = requests.put(responseUrl,
        data=json_responseBody,
        headers=headers)
        log.info("Status code: " + str(response.reason))
        status="SUCCESS"
        return status
    #Defind what happens if the PUT operation fails
    except Exception as e:
        log.error("send(..) failed executing requests.put(..): " + str(e))
        status="FAILED"
        return status

Lets look at the main function that will be executed. First we’ll initialize some of our variables. Next, we want to retrieve our input parameter (named Input1 and passed from CloudFormation), and then log it. After this there is a simple operation to concatenate the input with another string just to do something simple in our function. The next step is to provide some return data that will end up being our CloudFormation output. This is a JSON object so if you don’t need to return any custom info to CloudFormation, use an empty JSON object {}.

Below this, we’re calling a respond function (also located in our Lambda script which will retrieve info to send back to CloudFormation about the state of the Lambda script run. After this we return our data from the function.

def lambda_handler(event, context):
    #Initialize the status of the function
    status="SUCCESS"
    responseData = {}


    #Read and log the input value named "Input1"
    inputValue = event['ResourceProperties']['Input1']
    log.info("Input value is:" + inputValue)

    #transform the input into a new value as an exmaple operation
    data = inputValue + "Thanks to AWS Lambda"
    responseData = {"Message" : data} #If you need to return data use this json object

    #return the response back to the S3 URL to notify CloudFormation about the code being run
    response=respond(event,context,status,responseData,None)

    #Function returns the response from the S3 URL
    return {
        "Response" :response
    }

Lets look closer at the respond function. When we’re executing this function we need to send certain items back to CloudFormation so that the stack knows if it worked or not. Specifically it must return a response of SUCCESS or FAILED to a pre-signed URL. There are a list of response objects, specifically:

• Status (Required)
• Reason (Required if FAILED)
• PhysicalResourceId (Required)
• StackId (Required)
• RequestId (Required)
• LogicalResourceId (Required)
• NoEcho
• Data

The first part of this function builds the JSON object so that we can send it back to CloudFormation. We are then converting it to a string and logging the data for later reference. We set our responseURL which is passed to us from CloudFormation in the event parameter. After that we set the headers and then we try to do a PUT REST call with our return data. To do all of this, we had to import certain modules for our function which are seen in the full script, but none of these need to be provided in your zip file. It should be noted that this can also be done if you add your Lambda function in-line within your CFn template. if you use that method, there is a “cfn-response” module that can be called which eliminates the need to use the requests module.

def respond(event, context, responseStatus, responseData, physicalResourceId):
    #Build response payload required by CloudFormation
    responseBody = {}
    responseBody['Status'] = responseStatus
    responseBody['Reason'] = 'Details in: ' + context.log_stream_name
    responseBody['PhysicalResourceId'] = context.log_stream_name
    responseBody['StackId'] = event['StackId']
    responseBody['RequestId'] = event['RequestId']
    responseBody['LogicalResourceId'] = event['LogicalResourceId']
    responseBody['Data'] = responseData

    #Convert json object to string and log it
    json_responseBody = json.dumps(responseBody)
    log.info("Response body: " + str(json_responseBody))

    #Set response URL
    responseUrl = event['ResponseURL']

    #Set headers for preparation for a PUT
    headers = {
    'content-type' : '',
    'content-length' : str(len(json_responseBody))
    }

    #Return the response to the signed S3 URL
    try:
        response = requests.put(responseUrl,
        data=json_responseBody,
        headers=headers)
        log.info("Status code: " + str(response.reason))
        status="SUCCESS"
        return status
    #Defind what happens if the PUT operation fails
    except Exception as e:
        log.error("send(..) failed executing requests.put(..): " + str(e))
        status="FAILED"
        return status

See It In Action

Just so we can show some screenshots of the process, here is the input for my CloudFormation template as I’m deploying it through the AWS Console. See that I’ve got an input message of “Test Message” and I’m specifying information about my Lambda Function’s location.

You can also see my Lambda function neatly zipped up in my S3 bucket below.

Once the Lambda function has been deployed, we can see it in the Lambda Functions console.

If we look at the CloudWatch Logs for our function, we can see the data being returned to CloudFormation.

Lastly, we can see that in the CloudFormation template that we deployed, it has finished the creation and in the outputs tab, we can see the message that was returned to the stack. This output could be used for other stacks or purely informational.

Summary

Custom Resources might not be necessary very often, but they can let you do virtually anything you want within AWS. Maybe they execute a Lambda function to gather data, or maybe they trigger a Step Function that has tons of logic built into it to do something else magical. The world is your oyster now, what will you build with your new knowledge?