Connect AWS API Gateway directly to SNS using a service integration

AWS API Gateway is a powerful service for managing your REST APIs. It was released in 2015 as a way to make the newly-released AWS Lambda compute service accessible over HTTPS.

In this post, we'll discuss one of the more advanced API Gateway use cases -- using an AWS service integration to connect HTTP endpoints directly to other AWS services.

This allows you to skip your custom compute layer altogether. 🔥

By following the walkthrough in this post, you will be able to send the body of an HTTP request directly into an AWS SNS topic.

This post is intended to be a complete introduction to AWS service integrations with API Gateway, so we'll start with the basics before getting to the detailed walkthrough. Feel free to skip the earlier sections if you're already familiar with API Gateway.

In this post, we'll cover:

• Background and Terminology of API Gateway
  • When would I use API Gateway?
• Service proxy resources in API Gateway
• Using an API Gateway service proxy integration to SNS
  • Configuring an SNS Topic with a subscription to your cell phone
  • Setting up the basic API Gateway elements
  • Creating the service proxy integration
  • Deploying and using our API
• Conclusion and Recap

Buckle up, this is a long one. Let's get started!

Background and Terminology

Before we get too far, let's understand the basics of API Gateway and some core terminology.

AWS API Gateway is a managed service provided by AWS. The core use case of API Gateway is to connect web requests to custom compute. It is a key enabler of serverless architectures using AWS Lambda, and it provides a number of basic API utilities.

Some key terms to know with AWS API Gateway are:

• REST API: An API Gateway REST API is a group of resources and methods that are accessible via HTTP endpoints. A single API Gateway REST API uses the same domain name for all resources and methods contained within it.

• Resource: A Resource is an object that is made accessible over an API. A resource is associated with a particular path on a REST API. For example, if you have a users resource, it may be accessible at the /users path on your REST API.

• Method: A Method refers to a particular HTTP verb used with a web request. Example methods include GET, POST, and DELETE.

If you're following a RESTful API design, the combination of resource + method allows you to expose common actions on a particular object. For example, making a GET request to /users would return a list of all users. Making a GET request to /users/1234 should return information about the user with an ID of 1234. Making a DELETE request to /users/1234 would delete the user with ID 1234.

When would I use API Gateway?

API Gateway's main use case is in connecting web requests from clients to custom compute from AWS Lambda.

For example, if you're building a web application, you probably have some HTML, CSS, and Javascript that your users see when they navigate to your site in the browser. The Javascript in the frontend application makes HTTP requests to your backend. These backend requests can be to fetch existing data ("Who are the Twitter followers for user XYZ?") or they could be requests to save new data in the system ("User ABC wrote this tweet at 10:05 AM.").

You can use AWS Lambda to deploy stateless code to handle the logic in your backend API. API Gateway is the glue between your frontend in the browser and backend in Lambda. When your browser makes a request in the backend, the request will hit API Gateway first. API Gateway will authenticate the request, if needed. Then it will route the request to the proper AWS Lambda function for handling. When the AWS Lambda finishes processing, it will return the response to the requesting client.

In the meantime, API Gateway provides a number of additional benefits for builders of APIs. You can set up authorization in your API Gateway so that only properly authorized users can use your backend. You can configure caching in API Gateway to improve response times for your users. There are a number of other features, including rate limiting, request and response transformation, and payload compression.

Service Proxy Resources in API Gateway

In the standard case noted above, API Gateway is used to direct web requests to your custom logic in an AWS Lambda function. However, you can also use API Gateway to direct web requests to other AWS resources. This is known as a service proxy (sometimes called an AWS integration).

There are three main occasions where you may want to use AWS API Gateway as a service proxy:

1. You want a low-fuss ingest pipe from clients.

   The first use case for an API Gateway service proxy is to use it as a simple ingestion point into a system for further processing. This is my favorite use case for a service proxy. We'll walk through an example of it later on in this post.

   Imagine you have a large number of events from clients that you process in an asynchronous manner. Without the API Gateway service proxy, your architecture might look like this:

   

   The client makes an API request to your backend with the event payload. You have a Lambda function that handles the request, maybe does some light authorization and transformation, and then inserts the request into an SNS topic, an SQS queue, or a Kinesis stream for processing.

   Notice how little your Lambda function is doing. It might be doing nothing at all, just forwarding to a different AWS service. It could be doing authorization or light transformation, but we already saw that API Gateway can handle this basic functionality. There's no reason for your Lambda function to exist, and this is where the service proxy integration shines.

   

2. You want to skip Lambda altogether and use API Gateway as the entire interface to your data layer.

   A similar use case is to use API Gateway as a service proxy directly over a data store like AWS DynamoDB.

   If your API needs are simple and your authorization requirements are low, using API Gateway to proxy requests to DynamoDB can be a great fit. You can use elements in the request, such as the request path, to determine which item in DynamoDB should be retrieved and returned to the user.

   While this use case is similar to the first, I separate the two. In the first use case, it's a dumb pipe into a system where further processing happens asynchronously. This second use case is more CRUD-like, which often requires more complex requirements. There is usually more business logic in the Lambda in this example.

   I only recommend this approach if your API needs are very simple. If you're allowing users to create new resources and need to do any object validation or transformation before storing in DynamoDB, it can be tricky to write that logic in API Gateway mapping templates.

   Similarly, if you have complicated authorization logic, you may find it easier to handle that in AWS Lambda rather than in the mapping templates.

3. You want to provide a custom HTTP interface over an existing AWS service.

   A final use case of the service proxy is to wrap an existing AWS service in a new interface. AWS has a few examples of doing this in their documentation:

   • Create a REST API as an Amazon S3 Proxy in API Gateway
   • Create a REST API as an Amazon Kinesis Proxy in API Gateway

   In both of these examples, AWS shows how to expose a broad range of functionality around an AWS service via a custom API Gateway. For example, in the Kinesis walkthrough, it exposes both management-level API commands, like creating a stream, deleting a stream, or listing all streams, with data-level API commands, like inserting or reading records from a Kinesis stream.

   This use case might fit your needs if you want to provide a simpler interface for developers on your team, or if you want to migrate existing clients that you don't control to a new system. I haven't found much use for it, but that doesn't mean it won't fit your needs.

Now that we know the what and why about the API Gateway service proxy integration, let's walk through an example.

Using an API Gateway service proxy integration to SNS

In this section, we will set up an example API Gateway service proxy integration via the command line. The example will use the first use case for service proxies mentioned in the previous section -- a low-fuss ingest pipe from clients.

To follow along with this example, you will need the AWS CLI installed and your environment configured with AWS credentials.

The example will set up the following architecture:

There will be an SNS Topic that is forwarding messages to your cell phone. There will be an API Gateway REST API with a resource and method configuring to proxy requests to our SNS topic. With this architecture, a client can insert messages into our SNS Topic by making a POST request to our API Gateway endpoint.

We'll do this in three parts:

• First, we'll create an SNS topic and configure a subscription to send messages to your cell phone for testing.
• Then, we'll create the basic elements of our API Gateway REST API.
• Finally, we'll connect our REST API to our SNS topic via an API Gateway service proxy integration.

Let's get started.

Configuring an SNS Topic with a subscription to your cell phone.

First, we will set up our SNS Topic and configure a subscription. In this example, we'll use a subscription that sends a received message to our cell phone as an SMS message. This SMS subscription is solely so we can quickly see our example working. In a real application, your subscribers would likely be an SQS queue, a Lambda function, or an HTTP endpoint.

First, let's create our topic. You can do so with the following command in your terminal:

TOPIC_ARN=$(aws sns create-topic \
  --name service-proxy-topic \
  --output text \
  --query 'TopicArn')

This command creates an SNS Topic with the name "service-proxy-topic". We're also using the AWS CLI's output control to return just the value for the ARN for our topic and storing it to a TOPIC_ARN variable in our terminal. We'll need this later in the example.

You can see the value of your TOPIC_ARN by using the echo command:

echo $TOPIC_ARN

arn:aws:sns:us-west-2:955617200811:service-proxy-topic

Next, we'll create an SMS subscription to our SNS topic. Use the command below to create the subscription, substituting your own SMS device in the SMS_ENDPOINT variable. Make sure you include the country code in your number (1 if you're in the US).

SMS_ENDPOINT=+15555551234
aws sns subscribe --topic-arn $TOPIC_ARN \
    --protocol sms \
    --notification-endpoint $SMS_ENDPOINT

Confirm that your subscription is configured correctly by publishing a message to your SNS topic:

aws sns publish --topic-arn $TOPIC_ARN \
    --message 'This is a test'

You should receive a text on your mobile device.

Now that we have an SNS topic with a subscription that sends to our mobile device, let's set up our API Gateway with a service proxy.

Setting up the basic API Gateway elements.

The next step is to set up our basic API Gateway elements.

First, we will create a REST API in API Gateway. Run the following command in your terminal to create your REST API and assign the ID to an API_ID variable.

API_ID=$(aws apigateway create-rest-api \
    --name 'Service Proxy' \
    --output text \
    --query 'id')

Next, we want to create a resource on our REST API. Remember that a resource is essentially a path in our HTTP request that maps to a particular object. We want our SNS topic to be accessible at the /ingest path on our REST API.

Resources in an API are organized as a tree. Imagine you had an API that exposed two resources, Users and Orders. Your API resources might look like this:

/ <-- Root resource
  /users <-- Users collection resource
    /{id} <-- Users resource
  /orders <-- Orders collection resource
    /{id} <-- Orders resource

The root of your API is the / resource, which might return information on the available subresources in your API. Beneath that, you have the /users resources, which is a collection resource for all users. You also have the /{id} subresource under users, which allows you to act on a particular user by requesting something like /users/1234. Similarly, there is an /orders collection resource under the root, with a subresource of /{id} to act on a particular order.

The important takeaway here is that each resource (other than the root) has a parent resource. When creating a new resource in your REST API, you need to specify the parent resource under which it will be placed.

We want our SNS topic to be available at /ingest, so the parent resource is the root -- /.

You can fetch your resources from API Gateway using the get-resources command, which returns an array of all resources:

aws apigateway get-resources --rest-api-id $API_ID

{
  "items": [
    {
      "id": "d5miwy1dwl",
      "path": "/"
    }
  ]
}

When you run this command for your new API, it will only return one resource in the array -- the root -- as we haven't added any additional resources yet.

Let's store the value of that resource id into a ROOT_RESOURCE_ID variable:

ROOT_RESOURCE_ID=$(aws apigateway get-resources \
  --rest-api-id $API_ID \
  --output text \
  --query 'items[0].id')

Now that we have our root resource id, let's create our /ingest resource using the create-resource command. We'll save the ID of the resource in a INGEST_RESOURCE_ID variable.

INGEST_RESOURCE_ID=$(aws apigateway create-resource \
  --rest-api-id $API_ID \
  --parent-id $ROOT_RESOURCE_ID \
  --path-part ingest \
  --output text \
  --query 'id')

If we run our same get-resources command, we now see that there are two resources in our REST API:

aws apigateway get-resources --rest-api-id $API_ID

{
    "items": [
        {
            "id": "d5miwy1dwl",
            "path": "/"
        },
        {
            "id": "ig6rps",
            "parentId": "d5miwy1dwl",
            "pathPart": "ingest",
            "path": "/ingest"
        }
    ]
}

Finally, let's wire up a method for our resource. We want clients to send us data to be ingested into our topic, so we'll use a POST method.

aws apigateway put-method --rest-api-id $API_ID \
  --resource-id $INGEST_RESOURCE_ID \
  --http-method POST \
  --authorization-type NONE

{
    "httpMethod": "POST",
    "authorizationType": "NONE",
    "apiKeyRequired": false
}

Perfect. We've now created the basic elements of our API Gateway REST API, in addition to the SNS topic and subscription we created in the previous section. Next, we'll connect our two pieces by creating a service proxy integration.

Creating the service proxy integration

In this section, we will create a service proxy integration that sends the payload from a POST request to our /ingest resource to our Kinesis stream for ingestion.

We need to do a few things to make this happen:

• First, we'll create an IAM role that gives our API Gateway permission to publish messages to our SNS topic.
• Then, we'll build a request template to transform our incoming message into the format needed by our SNS topic.
• Finally, we'll create the integration to connect our endpoint to our SNS topic.

Creating our IAM role

AWS IAM is a robust permissions system that is deeply-integrated into AWS services. You can use IAM for fine-grained access control on multiple levels. This includes describing which services your team members can access and use and even which services can call other services within AWS.

We are going to use the latter in this example. We have a REST API in API Gateway, and we want to allow that REST API to publish messages to our SNS topic.

AWS is a giant topic in itself, so we'll just cover the basics here. Be sure to read the IAM user guide if you want a deeper dive.

First, we need to create an IAM role. When creating this role, we will include a trust policy that states what type of entity can assume the role. In our example, we want our role to be assumed by the API Gateway service.

Create the role with the proper trust policy using the following command and save the ARN to the ROLE_ARN variable:

ROLE_ARN=$(aws iam create-role \
  --role-name service-proxy-role \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": {
      "Effect": "Allow",
      "Principal": {"Service": "apigateway.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }
  }' \
  --output text \
  --query 'Role.Arn')

Second, we will update the inline policy on the role. With an IAM role, you grant it permission to perform certain actions. In this instance, we want to grant it permission to publish messages to our SNS topic.

Run the following command to add publish permissions to your SNS topic:

aws iam put-role-policy \
  --role-name service-proxy-role \
  --policy-name 'sns-publish' \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": {
      "Effect": "Allow",
      "Action": "sns:Publish",
      "Resource": "'$TOPIC_ARN'"
    }
  }'

Now that we have our IAM role configured, let's move on to creating a request template.

Creating a request template

In this section, we will build a request template for our API integration. Let's first understand why we need this.

The SNS Publish API call expects a POST request where the relevant parameters are passed in the body as form data (x-www-form-urlencoded). We need to pass in three parameters:

• Action (which must equal Publish for our use case)
• TopicArn
• Message

There are a few things we'd like to change about this endpoint for when our clients send a message to us:

1. We would like the request body to be JSON, rather than form-data, as this is more common for web APIs these days;

2. We don't want to require clients to specify "Action": "Publish" in their request body; and

3. We don't want clients to pass our SNS Topic ARN in their request body. We don't really want our clients to know about the Topic ARN or even the implementation detail that we're using SNS behind the scenes.

A request template is a way to convert an incoming HTTP request into a different format before sending to your backing integration. A request template uses the Velocity Template Language (VTL) for request templating. The full API reference for request and response templating in API Gateway can be found here.

Understanding all of the intricacies of API Gateway's mapping templates is outside the scope of this post. We'll just cover the basics. For our mapping template, we'll assemble our form data as follows:

"Action=Publish
&TopicArn=$util.urlEncode('<ourTopicArn>')
&Message=$util.urlEncode($input.body)"

This mapping template uses some static values, like the Action and our Topic Arn, as well as the dynamic value from each request body, to construct a Publish request to SNS.

We will use this mapping template in the next section when we create our integration in API Gateway.

Creating the integration in API Gateway

We are getting close to the moment of truth. We've created our IAM role. We have a mapping template to transform our incoming request into the format expected by the SNS service. Now, we just need to wire it all together.

First, we will create the integration between our HTTP endpoint and our AWS service. Use the call below to create the integration:

REGION=$(aws configure get region)
aws apigateway put-integration \
  --rest-api-id $API_ID \
  --resource-id $INGEST_RESOURCE_ID \
  --http-method POST \
  --type AWS \
  --integration-http-method POST \
  --uri 'arn:aws:apigateway:'$REGION':sns:path//' \
  --credentials $ROLE_ARN \
  --request-parameters '{
        "integration.request.header.Content-Type": "'\'application/x-www-form-urlencoded\''"
    }' \
  --request-templates '{
    "application/json": "Action=Publish&TopicArn=$util.urlEncode('\'$TOPIC_ARN\'')&Message=$util.urlEncode($input.body)"
  }' \
  --passthrough-behavior NEVER

There's a lot going on here, so let's walk through it.

The rest-api-id, resource-id, and http-method indicate for which API, resource, and method we are creating an integration. These are elements we created earlier.

The --type AWS portion indicates that the integration is a proxy to an AWS service.

With the --integration-http-method POST and --uri 'arn:aws:apigateway:'$REGION':sns:path//', we are indicating that we want to make a POST request to the SNS service. Additional docs on this are here, though they're somewhat hard to parse.

Next, we specify the --credentials property with the ARN of the IAM Role we created previously. Remember that this IAM Role has permission to Publish to our SNS topic and is able to be assumed by API Gateway.

The request-parameters argument allows us to specify additional information we want to send in the request to our integration. In this instance, we need to set the Content-Type header to application/x-www-form-urlencoded for the SNS service.

The request-templates argument allows you to construct the request body for the integration. You make a different template per Content-Type of the original request. In our example, we are only creating a template for the Content-Type of application/json. This template assembles the payload for the form data as described in the previous section.

Finally, the passthrough argument describes how to handle requests that don't have a Content-Type specified in the request templates. In our case, we want to require application/json, so we specify "NEVER" for the passthrough. Any other Content-Type values will be rejected by API Gateway.

We. are. so. close.

There are a few other little things we need to add so that our clients get relevant information back.

First, add an integration response. This is similar to the integration we just set up, but it's after the integration responds. You can add dynamic messages with VTL, but we'll just add a basic message of Message received..

Run the following command to add your integration response:

aws apigateway put-integration-response \
  --rest-api-id $API_ID \
  --resource-id $INGEST_RESOURCE_ID \
  --http-method POST \
  --status-code 200 \
  --selection-pattern "" \
  --response-templates '{"application/json": "{\"body\": \"Message received.\"}"}'

Second, add a method response. This is how API Gateway takes the integration response and assembles it into an HTTP-compatible response. Again, we'll just do the bare minimum to pass it back to the client:

aws apigateway put-method-response \
  --rest-api-id $API_ID \
  --resource-id $INGEST_RESOURCE_ID \
  --http-method POST \
  --status-code 200 \
  --response-models '{"application/json": "Empty" }'

All of our infrastructure is set up. We set up our integration to SNS with our request template. We have an integration response and a method response. In the next section, we'll deploy and test our system.

Deploying and using our API

We're finally here. We're ready to actually use our infrastructure. Let's try it out.

In this last section, we'll create a deployment for our REST API. Then we'll make an HTTP request to our API and verify that we received the message.

Before we can use our API, we need to deploy it. All of the changes we've made thus far are not live on an HTTP-accessible endpoint until we actually deploy.

To create a deployment, we make a CreateDeployment call. We need to associate a deployment with a particular stage for it to be accessible. Let's use prod, because YOLO..

aws apigateway create-deployment \
  --rest-api-id $API_ID \
  --stage-name prod

{
    "id": "fzygem",
    "createdDate": 1550342451
}

With our API Gateway deployed, we can now invoke our /ingest endpoint. The body of the message that we send to our endpoint will be passed along to our SNS topic.

We can use curl for invoking our endpoint:

curl -X POST https://$API_ID.execute-api.$REGION.amazonaws.com/prod/ingest \
  --data 'Hello, from your terminal!' \
  -H 'Content-Type: application/json'

{"body": "Message received."}

If it's working correctly, you should see the "Message received." response in your terminal.

Futher, you should have received an SMS message on your cell phone that you configured with a subscription to your SNS topic:

We did it! Data from a client to your SNS topic without using AWS Lambda as a middleman.

Conclusion and Recap

This was a meaty post, and there's a lot to take in. Let's do a basic recap of our process:

1. We created an SNS topic to receive messages and set up a subscription on that topic to forward all messages to a cell phone.

2. We set up our basic API Gateway resources, including a REST API, a resource (/ingest) and a method (POST) on that resource.

3. We made an IAM role that could be assumed by API Gateway and gave permission to publish messages to our SNS topic.

4. We configured an integration to our SNS topic from our REST API's resource and method. This integration included a request mapping template, a response mapping template, and a method response.

5. We deployed our REST API to make it live, then sent an HTTP request using curl. We received the message on our cell phone, indicating that it was successfully published to our SNS topic.

While the configuration is difficult at first, there are a number of benefits to this pattern. The biggest benefit is you don't have to write or pay for the execution of code whose sole job is to forward data from a client to a downstream service. You can connect these two directly.

This pattern works well for a number of services, such as AWS SNS shown here, but also AWS SQS, AWS Kinesis Data Streams, or AWS Kinesis Firehose.

You can also use additional features of API Gateway, such as custom authorizers to enable fine-grained authorization or usage plans for client rate-limiting.

Still have questions about API Gateway and service proxies after reading them? Hit me up via email or leave a comment below!