This post is about how to terraform up a serverless, scalable and resilient lambda API. I am using the 2019nCov-api as an example. In addition, we will have a look at terraform’s built-in functions to make a more reusable modules for api gateway and lambdas.

前面几篇都是写dev的, 今天这篇来写云架构和terraform. 如何用AWS api gateway，lambda和dynamodb做一个轻巧，可扩展的适应性强的数据接口（p.s. serverless, scalable and resilient 这个我不知道该怎么翻， 反正就是这个☁️云架构精致，小巧且耐用)

Use Case 用例

The use case is I want to build a simple API to allow user to retrieve coronavirus data. There are two major tables. One is DXY stats, another one is people’s travel data. And I want to allow people to query specific fields via url query parameters. Also I want to schedule the coronavirus data injection daily to dynamodb.

这个数据接口是用来提取新冠疫情的相关数据， 主要的数据是来源于丁香医生实时数据和同程软件查询工具。同时用户可通过设置参数来获取特定数据。丁香医生的数据是实时更新的，为了得到历史数据，我需要设定一个定时任务把数据录入 dynamodb。

Architecture Diagram ☁️架构图

Okie dokie, now let’s look at the tools we can use from AWS.

  • AWS API Gateway [☁️云网关]
  • AWS Cloudwatch Event [☁️云监控]
  • AWS Lambda [☁️云计算服务]
  • AWS Dynamodb [NoSQL☁️键/值和文档数据库]

👉 click here for the diagram

Let’s terraform it up

Terraform 是一个 IT 基础架构自动化编排工具，它的口号是 “Write, Plan, and create Infrastructure as Code”, 基础架构即代码。具体的说就是可以用代码来管理维护 IT 资源. 这个API比较难写的地方主要是这个AWS API Gateway [☁️云网关]， 这篇文章将着重于如何用terraform自带的function，用for each的方式来写。

As we can see from the diagram, the difficult part is the AWS API Gateway [☁️云网关]. We will need one API Gateway with two pathes, each path is using GET methods. Let’s use terraform’s functions to make a more reusable api gateway module. To create an AWS API gateway, you will need:

  • aws_api_gateway_rest_api
  • aws_api_gateway_resource
  • aws_api_gateway_method
  • aws_api_gateway_integration
  • aws_api_gateway_integration_response
  • aws_api_gateway_deployment

Okie, dokie Let’s look at the code. The key to make this more reusable is to pair the endpoints. 要用for each， 就得把他弄成个数列，这玩意只可意会不可言传， 看code吧:

Here is the output of the pairing 👇:

(base) pingzhous-MBP:lambda-tf-v12 pingzhouliu$ terraform state list module.api-gw
module.api-gw.aws_api_gateway_deployment.api-gw-module
module.api-gw.aws_api_gateway_integration.api-gw-module["GET dxy"]
module.api-gw.aws_api_gateway_integration.api-gw-module["GET travel"]
module.api-gw.aws_api_gateway_integration_response.api-gw-module["GET dxy 200"]
module.api-gw.aws_api_gateway_integration_response.api-gw-module["GET travel 200"]
module.api-gw.aws_api_gateway_method.api-gw-module["GET dxy"]
module.api-gw.aws_api_gateway_method.api-gw-module["GET travel"]
module.api-gw.aws_api_gateway_resource.api-gw-module["dxy"]
module.api-gw.aws_api_gateway_resource.api-gw-module["travel"]
module.api-gw.aws_api_gateway_rest_api.api-gw-module
module.api-gw.aws_lambda_permission.apigw_lambda["GET dxy"]
module.api-gw.aws_lambda_permission.apigw_lambda["GET travel"]

You can find the module’s main.tf and variables.tf here 👇

Now let’s call the api-gw module:

module "api-gw" {
  source = "./api-gw"

  api-gw-name = "2019-nCov-API-GW"
  stage_name = "v1"
  endpoints = [
    {
      path = "dxy"
      method = "GET"
      lambda = "dxy"
      request_parameters = { 
        "method.request.querystring.date" = true
        "method.request.querystring.country" = true
        "method.request.querystring.cityName" = true
        "method.request.querystring.provinceName" = true
        "method.request.querystring.all" = true
        }
    },
    {
      path = "travel"
      method = "GET"
      lambda = "travel"
      request_parameters = { 
        "method.request.querystring.date" = true
        "method.request.querystring.start" = true
        "method.request.querystring.stop" = true
        "method.request.querystring.all" = true
        }
    }
  ]
}

🥳 🥳 🥳 Here you go! Your API is up 🤗 🤗 🤗

pingzhous-MBP:lambda-tf-v12 pingzhouliu$ terraform output
endpoint_id = https://4mmhkv7z9e.execute-api.eu-west-1.amazonaws.com/v1/
execution_arn = arn:aws:execute-api:eu-west-1:900665556514:4mmhkv7z9e

$ curl -s https://4mmhkv7z9e.execute-api.eu-west-1.amazonaws.com/v1/dxy?all=yes | jq -r .[].date | sort | uniq
2020-02-08
2020-02-09
2020-02-10
2020-02-11
2020-02-12

$ curl -s https://4mmhkv7z9e.execute-api.eu-west-1.amazonaws.com/v1/dxy?date=2020-02-12 | jq .[78]
{
  "id": "3275",
  "date": "2020-02-12",
  "country": "中国",
  "provinceName": "重庆市",
  "cityName": "忠县",
  "confirmedCount": "19",
  "suspectedCount": "NULL",
  "curedCount": "6",
  "deadCount": "NULL",
  "msg": " No Man is an Island  🏝  没有人是一座孤岛 @pingzhou| 平舟 ⛵"
}

👉 click here to see more examples of this API

Next Topics:

  • Dynamodb Attribute [NoSQL☁️键/值和文档数据库如何设计属性]
  • AWS Cloudwatch Scheduler [用☁️云监控来做定时任务]
  • Run Python in Lambda with Boto3 [☁️云计算服务来跑python和AWS的python工具包]

Please feel free to leave some comments if you have any questions or have any other interesting use cases. If you have enjoyed reading this post, please feel free to buy me a (r'virtual|physical', 'coffee'). All the coffee from #2019nCov posts will be donated (if there are any).

Buy me a coffeeBuy me a coffee