Connecting...

W1siziisimnvbxbpbgvkx3rozw1lx2fzc2v0cy9zawduawz5lxrly2hub2xvz3kvanbnl2jhbm5lci1kzwzhdwx0lmpwzyjdxq

Deploying Scala Serverless Lambda using Travis CI by Chris Oh

W1siziisijiwmtkvmdmvmjivmtavntkvmjqvotg5l3blegvscy1wag90by0xmtq4odiwicgxks5qcgvnil0swyjwiiwidgh1bwiilci5mdb4otawxhuwmdnlil1d

What's the best way to deploy a Scala serverless lambda function into AWS? 

You can use Travis CI however, Software Engineer, Chris Oh came cross some obstacles doing this so he shares with us how to get around these, check it out!

 

 

Left to right: Scala, Travis CI, AWS Lambda, Serverless Framework

I have spent the last week learning about the best way to deploy a Scala serverless lambda function into AWS using Travis CI. I have hit so many roadblocks on the way that no one should go through what I did. The code in this post is here.

 

Prerequisite

I will assume you already know and have an Amazon Web Service (AWS) account. If you don’t, please go make one. You will also need a GitHub account.

High-Level Picture

  1. We will create a lambda function using the serverless framework.
  2. We will create an IAM role on AWS and provide it to Travis CI
  3. We will use Travis CI to auto-deploy to AWS every time we commit
Photo by The Roaming Platypus on Unsplash

 

Serverless Framework

The Serverless Framework is an open source tool to generate lambda function templates. It will generate the boilerplate needed to serve a basic lambda function. It can connect to any provider, such as Google Cloud Platform, Microsoft Azure, IBM OpenWhisk and so on. In this example, I will be deploying to AWS.

First, you will need to install the serverless cli on your machine by running

# Installing the serverless cli
npm install -g serverless
Once you have installed the framework, let’s create the aws-scala boilerplate! If you are interested in another language or different provider, check the list of templates here.
# Create a new Serverless Service/Project
serverless create --template aws-scala-sbt --path my-service

# Change into the newly created directory
cd my-service

You can already see the magic template appear once you open it in an editor.

Now the basic code it there, head over to the 'serverless.yml' file. We need to slightly edit the configuration.

Let’s change the service name to 'my-scala-lambda' just for the heck of it, under function->hello->handler change the 'hello.Handler' to 'hello.ApiGatewayHandler'. Also, let's add an 'events' so the serverless framework will also generate an API gateway for us to ping.

1  service: my-scala-lambda
2
3  provider:
4    name: aws
5    runtime: java8
6
7  package:
8    artifact: target/scala-2.12/hello.jar
9
10 functions:
11   hello:
12     handler: hello.ApiGatewayHandler
13     events:
14       - http:
15           path: users
16           method: get

Photo by Marvin Meyer on Unsplash

 

GitHub

TLDR : If you are a GitHub guru, git initialize this repo and push it to GitHub.

Let’s initialize this folder into a git repository and push it, so your peers can appreciate this fine code you wrote. I like to create the repo first in GitHub, so let’s do that. Create a public GitHub repo, and DO NOT Initialize this repository with a README.


The repo must be public and not with a README for the free tier
 

After creating follow the script under …or create a new repository on the command line. But add all files, not just the 'README.md'.

For me, it would be

echo "# aws-scala-travis-ci-guide" >> README.md
git init
git add .
git commit -m "first commit"
git remote add origin git@github.com:5tigerjelly/aws-scala-travis-ci-guide.git
git push -u origin master

Now refresh the GitHub repo website!

You will see all the lambda code pushed to GitHub!

Photo by Thomas Kvistholt on Unsplash

 

Amazon Web Service

Create an IAM role

We will need to create an IAM role for Travis so it will be able to create AWS resources and deploy our lambda code there. Let’s head to the AWS console. Go to Services -> IAM -> Users. Add a new user, set an appropriate name, like 'travis-ci' and check Programmatic access. Click on next: Permissions.

Create Policy

In the Attach existing policies directly tab, create a policy and paste the JSON below.
1  {
2      "Statement": [
3          {
4              "Action": [
5                  "apigateway:*",
6                  "cloudformation:CancelUpdateStack",
7                 "cloudformation:ContinueUpdateRollback",
8                  "cloudformation:CreateChangeSet",
9                  "cloudformation:CreateStack",
10                 "cloudformation:CreateUploadBucket",
11                 "cloudformation:DeleteStack",
12                 "cloudformation:Describe*",
13                 "cloudformation:EstimateTemplateCost",
14                 "cloudformation:ExecuteChangeSet",
15                 "cloudformation:Get*",
16                 "cloudformation:List*",
17                 "cloudformation:PreviewStackUpdate",
18                 "cloudformation:UpdateStack",
19                 "cloudformation:UpdateTerminationProtection",
20                 "cloudformation:ValidateTemplate",
21                 "dynamodb:CreateTable",
22                 "dynamodb:DeleteTable",
23                 "dynamodb:DescribeTable",
24                 "ec2:AttachInternetGateway",
24                 "ec2:AuthorizeSecurityGroupIngress",
25                 "ec2:CreateInternetGateway",
26                 "ec2:CreateNetworkAcl",
27                 "ec2:CreateNetworkAclEntry",
28                 "ec2:CreateRouteTable",
29                 "ec2:CreateSecurityGroup",
30                 "ec2:CreateSubnet",
31                 "ec2:CreateTags",
32                 "ec2:CreateVpc",
33                 "ec2:DeleteInternetGateway",
34                 "ec2:DeleteNetworkAcl",
35                 "ec2:DeleteNetworkAclEntry",
36                 "ec2:DeleteRouteTable",
37                 "ec2:DeleteSecurityGroup",
38                 "ec2:DeleteSubnet",
39                 "ec2:DeleteVpc",
40                 "ec2:Describe*",
41                 "ec2:DetachInternetGateway",
42                 "ec2:ModifyVpcAttribute",
43                 "events:DeleteRule",
44                 "events:DescribeRule",
45                 "events:ListRuleNamesByTarget",
46                 "events:ListRules",
47                 "events:ListTargetsByRule",
48                 "events:PutRule",
49                 "events:PutTargets",
50                 "events:RemoveTargets",
51                 "iam:CreateRole",
52                 "iam:DeleteRole",
53                 "iam:DeleteRolePolicy",
54                 "iam:GetRole",
55                 "iam:PassRole",
56                 "iam:PutRolePolicy",
57                 "iot:CreateTopicRule",
58                 "iot:DeleteTopicRule",
59                 "iot:DisableTopicRule",
60                 "iot:EnableTopicRule",
61                 "iot:ReplaceTopicRule",
62                 "kinesis:CreateStream",
63                 "kinesis:DeleteStream",
64                 "kinesis:DescribeStream",
65                 "lambda:*",
66                 "logs:CreateLogGroup",
67                 "logs:DeleteLogGroup",
68                 "logs:DescribeLogGroups",
69                 "logs:DescribeLogStreams",
70                 "logs:FilterLogEvents",
71                 "logs:GetLogEvents",
72                 "s3:CreateBucket",
73                 "s3:DeleteBucket",
74                 "s3:DeleteBucketPolicy",
75                 "s3:DeleteObject",
76                 "s3:DeleteObjectVersion",
77                 "s3:GetObject",
78                 "s3:GetObjectVersion",
79                 "s3:ListAllMyBuckets",
80                 "s3:ListBucket",
81                 "s3:PutBucketNotification",
82                 "s3:PutBucketPolicy",
83                 "s3:PutBucketTagging",
84                 "s3:PutBucketWebsite",
85                 "s3:PutEncryptionConfiguration",
86                 "s3:PutObject",
87                 "sns:CreateTopic",
88                 "sns:DeleteTopic",
89                 "sns:GetSubscriptionAttributes",
90                 "sns:GetTopicAttributes",
91                 "sns:ListSubscriptions",
92                 "sns:ListSubscriptionsByTopic",
93                 "sns:ListTopics",
94                 "sns:SetSubscriptionAttributes",
95                 "sns:SetTopicAttributes",
96                 "sns:Subscribe",
97                 "sns:Unsubscribe",
98                 "states:CreateStateMachine",
99                 "states:DeleteStateMachine"
100             ],
101             "Effect": "Allow",
102             "Resource": "*"
103        }
104     ],
105     "Version": "2012-10-17"
106 }

After creating this policy go back to the user creation, refresh the policies and select the one we just created. Skip tags, and create the user!

 

Download access key and secret access key

This page will never be shown to you ever again, so just download the CSV file. Once you have downloaded the key, we don’t need to do anything with AWS. We will come back later to check up on the status, but that will be it.

Photo by Cameron Venti on Unsplash
 

Travis CI

Head over to travis-ci.org, there you can deploy opensource projects for free. If you would like to deploy private projects, you will need to pay the upgraded version.

Sign in with GitHub, head to your settings, sync account and toggle the repo we created!

Now, when we push this repo to GitHub, Travis-CI will look for a '.travis.yml' file, which contains the configuration we want. A virtual machine will boot up and run the commands needed.

 

Create .travis.yml file

Head back to the code editor, and on the root level of the folder structure, create a new file titled

.travis.yml

Then you will need to paste this code.

1  language:
2    - scala
3  jdk:
4    - oraclejdk8
5  scala:
6    - 2.12.8
7  branches:
8  only:
9    - master
10 before_install:
11   - nvm install 6
12   - node --version
13   - npm --version
14 install:
15   - npm install -g serverless
16 script:
17   - sbt assembly
18 after_script:
19   - serverless deploy -v

I think this is the real take away of this post. I spent quite some time coming up with this. There are many versions of this file, but this is the one that really worked for me.

 

Add access keys

Now we need to add our AWS access key and secret access key. There are two different ways you can do it.

  1. Encrypt the key and add it to '.travis.yml' this will be public on GitHub, but no one is able to decode it.
  2. Set the environment variable in the Travis-ci website under settings.

This really depends on your personal taste. I tried both and prefer the first method. If I am collaborating with someone else on this project, and a new IAM needed to be created, the other person will have to be given permission to edit on Travis, if you start having larger teams, this becomes a hassle. There might be a team management tool in Travis, but it means I have to do the overhead. So its easier to stick the encoded variables right in Travis. You can find more details about Travis encryption here.

#install travis, you will need to have ruby
gem install travis

#inside your repo add your keys from the csv file
travis encrypt AWS_ACCESS_KEY_ID="<your key here>" --add
travis encrypt AWS_SECRET_ACCESS_KEY="<your key here>" --add

Once you run the code, look back at your '.travis.yml' a long string of cryptic text is added. This is your access key and secret access key. To be honest, it is okay to leave the access key as is, and only encrypt the secret access key, but what could be so bad about being extra careful.

Once the two keys are added, no one can even tell that the global variables set are AWS related.


Photo by Jakob Owens on Unsplash

Lights, Camera, Action!

Let's see if all this time spent was actually worth it!

git add .
git commit -m "adding travis.yml and secure keys"
git push

You will see the Travis come to life. It will spit out a lot of things. It will run 'sbt assemly' and create the target jar file, packaging everything. Then the serverless framework will take that jar, head over to AWS and deploy a cloud formation template to create all the resources needed. The jar file is uploaded into an S3 bucket, also auto-generated. It will create a lambda function and attach the API gateway.

This is what you will see if all the stars aligned. Go to the endpoint printed in the output. Make sure to grab the 'endpoints' link, not the 'ServiceEndpoint' link. We didn’t define anything for the root directory so you will get an error.
It works!
Photo by Aaron Burden on Unsplash
 

Summary

In this post, we covered a wide range of tools and frameworks. First, we created the lambda function code using the serverless framework. Next, we created a GitHub repo to host our code. Then, we created an AWS IAM role and gave the credentials to travis.yml. Last, we committed our code and saw the AWS resources getting created and outputting the endpoint we can use.

All the code in this post can be found hereHappy coding!'

 

This article was written by Chris Oh and originally posted on Medium.