Functional DevOps with Scala and Kubernetes by Joan Goyeau

It is not important to know about the business use case here, except that this system will have to be delivered in quick iterations to production.

I guess the first question is where are we going to run our applications?

Kubernetes

We are not going to deep dive in how Kubernetes works here but it’s important as a developer to understand some high level concepts which are well documented on https://kubernetes.io/docs/concepts/.

Ok now you might wonder how are we going to deploy on Kubernetes?

Orkestra

• We are functional developers convinced by its advantages over unsafe Yaml, Python or Groovy code. So it’s perfect here since the configuration is in Scala.
• It’s extendable with Scala/Java libraries.
• It runs on Kubernetes, so we can reduce operational management by using a managed Kubernetes cluster such as GKE or EKS. Plus we can reuse the compute power left on a cluster for DevOps tasks.
• It’s Highly Available, not that we are looking into 100% up time for an internal tool, but it let’s operational maintenance possible without impacting the usage of the tool during working hours.
• It’s Fully scalable, so we can run many jobs in parallel if the underlying Kubernetes cluster is large enough.

Continuous Integration

Continuous Integration is fairly easy and there is a tones of tools out there to do it like Travis, CircleCI, GitLabCI, Jenkins and many more.

All we want is probably for each pull requests execute a full compilation, the tests, some linter (like Scalafix, Scalafmt) and maybe check that the Contributor License Agreement has been signed. In any case it should be a matter of few commands.

Of course Orkestra can run those checks on pull requests. Let’s have a look at the code. Remember that at anytime you can get more documentation about Orkestra on https://orkestra.tech.

We first need to define a job that executes the checks:

object PullRequestChecks {
  lazy val board = JobBoard[GitRef => Unit](JobId("pullRequestChecks"), "Pull Request Checks")(
    Input[GitRef]("Git ref")
  )

  lazy val job = Job(board) { implicit workDir => gitRef =>
    Github.statusUpdated(Repository("myOrganisation/backend"), gitRef) { implicit workDir =>
      sh("./sbt test")
    }
  }
}

This is the first time we define a job so let’s take a moment to understand what’s going on here step by step:

We define a board that will represent the UI. There is different types of boards but we will be using a JobBoard here.
• In the JobBoard we define the signature of the function this job will be executing, here GitRef => Unit.
• We need to give a unique ID and a nice name for the display.
• Lastly we give the UI elements for the parameters form. Here only a Input[GitRef] with a nice name for the display.

Alright, let’s have a look at the UI and see how beautiful it is:

Continuous Integration was pretty simple but now that we have this so well checked code, we have to run it somewhere!

This arises some questions like:

• What will we be deploying? (our applications, databases? …)
• Where? (Kubernetes? AWS managed service? GCP managed service?…)
• How? (Docker? Yum?…)
• Configurations? (auto scaling?, secrets? environments?…)
• Extra tasks? (running migrations, copy data between environments, switching load balancers…)

Before doing any deployment we probably need an environment where to deploy our services and as we just saw actually multiple environments:

• Staging: where we will continuously deploy the master branch.
• Production: where we will deploy versions we are sure they work on staging. Our end users will use this environment, so it needs to be up at all times!

object CreateEnvironment {
  lazy val board = JobBoard[String => Unit](JobId("createEnvironment"), "Create Environment")(
    Input[String]("Environment name")
  )

  lazy val job = Job(board) { implicit workDir => environmentName =>
    Await.result(for {
      _ <- Kubernetes.client.namespaces.createOrUpdate(Namespace(environmentName))
      _ <- Elasticsearch.deploy(environmentName)
    } yield (), 1.minute)
  }
}

object DeployBackend {
  lazy val board = JobBoard[(String, String) => Unit](JobId("deployBackend"), "Deploy Backend")(
    Input[String]("Version"),
    Input[String]("Environment name")
  )

  lazy val job = Job(board) { implicit workDir => (version, environmentName) =>
    Await.result(DeployBackend(version, environmentName), 1.minute)
  }
}

We probably need also a job that publishes the artifact:

object PublishBackend {
  lazy val board = JobBoard[(GitRef, Boolean) => String](JobId("publishBackend"), "Publish Backend")(
    Input[GitRef]("Git ref"),
    Checkbox("Run checks")
  )

  lazy val job = Job(board) { implicit workDir => (gitRef, runChecks) =>
    Await.result(PublishBackend(gitRef, runChecks), 1.hour)
  }
}

Let’s also create a job that publishes and deploys straight away, so that if we want to deploy from source code we just have to do one action:

object PublishAndDeploy {
  lazy val board =
    JobBoard[(GitRef, Boolean, String) => Unit](
      JobId("publishAndDeployBackend"),
      "Publish and Deploy Backend"
    )(
      Input[GitRef]("Git ref"),
      Checkbox("Run checks", checked = true),
      Input[String]("Environment name")
    )

  lazy val job = Job(board) { implicit workDir => (gitRef, runChecks, environmentName) =>
    Await.result(for {
      version <- PublishBackend.job.run(gitRef, runChecks)
      _ <- DeployBackend.job.run(version, environmentName)
    } yield (), 1.hour)
  }
}

Again we need to update the Orkestra object to register our new jobs. At the same time we can add a GitHub hook for the automated deployment of the master branch to staging:

object Orkestra extends OrkestraServer with GithubHooks {
  lazy val board = Folder("Orkestra")(
    PullRequestChecks.board,
    CreateEnvironment.board,
    PublishBackend.board,
    DeployBackend.board,
    PublishAndDeploy.board
  )

  lazy val jobs = Set(
    PullRequestChecks.job,
    CreateEnvironment.job,
    PublishBackend.job,
    DeployBackend.job,
    PublishAndDeploy.job
  )
  
  lazy val githubTriggers = Set(
    PullRequestTrigger(Repository("myOrganisation/myRepo"), PullRequestChecks.job)(),
    BranchTrigger(Repository("myOrganisation/myRepo"), "master", PublishAndDeploy.job)("staging")
  )
}

• See that we are using a BranchTrigger to trigger the PublishAndDeploy job as soon as the master branch is updated.

Great, we now have a minimum viable CI/CD!

Extra jobs

This architecture already created us some repetitive tasks we will have to do from time to time.

One of them is copying a subset of data from the production environment to the staging environment. We probably want to override all staging data with fresh new real world data coming from production regularly, like every week.

Let’s write this copy data job:

object CopyData {
  lazy val board = JobBoard[(String, String) => Unit](JobId("copyData"), "Copy Data")(
    Input[String]("Source environment"),
    Input[String]("Destination environment")
  )

  lazy val job = Job(board) { implicit workDir => (source, destination) =>
    Await.result(copyData(source, destination), 1.minute)
  }
}

The good part of writing this job in Scala is that if your backend is written in a JVM language (Java, Scala, Groovy…) you can publish a jar of it, make the Orkestra project depend on it and call the functions directly. In this example we’ll assume that copyData() is a function defined in the backend code.

Again don’t forget to register the job and the board. And we will be adding a Cron trigger so that this job is ran every Monday at 5am (outside of developers working hours) so that we start the week on fresh data:

object Orkestra extends OrkestraServer with GithubHooks with CronTriggers {
  lazy val board = Folder("Orkestra")(
    PullRequestChecks.board,
    CreateEnvironment.board,
    PublishBackend.board,
    DeployBackend.board,
    PublishAndDeploy.board,
    CopyData.board
  )

  lazy val jobs = Set(
    PullRequestChecks.job,
    CreateEnvironment.job,
    PublishBackend.job,
    DeployBackend.job,
    PublishAndDeploy.job,
    CopyData.job
  )

  lazy val githubTriggers = Set(
    PullRequestTrigger(Repository("myOrganisation/myRepo"), PullRequestChecks.job)(),
    BranchTrigger(Repository("myOrganisation/myRepo"), "master", PublishAndDeploy.job)("staging")
  )

  lazy val cronTriggers = Set(
    CronTrigger("0 5 * * 1", CopyData.job)("prod", "staging")
  )
}

Slack integration