Connecting...

W1siziisimnvbxbpbgvkx3rozw1lx2fzc2v0cy9zawduawz5lxrly2hub2xvz3kvanbnl2jhbm5lci1kzwzhdwx0lmpwzyjdxq

Docker and managing Integration tests — A better approach by Afsal Thaj

W1siziisijiwmtgvmtivmdyvmtavntgvmtuvmjc2l2rvy2tlci5qcgcixsxbinailcj0ahvtyiisijkwmhg5mdbcdtawm2uixv0

Image credit Docker.

In today's article Software Engineer Afsal Thaj looks at the requirements and steps for ensuring that everything that you do with docker are first class citizens in the project that can run along with other test cases in your project when using Docker and managing Integration test. 

 

'As a summary of requirements, there shouldn’t be any more separate docker related orchestrations. Everything that you do with docker are first class citizens in the project that can run along with other test cases in your project. We can see bits and pieces of code everywhere trying to achieve this, but here we strive for the best possible way of doing it.

Let’s list down our requirements first.

 

Requirement 1

For certain test cases we need to run a docker-compose as the first step, which in turn depends on creating a few images, plus a few other custom steps.

 

Requirement 2

Separately being able to run tests that rely on docker under the hood, while running 'sbt test' should ideally avoid these slow running tests.

Although sbt 'it' can solve this to some extent, we need a better name-spacing management. Integration tests are in general refer to a wider context, while we prefer specific management of docker tests.

 

Requirement 3

As part of these requirements, we also prefer having the docker-compose.yml generated from sbt without polluting git history. In short, just about everything related to docker should come only through SBT. We need only 1 complexity, not more!

I am motivated to consider this as a requirement because I have had tough times managing the dockerFiles and compose yamls as separate hardcoded files. This will also avoid the complexities of overloaded context being passed to docker daemons and thereby accidentally slowing down the image creations. I initially solved this using a shell-script like a good pragmatic developer, but not anymore!

I haven’t found a proper resource anywhere detailing how we could achieve them nicely. So let us try now!

The main concepts involved:
1) Dockerish stuffs
2) Sbt Tasks
3) Sbt Scope
4) Sbt Tags for scala test cases (with respect to Specs2 and ScalaCheck).

Don’t worry, we will go step by step.

 

Step 1

Add the required plugins in plugins.sbt

addSbtPlugin("com.tapad" % "sbt-docker-compose" % "1.0.34")
addSbtPlugin("se.marcuslonnberg" % "sbt-docker" % "1.5.0")

 

Step 2

Let’s define a context that runs only docker related tests:

1. lazy val DockerTest = config("docker-int") extend Test

Ok, now we intend to run only those tests that depends on 'Docker' in this context. To avoid running other tests in this context, we use 'tags'.

Tests.Argument(
  TestFrameworks.Specs2, “include”, “DockerComposeTag”
)`

This is a snippet that I used in build.sbt that says, let me include only those tests with the tag “DockerComposeTag”. You can name the tag as you wish. Don’t worry too much on this now. We will get to know more on tags later.

Note: This work with any test frameworks: ScalaCheck, ScalaTest, Specs, Specs2, JUnit.

In the example above, the argument required for Spec2 is 'include' (or 'exclude' ) to include (or exclude) only those tests with the tag 'DockerComposeTag', and that in ScalaTest is 'n'(or '1') to include (or exclude) tests with the 'DockerComposeTag'.

1 lazy val myProject = 
2 (project in file("myProject"))
3   .enablePlugins(DockerPlugin, DockerComposePlugin)
4   .config(DockerTest)
5   .settings(inConfig(DockerTest)(Defaults.testTasks): _*)
6   .settings(testOptions in DockerTest := Seq(
7      Tests.Argument(TestFrameworks.Specs2, "include", "DockerComposeTag"))
8   )

This ensures that 'sbt docker-int:test' will run only those tests that are tagged with “DockerComposeTag”.

We will see how to tag a test case later on. Before that we need to ensure we are not running these long running test when we call simple 'sbt test'. To do that, we add 1 more level of setting as given below.

1 lazy val myProject = 
2   (project in file("myProject"))
3   .enablePlugins(DockerPlugin, DockerComposePlugin)
4   .config(DockerTest)
5   .settings(inConfig(DockerTest)(Defaults.testTasks): _*)
6   .settings(testOptions in DockerTest := Seq(
7      Tests.Argument(TestFrameworks.Specs2, "include","DockerComposeTag"))
8   )
9   // We exclude in other tests
10   .settings(testOptions in Test := Seq(
11     Tests.Argument(TestFrameworks.Specs2, "exclude", "DockerComposeTag"))
12   )

This ensures that 'sbt test' avoid long running tests.

 

Step 3

Now the next step is to create images, create docker-compose.yml, and 'dockerComposeUp' before we run the 'DockerTest'. We can achieve this using the addition of another setting to the above build configuration.

1 lazy val myProject = 
2   (project in file("myProject"))
3   .enablePlugins(DockerPlugin, DockerComposePlugin)
4   .config(DockerTest)
5   .settings(inConfig(DockerTest)(Defaults.testTasks): _*)
6   .settings(testOptions in DockerTest := Seq(Tests.Argument(TestFrameworks.Specs2, "include",
7     "DockerComposeTag")))
8   // We exclude in other tests
9   .settings(testOptions in Test := Seq(
10     Tests.Argument(TestFrameworks.Specs2, "exclude", "DockerComposeTag"))
11   )
12   // Add the settings into the project before it is being used.
13   .settings(DockerUtil.settings)
14   .settings((test in DockerTest) := {
15     (test in DockerTest).dependsOn(DockerUtil.dockerComposeUp).value
16   })

 

Step 4

Let us see the sample 'DockerUtil.scala' that is responsible for 'docker-compose up' before any 'DockerTest' is run.

1
2 // Take a look at the snippet. This is just a sample. You might have more/less steps as part of docker compose up
3 object DockerUtil {
4   lazy val makeDockerFile = taskKey[Seq[ImageName]]("Make a docker image for the app.")
5   lazy val makeDockerComposeFile = taskKey[File]("Make docker compose file for the project in resource managed location.")
6   lazy val dockerComposeUp = taskKey[File]("docker-compose up for the project")
7
8   def relativeDockerComposePath(resPath: File) = resPath / "docker" / "bin" / "docker-compose.yml"
9
10   def settings = Seq(
11     makeDockerFile := {
12       val artifact: File = (assembly in  Compile).value
13       val artifactTargetPath = s"/app/${artifact.name}"
14
15       val dockerfile = new Dockerfile {
16         from("openjdk:10-jre-slim")
17         add(artifact, artifactTargetPath)
18         cmd("java", "-jar", artifactTargetPath)
19       }
20      
21       val dockerImageNames = Seq(
22         ImageName(s"afsalthaj/myProject:latest"),
23         ImageName(s"afsalthaj/myProject:${version.value}")
24       )
25
26      DockerBuild(
27         dockerfile,
28         DefaultDockerfileProcessor,
29         streamImages,
30         BuildOptions(),
31         (target in docker).value,
32         sys.env.get("DOCKER").filter(_.nonEmpty).getOrElse("docker"),
33         Keys.streams.value.log
34       )
35
36       dockerImageNames
37     },
38
39     makeDockerComposeFile := {
40       val resourcePath = (resourceManaged in Compile).value
41
42       val imageName: ImageName = makeDockerFile.value(0)
43
44       val targetFile = relativeDockerComposePath(resourcePath)
45
46       // A sample content
47       val content =
48         s"""
49            |version: '2'
50            |services:
51            |  zookeeper:
52            |    image: confluentinc/cp-zookeeper:5.0.0
53            |    hostname: zookeeper
54            |    ports:
55            |      - '32181:32181'
56            | ......
57            
58            |  kafka:
59            |    image: confluentinc/cp-enterprise-kafka:5.0.0
60            |    hostname: kafka
61            |  .......
62            |
63            |  schema-registry:
64            |    image: confluentinc/cp-schema-registry:5.0.0
65            |
66            |  ....   
67            |  custom-application:
68            |    image: ${imageName}
69            |    hostname: my-project-app
70            |    depends_on:
71            |      - kafka
72            |      - schema-registry
73            |      - kafka-create-topics
74            |    ports:
75            |      - '7070:7070'
76    
77         """.stripMargin
78
79       IO.write(targetFile, content)
80       targetFile
81     },
82
83     dockerComposeUp := {
84       val file = makeDockerComposeFile.value
85       DockerComposePlugin.dockerComposeUp("testInstance", file.getAbsolutePath)
86       file
87     },
88   )
89
90 }

 

Step 5

Now the final step is to write test that are tagged with 'DockerComposeTag'.

1 import org.specs2.data.Tag
2 import org.specs2.matcher.MatchResult
3 import org.specs2.scalacheck.ScalaCheckFunction1
4 import org.specs2.specification.core.SpecStructure
5 import org.specs2.{ ScalaCheck, Specification }
6
7
8 class IntegrationTest extends Specification with ScalaCheck { self =>
9
10   // The test requires external environment. In this case
11   object DockerComposeTag extends Tag("DockerComposeTag")
12 
13   def is: SpecStructure =
14     s2"""Runs an actual kafka cluster, stream engine, and pushes the data continuously $test ${tag(DockerComposeTag)}"""
15
16   private def test: ScalaCheckFunction1[String, MatchResult[String]] =
17     prop { g: String =>
18       // Call kafka related things. Probably set minTestsOk=10, or a lower number. Property based on an actual environment
19       // does seem to have an impedence mismatch.But for that reason, we don't need to hardcode sample data for instance. 
20       // Upto you to decide on this. You can just rely on specs2 and not ScalaCheck when you do this test, if you need.
21       g must_=== g
22     }
23 }

 

Result:

'sbt docker-int:test' will run 'docker-compose up' as the first step, and runs only 'IntegrationTest'

'sbt test' will run all tests except 'IntegrationTest' and there won’t be any 'docker-compose up' or any sort of operations related to docker.

No more bash orchestration and separate compose files, or dockerFiles in the project.

Thanks, and have fun!'

 

This article was written by Afsal Thaj and posted originally on Medium.