Connecting...

W1siziisimnvbxbpbgvkx3rozw1lx2fzc2v0cy9zawduawz5lxrly2hub2xvz3kvanbnl2jhbm5lci1kzwzhdwx0lmpwzyjdxq

Testing Incrementally with ZIO Environment by John De Goes

W1siziisijiwmtkvmdmvmjavmtevmtgvmdyvodaxl3blegvscy1wag90by00ntk3otmuanblzyjdlfsiccisinrodw1iiiwiotawedkwmfx1mdazzsjdxq

Do you want to achieve testability? You can achieve it using environmental effects! 

In this article by John De Goes you'll see that environmental effects can provide easier testability without sacrificing teachability, abstraction, or type inference.

 

'In my last post, I introduced ZIO Environment, which is a new feature in ZIO that bakes in a high-performance, type-safe, and fully-inferred reader effect into the ZIO data type.

This capability leads to a way of describing and testing effects that I call environmental effects. Unlike tagless-final, which is difficult to teach, difficult to abstract over, and does not infer, environmental effects are simple, abstract well, and infer completely.

Moreover, while proponents of tagless-final argue that tagless-final parametrically constrains effects, my last post demonstrated this is not quite correct: not only can you embed raw effects anywhere in Scala, but even without leaving Scalazzi (the purely functional subset of Scala), you can lift arbitrary effects into any 'Applicative' functor.

The inability of tagless-final to constrain effects is more than just theoretical:

  • New Scala functional programmers use effect type classes like 'Sync' everywhere (which are themselves lawless and serve only to embed effects), and they embed effects using lazy methods, like 'defer' or 'point'.
  • Even some experienced Scala functional programmers embed effects in pure methods (for example, exceptions in the functions they pass to 'map', 'flatMap'), and some effect types encourage this behavior.

Tagless-final can be used by a well-trained and highly-disciplined team to constrain effects, but the same can be said for many approaches, including environmental effects.

After professionally and pedagogically wrestling with these issues for several years now, I’ve come to the conclusion there are just two legitimately compelling reasons to use tagless-final:

  1. Avoiding commitment to a specific effect type, which can be useful for library authors, but which is less useful for application developers (often it’s a hinderance!);
  2. Writing testable functional code, which is fairly straightforward with tagless-final because you can just create test instances for different effect type classes.

While testability is a compelling reason to use tagless-final, it’s not necessarily a compelling reason to choose tagless-final over other approaches—in particular, over environmental effects.

In this post, I’m going to show you how to use environmental effects to achieve testability. I hope to demonstrate that environmental effects provide easier and more incremental testability—all without sacrificing teachability, abstraction, or type inference.

 

A Web App

Let’s say we are building a web application with ZIO. Suppose the application was originally written with 'Future' or perhaps some version of 'IO' or 'Task'.

Later, the application was ported to ZIO’s 'Task[A]', which is a type alias for 'ZIO[Any, Throwable, A]'—representing an effect that requires no specific environment and that may fail with any 'Throwable'.

Now let’s say one of the functions in our application, called 'inviteFriends', invites the friends of a given user to the application by sending them emails:

def inviteFriends(userID: UserID): Task[List[Send]] =
  for {
    user    <- DB.lookupUser(userID)
    friends <- Social.getFriends(user.facebookID)
    resp    <- ZIO.foreach(friends) { friend =>
                 Email.invite(user, friend.email)
               }
  } yield resp

Portions of the 'Social', 'DB', and 'Email' objects are shown below:

object Social {
  ...
  def getFriends(fid: FacebookID): Task[List[FacebookProfile]] = ???
  ...
}
object DB {
  ...
  def lookupUser(uid: UserID): Task[UserProfile] = ???
  ...
}
object Email {
  ...
  def invite(user: UserProfile, friend: FacebookProfile): Task[Send] = ???
  ...
}

As currently written, our web application is not very testable. The function 'inviteFriends' makes direct calls to database functions, Facebook API functions, and email service functions.

While we may have automated tests for our web service, because our application interacts directly with the real world, the tests are actually system tests, not unit tests. Such tests are very difficult to write, they run slowly, they randomly fail, and they test much more than our application logic.

We do not have time to rewrite our application, and we cannot make it testable all at once. Instead, let’s try to remove dependency on the live database for the 'inviteFriends' function.

If we succeed in doing this, we will make our test code a little better, and after we ship the new code, we can incrementally use the same technique to make the function fully testable—fast, deterministic, and without any external dependencies.

 

Steps Toward Testability

To incrementally refactor 'inviteFriends' to be more testable, we’re going to perform the following series of refactorings:

  1. Introduce a type alias.
  2. Introduce a module for the database.
  3. Implement a production database module.
  4. Integrate the production module.
  5. Implement a test database module.
  6. Test the 'inviteFriends' function.

Each of these steps will be covered in the sections that follow.

 

Introduce A Type Alias

To simplify the process of refactoring our application, we’re going to first introduce a simple type alias that we can use in the definition of 'inviteFriends' and the functions that call it:

type Webapp[A] = Task[A]
Now we will mechanically update the 'lookupUser' function and any functions that call it to use the type alias:
def inviteFriends(userID: UserID): Webapp[List[Send]] =
  for {
    user    <- DB.lookupUser(userID)
    friends <- Social.getFriends(user.facebookID)
    resp    <- ZIO.foreach(friends) { friend =>
                 Email.invite(user, friend.email)
               }
  } yield resp

As an alternative to this technique, we could simply delete the return types entirely. However, it’s a good practice to place return types on top-level function signatures, so developers without IDEs can easily determine the return type of functions.

After this step, we are ready to introduce a service for the database.

 

Introduce a Database Module

The database module will provide access to a database service.

As discussed in my post on ZIO Environment, the database module is an ordinary interface with a single field, which contains the database service.

We can define both the module and the service very simply:

// The database module
trait Database {
  val database: Database.Service
}
object Database {
  // The database service
  trait Service {
    def lookupUser(uid: UserID): Task[UserProfile]
  }
}

Notice how we have decided to place just one method inside the database service: the 'lookupUser' method. Although there may be many database methods, we don’t have time to make all of them testable, so we will focus on the one required by the 'inviteFriends' method.

We are now ready to implement a production version of the service.

 

Implement Production Module

We will call the production database module 'DatabaseLive'. To implement the module, we need only copy and paste the implementation of 'Database.lookupUser' into our implementation of the service interface:
trait DatabaseLive extends Database {
  val database = 
    new Database.Service {
      // Implementation copy/pasted from
      // DB.lookupUser:
      def lookupUser(userID: UserID) = 
        ...
    }
}
object DatabaseLive extends DatabaseLive

For maximum flexibility and convenience, we have defined both a trait that implements the database module, which can be mixed into other traits, and an object that extends the trait, which can be used standalone.

 

Integrate Production Module

We now have all the pieces we need to replace the original 'DB.lookupUser' method, whose actual implementation now resides inside our 'DatabaseLive' module:

object DB {
  ...
  def lookupUser(uid: UserID): ZIO[Database, Throwable, UserProfile] = 
    ZIO.accessM(_.database lookupUser uid)
  ...
}

The 'lookupUser' method merely delegates to the database module, by accessing the model through ZIO Environment ('ZIO.accessM').

Here we don’t use the 'Webapp' type alias, because the functions in 'DB' will not necessarily have the same dependencies as our web application.

However, after enough refactoring, we might introduce a new type alias in the 'DB' object: 'type DB[A] = ZIO[Database, Throwable, A]'. Eventually, all methods in 'DB' might return effects of this type.

At this point, our refactoring is nearly complete. But we have to take care of one last detail: we have to provide our database module to the production application.

There are two main ways to provide the database module to our application. If it is inconvenient to propagate the 'Webapp' type signature to the top of our application, we can always supply the production module somewhere inside our application.

In the worst case, if we are pressed for time and need to ship code today, maybe we choose to provide the production database wherever we call 'inviteFriends'.

inviteFriends(userId).provide(DatabaseLive)

If we have a bit more time, we can push the 'Webapp' type synonym to the entry point of our purely functional application, which might be the main function, or it might be where our web framework calls into our code.

In this case, instead of using the 'DefaultRuntime' that ships with ZIO, we can define our own 'Runtime', which provides the production database module):

val myRuntime = 
  Runtime(DatabaseLive, PlatformLive)

The custom runtime can be used to run many different effects that all require the same environment, so we don’t have to call 'provide' on all of them before we run them.

Once we have this custom runtime, we can run our top-level effect, which will supply its required environment:

myRuntime.unsafeRun(effect)

At this point, we have not changed the behavior of our application at all—it will work exactly as it did before. We’ve just moved the code around a bit, so we can access a tiny effect through ZIO environment.

Now it’s time to build a database module specifically for testing.

 

Implement Test Module

We could implement the test database module using a mocking framework. However, to avoid all magic and use of reflection, in this post, we will build one from scratch.

For maximum flexibility, our test database module will track all calls to 'lookupUser', and supply responses using a 'Map', which can be dynamically changed by the test suite.

To support this stateful behavior, we will need a 'Ref', which is a concurrent-safe ZIO data structure that models mutable references. We will also need a simple (immutable) data structure to hold the state of the test database module.

We define the following test data structure, which is capable of tracking a list of 'UserID' values, and holding data that maps from 'UserID' to 'UserProfile'.

final case class TestDatabaseState(
  lookups : List[UserID],
  data    : Map[UserID, UserProfile]
) {
  def addLookup(uid: UserID): TestDatabaseState = copy(lookups = uid :: lookups)
}

Now we can define the service of our test database module. The service will require a 'Ref[TestDatabaseState]', so it can not only use test data, but update the test state:

final case class TestDatabaseService(ref: Ref[TestDatabaseState]) extends Database.Service {
  def lookupUser(uid: UserID): Task[UserProfile] = 
    for {
      _       <- ref.update(_.addLookup uid)
      data    <- ref.get.map(_.data)
      profile <- Task.fromEither(data.get(uid)
                   .fold(Left(new DBErr))(
                     Right(_)))
    } yield profile
}

Notice how the 'lookupUser' function stores the 'UserID' of every call in the 'lookups' field of the 'TestDatabaseState'. In addition, the function retrieves test responses from the map. If there is no response in the map, the function fails, presumably in the same way the production database would fail.

The test service must be placed in a module. In general, we should wait to create the module until the test suite, because then we will know the full set of dependencies for each test.

However, at this stage, the database service is the only dependency in our application, so we can make a helper function to create the test database module:

object TestDatabase {
  def apply(ref: Ref[TestDatabaseState]): Database = 
    new Database {
      val database: Database.Service = 
        new TestDatabaseService(ref)
    }
}

We now have all the pieces necessary to write a test of the 'inviteFriends' function, which will use our test database module.

 

Write the Test

To more easily test the 'lookupFriends' function, we will define a helper function. Given test data and input to the function, the helper will return the final test state and the output of the 'lookupFriends' function:

def runLookupFriends(data: Map[UserID, UserProfile], uid: UserID): Task[(TestDatabaseState, List[Send])] = 
  for {
    ref   <- Ref.make(TestDatabaseState(Nil, data))
    resp  <- lookupFriends(uid)
               .provide(TestDatabase(ref))
    state <- ref.get 
  } yield (state, resp)

The helper function creates a 'Ref' with the initial test data, uses the 'Ref' to create the 'TestDatabase' module, and then supplies the database module to the effect returned by 'lookupFriends'.

With this helper function, writing a test becomes quite simple:

class TestSuite extends DefaultRuntime {
  def testLookupFriends = {
    val (state, resp) = 
      unsafeRun {
        runLookupFriends(
          Map(TestUserID -> TestUserProfile),
          TestUserID
        )
      }
    
    (state.lookups must_=== List(TestUserID)) and
      (resp must_=== TestResponse)
}

This test for 'inviteFriends' is not perfect. It still interacts with a real Facebook API and a real email service. But compared to whatever tests already exist, at least this test does not interact with a real database.

Moreover, we were able to make this change in a minimally disruptive manner.

 

A Glimpse Beyond

After a little more refactoring, of course, we would succeed in making 'inviteFriends' fully testable. Even after the full refactoring, the code for 'lookupFriends' would not change.

Instead, our type alias for 'Webapp' would expand to include new environmental effects:

type WebappFX = 
  Database with Social with Email

type Webapp[A] = ZIO[WebappFX, Throwable, A]

Now all the methods in the 'DB', 'Social', and 'Email' objects would simply delegate to their respective modules using 'ZIO.accessM'.

Running our application would now look a little different:

val myRuntime = 
  Runtime(
    new  DatabaseLive 
    with SocialLive 
    with EmailLive, PlatformLive)
...
myRuntime.unsafeRun(effect)

Finally, testing 'lookupFriends' would be entirely fast, deterministic, and type-safe, without any dependencies on external systems, or use of any reflection.

 

Summary

Environmental effects make it easy to test purely functional applications—significantly easier and with less ceremony than tagless-final, with full type inference, and without false promises.

Morever, with environmental effects, we can make even just a single function testable, by pushing that function into an environmental effect. This requires just a few small changes that can be done incrementally without major disruption to the application.

This ability lets us make incremental progress towards better application architecture. We don’t have to solve all the problems in our code base at once. We can focus on making our application a little better each day.

While this post focused on ZIO Environment, if you’re using 'Future', Monix, or Cats IO, you can still use this approach with a 'ReaderT' monad transformer. With a monad transformer, you will lose type inference and some performance, but you will gain the other benefits in a more familiar package.

If you’d like to give ZIO Environment a try, hop over to the Github project page, and be sure to stop by the Gitter channel and say hello.

In future posts, I will cover how to provide partial dependencies, how to model services that require other services (the graph problem), how to hide implementation details, and how this approach differs from the classic cake pattern. Stay tuned for more!'

 

This article was written by John De Goes and originally posted on degoes.net