With Akka Cluster being such a powerful tool it has many benefits however it may not always be applicable in every project.
Why is this? SoftwareMill, Senior Software Engineer, Michał Matłoka gives us the reasoning behind why using Akka Cluster for interservice communication in a microservice architecture may not work in your favour.
'Akka Cluster is a very powerful tool allowing to model high performance stateful applications. Quite recently we have encountered a few clients with systems combining multiple different services into a single Akka Cluster. Let’s discuss today why this is a bad idea, and why Akka is not designed for such a scenario.
Let’s imagine that you have a system composed of 20 microservices. How would it work on a single Akka Cluster?
1. Who is the seed?
First of all, Akka Cluster requires seed nodes. Those are the initial contact points which other nodes will be joining. When you are deploying a microservice number 18, then which one is it going to join? We have seen scenarios where dedicated services were created, with a sole purpose of being seeds for other services. In such a scenario your services are tightly coupled, and nothing can be in practice deployed, when the special Seed Node service is down.
If you are using Kubernetes, Mesos or other similar systems, then Akka Management extensions allow to manage seeds discovery automatically for you. To read more about that take a look at Grzegorz Kocur’s blog post Running akka-cluster on Kubernetes.
2. Cluster state — deployments, split brain resolver
Each of the Akka Cluster node has a specific current state assigned.
When the node is being intentionally shut down, then you make sure it notifies the cluster about that fact. Akka Managment with Kubernetes usually does that automatically, but when you don’t use that package, the most common scenario is that you have some custom pre-stop hook. However, if you don’t notify the cluster that this shut down is intentional, then the node will go into the unreachable state. So what happens in case in which you shut down half of your microservices in the cluster? Well, it may consider this as a split brain. It may conclude it is a network partition and depending on the algorithm of a Split Brain Resolver the result may be that the remaining working part will get shut down, assuming that second part of the cluster is working and containing the majority.
Be aware that all of the services in the cluster are tied by the cluster state, so you need to make sure that lifecycle related operations are properly executed. This also means that often you just can’t shut down any part of your system, which means that services are again, tightly coupled.
Akka Cluster has a setting named Auto-Downing, making the unreachable nodes go automatically directly to status down. However it is not recommended to use it in the production environment especially when you are using Cluster Singleton, Sharding or Akka Persistence.
3. Binary compatibility
Let’s say that you reach the moment when you would like to use a new Scala/JDK version in a new fresh microservice. However, with single Akka Cluster it is not so simple. All nodes must be binary compatible! This concerns not only Akka versions in all of the services, but also Scala and Java versions if you are still using java serialization (you shouldn't).
Having the requirement of compatible Akka versions often leads to the creation of the 'common' project. Many people consider sharing code between services as an anti-pattern. You may say that it is only a small lib solving this one particular problem, but often it goes out of the control growing into the biggest project in the organisation.
4. At most once
Akka by default uses at-most-once message delivery semantics. It may not be the best choice for inter-service communication (depending on the project of course).
There is a special mix-in called AtLeastOnceDelivery, however you need Akka Persistence in order to be able to use it.
Currently, most developers use the standard untyped Actors. When you connect different services into a cluster, then usually you fetch the 'ActorRef' of the service located in a different applications by its path. This means that the “interface” for communication does not include any type information of expected messages. You need to switch back-and-forth between repositories to see what commands can be handled. This may lead to implementation errors, but makes the test writing process much more difficult. Some people may strive to share the code fetching 'ActorRef's' and wrap it in some traits or to introduce test Actor mocks. This, however, causes the infamous 'common' module to grow and introduces the coupling to the system.
6. Akka team says so
In general we recommend against using Akka Cluster and actor messaging between different services because that would result in a too tight code coupling between the services and difficulties deploying these independent of each other, which is one of the main reasons for using a microservices architecture.
I won’t deny that there are systems where different services are connected into a single Akka Cluster, they are working and making money. However, the choice of Akka Cluster as the main method of communication is not the best thing. It often leads to a tighter coupling of services, resulting often in the overhead related to development and management of deployments.
What can you do instead? Well, there are a lot of options. The good “old” REST HTTP APIs or the binary gRPC (there is even the Akka gRPC). However, you can also base your microservices architecture on messages, and use e.g. Apache Kafka.