A review of Scala Days New York 2018 by competition winner Craig Bordelon


Well done again to our competition winner of the Scala Days New York ticket. Craig Bordelon attended the conference of behalf of Signify Technology and he has given his review on what he enjoyed about Scala Days.

'Scala Days 2018 New York lived up to the hype. Hundreds of like-minded Scala developers, friends and sponsors descended upon midtown Manhattan to exchange info, learn from talks, and celebrate. There were three tracks exclusive of keynotes and the final panel session. All this and the food and t-shirts were great. Field notes from this observers talk attendance follows.

Martin Odersky with intro from Adrian Moors kicked off the conference with the topic of what to expect from and when to expect Scala 3. Scala 3 is based on the current work occurring in the doty project. At a fundamental level the changes are not that great.

For example, the Programming in Scala book for Scala 3 should remain largely the same once past the changes needed for Scala 2.13 collection classes.

At the same time Scala becomes more opinionated since its first release 14 years ago.

The compiler needs to become faster and constructs more consistent and sound at the expense of some 2.x abilities. Some features are removed such as type existentials. Union and intersection types are in with the latter replacing the “with” clause feature. Macros are currently too tied to the compiler internals. So with 3.0 everything is based on a new intermediate format called tasty. (typed abstract syntax trees).

Expect 2.13 soon and 2.14 (mid 2019) as the transitional releases and Scala 3 for mid 2020.

Martin presented the case for Scala 2 to 3 not to become another Python 2 to 3 situation (yeah a decade plus of 2 and 3 overlap -- see Wikipedia)

In the morning of day 2, Narkhede gave an impressive keynote describing the new real time data processing in many cases solved with the Kafka platform.

The past data centric platforms were the relational database (and apps), then data warehousing with Extract/Transform/Load (ETL) and old-style message queues. Now its real time and streaming.

Kafka is a sweet spot in terms of performance, persistence, and latency that was not before available. She gave motivating real-world examples from Norwegian government, Audi connected car, and Royal Bank of Canada.

Malawski gave terrific view of the new changes planned in Akka. Akka finally gets typed actors after years of experimenting. All actors will be based on FSM model and a Behaviour trait. Actors are now composable using Behaviour. The current implementation runs on top of the untyped actors but expect that to change.

Other changes to Akka include the new Akka remoting with artery (via config change) from which to expect 10x performance improvement.

ActorSelection is deprecated as is the current lifecycle methods. Lifecycle is done with a new Signal trait similar to actor messaging.

Finally, a Coursera course with Akka typed is planned.

Kelly Robinson gave a great talk of the modern features of Apache Spark, i.e. DataFrames and SQL support and the tradeoffs.

She presented the example of a large (size 500 million) corpus of password data.

Spark allows nice schema inference for DataFrames and other compelling features.

She looked at password length and other metrics via some simple Spark SQL usage. She searched for use of dog names and use of Scala in passwords.

Not surprisingly,  use of bad/simple passwords is quite high.


Dick Wall gave nice code heavy talk on using the Scala type system (compile time) to guard against errors with Geographic (lat/long etc) data. The take-aways were to use the language and compiler to strongly type the data to its coordinate reference system (CRS). The code shown made use of abstract types and context bounds with other implicits to pull off the smarts.

Belsey, Gjyshinca gave excellent talk on Morgan Stanleys platform for handling state change of small parts of object graphs with use of a performant cache. There is a cache overhead for change but otherwise a big benefit for referentially transparent functions.

The technique uses a compiler plugin to recognize certain annotations on functions and classes to rewrite the handling of the involved functions.

The execution of functions can occur in parallel or across machines on a grid when invalidated parts of the dependencies are updated.

The combination of scenario (change to graph), entity (class instance), function, and function param values forms the key into the cache. The platform manages all data/graph changes.

SBT 1.x is here as talk by Yokota and Wijnand presented. Hundreds of existing sbt plugins are also converted to 1.0 with a similar number still to be converted (or maybe not if some of these are not in use).

Sbt 1.x brings unified slash syntax instead of the confusing 0.x syntax for specifying scope components.

Sbt 1.x will use semantic versioning to keep backward compatibility in force.

The sbt server understands the language server protocol (LSP) to allow it to network into IDEs.

Think of sbt server as a hub for development.

It took many years but sbt is now 1.x as quite an accomplishment.

Whaling gave fascinating talk on the latest with the Scala native project and argued that it is one of the best ways to write native code though a bit more type safety.

Anytime one is writing with equivalently C APIs there are the dangers of segfaults and memory overrun/corruption.

Small programs of size less that 10K LOC that can be written without hundreds of unsupported Java classes is the target.

The Scala native Foreign function interface (FFI) makes it easy to access new C functionality.

The benefits potentially are several times smaller execution times and millisecond start-up.

However, don’t expect the Scala compiler or sbt with Scala native any time soon.

But for other tooling Scala native can be a great choice.

A new book on the subject is in the works.

Zeiger and Richard-Foy hit the major themes for the Scala collection library changes in 2.13. Depending on what your existing code is like you may have some migration needs.

It is recommended to first compile your code in 2.12 with -deprecation and clean up the issues related to collection classes.

In some cases it’s possible to rewrite code to still compile with 2.12 and 2.13. Then in other cases code must be changed specific to 2,13.

Stay tuned for scalafix 1.0 tool that via rules can help rewrite the simpler changes automatically.

Edelsen kicked off day 3 with example from nature that we can learn distributed algorithms from.

Algorithms that ant colonies use exhibit a near optimal solution to TSP (traveling salesman problem). Guaging bandwith in networks is like ant foresting. Bee colonies, emperor penguins, geese flight all exhibit algorithms that nature uses whereby individual nodes perform simple behaviour that then emerges at the flock level to a complex algorithm. Swarm intelligence can teach us how to design clusters. System level patterns are unpredictable from behaviour of its members but emerges from the whole. No leader, no global coordination. The problems nature could help us with include communication between millions of connected self driving cars on highways;  hundreds of robots working together for disaster clean-up etc.

Devore gave a talk on reactive and fast data starting with definition of microservices as a system of autonomous collaborative distributed services.

Much advice was presented.

Cluster your microservices for resilience.

Consistency and load times/latency are determined by SLA. (service level agreement). No right answer.

Micro service  should have its own data store to be autonomous.

CRUD can be ok if historical data is not needed otherwise use event sourcing to capture the additive “log” of events.

And From MS perspective, data from outside is past data. Its own data is now.

On the topic of fast data, one warning is don’t use your messaging backplane as source of truth. When building MS based systems feedback control is very important in your system design solved in Akka through supervisor strategy. Favour anomaly detection over simple threshold violations. When you see consistent anomaly then make a control change.

Norris gave a reasoned  talk on the six basic effects which in Scala are Option, Either, List, Reader, Writer, Sate. Starting with ordinary associativity of function composition and the identity function properties, Norris motivated the relationship between Kleisli arrow associativity and flatMap. Just having flatMap defined on a type is not sufficient for (Kleisli) associativity. Laws have to be obeyed which are typically easy to show for the six basic effects. So if one didn’t know the motivation of monads before this talk, this talk made it very accessible.

Dragos gave an enlightening talk on what slows down the Scala compiler.

The main things that slow the compilation beyond expected time is implicit search and whitebox macros. Scalac can compile average Scala code at 1200 LOC/sec but slows to 480 LOCs/sec for FP heavy project code (eg moderate use of cats/Scalaz).Type checking should take around 30% of compile time. Use of -verbose with Scalac to see times of each phase. There was a fun audience participation online quiz to answer compile time and other trivia questions for compilation and other Scala basics. (So when was Scala first announced? Ans 2004)

Warski gave nice introduction fo sttp which is a Scala http client library he created that reuses major backends such as Java’s HttpClient and Akka Http. Its a programmer friendly API with URI interpolator, compression, cookies, sending form data, file upload/download, multipart uploads, streaming, http authentication, redirect. No current support for WebSockets. Many examples were presented showing both synchronous and asynchronous (Futures) usage. Some backends may not support all functions. Determine for yourself if sttp is friendlier than the backends themselves. Its open source and documented.

Adrian Moors gave a nice overview of the coming releases of Scala 2.13, 2.14 and 3.0 (jokingly renamed 2.15).  3.0 is said to be 2.15 in that at its core the basics of Scala as a fusion of OO and FP remain unchanged.

Furthermore 2.13/14 will have laid a ground work to make 3.0 more seamless. Scala 2.13 will get collection library changes, smaller core library, faster compiler, some java 9 support and always more user friendliness.

Scala 2.14 is again a language focused release (even numbered releases in the 2.x series has been for some time). It will close the doty gap as the two compilers (scalac, dotyc) are zipped together further up the compiler phases. (They share a backend prior to this.) This release deprecates existentials, value classes, whitebox macros, package object inheritance and early initializers (and early initializers if not already deprecated in 2.13??). Trait parameters, opaque types, type lambda are added.

Implicits must have explicit return type.

Then with much celebration Scala 2.15 (aka 3.0) arrives as described in Martin Odersky’s keynote. Additions include Implicit function types, union types, transparent and principled meta programming, new macros (based on tasty), enum, @static (java interop).

The closing panel discussion hosted by Jon Pretty was mainly entertainment with a few serious questions from the audience.

We learned amongst other topics that doty type inference is more principled. In theory one should be able to write a specification for it.

Doty has no plans for runtime reflection. Use Java reflection.'

This article was written by Craig Bordelon