It just keeps getting better and better!
This month we're thrilled to welcome back our special guest John De Goes.
Who are our hosts?
G-Research is Europe’s leading quantitative finance research firm using machine learning, big data, and the most advanced technology
Schedule for the evening
6.00pm: Doors open, grab yourself a drink
6.40pm: Introduction to G-Research
7.00pm: James Prince, G-Research Software Engineer
Title: Partial results from ad-hoc Spark jobs
Running large Spark jobs?
Frustrated when there's a long-running straggler which kills the rest of the job? Wasting heaps of computing on jobs that fail and achieve nothing?
At GR, our researchers run many ad-hoc Spark jobs over large amounts of market data, data which is naturally skewed.
This unavoidable data skew means that our Spark tasks can have wildly different completion times and one long-running straggler delays inspecting the results of the tasks which have completed - or encounters failures which then cause the whole job to fail.
We propose a methodology for structuring Spark jobs which means that output partitions are available for consumption as soon as they are completed, and paves the way for implementing re-entrant jobs which are robust to random cluster failures
7.20pm: John De Goes
Title: Functional Programming in the Large
Almost all Scala developers practice functional programming in the small—using pattern matching, immutable data structures like collections, and occasionally even recursion instead of looping. Yet, for the longest time, functional programming “in the large” was deemed impractical or impossible. How could nothing but case classes and pure functions enable us to build web apps and data pipelines?
In this talk, John A. De Goes turns traditional wisdom on its head, arguing that functional programming is most useful not “in the small” (where performance sometimes requires procedural programming), but rather, “in the large”. Attendees will see how functional programming in the large tackles today's major architectural pain points (like error localization, testability, and modularity) and help developers ship better code, faster than ever before.
8.00pm: Grab some pizza and get chatting
Sign up as attending: https://www.meetup.com/Scala-in-the-City/events/269207480/