There are so many interesting design aspects of Flare and Lantern and we want to know more!
At Scala Days, Assistant Professor Tiark Rompf sparked the conversation on these design aspects and how they give us fresh new insight.
Flare & Lantern: Accelerators for Spark and Deep Learning
Frameworks like Spark and TensorFlow have commoditized cluster computing and training of neural networks. However, they leave precious performance on the table, especially when used together. Flare is a new back-end for Spark SQL that yields significant speedups by compiling query plans to native code. Lantern takes a fresh look at backpropagation, and provides a generic and performant differentiable programming framework in Scala, including efficient code generation for GPUs. This talk discusses interesting design aspects of Flare and Lantern and presents relevant case studies.