There are so many interesting design aspects of Flare and Lantern and we want to know more!
Flare & Lantern: Accelerators for Spark and Deep Learning
Frameworks like Spark and TensorFlow have commoditized cluster computing and training of neural networks. However, they leave precious performance on the table, especially when used together. Flare is a new back-end for Spark SQL that yields significant speedups by compiling query plans to native code. Lantern takes a fresh look at backpropagation, and provides a generic and performant differentiable programming framework in Scala, including efficient code generation for GPUs. This talk discusses interesting design aspects of Flare and Lantern and presents relevant case studies.