2016 USENIX Annual Technical Conference (USENIX ATC '16)

Big data analytics became a hot research topic nearly ten years ago, but since that time, a lot of things have changed. On the hardware side, trends such as the slowdown of processing with respect to I/O are starting to affect the design of big data systems. On the application side, big data systems are increasingly being used by non-programmers and require similar forms of interaction to "small data" analysis tools. Finally, big data systems are increasingly provided "as a service" on cloud infrastructure. I'll talk about these changes from the perspective of the Apache Spark project and from my experience at a company offering a cloud service for big data (Databricks).