Real-Time Big Data in D

Abstract:  "Big Data" applications, which process petabytes of data per year, are computationally demanding, yet they have different requirements from traditional performance-critical applications. A programming language which is well suited for CPU-bound code, isn't necessarily ideal for real-time Big Data.

At Sociomantic Labs, we consume a deluge of real time data every day using an all-D technology stack. This talk will explain how our use of D has been successful in our intensely competitive industry. Unfortunately, most existing D libraries were not designed for Big Data applications. I'll discuss the performance killers they contain, and propose some changes to make D the natural choice for Big Data tasks.

Benefits: In one sentence: D should consider Big Data as one of its key markets, so we should be ruthless about memory inefficiency.

1. An introduction to Big Data, why it is important, and how it differs from traditional high-performance code.
2. An update on Sociomantic's continued explosive growth, and the pivotal role D plays for us.
3. The impact of memory inefficiency on performance.
4. The performance killers in the language, and why Sociomantic can't use Phobos at present.
5. Techniques and idioms we use to minimize our memory footprint.
6. The consequences of using fibers.
7. What we can change to make Big Data an even bigger win for D.

Speaker Bio: Don is a Senior Software Developer at Sociomantic Labs, Berlin. He has contributed to the D language since 2005, mainly in the areas of mathematics and metaprogramming. Before joining Sociomantic as a full-time D programmer, he worked in the solar photovoltaic industry on numerical modelling, measurement, and industrial inkjet. His 'FastDelegate' library is well known in the C++ community. He has two sons, and a daughter with superpowers.