Apache Beam – Second London Meetup
They always say you should start the year on a positive note and that’s what we did last week: Datatonic and Qubit organised and hosted the second Apache Beam meetup. We were happy to welcome more than 30 data enthusiasts and very proud to be able to present 2 inspiring speakers for the night.
As usual, the first talk was a use case, focusing on both the business as well as how Beam was leveraged, while the second talk was a technical deep-dive.
Vadim Sobolevski, co-founder of FutureFlow, presented a fintech related use case. He proposed a framework to track uniquely identifiable units of money between different parties in different transaction, creating a graph with these monetary flows. The architecture incorporated several Google Cloud Platform components: Cloud Pub/Sub, Cloud Dataflow and Cloud Bigtable. He used these to build a solution that could scale to accommodate millions of transactions when the volumes increase as the business grows.
Vadim explaining what FutureFlow does
Next up Jean-Baptiste, software architect at Talend and part of the Apache Beam PMC, did an interactive intro on Beam and its concepts, good to brush up the Beam basics in case you missed the first meetup. We then digged a little deeper into the details of input and output. How I/O are actually DoFns, how we have to write them independent of the runner (we obviously want our Beam IO to work on Spark as well as Dataflow as well as the other runners) and what the future looks like with Splittable DoFns (using trackers and restrictions). If you want to learn more, this is a good resource to read. We wrapped up the night with some more technical discussions on topics like test driven development in Beam!
Jean-Baptiste taking a deep dive into IO in Beam
We also had a quick update from the community: there was a request to launch a hashtag; which we did: #ApacheBeamLondon. We pointed the audience to some interesting resources (e.g. the Beam newsletter — a great digest on everything that’s happening in Beam-land by Gris from Google) as well as a call for use cases (please reach out to us or Gris with interesting talks & blog posts!). Some metrics on the meetup itself: we saw a growth of >50% in both RSVPs as well as meetup members. Keep spreading the word!
We also asked for some feedback from the community: should we organise more practical sessions or more lecture-style sessions? Or perhaps should we oganise a hackathon where we let people present their problems and try to solve them together with some experts in the room? If you have suggestions or proposals, contact us!
Update from the Beam community
I want to thank the attendees and speakers and everyone who supports this meetup. Stay tuned for next sessions here. You can find the slides of the meetup here and we will soon be sharing a recording as well!
