Challenge
With the vision for instrumenting physical world, I was tasked with building a platform that enabled rapid prototyping & installation of sensors.
Solution
I built a data platform that provided an APIs & framework for:
- Defining sensor schema
- Collecting sensor data. The data was stored as timeseries in HBase.
- Stream collected data in realtime for visualization and installation validation.
- Perform real time and batch analytics on collected data using Spark.
- Build visualizations and reports for the analysis performed.
I presentation the design & architecture of the platform at Great International Developer Conference in 2015. Below are the slides:
The platform was built using the tech stack: Scala, Play! framework, Non-blocking I/O, Redis, HBase, Kafka, Spark, Hadoop, ElasticSearch. It not only garnered acclaim within the company but also earned a spotlight presentation by our CTO at the prestigious Strata Conference in London (slides 30 - 33):