I work on building end to end architecture for big data applications and running code on them( using Python, Java, Scala and Shell script) .I have built scalable big data systems from scratch including handling data from multiple structured/unstructured sources, web crawling, searching.
1 April 15– Present
• This project aims to re-target users who drop off from websites(ecommerce,hotels,flights,app) across other websites (publisher) on the internet.
• I have written more than 25 pig scripts which help to build the re-targeting model running everyday as a part of daily job in Azkaban, processing more than 2TB+ data on Hadoop cluster having more than 400 nodes.
• To support these pig scripts I had to design the complete backend(UDFs- 20+) in Java and python which process the input data in distributed fashion across cluster and integrated them in Pig script. .
July 2013 - March 2015
• Written bootstrap-code and scripts to automate the infrastrucutre on AWS cluster which installs hadoop, spark and shark on the cluster and loads 200+ GB of publically available data of wikipedia from s3 to HDFS using s3cmd and does caching automatically..
• Builds 10 machines scalable cluster from scratch in just 20 min with 200 Gb data cached in Spark..
• Later also extended the code in Scala, to other Spark applications like Spark Streaming - Twitter Sentimental Analysis- which displays popular #Tags and saves them on Hadoop and integrated it Zeppelin for real interactive data analytics of #Tags .
• Spark Mlib Recommendation algorithm - Collaborative Filtering with some sample data sets loaded while scaling itelf and run it over Spark cluster with just 1 click.
Aug- Nov 12
• Designed and created Credit Amendments and Loan Approval Process.
• Data insertion and routing is done in SQL and integration is done with API based on JAVA .
I have expertise in following applications:
I have given Technical Talks on Big-Data to many corporate executives online , mostly on the insights/technical aspects of Apache Spark. Most of prominent ones are :-
I'd love to hear from you. If you think I would be a good fit for your upcoming project, or would just like to just say hello, please fill out the form below.