Senior Data Engineer

• Mar 2023 - Present

Senior Data Engineer

• Dec 2022 - Feb 2023

Senior Data Engineer (Senior-1) - Big Data

• Oct 2022 - Nov 2022

Lead Analyst ( Data Engineer- Big Data )

• Sep 2021 - Oct 2022

Developed transformation framework that accepts configs as Json and converts the json to generate spark sql code to create target data. This is adopted in EY Solutions Hub as an accelerator. This accelerator will enable non spark users to easily define the ETL transformations using simple json or api request ========================== Worked for a Australian Online Retail Client to build a big data platform which would ingest from multiple source systems using Azure Services (… Show more Developed transformation framework that accepts configs as Json and converts the json to generate spark sql code to create target data. This is adopted in EY Solutions Hub as an accelerator. This accelerator will enable non spark users to easily define the ETL transformations using simple json or api request ========================== Worked for a Australian Online Retail Client to build a big data platform which would ingest from multiple source systems using Azure Services ( ADF, Databricks , Event Hub and Synapse Analytics ). I was able to setup the Ingestion Framework using Azure ADF and Databricks ( Preprocessing, Enrichment, Masking). I also setup entire CICD using Github actions and Terraform for Azure services involving ADF, Databricks and Synapse analytics ================================= Developed a realtime Data Quality Framework using Apache Flink, Kafka, Apache Pinot and Superset. This was a POC done to experiment real time time series DB's like Apache Pinot. Build Admin UI's using Flask and exposed API's to create DQ rules etc using FastAPI

Senior Analyst (Data Engineer - Big Data)

• Oct 2019 - Sep 2021

Worked for Australian Insurance Client to transform the way data engineering was done to a more modern cloud agnostic approach. Used Pyspark to build a standard framework with multiple reusable pip libraries created to enhance the developer usage. Used Databricks Delta lake to build efficient data lake on top on CEPH storage with capabilities like Time Travel, ACID, incremental changes etc. Used Airflow dags for the orchestration and Kubernetes for container orchestration of the… Show more Worked for Australian Insurance Client to transform the way data engineering was done to a more modern cloud agnostic approach. Used Pyspark to build a standard framework with multiple reusable pip libraries created to enhance the developer usage. Used Databricks Delta lake to build efficient data lake on top on CEPH storage with capabilities like Time Travel, ACID, incremental changes etc. Used Airflow dags for the orchestration and Kubernetes for container orchestration of the dockerized framework. I also conducted POC's to convert existing Greenplum based data warehouse development to a more structured dbt based approach. We created dbt projects for each data product area and orchestrated the entire flow with airflow.

Big Data Developer

• Jan 2017 - Oct 2019

⇨ Project Description : ● Part of the Analytical Platform team who calculates base facts and perform aggregation based on study submitted by Client ● These facts are used to calculate different useful metrics which help in understanding the sales performance of particular products based on their location in shelve/rack in a particular… Show more ⇨ Project Description : ● Part of the Analytical Platform team who calculates base facts and perform aggregation based on study submitted by Client ● These facts are used to calculate different useful metrics which help in understanding the sales performance of particular products based on their location in shelve/rack in a particular store --------------------------------------------------------------------------------------- ⇨ Technologies used : ● Used Apache Hive , Apache Spark extensively for fact calculations , aggregations and feature implementations ● Used HDFS as the data storage layer --------------------------------------------------------------------------------------- ⇨ Programming Languages : ● Proficient in Core Java ● Experienced programming Spark in both Scala and Python API's (PySpark) ● Good SQL knowledge

System Engineer

• Nov 2015 - Dec 2016

Worked as Java developer in an ERP project. Also worked in Internal TCS innovation group for website development in Ruby on Rails