Data Engineer

• Aug 2023 - Present

Data Analyst

• Sep 2022 - Aug 2023

*Successfully implemented automation using Pyspark, resulting in 70% reduction in the manual process* Leveraged scalable architecture concepts such as partitioning, caching, and serialization to optimize Spark code, and utilized tools such as Spark UI to fine-tune application performance and achieve significant improvements in execution time.* Working collaboratively with a full-stack developer by providing data discrepancies, data loss and resolved data discrepancies, and providing critical data insights and analysis to improve the overall data accuracy and reliability of the data. Show less

Associate data analyst

• Jun 2020 - Sep 2022

*Automated several data request jobs which has helped researchers to access NoSQL(MongoBD) data base as per client request using(pyspark/sql and databricks/EMR)*Created Daily alert mails to monitor the counts of newly harvested data. Weekly alert mails to test the loss of data between ETL-gateway prod using(pyspark/sql and databricks/EMR)*Scheduled several jobs using airflow Dag scheduler using(airflow/git)*Migrated the jobs from Databricks to EMR.*Took ownership of the data requirements from scratch to Production.*Worked along with ETL team in developing flexible solutions using(Pyspark/Python/EMR)*Got insights on 700M+ dataset using (Pyspark/sql/Databricks/Excel)*Strong Hands-on in designing and developing large scale Python-Spark based applications* In-depth knowledge on Pyspark3, RDD, Dataframe/Dataset, S3,csv,avro operations*Modelled Data as per the requirements. Show less