Senior Big Data Engineer

• Apr 2023 - Present

Senior Data Engineer - Lead Data Sciences

• Mar 2021 - Apr 2023

Wide knowledge and hands on experience in Azure cloud and distributed computing applications: — i. Azure Databricks ii. Azure Data Factory iii. Azure SQL Db iv. Azure DevOps v. Azure Datalake vi. Azure Synapse vii. Apache Sqoop viii. Apache Hadoop ix. Apache Spark x. Apache Hive xi. Apache Oozie xii. Airflow xiii. Impala xiv. Python xv. Shell Script ❖ Extensive project development experience in Azure/On Prem Bigdata environments for different ETL frameworks with Structured and Unstructured data. Exposure in both Cloudera and Hortonworks distributions ❖ Azure Data Engineering: Developed Data Ingestion Framework and custom transformation modules using Azure Databricks using PySpark, Azure Data Factory, Azure SQL DB, Azure Synapse, Azure Data Lake, Snowflake and Azure KeyVault with Azure DevOps for version control. ❖ Big Data On-Prem: Developed Data Ingestion Framework and custom transformation modules using Spark, Python, Shell Script, java, Sqoop, Hive, Impala and Oozie. Created Oozie workflows/coordinators for scheduling the ingestion/transformation python scripts across different layers. Performed data transformation using Impala, Hive, Java MapReduce, python, Shell script etc. ❖ Good knowledge of data warehousing techniques, Star/Snowflake schema, ETL, Fact and Dimensions tables. Experienced in implementing Type 1, Type 2 tables. ❖ An experienced team player with excellent interpersonal skills who could work independently under pressure can lead, motivate and influence others and can train and mentor subordinates. ❖ Strict adherence to quality in coding and best practices as per client standards. ❖ Worked in POC's with data sets for importing, cleaning, transforming and modeling using R as language Show less

Data Engineering Analyst/ Big Data Engineer

• Jul 2016 - Jan 2021

Worked on data warehousing projects from inception to implementation and delivered high quality solutions in a timely manner meeting and exceeding the business requirements. This involved working with the top management and executives, business sponsors and IT department. Responsible for defining the information architecture and technology infrastructure along with the design of multi dimensional data models and the Business Intelligence delivery solutions. Responsible for managing the data analysis, data sources, data mapping along with defining the meta data repository. Managed the evaluation of ETL, OLAP and Reporting tools and recommended the most suitable solutions depending on business needs. Experience in design and develop Big data solutions. Skilled in implementing Data Warehousing and Business Intelligence solutions. Skilled in performance tuning and optimization. Skilled in Hadoop and hadoop related echo systems like HDFS,Hive,Pig,Sqoop,Oozie, Flume, Kafka, Strom, Spark, ML lib and Mahot. Good understanding in NoSQL databases like HBASE and MongoDB. Experience in working EMR and EC2. Experience in design and develop predictive analytical models and reports using R. Show less