Lead Software Engineer

• nov 2021 - - Presente

Senior Data Engineer

• mar 2020 - nov 2021

TTEC’s Insights Platform is a cloud-based customer data platform that provides brands with a 360° view of their customers’ needs, behaviors, and preferences with the insights they need to deliver a great customer experience.• Developed Spark scripts using Python on Azure HDInsight for Data Aggregation, Validation and verified its performance.• Creating complex pipeline using Azure datafactory (ADF)• Creating and loading data in Databricks Delta lake table.• Creating mountpoints for Gen1 and Gen2 for blob storage.• Masking passwords using Azure Key-Vault • As Senior Data Engineer I need to Built pipelines to move data from Azure Blob to Data lake.• Utilized Azure HDInsight to monitor and manage the Hadoop Cluster.• Collaborated on insights with Data Scientists, Business Analysts and Partners.• As Senior Azure developer need to Built pipelines to move data from on-premise servers to Azure Data Lake.• Utilized Python Panda Frame to provide data analysis.• Enhanced and optimized Spark scripts and run data mining tasks.• Loaded data into Spark RDD and do in memory data Computation to generate the output response.• Involved in converting Hive/SQL queries into Spark Transformations using PySpark.• Optimizing of existing algorithms in Hadoop using Spark Context, Spark SQL, Data Frames and Pair RDD’s.• Experienced in performance tuning of Spark Applications for setting right Batch Interval Time, correct level of Parallelism and memory tuning.• Developed Hive queries to process the data and generate the data cubes for visualization.• Built specific functions to ingest columns into Schema for Spark Applications.• As Senior Cloud Data Engineer I have Experience in handling large data sets using Partitions, Spark in memory capabilities and efficient Joins, Transformations and other during ingestion process itself.• Built rest API by utilizing most efficient techniques using REDIS cache.• Working on Azure Synapse for advanced data Analysis.

Sr. Hadoop Developer

• mar 2019 - mar 2020

• Designing and building solutions for Ford ICI.s Hadoop environment• Write neat and clean code using Java, Scala or Python with proper unit test cases.• As Senior Hadoop Developer I need to BuiltBuild strong and lasting partnerships with the business.• Developing efficient solutions by utilizing Hadoop technologies like Hive, ELK, Spark, Pyspark, Nifi, Hbase and MongoDB• Coordinating and sharing working with onshore and offshore team to complete work on time• Writing complex hive quires to support business requirements.• Collaborate with cross-functional teams to define, design, and create solutions• Working closely with business teams to get the proper understanding of business needs• Interface with Business, IT, and business owners to prioritize improvement efforts

Spark and Hadoop Developer

• gen 2019 - mar 2019

• Responsible for design and development of analytic models, applications and supporting tools, which enable Data Scientists to create algorithms/models in a big data ecosystem.• Design, implement and delivery of an insights data pipeline supporting analytic products across the organization.• Design and integrate data from different sources.• As Hadoop Developer I need to Built and Develop code to automate the existing manual processes• Forms analytics platform components and/or processing components required to provide a business solution.• Engage with business stakeholders to design and own end-to-end solutions to empower data driven decision making.• Leverage data, technology and quantitative methods to form products that inject analytics and insights into daily workflow of teams.• Defines application scope and objectives, including impact to interfaces.• Ensures appropriate data testing is completed and meets test plan requirements. • Coordinates integration actions to ensure successful implementation.• Develop and support Extraction, Transformation and Load process ETL using Informatica PowerCenter Teradata and other related Technologies• Work independently and in collaboration with offshore onsite developers lead projects• Code and maintain custom or canned reports, system interfaces and extracts• Provide technical expertise in designing, documenting and coding new business initiatives• Translate complex business requirements to optimal technical solutions

Hadoop ETL Developer

• feb 2017 - dic 2018

• Responsible in Building distributed, reliable and scalable data pipelines to ingest and process data in real-time T-Mobile applications• Responsible for the documentation, design, development, and architecture of Hadoop applications• Implementing real time data processing using Spark and Kafka messaging services• Implementing real time data processing using Storm and RabbitMq messaging services • Implementing HBase logging for all real time and batch processing jobs for auditing purpose• Cleaning data as per business requirements using streaming T-Mobile API’s • Converting large amount of un structed data to structured data using Java or Scala• Improving existing Hadoop applications performance and throughput.• As Hadoop Developer I need to Transforming real time encrypted CDR data to CSV using Java• Parsing JSON to CSV and load it to Hive table using Scala• Exporting processed Hive table data to Teradata using Scala• Importing data from Amazon S3 to Hadoop and load data to Hive table using Scala• Importing of data from various data sources, performed transformations using Hive, MapReduce, and loaded data into HDFS. • Importing data from Oracle and MySQL to Hadoop using Sqoop. • Python script to parse WEB related data and loading the data into Production Hive tables• Responsible for writing daily Hive database monitoring Job using Shell script• Responsible in Scheduling and automation Jobs in control-M• Writing Phoenix queries on top of HBase tables for loading Tableau dashboard

HealthCare services data analysis

• feb 2016 - gen 2017

Healthcare services are appropriate for a major information arrangement. Electronic Health Record (EMR) alone gathers a tremendous measure of information. Throughout the most recent decade, pharmaceutical organizations have been collecting long stretches of innovative work information into restorative databases and along these lines, the patient records have been digitized. part of restorative information originating from different sources, guided choices can be produced using the bits of knowledge increased through huge information by utilizing different Machine Learning Algorithms. Generally, doctors utilize their judgment while settling on treatment choices, however over the most recent couple of years there has been a move towards proof-based medication. This project includes systematical audit of clinical information and settling on treatment choices dependent on the best accessible data.• Predicting monthly review of Health care institute based on each Quarter using Scala• Figuring patient's age and age gather from his date of birth given in EMR using Spark • Calculating the distribution of data for each patient using PySpark• Calculating monthly customer expenses on medical using Scala• Predicting customer expenses for next N visits using Spark MLlib

Exploring customer sentiments towards XBOX-ONE on Twitter – using Cloudera platform & SA

• ago 2015 - gen 2016

• The main aim of this project is to perform analysis on customers reaction towards Xbox One. • The analysis is performed on two different data sets: Before the release of XBOX One November 15th - 21st, and after the release of XBOX One November 22nd - 29th.• Sentiment analysis will then be performed and the results will be compared between the two data sets. • Tweets posted on XBOX One between the dates November 15th - November 29th.• Flume: To inject the data into HDFS• Hive or Map Reduce: To perform the operation required for analysis.• HDFS: Hadoop Distribute File System to store the data in the Data Nodes.