Senior Software Engineer - Analytics

• Feb 2021 - Present

Technology Lead-Hadoop

• Nov 2017 - Feb 2021

Client: Southern California Edison, Rosemead –CADeveloping applications using Hadoop Big Data technologies- Hue, Hive, Impala, Spark, Spark-Sql, Kafka, Map-Reduce, Scala, Hbase and Oozie.Developing Spark applications using Scala , RDD and SparkSql for processing of data.Importing data using Spark RDD and performing transformations , actions on RDD. Experience in creating external Hive tables,partitioning,bucketing , loading and aggregating data.Worked with various file-formats like Avro, Parquet and ORC.Used Spark-Sql to load structured and JSON data and create schema RDD and loaded into Hive Tables using Spark-Sql.Imported data from database source using Sqoop into HDFS.Experience in using Redwood scheduler to schedule the jobs.Creating HiveQL and Impala scripts for creating the tables and populating the data.Experience in using Hive Partitions, Bucketing in Hive and designed both Managed and External tables in Hive to optimize performance.Experience in understanding the data and designing/Implementing the enterprise platforms like Hadoop Data lake.Designed data Ingestion framework to load business validated data of around 3 million customers into Hadoop Data Lake to store Historical and Incremental data.Designing applications based on identified architecture and support Implementation design by resolving complex technical issues.Experience in Talend ETL Tool for handling huge volume of data.Involved in Customer discussions during requirement analysis with knowledge of requirement elicitation and analysis tools and techniques like story boarding, Agile user stories to understand system requirements and non-functional requirements like performance,security, availability and scalability etc.The Enterprise data lake is used to provide support for various uses cases including Analytics, processing, storing and Reporting of voluminous, rapidly changing, structured and unstructured data. Show less

Technology Lead-Hadoop

• Jun 2017 - Nov 2017

Client: Capital Group, Irvine –CADeveloping applications using Hadoop Big Data technologies- Hue, Pig, Hive, Impala,Spark, Map-Reduce, Kafka ,Hbase, Spark-sql and Oozie.Analyzing and Transformations of Data of Investments and Funds with Hive and Pig.Planning and Conducting collaboration sessions and requirements walkthrough with concerned stakeholders (Business SMEs, Architects, Tech Leads, Developers, QA).Experience in understanding the data and designing/Implementing the enterprise platforms like Hadoop Data lake.Creating HiveQL scripts for creating the tables and populating the data.Developing Map reduce jobs using apache commons components.Involved in Requirement Analysis, Design, and Development.Export and Import data into HDFS, HBase and Hive using Sqoop.Worked closely with the business and analytics team in gathering the system requirements.Load and transformation of large sets of structured and semi structured data. Show less

Technology Lead - Hadoop

• Sep 2016 - May 2017

Client: American Family Insurance , Madison WIRole: Technology Lead - Hadoop.American Family insurance provides term, universal and Whole life insurance; Personal and business Auto insurance, Personal umbrella insurance and Home insurance . This requires disparate data sources and high quality volume data loads in Hadoop Clusters. The objective of the project is to standardize the data quality checks to be performed on the data received from various source systems of Policy, Billing, Claims etc.Responsibilities:Experience in developing applications involving Hadoop Big Data technologies- Pig, Hive, Map-Reduce, Hbase, Spark and Oozie.Hands on Experience in Hortonworks Platform.Analyzing and Transformations of Data of Auto Insurance and Life Insurance Policies with Hive and Pig. Creating HiveQL scripts for creating the tables and populating the data.Developed Map reduce jobs using apache commons components.Supported code/design analysis, strategy development and project planning.Created reports for the BI team using Sqoop to export data into HDFS and Hive.Knowledge of Partitions, Bucketing concepts in Hive and designed both Managed and External tables in Hive to optimize performance.Analyzing/Transforming data with Hive and Pig.Developed Oozie workflow for scheduling the job.Processing large sets of structured, semi-structured and unstructured data and supporting systems application architecture.Hands on Experience in Maven, Git and Eclipse. Managed and reviewed Hadoop log files.Created HBase tables to load large sets of structured, semi-structured and unstructured data coming from UNIX, NoSQL and a variety of portfolios.Good Knowledge of Web Application Development.Good Experience in developing Model View Control (MVC) applications using Spring.Configured using Hibernate for the Database Connection and Hibernate Query Language (HQL) to add and retrieve the Information from the DatabaseUsed Log4j for loggingUsed ECLIPSE as IDE. Show less

Programmer Analyst

• Mar 2016 - Sep 2016

Role: Java Hadoop Developer. Hands on experience in installation, configuration, supporting and managing Hadoop Clusters using Apache, Cloudera (CDH3, CDH4).Experience in Spark, Scala, Kafka and Hadoop Ecosystem.Understanding business needs and functional specifications and map those to develop Spark SQL.Designed XSD,XSL to parse the XML structure file into in pipe delimited format to text file for facilitate effective querying on data.Created file based RDD's by using above textfile after parsing the above XML File and also the same RDD is processed to Data-Frames just to compare the performance between existed Map reduce methodology and Spark methodology.Developed RDD's and which were processed the data in Spark by writing pig scripts to analyse the data as per client requirement.Developed Pig Latin Scripts to extract data from the source system.Installed and configured MapReduce, HIVE and the HDFS;Assisted with performance tuning and monitoring.Collecting and aggregating large amounts of log data using Apache Flume and staging data in HDFS for further analysis.Very good understanding of Partitions, Bucketing concepts in Hive and designed both Managed and External tables in Hive to optimize performance.Solved performance issues in Hive and Pig scripts with understanding of Joins, Group and aggregation and how does it translate to MapReduce jobs.Developed UDFs in Java as and when necessary to use in PIG and HIVE queries.Developed UDFs in Java as and when necessary to use in PIG and HIVE queries.Created HBase tables to load large sets of structured, semi-structured and unstructured data coming from UNIX, NoSQL and a variety of portfolios.Analyzing/Transforming data with Hive and Pig.Proficient in developing Java Web Applications using JSP and Servlets, Struts 2.0,Hibernate and Spring and Oracle.⦁ Good Knowledge of Web Application Development.Used Log4j for logging.Used ECLIPSE as IDE. Show less

Senior Software Engineer

• Dec 2014 - Feb 2016

Client : GE CAPITAL. Role : Java Hadoop Developer. Hands on experience in installation, configuration, supporting and managing Hadoop Clusters using Apache, Cloudera (CDH3, CDH4).Helped design big data clusters and administered them.Adding/removing new nodes to an existing hadoop cluster. Installing and configuring hadoop eco system like sqoop, pig, hive.Experience in Hive,Pig,Spark,Scala,Kafka and Hadoop Ecosystem.Developed Pig Latin Scripts to extract data from the source system.Understanding business needs and functional specifications and map those to develop Spark SQL.Created file based RDD's by using above textfile after parsing the XML File and also the same RDD is processed to Data-Frames just to compare the performance between existed Map reduce methodology and Spark methodology.Developed RDD's and which were processed the data in Spark by writing pig scripts to analyse the data as per client requirement.Developed Pig Latin Scripts to extract data from the source system.Developed Map reduce jobs using apache commons components.Installed and configured MapReduce, HIVE and the HDFS;Assisted with performance tuning and monitoring.Experience in importing and exporting data using Sqoop from HDFS to Relational Database Systems.Collecting and aggregating large amounts of log data using Apache Flume and staging data in HDFS.Experience in working with Hive-Streaming, Partitions, Bucketing concepts in Hive and designed both Managed and External tables in Hive to optimize performance.Solved performance issues in Hive and Pig scripts with understanding of Joins.Developed UDFs in Java as and when necessary to use in PIG and HIVE queries.Created HBase tables to load large sets of structured, semi-structured and unstructured data coming from UNIX, NoSQL and a variety of portfolios.Analyzing/Transforming data with Hive and Pig.Proficient in developing Java Web Applications using JSP and Servlets, Struts 2.0,Hibernate and Spring and Oracle.Used ECLIPSE as IDE. Show less

SOFTWARE ENGINEER - Java Developer

• Dec 2011 - Nov 2014

Dedicated Software Engineer with 2 years 3 Months of experience in various domains such as Financial Services and Manufacturing. Worked over the entire SDLC and possess good development skills in Java, J2EE, Struts,Hibernate,Spring,Oracle SQL/PLSQL & Unix commands. Strong analytical and problem solving skills coupled with delivering software on time. ⦁ Developed Pig Latin Scripts to extract data from the source system.Adding/removing new nodes to an existing hadoop cluster.Finding the right joins logics and create valuable data sets for further data analysis.Analyzing/Transforming data with Hive and Pig.Developed Pig Latin Scripts to extract data from the source system.Responsible to develop custom udf’s in pig, hive.Experience in capacity planning/clustering of Hadoop storage system in HDFS.Learns new concepts quickly while working well under pressure and communicate ideas clearly and effectively. Dedicated to maintaining high quality, compliant standards. Proven emphasis on teamwork. Show less