Duc Hoa Nguyen
Lead Data Engineer | Engineering Manager ‐ Data Platform, ETL, Log Management at VNLIFE- Claim this Profile
Click to upgrade to our gold package
for the full feature experience.
Topline Score
Bio
Experience
-
VNLIFE
-
Vietnam
-
IT Services and IT Consulting
-
1 - 100 Employee
-
Lead Data Engineer | Engineering Manager ‐ Data Platform, ETL, Log Management
-
Jan 2020 - Present
* Led the charge in revamping and modernizing the organization’s data platform and warehouse. Pioneered the Data Lake and Delta Lake initiative, and near real‐time stream processing. This effort significantly optimized and expedited the development and delivery velocity of new ETL jobs, achieving a remarkable 3x improvement in efficiency. • Enhanced data team’s productivity through the introduction of best data engineering practices, documentation and knowledge sharing within the team, fostering an environment of continuous improvement. This led to a marked improvement in codebase quality, reduced rate of defects & a substantial increase in critical path test coverage. * Architected and implement an authorization service that centralized access control for the organization’s data platform that simplified data security and governance. Defined once, secure everywhere with Open Policy Agent (OPA) and integrated into our query engine (Trino). * Spearheaded the design and implementation of structured logging event collector, enrichment, and serving platform. This platform serves as the centralized source of truth and provides an additional avenue for application performance management metrics exploration. * Provided mentorship & guidance to junior team members, helping them develop technical skills & contributing to their professional growth. Show less
-
-
-
Grab
-
Singapore
-
Software Development
-
700 & Above Employee
-
Lead Software Engineer - Trust, Identity, Safety & Security
-
May 2016 - Dec 2019
* Orchestrated, executed the architecture and implementation of a streaming service and pipeline. This system enabled real‐time sessionalization, featurization, and the generation of transactional scores & verdicts for critical fraud detection & safety application. * Designed & developed a service to replace existing error‐prone implementation to optimize data transfer & provide observability of controlplane for distributed workers. Achieved 2‐5 times faster transfer speeds while accommodating various source storages, file formats, insert operations, multi‐tenancy, and different database sinks. * Implemented & optimized end‐to‐end data pipelines for syncing, parsing, and wrangling logs to support analysis and pattern learning. These pipelines were tightly integrated with the machine learning team’s workflow and leveraged Airflow scheduling on top of Amazon EMR clusters * Conducted regular performance tuning and optimization of ETL workflows to ensure data pipelines were operating at desired efficiency. Show less
-
-
-
Cloudera
-
United States
-
Software Development
-
700 & Above Employee
-
Solutions Consultant
-
Aug 2015 - Apr 2016
* Advised clients on data architecture within the Hadoop stack, delivering services to optimize data solutions. * Executed large cluster deployments encompassing installation, workload tuning, both on‐premises & on cloud‐based platforms. * Performed data migration & plumbing from existing data warehouses to managed HDFS cluster, improving data accessibility & efficiency. * Collaborated with Data Science teams to facilitate data‐driven use‐cases by providing critical support in data preparation, feature extraction, and operationalizing machine learning models (via PMML, Kafka, and Spark). Show less
-
-
-
StarHub
-
Singapore
-
Telecommunications
-
700 & Above Employee
-
Lead System Analyst
-
Jul 2015 - Aug 2015
-
-
Senior System Analyst
-
Feb 2014 - Jul 2015
* Automated the ETL (Extract, Transform, Load) process, streamlining the flow of data from various cell towers & GPS sources into data martsfor dashboard analytics. Data ingestion, cleaning, archiving, and transformation were done through HBase, Pig, Parquet & GreenPlum.* Produced user‐defined functions that simplified and expedited windowed data analysis, exploration, and computation, enabling businessanalysts to extract actionable insights from large datasets to drive malls footfall analysis and Customer 360 use‐cases.* Provided mentorship and guidance to junior engineers, facilitating their professional growth and helping them develop strong technical skills. Show less
-
-
-
Savant Degrees
-
Singapore
-
IT Services and IT Consulting
-
1 - 100 Employee
-
Web Developer / Software Engineer
-
Jun 2010 - Sep 2013
• Developed architecture and lead the building of web products ranging from Portfolio Management, Resources Planning System, Donation Platform to E-commerce solutions. • Writing abstraction and API wrapper for 3rd-party services. Payment gateway integration. • Researched, integrated and aligned new technologies and best practices into daily engineering. • Engineered and further developed in-house framework and API which reduced project bootstrap and development time. • Organized technical training for internal audiences. Show less
-
-
Education
-
National University of Singapore
Bacher of Computing (Honour), Information System - School of Computing