Data Engineer IV

• Oct 2022 - Present

• Developing and maintaining generic frameworks in GCP (Airflow/Composer, BigQuery, App Engine, Cloud Functions) with main use of Python and SQL/DDL/DML for other Data Engineers to utilize for their implementations. • Implementing scheduled orchestrations in Airflow (Composer) for data refreshes of Raw Data Lakes and Data warehouses• Building and maintaining of App Engine instances responsible for data ingestion from rest API’s. Use of Python and Flask as well as data relevant libraries, i.e Pandas, Marshmallow.• Implementation of event-based processing by use of Cloud Functions.• Maintenance and development of Data Lakes including end-to-end ingestion and staging processes. Use of Cloud Storage Buckets and BigQuery.• Designing data warehouse tables in BigQuery to conform to Compute and Cost optimization as well as maintaining the standard of service as a single version of the truth for finalized reporting.• Data ingestion using StreamSets, DataStream, PubSub & Dataflow for CDC.• NOSQL Document-based data storage implementations for application-processing metadata.• Management of Cloud IAM roles for the team. I.e granting roles and access to individuals/groups and maintaining security standards among peers for auditing purposes.• CI/CD of software and implementations by use of GIT repositories.

Data Engineer III

• Nov 2021 - Sep 2022

Data Analyst

• Jun 2021 - Oct 2021

• Publishing monthly KPI dashboard and analysis on company’s performance metrics • Track and analyse metadata. • Analyze and construct CAC vs LTV metrics for Investment purposes • Testing code developed by the dev team. • Assisting in deployment of new code and testing back-end environments. • Find bottle necks and identify improvements. • Dev of a recommendation engine (tensor flow deep learning with GPU training), see Neural Collaborative Filtering (Xiangnan He et al., 2017). • SharePoint scrape/crawl scripts to extract relevant data from documents. • Use unsupervised learning (k-means clustering) to gain insights.

Data Engineer

• Aug 2019 - May 2021

Designing, implementing, supporting, testing, documenting, and troubleshooting complete end to-end data ingestion processes. Designing, implementing, supporting, testing, documenting, and troubleshooting complete end to-end data ingestion processes.