Senior Data Engineer

• Feb 2022 - Present

At Extra Card, I've been working on designing data pipelines and setting up infrastructure necessary to ingest, procure and transform internal and external data to support critical downstream applications. Some of key contributions since my start in Data Engineering and CICD are as follow: • Leading CICD practices for Data team setting up AWS services and Prefect as orchestration tool. • Testing, deploying internal applications and Machine learning models with Prefect agents and blocks. • Deploying lambda functions in container images using serverless AWS SAM, ECR,Cloudformation and DynamoDB for storage • Triggering ,scheduling and alerting with SNS, Eventbridge, Cloudwatch • Security best practices on AWS with Secret Manager and KMS encryption • Continuous ETL and modeling on AWS Redshift and AWS S3 • Leading/designing dbt workflows, source freshness configurations, data quality checks and logging • Putting in place CI/CD best practices and flow diagram on dbt cloud • Setting up Fivetran to ingest raw data into Dev with normalization in Snowflake • Configuring Github Action as default CD process along with dbt for PR • Setting up Airbyte docker container on AWS EC2 ingesting PostgresQL into Snowflake • Documenting best practices on Git and Github with branching and merging use cases • Testing some use cases with focus on orchestration using Prefect vs Kubernetes • Configuring External and Internal stage and integration between AWS S3 and Snowflake and applying access policy and data masking. • Setting up Monitoring tools like DataDog and notifications on data pipelines across datawarehouse • Managing over 250 dbt models and macros to help easing data science and BI predictive models • Managing Metabase internal dashboards to understand updates on customer churn rate, customer CAC and other valuable metrics to company’s growth.