Experience

HR DataHub

• United Kingdom
• Technology, Information and Internet
• 1 - 100 Employee

• Data Engineer

  • May 2023 - Present



Amazon

• United States
• Software Development
• 700 & Above Employee

• Data Engineer

  • Feb 2022 - Aug 2022

  Senior Data Engineer in a team of thirty data engineers and scientists assisted new project to build an umbrella team with the objective of finding new insights into new candidates working for Amazon delivery department. Assisted manager in building the new candidata team, for the first three months there were no other data engineers Worked with data scientists to design and implement mission critical data pipelines cleansing and transforming data for stakeholders using AWS Glue (PySpark), Athena (data exported as parquet files), Lambda functions, Redshift and S3. Jobs were scheduled hourly to consolidate data from various sources including Redshift to provide insights for each candidate the behaviour, concerns, performance, number of hours worked, average salary and various scores calculated by data scientists etc Designed and built lambda functions in conjunction with Athena and S3 to export and copy data from one AWS account to another via S3 to different teams across Amazon as per client requirements Assisted infrustructure engineer in making modifications to infrustructure using Cloud Formation to support continuous deployment process, permissions, scheduling etc (code was deployed as containers) Implemented views in Redshift joining various tables and worked with different data scientist teams in order to deal with challenges around nuances between tables from different sources and structure Worked with external teams and data scientists in validating the integrity of source data and escalating when finding issues Show less



Cynozure Group

• United Kingdom
• Business Consulting and Services
• 1 - 100 Employee

• Data Engineer

  • Jan 2021 - Jun 2021

  - Migrated across production data to a lab environment for usage by data scientists - Built views on top of tables summarising orders, products, customer consumption etc as per requirements by data scientists and analysts - Assisted in building parts of CI/CD process using bash & Jenkins - Built Python processes consuming flat files from a secure FTP server in order to create fact and dimension tables using Pandas - Migrated across production data to a lab environment for usage by data scientists - Built views on top of tables summarising orders, products, customer consumption etc as per requirements by data scientists and analysts - Assisted in building parts of CI/CD process using bash & Jenkins - Built Python processes consuming flat files from a secure FTP server in order to create fact and dimension tables using Pandas



IG Group

• United Kingdom
• Financial Services
• 700 & Above Employee

• Data Science Engineer

  • Jun 2020 - Dec 2020

  Senior Data Engineer as part of the Risk team building and maintaining mission critical pipelines to meet regulatory requirements (for example Capital Requirements Regulations). Daily responsibilities included developing pipelines, maintaining existing infrustructure and processes, supporting risk team data scientists and upper management, resolving pipeline issues. Played leading role working with Data Scientists to develop and deploy the Stress Test pipeline covering liquidity risk across the entire business factoring in scenarios such as fluctuations in FX, commodity and equity prices. - Process developed in python and orchestrated using Luigi - Challenges included running process on hourly accross 120 scenarios in a reeasonable amount of time. Multi processsing technique was used to spawn a process for each scenario. - Blackscholes Options pricing had proven to be expensive in terms of resources and timing. To overcome this challenge an execute once and use across all scenarios approach was taken as opposed to executing this expensive code for each scenario. - Aggregations were performed accross all trades were exported as snapshots to the MongoDB datawarehouse Worked with stakeholders to develop pipelines for newsfeeds, FX prices, PRR, CRR reports. Maintained existing codebase, developing new features as per requirements, backfilling data, maintaining infrustructure Deployed code changes to production and development environment using Bamboo continous deployment Resolved pipeline failures & issues to avoid regulatory breaches and heavy fines (Python – pandas etc, Conda packaging & environments, bamboo, MongoDB, Oracle, BitBucket,Jira, Linux) Show less



VEON

• Netherlands
• Telecommunications
• 700 & Above Employee

• Data Engineer

  • Jul 2018 - Dec 2018

  Worked in Business Intelligence App Analytics Team and single handedly built all batch based ETL fully configurable processes which were able to run as both manually via command line (specify date ranges to integrate) and Micro services (Docker images run daily) on Veon's Google Cloud Production, UAT and Dev servers. - Fully configurable Python based process which aggregated daily,weekly & Monthly data broken down by country from Firebase raw tables (using Google API) providing metrics such as daily active users, app deletions, message usage rate, number of first time users, onboarding funnel, chat funnel, data rate funnels, offers funnels which describe user journeys and where bottle necks were within each metric. - IOS & Google ratings & installs insight processes (Python, BigQuery, Docker) built around App Annie and Google API (Google Analytics) providing stakeholders with insights on overall and daily ratings average for both platforms using raw data. Data from both differed structurally so there were also challenges in transforming the data in a view. - App Flyer API integration process (Python, BigQuery, Docker) proviving insights on campaign metrics such as daily, weekly and monthly number of impressions, clicks, conversion rates. - Transformed Crash Analytics data from various sources for both iOS and Google platforms thus providing insights into daily crashes by number of actives users and severity of these crashes across all countries. - Migrated legacy data transformation Python scripts over to Pyspark for better performance on larger datasets. - Assisted with building of Tableau dashboards used by stakeholders to better understand customer experience, onboarding funnels etc. - Utilised Amazon Glue to infer data structure (crawler) within S3 and Athena to perform SQL analysis. Show less



Virgin Money

• United Kingdom
• Financial Services
• 700 & Above Employee

• Data Engineer

  • Nov 2017 - Jul 2018

  Lead engineer in designing and building state of the art mission critical Operational Risk framework designed to remove or mitigate business key risk areas across the whole of the Digital bank from the ground up. Additional tasks included building out data pipelines and supporting the Data Science team. - Designed and built Operational Risk data model based on requirements and framework guidelines defined by ourselves and Oliver Wyman - Designed and built out the Operational Risk data integration ETL process transforming key risks data across the business defined in spreadsheets such as controls, failure modes / failure mechanisms (FMEA’s), cause, incidents, appetites etc. (qualitative and quantitative data) into a database fully describing all relationships between entities thus giving an accurate view of the business vulnerabilities for stakeholders to assess (Python, PostgreSQL). - Designed and built dashboards from scratch as per requirements thus giving an intuitive and comprehensive overall view of business risk appetites across all areas such as systems, data and information management, innovation, financial crime, customer’s appetite, processes etc. down to more granular information such as corresponding failure mechanisms, measurement nodes, and controls associated with key risk areas all illustrated in graphs, heat maps, tables etc (Board BI). - Implemented views in the backend database mapping Digital bank risks back to core bank thus providing senior staff within core bank information on vulnerabilities and associated costs through their lens (Board BI). - Integrated event data from the 10x platform to measure performance of controls across the bank before working with data scientists to assess whether controls were adequate or not. - Worked with data scientists to integrate YouGov data before transforming this in a manner suitable to be used in models to best describe tech savvy customers, people more likely to switch bank accounts etc (Spark). Show less



Omnicom Media Group

• United States
• Advertising Services
• 700 & Above Employee

• Data Engineer

  • Jun 2016 - Sep 2017

  Played leading role in providing Analysts and Traders with mission critical high volume performance data across multiple DSP's to help make better trading decisions. - Automated processes to extract, clean and ingest mission critical Log Level & Reporting data from various sources. Ensuring integrity and aggregating performance metrics for traders to help optimize campaigns for Appnexus, Trade Desk, Amazon, Google DBM etc. using API, FTP, S3 buckets etc (Python, Reshift, Amazon AWS, Rest API / HTTP). - Introduced best practices for optimizing SQL queries / tables to improve performance (Vacuuming, - Sort / Distributed Keys, Execution Query Plan etc). - Single handedly Maintained databases with terabytes of data to ensure healthy performance (vacuuming, unloading, removing redundent tables etc). - Built processes to create segments to help target key audiences (Python, API). - Worked with DSP's to help improve consistency and integrity of data. From observing the data, noticed cases where Omnicom were being overcharged and helped reclaim over £8000 per Month for over 9 Months. - Created Ad-Hoc reports for clients measuring performance, brand safety and viewability. - Created backup process to unload data from key tables every Month (Python, SQL, EC2). - Worked with Analysts to provide more in depth meaningful information from the data. - Worked with developers to provide front end data and responsive querying. - Built processes providing reports to managers & traders covering monthly revenue, profit and various performances metrics across all DSP's for each advertisers (Python, SQL). Show less



Angloinfo

• United Kingdom
• Online Audio and Video Media
• 1 - 100 Employee

• Data Engineer

  • Jul 2015 - Jul 2016

  Played leading role to migrate all data and image related content across from old website to the new improved and more maintainable version of the world's largest Expat information site. - Designed and implemented all aspects of data migration, aggregation and cleaning of Directories business listings ( free and paid for), Classifieds, Discussions, Events, Guides, Blogs, Members, movies, CMS and geo location data across all regions and mapping these to their corresponding categories using bulk loading as well as propriety API calls to populate the database and Elastic Search (Bash,PHP,Python,MySQL,Ubuntu). - Designed and implemented scripts to migrate across all images from old website to S3 servers and mapping these to corresponding content of the website (S3, Bash,PHP,Python,MySQL,Regular expressions). - Designed and implemented post migration scripts to links Members to Discussions,blogs and members as well as transformation / substitution of HTML as per requirements using regular expressions. ( Bash,PHP,Python,MySQL,Regular expressions). - Optimised SQL queries using execution plan (explain in this case) and adding indexes where needed to reduce query time (MySQL). - Responsible for deploying changes to development and production site. Show less



Just Eat Takeaway.com

• Netherlands
• Software Development
• 700 & Above Employee

• Data Engineer

  • Feb 2014 - Oct 2014

  Played a key role building process which provided daily Ecom data (such as orders, customer data, Google Analytics, Cybersource, Google Spreadsheets etc) from various sources in order to help Analysts within the Business Intelligence team to target right products to customers (Python, Redshift, Amazon AWS). - Designed and implemented the automated email data feed process used to target new offers to customers based on previous orders as well as rate your meal emails. The feed would run on a daily basis sending across to eCircle (TeraData) CSV files containing information on updated and new customers from various different countries and their corresponding new meal offers via FTP (Python, Redshift, Amazon S3 servers). - Daily Maintenance of Ecom data warehouse . Mentoring and training up of team members. Show less



BlisMedia

• Software Engineer

  • Jun 2012 - May 2013

  Played a key role within a team to design, develop and maintain a mission critical Real Time Bidding platform (RTB, also known as the High Frequency trading of advertising). Written in C++ on top of an Apache HTTP server and using industry standard platform and tools such as Ubuntu (Linux) and GDB for debugging, - Integrated various Supply Side Platforms (SSP) into bidder such as Rubicon,Smaato,MobClix,MoPub, Google AdX and AdMob in C++ using Ubuntu (Linux). - Redesigned and Implemented Reporting solution which gave feedback on hourly Volumes of Requests, Responses, Wins, Losses and clicks grouped by location, device, SSP and various other metrics. Show less



Credit Suisse

• Switzerland
• Banking
• 700 & Above Employee

• System Engineer

  • Jun 2010 - Apr 2011

  Supported day to day activities of the monitoring environment for a global team, such as testing, patching, configuration management, power down/DR exercises (in UNIX) and client support queries. - Utilising industry standard monitoring / messaging solutions such as OWL within trade applications to monitor state of processes. - Designed and implemented mission critical wall-street web service communication stub (PERL) in a UNIX environment, interfacing monitoring system to distributed Credit Suisse Market Data Server processes (MDS) via SOAP over HTTP (UNIX / Solaris). - Designed and implemented various automated Control-M (PERL) scripts to restore Disaster Recovery environments. - Designed and implemented JMX agent features (Java) to monitor various java platform health attributes such as memory utilisation (UNIX / Solaris). Show less



Nokia

• Finland
• Telecommunications
• 700 & Above Employee

• Software Engineer

  • Jan 2007 - Apr 2010

  Designed and implemented various UI features for handsets as per requirements to further enhance user experience (written in C). One such example was the Japanese text editor which was widely used in many applications within devices. Exposure to full software lifecycle including design, implementation and debugging. Designed and implemented various UI features for handsets as per requirements to further enhance user experience (written in C). One such example was the Japanese text editor which was widely used in many applications within devices. Exposure to full software lifecycle including design, implementation and debugging.