Big Data Engineering Lead – Wallet & Apple Pay – NYC

Job Overview

Key Qualifications

10+ years ETL, BI & Data Analytics

Confirmed experience with Big Data systems, pipelines and data processing

Practical hands-on experience with technologies like Apache Hadoop, Apache Pig, Apache Hive, Apache Sqoop & Apache Spark

Ability to understand API Specs, identify relevant API calls , extract data and implement data pipelines & SQL friendly structures

Expertise in Python, Unix Shell scripting and Dependency driven job schedulers

Expertise in Core JAVA, Oracle, Teradata and ANSI SQL

Familiarity with rule based tools and APIs for multi stage data correlation on large data sets is a plus

Excellent time leadership skills with the ability to manage work to tight schedules and handle the pressure of executive requests and product launches

Experience with mentoring and leading data engineers

Description

Translate business requirements by business team into data and engineering specificationsBuild scalable data sets based on engineering specifications from the available raw data and derive business metrics/insights Work with engineering and business partners to define and implement the data engagement relationships required with partners Understand and Identify server APIs that needs to be instrumented for data analytics and reporting and align the server events for execution in already established data pipelines Process, clean and validate the integrity of data used for analysis Develop Python and Shell Scripts for data ingestion from external sources for business insights Work hand in hand with the DevOps team and develop monitoring and alerting scripts on various data pipelines and jobs Mentor a team of hardworking engineers

Education & Experience

Minimum of bachelor’s degree, preferably in Computer Science, Information Technology or EE, or relevant proven experience is helpful

View More