Big Data Engineer – Wallet & Apple Pay – Austin

Job Overview

Key Qualifications

5+ years of professional experience with Big Data systems, pipelines and data processing

Practical hands-on experience with technologies like Apache Hadoop, Apache Pig, Apache Hive, Apache Sqoop & Apache Spark

Ability to understand API Specs, identify relevant API calls , extract data and implement data pipelines & SQL-friendly data structures

Identify Data Validation rules and alerts based on data publishing specifications for data integrity and anomaly detection

Understanding on various distributed file formats such as Apache AVRO, Apache Parquet and common methods in data transformation

Expertise in Python, Unix Shell scripting and Dependency driven job schedulers

Expertise in Core JAVA, Oracle, Teradata and ANSI SQL

Familiarity with rule based tools and APIs for multi stage data correlation on large data sets is a plus

Description

Translate business requirements by business team into data and engineering specificationsBuild scalable data sets based on engineering specifications from the available raw data and derive business metrics/insights Understand and Identify server APIs that needs to be instrumented for data analytics and align the server events for execution in already established data pipelines Explore and understand sophisticated data sets, identify and formulate correlational rules between heterogenous sources for effective analytics and reporting Process, clean and validate the integrity of data used for analysis Develop Python and Shell Scripts for data ingestion from external data sources for business insights

Education & Experience

Minimum of bachelor’s degree, preferably in Computer Science, Information Technology or EE, or relevant proven experience is preferred

View More