Data Engineer

Job Overview

Key Qualifications

Proficient in SQL and at least one programming language (Java/Python preferred)

6+ years of experience in software and/or data engineering

3+ years of Spark development.

6+ years of experience in Big Data Technologies (Hadoop, MapReduce, Hive etc…). Spark experience preferred.

Experience on Kubernetes, Docker preferred.

Description

• Develop and automate large scale, high-performance, scalable platform (batch and/or streaming) to drive faster analytics• Ability to design large-scale, complex applications and frameworks with excellent run-time characteristics such as low-latency, fault-tolerance and availability• Experience in building and maintaining custom frameworks to support engineering/analytics needs• Knowledge of continuous integration, testing methodologies, TDD and agile development methodologies.• Partner with analytic consumers and data scientists to build and improve new/existing constructs and solve data engineering problems @ scale.• Good knowledge of Data formats (Parquet, ORC etc.) and consensus management systems• Exposure to structured or unstructured storage and distributed caching.• Deploy inclusive data quality checks to ensure high quality of data.• Experienced Engineer or Contributor or Committer to open source technologies is plus. • Evangelize high quality software engineering practices towards building data infrastructure and pipelines at scale. • Structured thinking with ability to easily break down ambiguous problems and propose impactful solutions.• Communication Strong documentation and technical writing skills. • Attention to detail and effective verbal/written communication skills.

Education & Experience

BS/MS in Computer Science Quantitative Finance, Math, Physics or a related Engineering degree

View More