Data Engineer (TS Clearance)

Job Overview

Job Type: Full TimeSalary: $100,000 – 110,000 / yearAn Active Top Secret (TS) Clearance is Required.We are looking for a data engineer /developer to build next generation data platform from ground up on AWS as part of a small focused team. Our ideal candidate will have strong knowledge of computer science fundamentals with programming experience in Scala / Python and Spark. The right candidate for this role will identify this challenge as a unique and valuable opportunity to help drive our global technology transformation, so if this sounds interestingResponsibilities:+ Build complex data ingestion pipelines using Scala, Spark, Parquet and S3+ Design scalable processes in event driven architecture to support Fixed Income applications+ Develop near real time streaming analytics using Kafka/Kinesis+ Act as Subject Matter Expert and help rest of the team in leveraging the platform and migrating applications to it+ Establish end to end data lineage and data catalogue. Work with data governance team to setup data quality checks and metrics+ Create self-service notebook environment with Zeppelin/Jupyter for exploratory data analytics and rapid interactive development+ Troubleshoot any performance issues and ensure efficient data organization+ Build efficient web-based tools for monitoring and trackingQualifications:+ An Active Top Secret (TS) Clearance is Required.+ Bachelor’s degree in computer science or related field required. Master’s degree preferred+ 1+ years of experience in programming with Scala, Python or Java+ 1+ years of experience in Scala, Spark and functional programming. Deep knowledge of spark internals such as partitioning, DAG, lazy evaluation.+ Strong experience with relational data bases, SQL, and query optimization. Knowledge of data warehousing, dimensional data model and business intelligence is a plus.