Lead Data Engineer with Kafka, Spark, and Hadoop

Byteware Inc.
Byteware Inc.

Job Overview

As a Lead Engineer, yoursquoll take the lead as youhellip Use your technical expertise to apply and maintain knowledge of current and emerging technologies within your specialized area(s) of the technology domain. Evaluate new technologies and participates in decision-making, accounting for several factors such as viability within the clientrsquos technical environment, maintainability, and cost of ownership. Initiate and execute research and proof-of-concept activities for new technologies. Lead or set strategy for testing and debugging at the platform or enterprise level. In complex and unstructured situations, serve as an expert resource to create and improve standards and best practices to ensure high-performance, scalable, repeatable, and secure deliverables Lead the design, lifecycle management, and total cost of ownership of services. Participate in planning services that have enterprise impact. Provide suggestions for handling routine and moderately complex technical problems, escalating issues when appropriate. Gather information, data, and input from a wide variety of sources identify additional resources when appropriate, engage with appropriate stakeholders, and conduct an in-depth analysis of information. Develop plans and schedules, estimate technical pre-requisites for project requirements, and define milestones and deliverables. Monitor workflow and risks play a leadership role in mitigating risks and removing obstacles. Lead and participate in complex construction, automation, and implementation activities, ensuring successful implementation with architectural and operational requirements met. Establish new standards and best practices to monitor, test, automate, and maintain IT components or systems. Serve as an expert resource in disaster recovery and disaster recovery planning. Stay current with the clientrsquos technical capabilities, infrastructure, and technical environment. Develop fully attributed data models, including logical, physical, and canonical. Influence data standards, policies, and procedures. Install, configure, andor tune data management solutions with minimal guidance. Monitor data management solution(s) and identify optimization opportunities You are equally happy coding and leading a team to implement a solution You have a track record of innovation and expertise in Data Engineering Yoursquore passionate about craftsmanship and have applied your expertise across a range of industries and organizations You have a deep understanding of data modeling and experience with data engineering tools and platforms such as Kafka, Spark, and Hadoop You have built large-scale data pipelines and data-centric applications using any of the distributed storage platforms such as HDFS, S3, NoSQL databases (Hbase, Cassandra, etc.) and any of the distributed processing platforms like Hadoop, Spark, Hive, Oozie, and Airflow in a production setting Hands-on experience in MapR, Cloudera, Hortonworks andor cloud (AWS EMR, Azure HDInsights, Qubole etc.) based Hadoop distributions You are comfortable taking data-driven approaches and applying data security strategy to solve business problems Yoursquore genuinely excited about data infrastructure and operations with familiarity working in cloud environments Working with data excites you you have created Big data architecture, you can build and operate data pipelines, and maintain data storage, all within distributed systemsLead Data Engineer with Kafka, Spark, and Hadoop 1

View More
Job Detail
Shortlist Never pay anyone for job application test or interview.