Data Engineer

DiverseLynx
DiverseLynx

Job Overview

Responsibilities:

  • Design and Build distributed, scalable, and reliable data pipelines that ingest and process data at scale.
  • Extensive Data Analysis and Data exploration
  • Explore new data sources and data from new domains
  • Build feature computation pipelines for various Client models.
  • Candidate Profile:

  • 8-10 years of hands-on programming experience with 4+ years in Hadoop platform
  • Proficiency in data analysis and strong SQL skills.
  • Knowledge of various components of Hadoop ecosystem and experience in applying them to practical problems Hive/Impala/Spark.
  • Strong experience with spark.
  • Experience with Python
  • Proficiency with shell scripting
  • Experience in data warehousing, ETL tools , MPP database systems
  • Experience working in HIVE & Impala & creating custom UDFs and custom input/output formats /serdes
  • Ability to acquire, compute, store and process various types of datasets in Hadoop platform
  • Excellent written and verbal communication skills
  • Experience working with Scala or Java is preferred.
  • Understanding of various Visualization platforms (Tableau, Qlikview, others) is nice to have.
  • Top skill sets / technologies:

  • Hive/Impala/MapReduce/Spark
  • Unix/Shell Scripting
  • Python
  • ETL/Data warehousing
  • SQL
  • Scala/Java
  • View More
    Job Detail
    Shortlist Never pay anyone for job application test or interview.