Azure Data Engineer

Job Overview

As a Data Engineer on the Data Science and Platform team, you will middot Develop and maintain data pipelines including solutions for data collection, management and usage. middot Work closely with Data Scientists to optimize and reengineer model code to be modular, efficient and scalable, and to deploy models to production. middot Maintain and enhance our data and computation platform. middot Develop and implement solutions for data quality validation and continuous improvement. middot Manage, execute and monitor weekly and monthly production operations resolve and escalate production issues as appropriate. middot Partner with business stakeholders to understand business and technical requirements, plan and execute projects, and communicate status, risks and issues. middot Perform root cause analysis of system and data issues and develop solutions as required. Qualifications Required BABS in business, computer science or other technical field or 5+ years of appropriate work experience Proficient in Azure SQL Data Warehouse, T-SQL and database architecture. Implement data warehouses and analysis services as part of a modern data warehouse (MDW) design Experience working with cloud-based technologies, including relational databases, data warehouse, big data (Spark), orchestrationdata pipeline tools, data frames concepts, Spark Streaming and Spark SQL Experience with Azure Analytics stack, Azure Data Lake, Azure Databricks, Azure Data Factory. Experience with ETL data pipelining concept. Proficient in Power BI Experience working in software engineering, and can demonstrate best practices for project management, quality control, and product development. Proven track record of collaborative development in an agile team environment. Experience with Python, Scala. Expertise in Pyspark or Scala is a must. Hands on experience of writing notebooks in Databricks using python for complex data aggregations, transformations, schema operations. Preferred middot Experience with Cosmos DB middot Significant experience with utilizing Azure Analysis Services middot Experience with Airflow middot Elastic Search middot Experience in Java middot Experience building APIs and services using REST