Responsible for maintaining scalable and reliable data pipelines that support data operations for Reporting, Analytics, Applications, and Data Science by gathering and processing raw data at scale. Delivers solutions by developing, testing, and supporting cloud-based streaming applications. Develops data set processes for data modeling, mining, and consumption.
MAJOR DUTIES AND RESPONSIBILITIES
Actively and consistently supports all efforts to simplify and enhance the customer experience.
Create and maintain scalable, reliable, consistent and repeatable pipelines that support data operations for Reporting, Analytics, Applications, and Data Science.
Gather and process real-time data at scale (including writing APIs, stream processing jobs, and aggregation jobs).
Expertly use cloud-based tools to ingest and process data.
Profile data to measure quality, integrity, accuracy, and completeness.
Manage life cycle of multiple data sources.
Increase speed to delivery by implementing workload/workflow automation solutions.
Interact with various stakeholders to understand their business needs, communicate project status and develop relationships to ensure satisfaction
Perform other duties as assigned.
Skills/Abilities and Knowledge
Ability to read, write, speak and understand English
Ability to use a wide variety of open source technologies and cloud services
Strong coding experience using Scala or Java
Strong background in Linux/CentOS installation and administration
Strong knowledge in data storage that demonstrates knowledge of when to use a file system, relational database, or NoSQL variant
Strong experience with Spark or Hadoop/Hive
Experience receiving, converting, and cleansing big data
Ability to identify and resolve end-to-end performance, network, server, and platform issues
Attention to detail with the ability to effectively prioritize and execute multiple tasks
In-depth knowledge of Agile development methodologies
Bachelor’s degree in an engineering discipline or computer science
Related Work Experience
3-5 years of designing and building Scala or Java applications
2+ years of designing and building cloud native applications
2+ years of Linux/Unix/CentOS system admin
1+ year(s) of designing and building Kafka or Kinesis applications
Associated topics: data integrity, data manager, data management, data warehousing, database, database administrator, hbase, mongo database administrator, sybase, teradataView More