Sr. Big Data Engineer

Allstate Insurance
Allstate Insurance

Job Overview

Where good people build rewarding careers.

Think that working in the insurance field can’t be exciting, rewarding and challenging? Think again. You’ll help us reinvent protection and retirement to improve customers’ lives. We’ll help you make an impact with our training and mentoring offerings. Here, you’ll have the opportunity to expand and apply your skills in ways you never thought possible. And you’ll have fun doing it. Join a company of individuals with hopes, plans and passions, all using and developing our talents for good, at work and in life.

Job Description

LOCATION: Irving, TX OR Charlotte, NC OR TEMPE, AZ

Information Services Group provides technology for Allstate’s information strategy, which gives a significant competitive advantage to the company. Users across Allstate have the ability to tap into massive amounts of data to understand what consumers want in order to make sound business decisions, drive product innovation and improve the customer’s experience.

The Big Data Engineer – Senior Consultant II is accountable for leading a technical team of developers on the Enterprise Data Quality team to deliver innovative data quality solutions that generate business value, specifically focused on Hadoop-based, non-relational data. This role requires in-depth data experience to translate unique business requirements into technical data quality specifications that ensure data is fit-for-purpose. Engage with strategic partners to optimize the data quality function and identify strategic capabilities to keep up with data and insurance industry trends.

Job Description

Key Responsibilities

* Serve as a subject matter expert and an advocate of the data quality strategic vision and enterprise data strategy
* Design and drive the implementation of end-to-end data quality solutions for multiple data ecosystems that leverage non-relational data
* Consistently contribute to process improvements and help lead the team to deliver working solutions and data quality standards
* Participate and review documentation for self-service data quality practices, the process framework, and internal guides
* Collaborate with various business and technical teams to gather requirements around data quality rules and propose the optimization of these rules if applicable, then design and develop these rules in Informatica Data Quality, Informatica Data Engineering Quality (DEQ), or other home-grown technologies
* Perform thorough data profiling with multiple usage patterns, root cause analysis and data cleansing and develop scorecards utilizing Informatica, Excel and other data quality tools
* Configure and manage connection process
* Develop “matching” plans, help determine best matching algorithm, configure identity matching and analyze duplicates
* Build workflows scorecards in Informatica Data Engineering Quality (DEQ) to support data remediation
* Run data quality jobs (address standardization and validation, email cleanups, name cleanup, parsing, etc.) utilizing DEQ and other ETL tools
* Serve as the primary resource to team members and data stewards for training, problem resolution, data profiling etc.
* Analyze and provide data metrics to target personas to provide insight into data quality and help prioritize remediation efforts
* Identify and resolve issues, bugs, and impediments

Job Qualifications

Education and Experience

* Bachelor’s Degree in Math, Statistics, Finance, Business, Economics, IT or related field, or equivalent experience
* 5-7+ years of data and/or analytics experience

Required Skills

* Experience with various data types (i.e. relational, unstructured, semi-structured, hierarchical, linked “graph” data, streaming, biometrics)
* Experience with development, management, and manipulation of large, complex datasets
* Experience with database & ETL technologies
* Demonstrated knowledge of data management competencies and implementation
* Proficient at writing SQL queries and verifying the results
* Familiar with organizational change management strategies
* Strong communication skills, both written and verbal
* Excellent time-management and organization skills
* Ability to operate in a rapidly changing, high-visibility environment
* Desire to learn and ability to learn quickly
* Advanced analytical and problem-solving skills
* Strong decision-making skills
* Ability to draw meaningful conclusions and recommendations

Job Qualifications


* Experience working within agile methodologies
* Working knowledge of cloud infrastructure (AWS/Azure/Google)
* Intermediate technical knowledge related to various data platforms
* Administration experience on Hadoop, HDFS, YARN, Spark, Sentry/Ranger, HBase and Zookeeper
* Experience using Informatica Data Quality, Informatica Data Engineering Quality, Informatica Analyst
* Experience in configuring & troubleshooting the components in the Hadoop ecosystem Spark, Solr, Scala, Kafka etc.
* Ability to communicate and present advanced technical topics to general audiences
* Ability to code in languages such as Python, Perl, Java, C, R, SQL, XSLT
* Understanding of analytics techniques such as predictive modeling, machine learning, neural networks

Certificates, Licenses, Registrations

* Preferred: Certified Data Management Professional Certification
* Preferred: Informatica Data Quality or Data Engineering Quality (or other Informatica tool) Certification
* Preferred: non-relational data store certification

The candidate(s) offered this position will be required to submit to a background investigation, which includes a drug screen.

Good Work. Good Life. Good Hands®.

As a Fortune 100 company and industry leader, we provide a competitive salary – but that’s just the beginning. Our Total Rewards package also offers benefits like tuition assistance, medical and dental insurance, as well as a robust pension and 401(k). Plus, you’ll have access to a wide variety of programs to help you balance your work and personal life — including a generous paid time off policy.

Learn more about life at Allstate. Connect with us on Twitter, Facebook, Instagram and LinkedIn or watch a video.

Allstate generally does not sponsor individuals for employment-based visas for this position.

Effective July 1, 2014, under Indiana House Enrolled Act (HEA) 1242, it is against public policy of the State of Indiana and a discriminatory practice for an employer to discriminate against a prospective employee on the basis of status as a veteran by refusing to employ an applicant on the basis that they are a veteran of the armed forces of the United States, a member of the Indiana National Guard or a member of a reserve component.

For jobs in San Francisco, please click “here” for information regarding the San Francisco Fair Chance Ordinance.

For jobs in Los Angeles, please click “here” for information regarding the Los Angeles Fair Chance Initiative for Hiring Ordinance.

To view the “EEO is the Law” poster click “here”. This poster provides information concerning the laws and procedures for filing complaints of violations of the laws with the Office of Federal Contract Compliance Programs

To view the FMLA poster, click “here”. This poster summarizing the major provisions of the Family and Medical Leave Act (FMLA) and telling employees how to file a complaint.

It is the Company’s policy to employ the best qualified individuals available for all jobs. Therefore, any discriminatory action taken on account of an employee’s ancestry, age, color, disability, genetic information, gender, gender identity, gender expression, sexual and reproductive health decision, marital status, medical condition, military or veteran status, national origin, race (include traits historically associated with race, including, but not limited to, hair texture and protective hairstyles), religion (including religious dress), sex, or sexual orientation that adversely affects an employee’s terms or conditions of employment is prohibited. This policy applies to all aspects of the employment relationship, including, but not limited to, hiring, training, salary administration, promotion, job assignment, benefits, discipline, and separation of employment.Sr. Big Data Engineer 1

View More
Job Detail
Shortlist Never pay anyone for job application test or interview.