DataBricks Engineer; Insurance

TECHEAD is celebrating over thirty-five years of incredible heritage, talent, and accomplishments! To learn more about TECHEAD, visit us at or on
Glassdoor.

As an IT team member on the data solutions and data engineering team, you will be joining a team that is transforming data solutions and capabilities in the Finance / Insurance Field. You will be focused on improving out of the box Data Bricks capabilities and executing project related scope in Data Bricks.

You will be improving out of the box Data Bricks capabilities with Data Bricks dashboards, frameworks, and infrastructure Delta tables. An example is building a framework so other teams can collect the needed data for a Data Bricks dashboard heat map to show what pipelines are current to an SLA in a data streaming or micro batch environment.

You will be using the medallion architecture with many data ingestion (bronze), standardization (silver) and curation (gold) functions to improve the data usability with ER and dimensional modeling to create better data structures, values, and derived attributes over legacy data solutions. A specific scope is leveraging underwriting data to improve understanding of future policy costs. This includes using AI to extract data from images and pdfs which was the source of many medical documents that are received about a persons health before approving a policy.

• You will be expected to perform the entire SDLC process from high level requirements definition, build, testing, and production support.
• Working with business users on data requirements.
• Teaming with architects and a tech lead on the design for specific project scope.

• Performing all roles in the SDLC process build and testing are the core roles for this position.
• Creating documentation, training, and consulting with other teams so they know how to leverage frameworks that you have built or enhanced as mentioned above in the example of a SLA data freshness framework for all pipelines.
• Implement and maintain data security policies and procedures, ensuring compliance with industry and company standards and regulations.
• Work with downstream and upstream teams including onshore/offshore vendor teams.
• Ability to quickly upskill in any additional cloud technologies as required.
• Ability to assume new challenges with respect to different technologies or features in Data Bricks such as Intelligent Document Processing, Genie coding, and decide on best practices on how to use these features so others can leverage your work and experience.

• Minimum 3-5 years’ experience full time using Databricks in Azure or AWS using the medallion architecture and data models.
• Using or creating coding frameworks or functions for common data pipeline standardization processes such as standardizing missing values, dates, reference data management, and data repair or imputation for missing values.
• Creating derived business values with simple to complex logic.
• Creating standard dimensions and fact tables with SCD2 time tracking.
• Data Bricks orchestration skills with batch and streaming pipelines
• Creating unit tests, performing SIT, and collaborating with users on UAT test results debugging
• Data Bricks compute and storage optimization skills
• Comprehensive understanding of automation and deployment tools such as CI/CD and Gitlab.
• Strong understanding of Agile and development methodologies including story creation and defining technical tasks with effort estimates.

• Five years of experience with Data Bricks in a transformation environment where you owned and built significant capabilities from scratch
• Experience with insurance business…