Director, Data Engineer - PGIM Technology; Hybrid - Newark, NJ

Director, Data Engineer - PGIM Technology (Hybrid - Newark, NJ)

As the Director, Data Lead for the PGIM Product Technology, you will engage in requirements gathering, lead design, architect and partner with business owners, technology leads and engineering teams to support shift to Microsoft cloud platforms for data processing and management. This candidate will work to define and enforce standards while also overseeing end-to-end data engineering including sourcing, modeling, security, governance, and business intelligence functions.

You will design and lead engineers to implement solutions in accordance with industry best practices. Your expertise in data processing and data analytics using the Microsoft Fabric platform will be crucial in delivering successful data solutions. The candidate will play a key role to bridge solutions, communication, timelines, and challenges/solutions to project stakeholders for multiple, concurrent overlapping implementation initiatives.

This role is based in our office in Newark, NJ. Our organization follows a hybrid work structure where employees can work remotely and from the office, as needed, based on demands of specific tasks or personal work preferences. This position is hybrid and requires your on-site presence on a reoccurring weekly basis 3 days per week.

Design and Implement Configurable Data Ingestion ETL framework including Data Quality, transformation and modelling. Implement Lakehouse solutions using Microsoft Fabric and paving the road to PGIM future state using Azure services and Fabric Infrastructure.

Develop and deploy data integration solutions, including data ingestion, transformation, and delivery to consumers within Microsoft Fabric, while utilizing optimal storage formats and layers.

Work with Infrastructure/Devops teams to come up with CI/CD pipelines as a framework that can be extended across organization.

Architect solutions that incorporate BI tools like Power BI ensuring seamless data integration for analytics and reporting.

Ensure data accuracy, consistency, and integrity across the data warehouse ecosystem by crafting robust solution designs.

Design and Build AI solutions leveraging search, embeddings, retrieval, ranking and Retrieval-Augmented Generation (RAG).

Set up roadmap for AI as a service to consuming systems and business stakeholders.

Establish a workspace architecture strategy through design and implement of data warehouse and lakehouse solutions allowing data sharing between different teams avoiding duplication, yet adhering to security and governance rules.

Lead a small team to design/build and monitor data and AI solutions.

Ensure teams follow best practices and standards for security, compliance, scalability, reliability, and performance.

Serve as SME in the Fabric space to technology partners, provide hands on training to development teams, and mentor business areas to maximize Fabric AI Capabilities.

Create and maintain comprehensive documentation (Playbooks) of solution architectures to internal and external stakeholders covering Data Fabric architecture, processes, and procedures.

Collaborate closely with our business and technology partners including data analysts, architects, engineers, and client stakeholders to gather requirements and translate them into scalable Microsoft Fabric solutions.

Proactively seek out new knowledge and adapt to new trends, technical solutions, and patterns that improve the availability, reliability, efficiency, observability, and performance of products while also driving consistency in monitoring and operations at scale, and share knowledge with other engineers.

Bachelor’s or Master’s degree in Computer Science, Data Engineering or related discipline.

10+ years of experience in cloud computing with Microsoft Azure leading migrations, integrations, and deployments. 10+ years of experience as a Data Architect, with demonstrated expertise in designing, managing, and deploying client solutions.

Strong knowledge of Fabric workloads and features including Notebooks, Deployment Pipelines, Workspace architecture, Workspace Security, Data Warehouse/Lakehouse…