Senior Data Engineer

Location: Richmond

Anticipated End Date:

Position Title:

Senior Data Engineer

Job Description:

Senior Data Engineer

Hybrid 1:
This role requires associates to be in-office 1 - 2 days per week, fostering collaboration and connectivity, while providing flexibility to support productivity and work-life balance. This approach combines structured office engagement with the autonomy of virtual work, promoting a dynamic and adaptable workplace. Alternate locations may be considered if candidates reside within a commuting distance from an office.

Please note that per our policy on hybrid/virtual work, candidates not within a reasonable commuting distance from the posting location(s) will not be considered for employment, unless accommodation is granted as required by law.

The Senior Data Engineer serves as the technical lead and platform owner for the security data lake supporting Application Security, Vulnerability Management, and related cybersecurity functions. This role defines the platform architecture, engineering standards, data models, integration patterns, governance controls, and technical roadmap necessary to deliver reliable and actionable security data. The role is also responsible for the design and delivery of secure, scalable, and governed data ingestion, transformation, enrichment, storage, and consumption capabilities across Microsoft Fabric and supporting Azure services.

How you will make an impact

* Leads the architecture, engineering, and ongoing operation of the security data lake and associated data products supporting multiple security technologies and data sources. Establishes the technical roadmap, engineering standards, reusable design patterns, and platform operating model.

* Acts as a subject matter expert among peers, with manager and senior management on all matters pertaining to data platforms and the supporting architecture.

* Develop and maintain the end-to-end architecture of the security data lake, including ingestion, storage, transformation, enrichment, semantic, and consumption layers, based on business, security, compliance, scalability, and reliability requirements.

* Ensure data pipelines comply with security policies and regulatory requirements (e.g., HIPAA, NIST, SOC
2)

* Develop testing plans to ensure quality of implementation, including deployment validation, data quality checks, reconciliation, and repeatable testing of data pipeline changes.

* Determine and perform complex configuration changes to meet business and information security requirements, including Microsoft Fabric work spaces, data pipelines, connections, service principals, secrets, and environment-specific configuration.

* Provide trouble resolution on complex problems and lead implementations for data pipelines, Microsoft Fabric, Azure Dev Ops, REST API, and source-system integration issues.

* Provide technical support to business and technology associates, in risk assessments and implementation of appropriate information security procedures, standards and technologies.

* Represent major upgrades and reconfigurations in change control; design and analyze vendor and platform services meeting business and information security requirements, including governed promotion across development, test, and production environments.

* Represent infrastructure security support in significant projects and perform the most complex operations and administration tasks.

* Respond to level 3 & 4 change and problem requests without supervision;

* Lead level 1 & 2 incident recoveries and root cause analysis.

* Coordinate and prepare the reporting of data security events and incidents.

* Perform capacity analysis; recommend and implement capacity increases for data platform performance, scalability, and reliability.

* Maintains relationship with key vendors;

* Serve as the technical escalation for results of preventative maintenance routines;

* Supervise preventative maintenance;

* Creates presentations and seeks IT management approval and acceptance of significant replacements or reconfigurations of the data pipeline.

Minimum Requirements:

Requires BS/BA in information Technology or related field of study and a minimum of 8 years experience in systems administration and security aspects of information systems, access management and network security technologies, network communications, computer networking, telecommunications, systems development and management, hardware, software, data, and people; experience with multiple technical and business disciplines required; or any combination of education and experience, which would provide an equivalent background.

Preferred Skills, Capabilities and Experiences:

* Experience with architecting and operating Microsoft Fabric, Apache Spark, or comparable enterprise lakehouse platforms

* Experience in the design, development, testing, deployment and maintenance of scalable data pipelines for structured and unstructured data.

* Experience with SQL, Python, PySpark, Spark, Delta Lake, Parquet, REST APIs, JSON, Git, Azure Dev Ops,…