Data Engineer - Managing Consultant

## Data Engineer - Managing Consultant Apply locations:
US - VA, McLean time type:
Full time posted on:
Posted Todayjob requisition :
41103
** Job Family:
** Data Engineering & Architecture Consulting
*
* Travel Required:

** Up to 10%
*
* Clearance Required:

** Ability to Obtain Public Trust
*
* What You Will Do:

*** Design, build, and optimize scalable, production-grade data ingestion, transformation, and analytics-ready pipelines using Databricks (Delta Lake, Delta Live Tables, Auto Loader) and AWS services, enabling trusted, timely data access across enterprise use cases.
* Engineer standardized, repeatable data pipelines supporting batch and near real-time processing, integrating legacy data sources and modern cloud-native services to advance enterprise data availability and eliminate data access gaps.
* Execute full delivery lifecycle by supporting intake, discovery, source profiling, and technical design, while maintaining requirements traceability and aligning solutions to Architecture Review Board (ARB) and governance expectations.
* Implement and maintain governed data pipelines with embedded metadata, lineage, and data quality controls, ensuring pipelines meet defined technical, security, and documentation requirements before production deployment.
* Develop and operationalize data engineering frameworks that incorporate observability, monitoring, alerting, and resilient error handling to maintain platform stability and support ≥99.9% availability targets.
* Partner with platform engineering and cloud operations teams to integrate pipelines with AWS services (S3, Glue, Kafka/Kinesis, APIs), enabling secure, scalable data movement and cross-platform interoperability.
* Enable governed analytics and self-service data consumption through curated datasets, semantic layers, and SQL warehouse integration, supporting enterprise reporting, dashboards, and advanced analytics use cases.
* Apply security and compliance controls aligned to IRS cybersecurity policies, including RBAC/ABAC, data masking, encryption, and audit logging to protect sensitive data and maintain regulatory compliance.
*
* What You Will Need:

*** Bachelor’s degree is required
* Minimum EIGHT (8) years of experience in data engineering within cloud-based environments,
* Minimum FOUR (4) years of hands-on Databricks experience designing scalable data pipelines.
* Advanced proficiency in Python, PySpark, and SQL for large-scale data processing and pipeline development in lakehouse architectures.
* Demonstrated experience building and optimizing data pipelines leveraging Delta Lake, medallion architecture, and modern data ingestion frameworks (batch and streaming).
* Strong experience with AWS data platforms and services (e.g., S3, IAM, VPC, Glue, streaming frameworks) and integration with enterprise data ecosystems.
* Experience delivering production-ready data solutions incorporating metadata management, lineage, data quality, and observability frameworks.
* Familiarity with Dev Sec Ops  and CI/CD pipeline implementation, including automation, testing, and deployment within cloud data environments.
* Knowledge of data governance, security, and compliance requirements within regulated environments, including FISMA and FedRAMP High
** What Would Be Nice to Have:
*** Experience supporting federal data platforms or large-scale enterprise data modernization efforts, particularly within IRS, Treasury, or similar regulated environments.
* Hands-on experience with Databricks Unity Catalog, Delta Sharing, Genie, and Clean Rooms for governed data access and collaboration.
* Experience implementing streaming and near real-time data pipelines using Kafka, Kinesis, Event Bridge, or similar technologies.
* Familiarity with Informatica EDC/Axon or enterprise metadata/catalog tooling.
* Experience with performance optimization techniques (Photon, Z-ordering, liquid clustering) to improve large-scale data workloads.
* Exposure to CI/CD automation using Terraform, Git Hub Actions, and Databricks Asset Bundles for infrastructure and pipeline deployment.
* Advanced cloud or Databricks certifications (AWS, Databricks) in good standing.
*
* What We Offer:

** Guidehouse offers a…