Data & Software Engineer Information Technology McLean​/Chantilly, VA

The Data & Software Engineer works with a small team to build complex data flows for a custom application. Successful candidate will have advanced Python programming skills, familiarity with Java, an understanding of data security, privacy, governance, and compliance principles, and a demonstrated history of building production data pipelines and ETL workflows at scale.

The candidate must have experience:

• Building end‑to‑end data pipelines leveraging Python.
• Using orchestration tools to deploy data pipelines, including configuring and updating Spark jobs.
• Containerizing and deploying applications in cloud environments like AWS.
• Working with MySQL and PostgreSQL, including performance tuning, schema design, and query optimization for complex analytical workloads.
• Leveraging industry‑standard tools for code control (Git, IaC control, etc.).
• Working with data catalogs, tracking data lineage, and handling a variety of data formats, including geospatial.
• Using Bash scripting for automation and data processing tasks.
• Integrating AI/ML services and models.

• Work with stakeholders to understand data requirements, assess feasibility, and design appropriate solutions with minimal oversight.
• Leverage strong problem‑solving and debugging skills for data quality issues, pipeline failures, and performance bottlenecks.
• Leverage a background in large‑scale data migration or platform modernization efforts.
• Contribute to data engineering documentation, best practices, and design patterns.

• TS/SCI FSP Clearance.
• Minimum of 5 years experience:
  • Demonstrated experience building production data pipelines and ETL/EL workflows at scale.
  • Proficiency with Apache Spark and PySpark for distributed data processing.
  • Advanced Python programming skills including data manipulation libraries (Pandas, Num Py) and data engineering best practices.
  • Understanding of data security, privacy, governance, and compliance principles.
  • Experience with workflow orchestration tools (such as Step Functions, Airflow).
  • Familiarity with containerization (such as Docker or Podman) and deploying data applications in cloud environments.
  • Experience with AWS services (S3, Lambda, Step Functions).
  • Experience with PostgreSQL and MySQL in production environments, including performance tuning and schema design.
  • Demonstrated experience with SQL and query optimization for complex analytical workloads.
  • Experience with version control (Git) and CI/CD practices for data pipelines.
  • Demonstrated ability to work with stakeholders to understand data requirements, assess feasibility, and design appropriate solutions with minimal oversight.
  • Strong problem‑solving and debugging skills for data quality issues, pipeline failures, and performance bottlenecks.

• Experience with data lakehouse architecture using Apache Iceberg.
• Hands‑on experience configuring, deploying, and integrating data platform components:
  • Apache Ranger (access control and data governance).
  • Trino (distributed SQL query engine).
  • Data catalogs (Unity Catalog OSS, Apache Polaris, etc.).
  • Apache Superset (data visualization and dashboarding).
• Proficiency with Bash scripting for automation and data processing tasks.
• Experience with Infrastructure as Code (Terraform or Cloud Formation) for data infrastructure.
• Familiarity or experience with tracking data lineage and associated tooling such as Open Lineage.
• Familiarity or experience with Java.
• Familiarity with data quality frameworks, testing methodologies, and validation strategies.
• Background with large‑scale data migrations or platform modernization efforts.
• Experience integrating AI/ML services and models (translation, OCR, speech‑to‑text, NLP, language detection, topic modeling), LLMs, and retrieval‑augmented generation pipelines.
• Familiarity with geospatial data processing (H3, PostGIS, or similar).
• Contributions to data engineering documentation, best practices, and design patterns.
• Experience with No
  
  SQL databases (DynamoDB, etc.).