Senior Data Engineer

Optomi, in partnership with a premier financial regulatory organization, is seeking a Data Engineer for their Tysons Corner, VA or Rockville, MD location! This position sits within a highly visible surveillance and analytics organization focused on fraud detection, market monitoring, and regulatory technology initiatives. The ideal candidate will operate across two equally important domains: large-scale data engineering and hands-on generative AI development.

This engineer will help build and maintain Spark-based pipelines processing massive regulatory datasets while also developing LLM-powered assistants, agent frameworks, and AI-driven solutions that empower business users to interact with data more efficiently. The right candidate will enjoy working on cutting-edge AI initiatives while supporting one of the largest financial data ecosystems in the cloud.

• Working on a highly visible initiative supporting fraud detection, surveillance analytics, and regulatory technology modernization.
• Building enterprise-grade GenAI solutions that solve real business problems rather than simply consuming off-the-shelf AI tools.
• Working hands‑on with Spark, Databricks, AWS cloud services, and distributed data processing technologies.
• Collaborating directly with senior architects and engineering leaders to bring AI concepts and prototypes into production.

• 5+ years of experience in Data Engineering, Big Data Engineering, or similar software engineering disciplines.
• Strong experience building and maintaining data pipelines using Apache Spark, PySpark, and SQL.
• Experience working with Databricks and distributed data processing environments.
• Strong Python development skills for data engineering, automation, and AI‑related workloads.
• Experience with AWS data platform technologies including S3, EMR, Lambda, Glue, and related services.
• Experience with SQL query engines such as Hive, Trino/Presto, or similar technologies.
• Hands‑on experience building solutions with LLMs rather than simply using AI tools for productivity.
• Experience with agent frameworks such as Lang Chain, Lang Graph, AWS Strands, or equivalent platforms.
• Experience implementing RAG architectures, prompt engineering, memory management, and context‑aware workflows.
• Experience with CI/CD pipelines utilizing Jenkins, Git Hub Actions, Git Lab CI, or similar technologies.
• Experience in highly regulated industries such as Financial Services, Banking, Insurance, Healthcare, or Government is preferred.

• Build and maintain scalable data pipelines using Spark, PySpark, SQL, and cloud‑native AWS technologies.
• Design and optimize ETL/ELT processes supporting large‑scale surveillance and regulatory datasets.
• Develop solutions for leveraging AWS services including S3, EMR, Lambda, Glue, and related cloud data platforms.
• Write and optimize complex SQL queries utilizing joins, aggregations, window functions, and analytical processing techniques.
• Troubleshoot production pipelines, investigate data quality issues, and resolve cloud‑based processing failures.
• Develop LLM‑powered agent systems capable of generating structured outputs and performing business‑driven tasks.
• Build AI assistants that leverage RAG architectures, enterprise knowledge sources, and large‑scale data repositories.
• Implement agent workflows using Lang Chain, Lang Graph, AWS Bedrock, or similar frameworks.
• Integrate AI systems with APIs, data catalog services, enterprise data lakes, and structured data platforms.
• Utilize AI‑assisted development tools while applying rigorous engineering standards, testing practices, and validation techniques.
• Build and maintain CI/CD automation pipelines while contributing to platform reliability, code quality, and operational excellence.
• Partner with architects, engineering teams, and business stakeholders to deliver scalable data and AI solutions that support future reporting, dashboarding, and business intelligence initiatives.