Artificial Intelligence Engineer

The AI / Agentic AI Engineer works across: (1) large-scale data pipeline development processing market events in a cloud environment, and (2) primarily, design and development of agentic AI systems including LLM-powered regulatory data assistants, MCP servers, and agent harness architectures. This position contributes to overall product quality throughout the software development lifecycle.

• Build and maintain ETL/ELT pipelines using Apache Spark, Hive, and Trino across S3-based data lake environments
• Develop and optimize SQL for large-scale surveillance datasets including window functions, multi-table joins, and complex aggregations
• Build and engineer big data systems (EMR-on-EC2, EMR-on-EKS) and develop solutions on analytical platforms (Sage Maker, Domino, Dataiku)
• Participate in data quality monitoring, anomaly detection, and production incident investigation
• Develop AI agent systems using AWS Bedrock and agent frameworks (Strands Agents SDK, Lang Chain/Lang Graph, or equivalent)
• Build agent harness architectures combining LLM reasoning with deterministic execution - skill/RAG-based SQL generation and structured output validation
• Implement agent memory, context management, and tool integration (MCP servers, API connectors, data catalog lookups) across the data lake
• Build evaluation frameworks for agent accuracy - paraphrase robustness, routing precision, and structural consistency
• Stay informed of advances in LLM frameworks (Lang Graph, Google ADK, AWS Strands) and emerging AI capabilities
• Write clean, well-tested code; contribute to CI/CD Jenkins pipelines and infrastructure-as-code on AWS
• Ensure secure handling of RCI and sensitive regulatory data across both data pipelines and agent outputs - auditable execution traces
• Adhere to CLIENT and team standards for secure development practices and technology policies
• Partner across teams, communicate technical information at the appropriate level, and maintain documentation on Confluence/Wiki
• Actively learn from senior team members; contribute to process improvement in line with CLIENT's values of collaboration, expertise, innovation, and responsibility

• Experience building data pipelines using Apache Spark (PySpark preferred) and SQL
• Experience with SQL query engines (Hive, Trino/Presto, or similar) and cloud data platforms (AWS S3, EMR, Lambda)
• Understanding of common issues like data skew and strategies to mitigate it, working with large data volumes, and troubleshooting job failures due to resource limitations, bad data, and scalability challenge
• Real-world experience with debugging and mitigation strategies

• Practical experience building LLM-powered agent systems that use tools and produce structured outputs (not just chatbot interfaces)
• Hands-on experience with at least one agent framework:
  Lang Chain, Lang Graph, AWS Strands, or equivalent
• Working knowledge of prompt engineering, RAG architectures, and context/memory management
• Experience with foundation model APIs (Anthropic Claude, Amazon Nova, OpenAI, or similar)
• Memory Architecture:
  Understanding of agent memory tiers - working memory, episodic memory, semantic memory - and strategies for context persistence, pruning, and retrieval across session
• Agent Harness Design:
  Familiarity with harness patterns that wrap LLM reasoning with deterministic guardrails, tool routing, verification loops, and graceful degradation
• Hands-on experience with AI development tools (Git Hub Copilot, Q Developer, ChatGPT, Claude, etc.)
• Experience with spec-driven development - using structured specifications to guide AI code generation, review, and validation
• Ability to leverage AI pair programming for code suggestions, debugging, refactoring, and automated test generation
• Experience with AWS services like S3, EMR, EMR on EKS, Lambda, Bedrock, Step Functions, etc
• Hands-on experience using S3 with Spark (e.g., dealing with file formats, consistency issues
• Familiarity with AWS Bedrock for foundation model invocation, knowledge bases, guardrails, and agent orchestration
• Exposure to Google Cloud Vertex AI (model garden, grounding, agent builder) or equivalent managed AI platforms
• Familiarity with AWS monitoring and logging tools (Cloud Watch, Cloud Trail) for production workloads

• Proficiency in Python for data engineering and automation
• Ability to write clean, modular, and performant code
• Experience with functional programming concepts (e.g., immutability, higher-order functions)
• Strong understanding of collections, concurrency, and memory management

• Proficiency with SQL window functions, multi-table joins, and aggregations
• Ability to write and optimize complex SQL queries
• Experience handling edge cases like NULLs, duplicates, and ordering

• AWS Bedrock Agent Core (memory, identity, tool gateway)
• Model Context Protocol (MCP) server development and integration
• Agent evaluation harnesses and agentic patterns (draft-verification,…