Senior Software Engineer AI

Senior AI/ML Engineer

We're looking for a Senior AI/ML Engineer to design, build, and optimize data pipelines that power our next-generation AI-driven accounting agents. You'll lead the development of scalable, high-performance data infrastructure while collaborating closely across teams.

• Lead data pipeline development:
  Build and maintain PySpark ETL pipelines with high data quality and performance
• Manage integrations:
  Establish robust connections to client data sources via APIs and tools like Five Tran, Plaid, and Black Line's own internal connector ecosystem
• Ensure reliability:
  Monitor pipeline performance, automate testing, and validate data accuracy
• Optimize for scale:
  Implement performance improvements (e.g., CDC mechanisms, indexing strategies) for large-scale datasets
• Collaborate & innovate:
  Work with business stakeholders to refine data requirements and integrate cutting-edge AI and big data technologies

Leadership and Strategy

• Partner with data science, security, and product teams to set evaluation and governance standards (Guardrails, Bias, Drift, Latency SLAs).
• Mentor senior engineers and drive design reviews for ML pipelines, model registries, and agentic runtime environments.
• Lead incident response and reliability strategies for ML/AI systems.
• Collaborate with development teams to integrate AI solutions into existing workflows and applications.
• Ensure seamless integration with different platforms and technologies.
• Define and manage MCP Registry for agentic component onboarding, lifecycle versioning, and dependency governance.
• Build CI/CD pipelines automating LLM agent deployment, policy validation, and prompt evaluation of workflows.
• Develop and operationalize experimentation frameworks for agent evaluations, scenario regression, and performance analytics.
• Implement logging, metering, and auditing for agent behavior, function calls, and compliance alignment.
• Create scalable observability systems-tracking conversation outcomes, factual accuracy, latency, escalation patterns, and safety events.
• Architect end-to-end guardrails for AI agents including prompt injection protection, identity-aware routing, and tool usage authorization.
• Collaborate cross-functionally to standardize authentication, authorization, and session governance for multi-agent runtimes.
• Architect and standardize model registries and feature stores to support version tracking, lineage, and reproducibility across environments.
• Lead the deployment of machine learning models into production environments, ensuring scalability, reliability, and efficiency.
• Collaborate with software engineers to integrate machine learning models into existing applications and systems.
• Implement and maintain APIs for model inference.
• Design and manage training infrastructure including distributed training orchestration, GPU/TPU resource allocation, and automatic scaling.
• Implement CI/CD for model workflows using pipelines integrated with model validation, bias checks, and rollback automation.
• Build standardized experimentation frameworks for reproducible training, tuning, and deployment cycles (MLflow, W&B, Kubeflow).
• Manage and optimize the infrastructure required for machine learning operations in cloud.
• Work closely with other teams to ensure the availability, security, and performance of machine learning systems.
• Implement robust monitoring solutions for deployed machine learning models to detect issues and ensure performance.
• Collaborate with data scientists and engineers to address and resolve model performance and data quality issues.
• Conduct regular system maintenance, updates, and optimizations to ensure optimal performance of machine learning solutions.
• Develop and maintain automation scripts and tools for managing machine learning workflows.
• Implement orchestration systems to streamline the end-to-end machine learning lifecycle, from data preparation to model deployment.
• Collaborate with data scientists to understand model requirements and constraints for deployment.
• Facilitate the transition of machine learning models from research to production, ensuring scalability and efficiency.
• Identify and implement optimizations to…