Senior Associate - Security Data Lake Engineer

Location Designation: Hybrid - 3 days per quarter

As part of Technology, you'll have the opportunity to contribute to groundbreaking initiatives that shape New York Life's digital landscape. Leverage cutting‑edge technologies like Generative AI to increase productivity, streamline processes, and create seamless experiences for clients, agents, and employees. Your expertise fuels innovation, agility, and growth — driving the company's success.

New York Life is seeking a Security Data Lake Engineer to design, implement, and optimize the enterprise‑scale security data lake and advanced analytics capabilities. This role will serve as a technical specialist responsible for building scalable data pipelines, enabling data science‑driven use cases, and ensuring the long‑term health and cost‑effectiveness of the data lake.

The Security Data Lake Engineer will collaborate with SOC Analysts, Threat Intelligence, Data Scientists, and other Cybersecurity teams to unlock new detection and analytics capabilities, including AI‑driven anomaly detection, semantic search, and retrieval‑augmented generation (RAG). This role is ideal for an engineer who thrives at the intersection of big data, cloud services, and applied data science, with a focus on enabling cybersecurity outcomes.

• Architect, build, and manage data pipelines across AWS services (S3, Kinesis, Lambda), APIs, and Cribl
• Ensure data lake performance and usability through indexing, query optimization, and aggregation pipeline management for 3.5TB+ daily volumes
• Act as the dedicated owner to maintain data quality and prevent “data swamp” degradation
• Establish a Data Lake Center of Excellence (C4E) to empower decentralized analytics development while maintaining governance and standards
• Collaborate with security teams to define, develop, and deploy data‑driven use cases
• Build ML pipelines for anomaly detection, vector search, semantic search, and other advanced security analytics
• Integrate and optimize Elasticsearch, vector databases, and KNN/percolator search capabilities
• Develop and maintain Jupyter Notebook workflows for data wrangling, modeling, and analysis
• Apply machine learning and natural language processing techniques, including retrieval‑augmented generation (RAG)
• Partner with AI adoption initiatives to ensure secure and effective use of enterprise data for AI/ML
• Stay current with emerging technologies and techniques in data science, big data, and cloud analytics
• Work closely with SOC Engineers, Threat Hunters, and data platform teams to integrate log sources and telemetry into the data lake
• Provide guidance and mentorship across Cyber teams on building analytics use cases
• Serve as the in‑house expert to reduce reliance on external consulting spend, ensuring institutional knowledge is retained within NYL

• 5+ years of hands‑on experience in data engineering, big data platforms, or applied data science
• Proficiency with AWS services (S3, Kinesis, Lambda, Bedrock) and enterprise‑scale analytics design
• Strong programming skills in Python and/or Scala; experience with Jupyter Notebooks
• Experience with indexing, query optimization, and data pipeline optimization at scale (3.5TB+ daily)
• Advanced knowledge of Elasticsearch including advanced search methods (KNN vector search, percolator reverse search), and performance tuning, platform maintenance, etc.
• Experience applying ML techniques for anomaly detection, semantic search, embeddings, and NLP

• Familiarity with Cribl for data routing and transformation
• Advanced degree in mathematics, computer science, or related empirical sciences
• Previous experience in cybersecurity environments or willingness to learn security context quickly

• Bachelor’s degree in Computer Science, Data Science, Cybersecurity, or related field; advanced degree preferred
• Certifications in AWS Data Engineering, Machine Learning, or Big Data technologies a plus

Salary Range: $121,000-$172,500

Overtime eligible:
Exempt

Discretionary bonus eligible:
Yes

Sales bonus eligible:
No

Actual base salary will be determined based on several factors but not limited to individual’s experience, skills, qualifications, and job location. Additionally, employees are eligible for an annual discretionary bonus. In addition to base salary, employees may also be eligible to participate in an incentive program.