Senior Associate - Elasticsearch Engineer

Location and Designation

Hybrid – 3 days per quarter

As part of Technology, you will help shape New York Life ’s digital landscape. Leveraging cutting‑edge technologies such as Generative AI, you’ll increase productivity, streamline processes, and create seamless experiences for clients, agents, and employees. This mid‑level engineer (3–5years experience) in the Security Data Platform team will support the day‑to‑day operation, performance, and reliability of the enterprise Elasticsearch platform that serves as the core of our security data lake.

The engineer will manage a large Elasticsearch
9.x cluster (40+ nodes), build and maintain ingest pipelines that normalize high‑volume security log sources to Elastic Common Schema (ECS), design and tune index templates and data‑stream life cycles across hot/warm/cold/frozen tiers, and ensure the platform meets performance SLAs and compliance‑driven retention requirements. This role works closely with the log routing layer (Cribl) and the SOC engineering team.

• Hands‑on experience operating and troubleshooting multi‑node Elasticsearch clusters (40+ nodes) including shard allocation, recovery tuning, back pressure diagnosis, and node‑level resource management
• Strong understanding of Index Lifecycle Management (ILM) policies across hot/warm/cold/frozen tiers, searchable snapshots, and frozen‑tier index restoration workflows
• Experience building and maintaining ingest pipelines using native Elasticsearch processors (grok, set, rename, convert, script, pipeline chaining) with a preference for processor‑based approaches over Painless where possible
• Working knowledge of Painless scripting for ingest‑time field transformations, conditional logic, and data normalization
• Proficiency with index templates, component templates, and data‑stream architecture — including mapping conflicts, dynamic templates and failure‑store indices
• Familiarity with Elastic Common Schema (ECS) field mapping conventions and how to apply them to security log sources during ingest
• Experience with data‑stream rollovers, reindexing operations, and mapping migration strategies for live production data
• Ability to write and optimize ES|QL and KQL queries for security use cases, and build/maintain Kibana dashboards and data views
• Experience monitoring and tuning search performance including slow‑query log analysis, shard sizing strategies, query profiling, and understanding the impact of mapping choices on query efficiency
• Familiarity with cluster health and performance monitoring via Kibana Stack Monitoring and Devtools for diagnosing allocation and performance issues

• Experience with cross‑cluster search (CCS) and remote cluster configuration in multi‑cluster architectures
• Familiarity with Terraform‑managed Elasticsearch resources (roles, API keys, index templates, data views)
• Exposure to Cribl Stream or similar log routing/transformation platforms feeding into Elasticsearch via HEC or Elasticsearch output
• Understanding of compliance‑driven data retention requirements (e.g., NYDFS, NAIC) and how they map to ILM/tier policies
• Experience with Elastic Security app, detection rules, or security‑focused Kibana content
• Experience with Elastic Cloud cost management including deployment sizing, autoscaling behavior, data‑tier cost optimization, and identifying savings opportunities through shard consolidation, ILM tuning, or field reduction at ingest
• Understanding of capacity planning – forecasting storage and compute needs based on ingest rates, retention requirements, and query workload patterns

Salary Range: $124,000–$177,000

Overtime eligible:
Exempt

Discretionary bonus eligible:
Yes

Sales bonus eligible:
No

The actual base salary will be determined based on several factors but not limited to individual’s experience, skills, qualifications, and job location. Employees may also be eligible to participate in an incentive program.

We provide a full package of benefits for employees – and have unique offerings for a modern workforce, including leave programs, adoption assistance, and student loan repayment programs. Based on feedback from our employees, we continue to refine and add benefits to our offering, so that you can flourish both inside and outside of work.

Job Requisition