Senior Databricks Administrator IRC

Description

Our client is concerned with excessive charging attributed to the Databricks Lakehouse environment. Global Logic carried out a review of the Databricks environment and made a number of recommendations. Client would now like to implement some of the recommendations. These will be delivered in two work streams:

• Databricks Environment Changes - changes which can be made to Databricks environment setup including data deletion and control of instances
• Pipeline Rebuild POC - to understand the cost impact of some specific changes (e.g. use of SCD Type 2 vs. Type
  1), rebuild one of the data pipelines to quantify the effect of these changes and then extrapolate across other pipelines.

#LI-TZ1

• 5+ years in development roles, 3+ years specialized in Data Platform
• Deep expertise in the Databricks Admin console and APIs
• Strong understanding of cloud-native IaC (Infrastructure as Code) like Terraform or ARM/Cloud Formation for automating instance provisioning
• Skilled in scripting (Python, Bash) for identifying and deleting stale assets.
• Possesses a "Fin Ops" mindset, with experience in managing cloud costs via reserved instances, spot capacity, and instance-type selection.

• Owns the technical implementation of Databricks environment configuration changes.
• Reviews current environment setup, including cluster configuration, instance types, and data storage parameters, to identify cost optimisation opportunities.
• Designs and implements environment-level policies for data retention, cluster management, and replication control in line with AA governance standards.
• Ensures all configuration changes are executed safely within non-production environments, with no impact to production workloads.
• Provides expert input on automation, scaling, and monitoring strategies to reduce ongoing compute and storage costs.
• Supports knowledge transfer to The AA's Platform and Dev Ops teams regarding implemented configuration changes and policies.

• Culture of caring. At Global Logic, we prioritize a culture of caring. Across every region and department, at every level, we consistently put people first. From day one, you'll experience an inclusive culture of acceptance and belonging, where you'll have the chance to build meaningful connections with collaborative teammates, supportive managers, and compassionate leaders.
• Learning and development. We are committed to your continuous learning and development. You'll learn and grow daily in an environment with many opportunities to try new things, sharpen your skills, and advance your career h our Career Navigator tool as just one example, Global Logic offers a rich array of programs, training curricula, and hands‑on opportunities to grow personally and professionally.
• Interesting & meaningful work. Global Logic is known for engineering impact for and with clients around the world. As part of our team, you'll have the chance to work on projects that matter. Each is a unique opportunity to engage your curiosity and creative problem‑solving skills as you help clients reimagine what's possible and bring new solutions to market. In the process, you'll have the privilege of working on some of the most cutting‑edge and impactful solutions shaping the world today.
• Balance and flexibility. We believe in the importance of balance and flexibility. With many functional career areas, roles, and work arrangements, you can explore ways of achieving the perfect balance between your work and life. Your life extends beyond the office, and we always do our best to help you integrate and balance the best of work and life, having fun along the way!
• High‑trust organization. We are a high‑trust organization where integrity is key. By joining Global Logic, you're placing your trust in a safe, reliable, and ethical global company. Integrity and trust are a cornerstone of our value proposition to our employees and clients. You will find truthfulness, candor, and integrity in everything we do.
• About Global Logic Global Logic, a Hitachi Group Company, is a trusted digital engineering partner to the world's largest and most forward‑thinking companies. Since 2000, we've been at the forefront of the digital revolution - helping create some of the most innovative and widely used digital products and experiences. Today we continue to collaborate with clients in transforming businesses and redefining industries through intelligent products, platforms, and services.