Senior Big Data Consultant

Job Title:

Senior Big Data SME (Subject Matter Expert)

Location:

Remote

Experience:

9 to 12 years

Education:

Bachelors in Science (IT/Computer Science/Engineer)

Employment Type:

Full-Time

About Us

:

We are a dynamic team focused on building scalable, high-performance data platforms across the cloud ecosystem (Azure, AWS, GCP). As part of an exciting growth phase, we are looking for a Senior Big Data SME to join our team. This role will allow you to leverage your expertise in cloud technologies, big data, and data engineering to drive innovation and excellence in data architecture and solutions

.
Job Descriptio

n:

We are seeking a Senior Big Data Subject Matter Expert (SME) to help us architect, build, and optimize large-scale data platforms using leading cloud technologies and Big Data tools. As an SME, you will bring deep technical expertise and strategic vision to our cloud data initiatives, guiding the team on best practices, ensuring the successful delivery of data projects, and improving operational efficiency.

You’ll work closely with engineers, architects, and leadership to ensure the implementation of cutting-edge solution

s.
Key Responsibiliti

- es:
Architect and Design Scalable Data Platforms:
Leverage your expertise in Azure, AWS, and GCP to design and build scalable, high-performance data platforms for analytics and real time process
- ing.

Cloud Technologies Expertise:
Drive the adoption of cloud-native tools and services like Azure Synapse, AWS Glue, Redshift, and Google Big Query, and ensure alignment with industry best practi
- ces.

Big Data & Streaming Solutions:
Provide hands-on support and guidance on big data technologies including Apache Spark (PySpark, Scala, Java), Hadoop (HDFS, YARN, Hive, Impala), and streaming frameworks like Kafka, Kinesis, and Fl
- ink.

Data Pipelines & ETL Frameworks:
Lead efforts in creating and optimizing ETL pipelines using tools like DBT, Apache Airflow, and cloud-native data orchestration tools (e.g., AWS Glue, Azure Data Facto
- ry).Containerization & Orchestration:
Lead efforts in containerizing applications using Docker and orchestrating them with Kubernetes (AKS, EKS, GKE) for scalable and reliable deploym
- ent.

Data Security & Governance:
Ensure data security, governance, and compliance using IAM, RBAC, and data lineage tracking tools (Azure Purview, AWS Glue Data Catal
- og).CI/CD & Dev Ops:
Contribute to the development of CI/CD pipelines with Terraform, Jenkins, Azure Dev Ops, and Git Hub Actions to automate the deployment of cloud infrastructure and data workfl
- ows.

Cost Optimization:
Implement strategies for optimizing cloud resource utilization and reducing operational costs, particularly in storage (AWS S3, Azure Blob, GCS) and compute resour
- ces.

Team Collaboration & Mentorship:
Provide mentorship and guidance to junior engineers, and lead initiatives to build and maintain high-performing teams through knowledge-sharing sessions and best practice adopt

ion.
Requirem

- ents:

Experience:

9+ years in Data Engineering or related roles, with a focus on cloud technologies and big data solut

ions.
Technical Expe

- rtise:

Deep knowledge of cloud platforms such as Azure, AWS, and GCP. Hands-on experience with Big Data technologies like Apache Spark, Hadoop, Scala, Mapreduce, Kafka, and
- Flink.

Solid understanding of ETL frameworks (DBT, Apache Airflow, AWS Glue, etc.). Expertise in containerization with Docker and Kubernetes (AKS, EKS, GKE). Proven experience in Data Warehousing & Modeling (Snowflake, Redshift, Big Query, Syn
- apse).Strong background in Data Security and Governance (IAM, RBAC, encryption, data lineage).

Experience with CI/CD pipelines using Terraform, Git Hub Actions, and Azure Dev Ops. Programming & Scripting S
- kills:

Proficiency in Python, Bash, or Power Shell for automation tasks. Cloud Architecture:
Experience designing hybrid/multi-cloud architectures to ensure high availability and fault tolerance across Azure, AWS, an
- d GCP.Leadership & Mentorship:
Proven ability to lead teams, mentor junior engineers, and collaborate effectively with cross-functional

teams.
Preferred

- Skills:

Familiarity with Machine Learning pipelines and predictive ana
- lytics.

Experience with Serverless Computing (AWS Lambda, Azure Functions, Google Cloud Func

tions).