Sr Big Data​/OpenShift API Developer

Sr Big Data / Open Shift API Developer

Location:

Charlotte, NC (Hybrid – 3 days office in a week)

Platform Management

• Deploy, configure, maintain, and support containerized Apache Spark applications on Open Shift and Google Cloud.
• Manage Spark workloads running on Kubernetes and Open Shift clusters.
• Configure pods, deployments, services, routes, secrets, config maps, resource quotas, and name spaces.
• Define resource requests and limits for Spark drivers, executors, and supporting services.
• Support cluster capacity planning, scaling, patching, upgrades, and environment management.
• Maintain Helm charts and deployment templates for repeatable application releases.

Data Pipeline Development

• Design and implement large-scale batch and streaming data processing workflows using Apache Spark and Apache Airflow.
• Develop robust, reusable, and production-ready PySpark, Scala, or Java applications.
• Build Airflow DAGs for scheduling, orchestration, dependency management, retries, and alerting.
• Implement data ingestion, transformation, validation, enrichment, and publishing processes.
• Support both real-time and batch data processing use cases.
• Apply data engineering best practices for reliability, scalability, data quality, and maintainability.

API and Microservices Development

• Design, develop, test, and maintain RESTful APIs and microservices supporting data and analytics platforms.
• Define API contracts, request/response schemas, authentication mechanisms, error handling, and service-level requirements.
• Integrate APIs with Spark applications, data platforms, databases, messaging systems, and cloud services.
• Develop automated unit, integration, and API tests.
• Apply microservices architecture principles, including loose coupling, scalability, resiliency, and observability.

Performance Optimization

• Tune Spark jobs for performance, scalability, and efficient resource utilization.
• Optimize partitioning, joins, caching, serialization, shuffle operations, file formats, and executor configuration.
• Use Open Shift and Kubernetes resource-management capabilities to improve application performance.
• Configure horizontal and vertical scaling, autoscaling, and workload scheduling where appropriate.
• Analyze Spark execution plans, application logs, cluster metrics, and resource-consumption patterns.
• Identify and resolve bottlenecks across applications, infrastructure, networking, and storage layers.

Integration

• Integrate Apache Spark with data sources and platforms such as:
• Apache Kafka
• Amazon S3
• Google Cloud Storage
• Azure Blob Storage
• HDFS
• Relational and No
  
  SQL databases
• Data lakes and data warehouses
• Develop reliable ingestion and publishing patterns for batch and streaming data.
• Implement schema management, checkpointing, offset handling, retry logic, and failure recovery.
• Support secure connectivity between applications, cloud platforms, messaging systems, and databases.

CI/CD Implementation

• Build and maintain CI/CD pipelines for deploying Spark applications, APIs, and supporting services to Open Shift.
• Use tools such as Git Hub Actions, Sonar Qube, Harness, Docker, Helm, and Git.
• Automate build, code-quality analysis, security scanning, testing, packaging, and deployment processes.
• Implement environment-specific configurations and promotion strategies across development, QA, staging, and production.
• Manage container images, registries, versioning, release tagging, and rollback procedures.
• Establish deployment standards and support automated release management.

Monitoring and Troubleshooting

• Monitor Open Shift cluster health, Spark job performance, application availability, and resource utilization.
• Use Open Shift monitoring capabilities and tools such as Prometheus, Grafana, and centralized logging platforms.
• Develop dashboards, alerts, health checks, and operational metrics.
• Troubleshoot complex issues involving Spark applications, APIs, containers, Kubernetes resources, networking, storage, and integrations.
• Perform root-cause analysis and implement proactive remediation.
• Participate in incident, problem, and change-management activities.
• Provide production support and participate in on-call rotations when required.

Security and Compliance

• Implement…