Principal Software Engineer - AWS Big Data Analytics

Technical Skills

• Cloud Knowledge: A deep understanding of cloud computing concepts, especially related to Amazon Web Services (AWS).
• Infrastructure Design:
  Ability to design and implement cloud solutions, considering scalability, security, and performance.
• Experience:
  
  Practical firsthand experience with cloud platforms, preferably AWS.
• Best Practices:
  Familiarity with best practices for deploying and managing cloud resources.
• Cloud Governance:
  Knowledge of governance principles within the AWS environment.
• Amazon EMR:
  In-depth knowledge of EMR, including cluster setup, configuration, and management.
• EC2 Instances:
  Understanding of EC2 instance types and their suitability for EMR clusters.
• S3:
  Proficiency in using Amazon S3 for data storage and EMR input/output.
• IAM:
  Ability to manage permissions and access control using Identity and Access Management.
• Lambda:
  Knowledge of serverless computing for automating tasks.
• Data Pipelines:
  Experience with creating data pipelines using AWS services.
• Lake Formation:
  Experience with Glue Catalog or Lake Formations
• Data Storage Fundamentals: A solid understanding of various data storage options, including Amazon S3, Amazon RDS, Amazon Redshift, and DynamoDB.
• Database Design and Optimization:
  Proficiency in designing efficient database schemas, indexing strategies, and query optimization.
• Big Data Technologies:
  Familiarity with big data tools like Amazon EMR, Athena, Lake Formation, Data Sync, and Glue for processing large datasets.
• Cluster Sizing:
  Ability to determine the right cluster size based on workload requirements.
• Scaling:
  Know-how of scaling EMR clusters dynamically based on demand.
• Monitoring and Troubleshooting:
  Proficiency in monitoring cluster health, identifying bottlenecks, and resolving issues.
• Cost Optimization:
  Strategies to tag resources with an eye on optimizing costs and observability.

• Encryption:
  Understanding of data encryption at rest and in transit.
• Security Groups and VPC:
  Knowledge of network security and virtual private clouds.
• Compliance Controls:
  Ensuring compliance with industry standards and regulations.

• Langauge Proficiency:
  Bash,Python, R, Spark, SQL in scripting languages for automating tasks.
• MLOPs:
  Ability to collaborate with the business to optimize Data Engineering process, and data lifeycle management
• Cloud Formation/Terraform:
  Ability to create and manage infrastructure as code.
• Infrastructure as Code (IaC):
  Ability to assist Dev Ops engineers to develop proper Terraform templates used to provision AWS analytics infrastructure.

• Snapshotting:
  Familiarity with taking EMR cluster snapshots for backup and recovery.
• High Availability:
  Implementing strategies for fault tolerance and disaster recovery.

• Communication:
  Effective communication with stakeholders, developers, and data engineers.
• Problem-Solving:
  Analytical thinking to address complex issues.
• Adaptability:
  Keeping up with evolving technologies and best practices.
• Decisiveness:
  Make informed decisions, especially when dealing with complex architectural choices.
• Business Acumen:
  Understand business requirements and align technical solutions accordingly.
• Continuous Learning: A zeal for staying updated with evolving cloud technologies.

• Experience:
  
  Senior AWS Cloud Architects must have 3 to 5 years of firsthand experience in designing and building cloud applications.
• Certifications:
  
  An AWS Professional Solutions Architect certification is desired for senior roles

• Aligns business strategy with software solutions.
• Proposes, designs & codes software solutions to address complex business needs. Oversees technical and procedural documentation required.
• Leads complex problem solving.
• Provides technical guidance and support to colleagues and solution development.
• Displays an innovative approach to apply modern principles, methodologies and tools to advance business initiatives and capabilities.

Application Design, Architecture – Knowledge of application design activities, tools and techniques; ability to utilize these to convert business requirements and logical models…