Pyspark Architect

Responsibilities

• Participate in customer discussions to gather and analyze system requirements for performance, security and scalability.
• Provide suggestions on solution architecture, contribute to tech framework development, support enterprise architecture definition, and drive design creation
• Identify suitable technologies and tools available in the market, and perform a comparative analysis in line with client's business requirement, thereby identifying optimal architectural solution options
• Participate in developing integration strategies based on validation of proof of concept of proposed architectural solutions
• Facilitate a smooth transition by ensuring knowledge transfer and adherence to quality standards, while contributing to the build and deployment automation strategy
• Collaborate with development teams to resolve complex technical issues using solution deployment and roll out methods, during build, deployment and support phases
• Analyze system performance basis architectural solutions implemented and support in identifying potential risks in the system landscape
• Create artifacts of architectural frameworks and technical solutions, thereby enriching team's knowledge repository

• The ability to analyse, design, and develop scalable and secure architectures.
• A knack for assessing multiple technologies, tools and frameworks to recommend optimal solutions.
• Understanding of integration methodologies, proof-of-concept validation and deployment strategies.
• The ability to document architectural frameworks and solutions.
• Proactive communication with internal teams and clients, ensuring alignment with business objectives.
• A knack to keep with up technical advancements to ensure relevant and effective recommendations.
• The ability to maintain standards and ensure smooth transition through knowledge transfer

• Strong experience in technology consulting, enterprise and solutions architecture and architectural frameworks
• Strong experience in Cloudera Hadoop ecosystem, i.e. Hadoop, Hive, Scala, SPARK, Sqoop, Flume, Kafka, Python, No
  
  SQL database technologies
• Good Understanding and knowledge of Hadoop architecture and various components such as high availability architecture, experience in workload management, scalability and distributed platform architecture
• Experience with design and implementation of ETL/ELT framework for complex warehouses/marts. Knowledge of large data sets and experience with performance tuning and troubleshooting
• Background in all aspects of software engineering with strong skills in parallel data processing, data flows, REST APIs, JSON, XML, and micro service architecture

• Experience in working in any hyperscalers, AWS, Azure, GCP, Databricks and Snowflake
• Experience in data virtualization/data federation for creating data product (e.g. in Starburst) in heterogenous environment (Hadoop, RDBMS, No
  
  SQL etc.)
• Hands-on development mentality, with a willingness to troubleshoot and solve complex problems