Data Engineer

Reporting to the Manager, BI Development, the Data Engineer will be responsible for integrating enterprise wide data from Cofense’s Cyber Security based Applications, Micro-services and other disparate data sources.

• Work with Architects & Cloud Systems Engineers in designing Data Platform and Architecture
• Substantial experience with SQL and no-SQL OLTP databases and OLAP data warehousing technologies, especially AWS Aurora
• Experience data modeling and building data pipelines for multi-product and/or multi-department organizations
• Develop Data Pipelines on Cloud Technologies like Azure/AWS with well-defined tool frameworks
• Able to develop ETL code to stream data from disparate (structured and semi-structured) SaaS product data stores to Data Lake/Data Warehouse using Python, Azure/AWS Data Lake services
• Ability to write complex SQL scripts and automate them using Python
• Develop test cases and unit tests for key implementations of Data Platform by adhering to software engineering best practices and standards
• Secure data end-to-end by complying data privacy rules while developing processes to move data across Applications/Data Sources and Data Lake/Data Warehouse, as well as while delivering data through SQL clients and BI tools.
• Experience building adhoc reporting APIs and service layer on top of underlying OLTP and OLAP databases
• Help integrate Data Platform with BI tools like Power BI, Tableau, Splunk etc.
• Ability to develop and interpret Entity Relationship Diagrams (ERD) across data sets in relational database systems as well as non-relational Data stores
• Able to do Data Mining and Identify trends, patterns, anomalies in complex data sets across multiple data sources/systems and present results without ambiguity
• Develop data transformations to generate Facts, Summaries, Key metrics by applying business rulesets and aggregations using Python, SQL and other transformation tools
• Able to review current processes related to data ingestion, transformation and statistical analysis and re-engineer them
• Collaborate with business users across Cofense’s departments in defining requirements, prioritize project work and deliver them timely
• Other duties as assigned

Knowledge,

Skills and Abilities

Required

• Expertise in SQL skills for data transformations, statistical analysis, and troubleshooting across more than one Database Platforms (MySQL, PostgreSQL, Redshift, Azure SQL Warehouse etc.)
• Expert in writing complex SQL scripts and automate them using Python
• Knowledge of Data management on No
  
  SQL DBs like DynamoDB, Mongo, and know-how of Big Data tools Hadoop, Spark, Kafka/Kinesis/SQS/Azure Queues or other messaging tools is huge plus
• Analytical skills, with good at finding data trends/outliers, anomalies, and articulate complex information or data points with Business Users, Management, and individuals
• Enthusiasm to work with lot of data across disparate data sources and Databases
• Has strong sense of engineering craftsmanship, takes pride in the code they write.
• Has a sense of intellectual curiosity and a burning desire to learn is self-driven, actively looks for ways to contribute, and knows how to get things done
• Is deliriously customer-focused both internal and external customers
• Sees big picture impact and relationships among and across work units
• Identifies complex technical problems and tries to resolve with minimal help

Education and/or

Experience:

• Bachelor’s degree in Computer Science or Math, Data Analytics, Data Sciences, BI or demonstrated industry experience preferred
• Over 5 years of proven experience in data architecture, data modelling, and lifecycle management
• Hands-on experience with relational and cloud databases, including Azure SQL Database, Microsoft SQL Server, Amazon Aurora, and Amazon Redshift
• Strong background in developing Python and data technologies
• Practical experience designing and developing ETL data pipelines and applications using SQL and Python
• Strong expertise in writing complex SQL queries for data transformation and automating processes using Python Experience building and consuming RESTful APIs using Python libraries
• Proficient in integrating data platforms with BI tools such as Power BI and Splunk,…