Senior Data Engineer

Job ID26WD
101542

Job

Requisition  Overview Autodesk is looking for a skilled and innovative Senior Data Engineer to join our Compliance team, which builds data-driven systems to detect and address non-compliant usage of Autodesk software. This is a key strategic initiative that leverages large-scale data processing and analytics to drive business  this role, you will design, build, and optimize scalable data pipelines and distributed processing systems that handle billions of records.

You will focus on improving platform reliability, performance, scalability, and operational efficiency while enabling business-critical analytics.

We are looking for an engineer who combines strong expertise in big data technologies with hands-on experience using AI-powered development tools to accelerate engineering workflows, improve code quality, and deliver solutions faster.

The ideal candidate is technically strong, self-driven, comfortable owning complex engineering initiatives, and passionate about leveraging modern technologies and AI-assisted development practices to solve challenging data engineering problems.

Our technology stack includes:
Python, SQL, Apache Spark, PySpark, Spark SQL, Apache Airflow, Hive, Presto/Trino, Snowflake, AWS (S3, EMR, Glue), Git, Jenkins, Power BI, Looker, and AI-assisted development tools such as Git Hub Copilot, Cursor, ChatGPT, Claude, or equivalent solutions.

Responsibilities Leverage AI-powered development tools to accelerate coding, debugging, refactoring, test generation, documentation, and troubleshooting of complex data engineering solutions

Integrate AI-assisted development practices into engineering workflows to improve productivity, code quality, and delivery speed

Evaluate and validate AI-generated solutions for correctness, performance, maintainability, and security while following responsible AI practices

Identify opportunities to automate repetitive engineering tasks and improve developer efficiency through AI-powered solutions

Design, develop, and optimize large-scale batch and near-real-time data processing pipelines using Apache Spark, PySpark, and Spark SQLBuild scalable and efficient ETL/ELT pipelines capable of processing billions of records across distributed data platforms

Optimize distributed processing workloads through partitioning, efficient joins, query tuning, resource management, and performance analysis

Develop and maintain data processing solutions using technologies such as Hive, Snowflake, AWS S3, EMR, and Glue Build and maintain production-grade data workflows using Apache Airflow, including DAG development, scheduling, dependency management, retries, and failure recovery

Implement data quality checks, pipeline monitoring, observability, and alerting to ensure reliable data processing

Design reusable, modular, and fault-tolerant pipelines supporting incremental processing, backfills, and operational scalability

Troubleshoot production issues and continuously improve pipeline reliability, performance, and cost efficiency

Develop and maintain scalable, cloud-native data infrastructure on AWSAutomate infrastructure provisioning, deployments, and operational processes using Infrastructure as Code and CI/CD practices

Implement automated testing, version control, code reviews, and software engineering best practices

Own the end-to-end delivery of data engineering solutions, from technical design and implementation to deployment and production support

Partner with product managers, data scientists, analysts, and business stakeholders to translate business requirements into scalable technical solutions

Design efficient data models and curated datasets to support compliance detection, product usage analytics, and business reporting

Define and promote data engineering standards, reusable frameworks, and architectural best practices

Communicate technical designs, trade-offs, and engineering outcomes effectively to stakeholders

Minimum Qualifications 4–7 years of relevant experience in data engineering, big data systems, or software engineering, building and operating production-grade data platforms3+ years of hands-on experience with Apache Spark, including PySpark, Spark SQL, and Data Frame APIs3+ years of strong programming experience in Python and SQL, with demonstrated ability to write clean, modular, maintainable, and testable code Strong understanding of distributed computing fundamentals, including parallel processing, partitioning, shuffling, data skew, fault tolerance, and Spark…