Airflow Expert

Python Scripting Expertise

Expert-level proficiency in Python (3.x) for data engineering, with strong experience in Scripting, pandas, requests, SQL Alchemy, custom libraries, and building reusable, production-grade code.

• Deep understanding of core concepts, including DAGs, tasks, operators, sensors, executors, hooks, and connections.
• Hands-on experience with Airflow architecture, scheduling, task dependencies, and extensibility through custom operators and plugins.
• Advanced configuration and debugging skills, including airflow.cfg and environment variables, logging setup (local and remote to GCS), and troubleshooting using dag.test(), CLI commands, log analysis, and task isolation.
• Proven ability to handle production issues such as scheduler delays, worker failures, resource constraints, and high-availability best practices.

Proficiency with Git and Git Hub workflows, including branching strategies (Git Flow/Git Hub Flow), pull requests, code reviews, issue tracking, Git Hub Actions for CI/CD, and collaborative development.

Solid hands-on experience with GCP services, particularly Cloud Composer (managed Airflow), Big Query (advanced SQL and Scripting), Cloud Storage, Dataflow, Pub/Sub, IAM, VPC, and Cloud Monitoring/Logging for pipeline observability.