| Service | Purpose | Key Highlights |
|---|---|---|
| AWS Step Functions | Workflow Orchestration | Serverless workflow engine to chain Lambda, Glue Jobs, EMR steps, ECS tasks, or any API call. Supports branching, retries, error handling, and parallel execution. Equivalent to ADF pipelines. |
| AWS Managed Workflows for Apache Airflow (MWAA) | Data Pipeline Orchestration (Python DAGs) | Fully managed Apache Airflow. Ideal for complex DAG-based scheduling (cross-account, multi-step ETL). You can orchestrate Glue, EMR, and Redshift tasks via Airflow operators. |
| AWS Glue Workflows | Glue-native ETL orchestration | Visual or programmatic orchestration for Glue Jobs, Crawlers, and Triggers. Perfect for pipeline sequencing within AWS Glue ecosystem. |
| AWS CloudWatch | Monitoring & Logging | Collects metrics, logs, and alarms for every AWS service. Central to data observability - you can monitor Lambda executions, Glue job logs, Kinesis throughput, or Redshift query latency. |
| AWS CloudTrail | Governance & Auditing | Tracks every API call or configuration change for compliance and lineage tracking. |
Created
November 4, 2025 12:54
-
-
Save Anshuman-02905/c80694457ba10053e73ca777d6d4dd7d to your computer and use it in GitHub Desktop.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment