Skip to content

sfn-glue-sync-cdk: Update aws-cdk-lib and replace deprecated Python shell 3.6 with 3.9 #3214

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kakakakakku wants to merge 5 commits into
base: main
Choose a base branch
from
Open

sfn-glue-sync-cdk: Update aws-cdk-lib and replace deprecated Python shell 3.6 with 3.9 #3214

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
660aa9b
sfn-glue-sync-cdk: Update aws-cdk-lib
kakakakakku Jul 6, 2026
0db9974
sfn-glue-sync-cdk: Replace deprecated definition with definitionBody …
kakakakakku Jul 6, 2026
928c76c
sfn-glue-sync-cdk: Replace deprecated Python shell 3.6 with 3.9
kakakakakku Jul 6, 2026
d771875
sfn-glue-sync-cdk: Add note about Python shell version in README
kakakakakku Jul 6, 2026
3f82898
sfn-glue-sync-cdk: Update README.md
kakakakakku Jul 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 44 additions & 35 deletions sfn-glue-sync-cdk/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
# AWS Step Functions workflow to integrate with AWS Glue Job using CDK.

This CDK application deploys a Step Functions workflow, that takes in a payload and trigger a AWS Glue job synchronously. In this pattern, the state machine does wait for Glue job to finish. The application contains the minimum IAM resources required to run the workflow and Glue job.


This CDK application deploys a Step Functions workflow, that takes in a payload and trigger a AWS Glue job synchronously. In this pattern, the state machine does wait for Glue job to finish. The application contains the minimum IAM resources required to run the workflow and Glue job.

Learn more about this pattern at Serverless Land Patterns: https://serverlessland.com/patterns/sfn-glue-sync-cdk

Expand All @@ -13,7 +11,7 @@ Important: This application uses various AWS services and there are costs associ
* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
* [AWS Cloud Development Kit](https://docs.aws.amazon.com/cdk/latest/guide/getting_started.html) (AWS CDK >= 1.124.0) Installed
* [AWS Cloud Development Kit](https://docs.aws.amazon.com/cdk/v2/guide/getting-started.html) installed

## Deployment Instructions

Expand Down Expand Up @@ -52,14 +50,16 @@ Important: This application uses various AWS services and there are costs associ
The CDK stack deploys the resources and the IAM permissions required to run the application.
This stack will deploy S3 bucket to store glue job scripts, python-shell glue job and state function which executes 'glue:startJobRun.sync' step.


Step Functions supports AWS Glue through the service integration pattern.
You can call the `StartJobRun` API from a Task state with
[Run a Job (.sync)](https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html#connect-sync) integration pattern.

Task `GlueStartJobRun` defined in the `app.py` triggers a glue job and wait for job completion before transitioning to next step.
Python-shell glue job is defined with `glue.CfnJob` in `app.py` which triggers a python script `hello.py`


> [!NOTE]
> This pattern uses a Python shell job (Python 3.9), which is the latest Python version available for Python shell jobs. If you need a newer Python version, consider other options described in [Migrate from AWS Glue Python shell jobs](https://docs.aws.amazon.com/glue/latest/dg/pyshell-migration.html).

## Steps for Testing

* Start the Step Function execution with the sample event payload.
Expand All @@ -79,30 +79,33 @@ aws stepfunctions start-execution --state-machine-arn "{StateMachineArn}" --inpu

```json
{
"executionArn": "arn:aws:states:us-east-1:<*AccountNumber*>:execution:GlueJobStateMachineE759615B-t1nYjCro8h5F:36bd81e5-787a-4e34-ac6a-1f21064802ff",
"startDate": "2021-11-17T22:37:02.301000-06:00"
"executionArn": "arn:aws:states:us-east-1:000000000000:execution:GlueJobStateMachineE759615B-at44lw4ZXLFV:132bb420-2b55-4fa4-9085-63a0e226e2d7",
"startDate": "2026-07-06T23:55:20.379000+09:00"
}
```

Note the `executionArn` from the above output and run the following CLI command to get the status of the execution.

```bash
aws stepfunctions describe-execution --execution-arn "{executionArn}"
aws stepfunctions describe-execution --execution-arn "{executionArn}"
```

### Get execution status output:

```json
{
"executionArn": "arn:aws:states:us-east-1:<*AccountNumber*>:execution:GlueJobStateMachineE759615B-t1nYjCro8h5F:36bd81e5-787a-4e34-ac6a-1f21064802ff",
"stateMachineArn": "arn:aws:states:us-east-1:<*AccountNumber*>:stateMachine:GlueJobStateMachineE759615B-t1nYjCro8h5F",
"name": "36bd81e5-787a-4e34-ac6a-1f21064802ff",
"executionArn": "arn:aws:states:us-east-1:000000000000:execution:GlueJobStateMachineE759615B-at44lw4ZXLFV:132bb420-2b55-4fa4-9085-63a0e226e2d7",
"stateMachineArn": "arn:aws:states:us-east-1:000000000000:stateMachine:GlueJobStateMachineE759615B-at44lw4ZXLFV",
"name": "132bb420-2b55-4fa4-9085-63a0e226e2d7",
"status": "RUNNING",
"startDate": "2021-11-17T22:37:02.301000-06:00",
"startDate": "2026-07-06T23:55:20.379000+09:00",
"input": "{\"message\": \"Hello from sfn glue step\"}",
"inputDetails": {
"included": true
}
},
"redriveCount": 0,
"redriveStatus": "NOT_REDRIVABLE",
"redriveStatusReason": "Execution is RUNNING and cannot be redriven"
}
```

Expand All @@ -118,24 +121,28 @@ aws glue get-job-runs --job-name "cdk-test-glue-python-job"
{
"JobRuns": [
{
"Id": "jr_c14390561406a75616f270734b658e17f7175895f3fc3c289279b708840e5584",
"Id": "jr_3d888acaf38939653452dcfe972966c87b6b325c6d44d91f629406d03c79c3f4",
"Attempt": 0,
"JobName": "cdk-test-glue-python-job",
"StartedOn": "2021-11-17T22:37:02.488000-06:00",
"LastModifiedOn": "2021-11-17T22:37:06.008000-06:00",
"JobRunState": "RUNNING",
"JobMode": "SCRIPT",
"JobRunQueuingEnabled": false,
"StartedOn": "2026-07-06T23:55:20.582000+09:00",
"LastModifiedOn": "2026-07-06T23:56:38.460000+09:00",
"CompletedOn": "2026-07-06T23:56:38.460000+09:00",
"JobRunState": "SUCCEEDED",
"Arguments": {
"--message": "Hello from sfn glue step"
},
"PredecessorRuns": [],
"AllocatedCapacity": 0,
"ExecutionTime": 45,
"ExecutionTime": 71,
"Timeout": 6,
"MaxCapacity": 0.0625,
"LogGroupName": "/aws-glue/python-jobs",
"NotificationProperty": {
"NotifyDelayAfter": 6
}
},
"GlueVersion": "5.1"
}
]
}
Expand All @@ -151,44 +158,46 @@ aws logs tail "/aws-glue/python-jobs/output"
### Log output:

```log
2021-11-18T04:40:18.556000+00:00 jr_c14390561406a75616f270734b658e17f7175895f3fc3c289279b708840e5584 python-shell glue job message - Hello from sfn glue step

2026-07-06T14:56:28.170000+00:00 jr_3d888acaf38939653452dcfe972966c87b6b325c6d44d91f629406d03c79c3f4 python-shell glue job message - Hello from sfn glue step
```

Re-run the following CLI command to get the latest status of the staefunction execution and confirm the `status` is `SUCCEEDED`.
Re-run the following CLI command to get the latest status of the staefunction execution and confirm the `status` is `SUCCEEDED`.

:information_source: It might take some time for the statefunction execution to complete after glue job is completed.

```bash
aws stepfunctions describe-execution --execution-arn "{executionArn}"
aws stepfunctions describe-execution --execution-arn "{executionArn}"
```

### Get execution status output:

```bash
{
"executionArn": "arn:aws:states:us-east-1:<*AccountNumber*>:execution:GlueJobStateMachineE759615B-t1nYjCro8h5F:36bd81e5-787a-4e34-ac6a-1f21064802ff",
"stateMachineArn": "arn:aws:states:us-east-1:<*AccountNumber*>:stateMachine:GlueJobStateMachineE759615B-t1nYjCro8h5F",
"name": "36bd81e5-787a-4e34-ac6a-1f21064802ff",
"executionArn": "arn:aws:states:us-east-1:000000000000:execution:GlueJobStateMachineE759615B-at44lw4ZXLFV:132bb420-2b55-4fa4-9085-63a0e226e2d7",
"stateMachineArn": "arn:aws:states:us-east-1:000000000000:stateMachine:GlueJobStateMachineE759615B-at44lw4ZXLFV",
"name": "132bb420-2b55-4fa4-9085-63a0e226e2d7",
"status": "SUCCEEDED",
"startDate": "2021-11-17T22:37:02.301000-06:00",
"stopDate": "2021-11-17T22:41:08.340000-06:00",
"startDate": "2026-07-06T23:55:20.379000+09:00",
"stopDate": "2026-07-06T23:57:24.983000+09:00",
"input": "{\"message\": \"Hello from sfn glue step\"}",
"inputDetails": {
"included": true
},
"output": "{\"AllocatedCapacity\":0,\"Arguments\":{\"--message\":\"Hello from sfn glue step\"},\"Attempt\":0,\"CompletedOn\":1637210428964,\"ExecutionTime\":197,\"Id\":\"jr_c14390561406a75616f270734b658e17f7175895f3fc3c289279b708840e5584\",\"JobName\":\"cdk-test-glue-python-job\",\"JobRunState\":\"SUCCEEDED\",\"LastModifiedOn\":1637210428964,\"LogGroupName\":\"/aws-glue/python-jobs\",\"MaxCapacity\":0.0625,\"NotificationProperty\":{\"NotifyDelayAfter\":6},\"PredecessorRuns\":[],\"StartedOn\":1637210222488,\"Timeout\":6}",
"output": "{\"AllocatedCapacity\":0,\"Arguments\":{\"--message\":\"Hello from sfn glue step\"},\"Attempt\":0,\"CompletedOn\":1783349798460,\"ExecutionTime\":71,\"GlueVersion\":\"5.1\",\"Id\":\"jr_3d888acaf38939653452dcfe972966c87b6b325c6d44d91f629406d03c79c3f4\",\"JobMode\":\"SCRIPT\",\"JobName\":\"cdk-test-glue-python-job\",\"JobRunState\":\"SUCCEEDED\",\"LastModifiedOn\":1783349798460,\"LogGroupName\":\"/aws-glue/python-jobs\",\"MaxCapacity\":0.0625,\"NotificationProperty\":{\"NotifyDelayAfter\":6},\"PredecessorRuns\":[],\"StartedOn\":1783349720582,\"Timeout\":6}",
"outputDetails": {
"included": true
}
},
"redriveCount": 0,
"redriveStatus": "NOT_REDRIVABLE",
"redriveStatusReason": "Execution is SUCCEEDED and cannot be redriven"
}
```

## Cleanup
1. Run the given command to delete the resources that were created. It might take some time for the CloudFormation stack to get deleted.

1. Run the given command to delete the resources that were created. It might take some time for the CloudFormation stack to get deleted.

:warning: This will delete S3 bucket defind as part of this CDK stack(S3 bucket name is provided in the CDK stack outputs)!
:warning: This will delete S3 bucket defined as part of this CDK stack(S3 bucket name is provided in the CDK stack outputs)!

```bash
cdk destroy
Expand All @@ -210,4 +219,4 @@ aws stepfunctions describe-execution --execution-arn "{executionArn}"
----
Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.

SPDX-License-Identifier: MIT-0
SPDX-License-Identifier: MIT-0
15 changes: 7 additions & 8 deletions sfn-glue-sync-cdk/app.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,23 +20,22 @@ class SfnGlueCdkStack(Stack):
def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)

# Glue job execution IAM Role
# Glue job execution IAM Role
glue_job_role = iam.Role(
self,
'Glue-Job-Role',
assumed_by=iam.ServicePrincipal('glue.amazonaws.com'),
managed_policies = [iam.ManagedPolicy.from_aws_managed_policy_name('service-role/AWSGlueServiceRole')]
)



S3_BUCKET_NAME = "MyCdkGlueJobBucket"

# S3 Bucket to host glue scripts
bucket = s3.Bucket(self, S3_BUCKET_NAME, versioned=True,removal_policy=RemovalPolicy.DESTROY,
auto_delete_objects=True, block_public_access=s3.BlockPublicAccess.BLOCK_ALL)

# asset to sync local scripts folder with S3 bucket
asset = s3deploy.Source.asset("./resources/glue-scripts")
asset = s3deploy.Source.asset("./resources/glue-scripts")

# Sync local scripts with S3 bucket
s3deploy.BucketDeployment(self, "DeployGlueJobScripts",
Expand All @@ -55,7 +54,7 @@ def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
role=glue_job_role.role_arn,
command=glue.CfnJob.JobCommandProperty(
name='pythonshell',
python_version='3',
python_version='3.9',

@kakakakakku kakakakakku Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: python_version='3' means Python 3.6, which is already deprecated. I updated it to '3.9', the latest version available for Python shell jobs. No newer Python version has been released for Python shell jobs since then.

See https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html

The same change was made and approved in PR below.

script_location=scriptLocation
))

Expand All @@ -66,15 +65,15 @@ def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
arguments=sfn.TaskInput.from_object({
"--message": sfn.JsonPath.string_at("$.message")
}),
timeout=Duration.minutes(6),
task_timeout=sfn.Timeout.duration(Duration.minutes(6)),
notify_delay_after= Duration.minutes(6)
)

# State Function defination
# State Function definition
definition = glue_task
state_machine = sfn.StateMachine(
self, "GlueJobStateMachine",
definition=definition,
definition_body=sfn.DefinitionBody.from_chainable(definition),

@kakakakakku kakakakakku Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

[WARNING] aws-cdk-lib.aws_stepfunctions.TaskStateBaseOptions#timeout is deprecated.
  use `taskTimeout`
  This API will be removed in the next major release.
[WARNING] aws-cdk-lib.aws_stepfunctions.StateMachineProps#definition is deprecated.
  use definitionBody: DefinitionBody.fromChainable()
  This API will be removed in the next major release.

timeout=Duration.minutes(10)
)

Expand Down
2 changes: 1 addition & 1 deletion sfn-glue-sync-cdk/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
aws-cdk-lib==2.13.0
aws-cdk-lib==2.260.0
constructs>=10.0.0,<11.0.0