Questions tagged [aws-step-functions]
AWS Step Functions is a visual workflow service that helps developers use AWS services to build distributed applications, automate processes, orchestrate microservices, and create data and machine learning (ML) pipelines.
1,599
questions
62
votes
3
answers
50k
views
Airflow versus AWS Step Functions for workflow
I am working on a project that grabs a set of input data from AWS S3, pre-processes and divvies it up, spins up 10K batch containers to process the divvied data in parallel on AWS Batch, post-...
- 1,779
45
votes
2
answers
49k
views
AWS SQS trigger Step Functions
Quick question: Is it possible to trigger the execution of a Step Function after an SQS message was sent?, if so, how would you specify it into the cloudformation yaml file?
Thanks in advance.
- 915
39
votes
2
answers
33k
views
AWS Lambda vs AWS step function
I am designing an application for which input is a large text file (size ranges from 1-30 GB) uploaded to S3 bucket every 15 min. It splits the file into n no of small ones and copy these files to 3 ...
- 635
39
votes
5
answers
23k
views
How to concatenate constant string with jsonpath
I have AWS step machine and one of the step is used to notify failure using SNS service. I want to select some metadata from input json into outgoing message. So i am trying to concatenate constant ...
- 32.4k
39
votes
7
answers
63k
views
AWS Step Function - Wait until an event
I have a use case where I have a AWS Step function that is triggered when a file is uploaded to S3, from there the first step runs an ffprobe to get the duration of the file from an external service ...
- 3,468
38
votes
6
answers
27k
views
Nested Step Function in a Step Function: Unknown Error: "...not authorized to create managed-rule"
I have a Step Function (Parent) created in a SAM/CloudFormation template that, among other things, calls another Step Function (Child). I'm following the instructions on calling Child, from Parent, ...
- 937
38
votes
7
answers
30k
views
AWS step functions and optional parameters
I would like to have a default value for a parameter passed into a step function
e.g.,
"Parameters": {
"foo.$": "$.foo" OR "bar" if "$.foo" not specified
}
is there a way to do this natively ...
- 18.4k
35
votes
3
answers
9k
views
AWS Data Pipeline vs Step Functions
I am working on a problem where we intend to perform multiple transformations on data using EMR (SparkSQL).
After going through the documentation of AWS Data Pipelines and AWS Step Functions, I am ...
- 659
34
votes
5
answers
24k
views
Api gateway get output results from step function?
I followed tutorial on creating and invoking step functions
I'm getting output in my GET request of api as
{
"executionArn": "arn:aws:states:ap-northeast-1:123456789012:execution:HelloWorld:...
- 7,658
27
votes
4
answers
29k
views
How to specify multiple result path values in AWS Step Functions
I have a AWS Step Function State formatted as follows:
"MyState": {
"Type": "Task",
"Resource": "<MyLambdaARN>",
"ResultPath": "$.value1"
"Next": "NextState"
}
I want to add a second ...
- 4,368
25
votes
2
answers
20k
views
Pass multiple inputs into Map State in AWS Step Function
I am trying to use AWS Step Functions to trigger operations many S3 files via Lambda. To do this I am invoking a step function with an input that has a base S3 key of the file and part numbers each ...
- 11.4k
24
votes
5
answers
27k
views
Parallel States Merge the output in Step Function
Is it possible to have following kind of Step Function graph, i.e. from 2 parallel state output, one combined state:
If yes, what would json for this looks like? If not, why?
- 4,924
24
votes
2
answers
31k
views
Step function exceeding the maximum number of characters service limit
My state in a step function flow returns an error of state/task returned a result with a size exceeding the maximum number of characters service limit.. In the step function documentation, the limit ...
- 633
24
votes
3
answers
10k
views
Why would a step function cancels itself when there are no errors
I have a step function (illustrated below) and I was working out the few remaining issues when all of a sudden I hit a rather unexpected result:
All tasks that have completed, completed successfully ...
- 8,863
23
votes
4
answers
56k
views
Pass and use input (parameters) to a lambda task from a step function
I have a simple step function launching a lambda and I am looking for a way to pass parameters (event / context) to each of several consequent tasks. My step function looks like this:
{
"Comment": ...
21
votes
6
answers
35k
views
How to use jsonPath inside array in AWS Step Functions
I am writing an AWS step function, and for one of the steps, I wish to call a lambda that accepts an array as one of the inputs. However, if I try to pass in a JsonPath into the array, I get
The ...
- 3,093
20
votes
4
answers
19k
views
Passthrough input to output in AWS Step Functions
How can I passthrough the input to a Task state in an AWS Step Functions to the output?
After reading the Input and Output Processing page in the AWS docs, I have played with various combinations of ...
- 33.1k
17
votes
3
answers
12k
views
Passing data to step function catch
I'm using step functions on AWS. Consider the state machine made up of lambdas:
"StartAt": "Metadata",
"States": {
"Metadata": {
"Type": "Task",
...
- 3,563
17
votes
4
answers
15k
views
Unit testing AWS: step function
I am beginner in AWS and i have created my first AWS step function given below, Now next step is to unit test this step function. I independently unit tested my lambda function now i got stuck and ...
- 1,378
17
votes
2
answers
14k
views
Can a lambda in an AWS Step Function know its current Execution Id of AWS Step Function?
For a lambda executed within a step function, I want to know the current execution id of AWS State Machine within that. Is there a way to know this within the step function?
- 339
16
votes
4
answers
38k
views
AWS Step Function - Adding dynamic value to Pass state type
I have the following state defined in my state machine.
"loop":{
"Type": "Pass",
"Result":{
"totalCount": "$.newFieldsResponse.body.count",
"currentCount": 0,
"...
- 371
16
votes
4
answers
27k
views
SerializationException:Start of structure or map found where not expected: API Gateway to Step function
I am using the standard blog tutorial on integrating api gateway with step functions from here:
https://docs.aws.amazon.com/step-functions/latest/dg/tutorial-api-gateway.html
My step function expects ...
- 2,333
16
votes
2
answers
9k
views
What happens when I stop an execution in AWS Step Functions?
I was reading the API documentation an there's an action called StopExecution.
https://docs.aws.amazon.com/step-functions/latest/apireference/API_StopExecution.html
I was wondering what it does. Yes,...
- 461
15
votes
4
answers
11k
views
Can a lambda in an AWS Step Function know the name of the step it is in?
For a lambda executed within a step function, I kind of expected that I could get the name of the current step from the lambda context, but it doesn't seem to be that simple.
Is there any way to get ...
- 1,159
15
votes
3
answers
5k
views
What is the AWS Service Principal value for stepfunction?
I'm writing the terraform for creating an IAM role for AWS StepFunctions.
What should be the value for Principal in assume_role_policy
{
"Version": "2012-10-17",
"Statement": [
{
"...
- 1,779
15
votes
3
answers
16k
views
AWS Step Functions - Pass input to another task
How can I pass my input to my output in a task in AWS Step Functions?
I'm aware of this question, and the docs:
If the value of ResultPath is null, that means that the state’s own raw output is ...
- 4,044
15
votes
2
answers
10k
views
How to get Task Token in AWS Lambda Function
I have an AWS Step Function with the handlers implemented in Java.
My step function definition:
definition:
Comment: Steps for issuing a card
StartAt: RecipientFraudChecks
States:
...
- 4,808
15
votes
2
answers
4k
views
How to pass sensitive information securely in AWS Step Functions
Using AWS Step Functions to string together Lambdas is interesting, but is there a way to disable/hide logging on the execution details screen? Private information being handed from one lambda to ...
- 803
15
votes
2
answers
9k
views
how to access error information in Step Function from previous Function in Catch state
I am wondering how to access the cause of my custom exception raised inside my lambda function. I need to access it at the end of my Step Functions workflow, as shown below.
The diagram below is an ...
- 954
14
votes
5
answers
9k
views
How to stop all running Step Functions of a specific state machine?
I accidentally started very many step functions and now wish to terminate all of them.
Any smart ways to do this using the CLI or web console?
- 4,648
14
votes
2
answers
5k
views
AWS SNS vs AWS Step Functions
What's the better option to coordinate tasks between microservices?
For example, if I have a microservice that handles customer information and need to notifies other microservices, is it better to ...
- 7,936
14
votes
2
answers
29k
views
Invoking AWS Step function from Lambda in python
So I am trying to invoke a simple step function I wrote using a Lambda in python. I am using boto3 for this purpose
client = boto3.client('stepfunctions')
response = client.start_execution(
...
- 1,174
13
votes
2
answers
15k
views
Getting AWS State machine execution ARN inside the step function
I need to get the ARN of the current execution for my state machine inside the state machine itself. Is it possible?
So can I create a state machine(say from a lambda function) and then take its ...
- 1,524
13
votes
3
answers
11k
views
AWS Step Functions with dynamic steps
I am exploring AWS step functions, however, I am not able to find the answer to a question that my use case requires. Let us say we have 10 tasks we can do in the system overall. Sometimes we have to ...
- 5,609
13
votes
2
answers
4k
views
Can AWS step function executes more than 25000 times?
I am currently evaluating AWS state machine that can process single document. The state machine would take 5-10 mins to process a single document.
{
"Comment":"Process document",
"StartAt": "...
- 32.4k
13
votes
2
answers
19k
views
How to limit concurrency of a step in step functions
I have a state machine in AWS. I want to limit concurrency of a task (created via lambda) to reduce traffic to one of my downstream API.
I can restrict the lambda concurrency, but the task fails ...
- 489
13
votes
2
answers
891
views
How can I return the result of a mapreduce operation to an AWS API request
I have a program that performs several thousand monte-carlo simulations to predict a result; I can't say what they really predict, so I'm going to use another example from "the indisputable existence ...
- 570
12
votes
2
answers
22k
views
How to invoke an AWS Step Function using API Gateway
How do I invoke an AWS Step Function using an API Gateway POST request, and the request's JSON payload to the Step Function ?
- 908
12
votes
3
answers
10k
views
AWS Step Function: Function .length() returned error in variable field in Choice state
I have a Choice state in AWS Step Function that would compare an array's length from Input and make decision to go next state.
However, the length() function, to get the array's length, returned an ...
- 1,028
12
votes
2
answers
16k
views
Using an amazon step function, how do I write a choice operator that references current time?
In the aws-step function documentation, it seems possible to write a 'choice' state comparing variables from the current state. However, is it possible to write a timestamp comparison referencing the ...
- 2,367
12
votes
2
answers
8k
views
Is there a way to get Step Functions input values into EMR step Args
We are running batch spark jobs using AWS EMR clusters. Those jobs run periodically and we would like to orchestrate those via AWS Step Functions.
As of November 2019 Step Functions has support for ...
- 1,356
12
votes
2
answers
25k
views
Retry logic in AWS step function
I am testing the retry logic of the step function.
Theoretically following step function should have been retried to execute the lambda 3 times if it fails.
{
"StartAt": "Bazinga"...
- 1,987
12
votes
4
answers
33k
views
What is the best way to run python scripts in AWS?
I have three python scripts, 1.py, 2.py, and 3.py, each having 3 runtime arguments to be passed.
All three python programs are independent of each other. All 3 may run in a sequential manner in a ...
- 380
12
votes
3
answers
5k
views
Is there a way of running AWS Step Functions locally when defined by CDK?
AWS Step Functions may be run in a local Docker environment using Step Functions Local Docker. However, the step functions need to be defined using the JSON-based Amazon States Language. This is not ...
- 11.4k
12
votes
1
answer
4k
views
How can I define step function execution name from an EventBridge rule?
I'm using the EventBridge to trigger a step function. My EventBridge rule in the CloudFormation template looks as follows:
JobStepFunctionTrigger:
Type: AWS::Events::Rule
Properties:
...
- 1,946
11
votes
3
answers
21k
views
Loop inside a Step Function
I am trying to call a couple of steps in my step function in a loop but I am unable to get my head around how I need to do this. Here's what I have till now: I need to add another lambda function(...
- 4,318
11
votes
2
answers
8k
views
AWS Step Functions and Fargate task: container runtime error does not cause pipeline to fail
I have a pipeline defined in AWS Step Functions. One step is defined as Fargate Task, which pulls a docker image and run some python code. I've surprisingly found that if the container running in the ...
- 4,371
11
votes
5
answers
8k
views
Propogating error message through Fail state in aws Step Functions
I am using aws Step Functions to manage a workflow. I am using Fail states to handle errors within the workflow. I would like to propagate some of the json from the Step Function workflow so that a ...
- 579
11
votes
1
answer
15k
views
Anyone has experience with triggering step function with S3 event?
I'm learning about step function and specifically, I'm trying to figure out how to trigger state machine execution with S3 event. I was reading this post: https://docs.aws.amazon.com/step-functions/...
- 1,077
11
votes
4
answers
8k
views
How can i call AWS Step Functions by API Gateway? [closed]
I would like to know how to make API Gateway call a Step Function and execute it.
- 361