45

Quick question: Is it possible to trigger the execution of a Step Function after an SQS message was sent?, if so, how would you specify it into the cloudformation yaml file?

Thanks in advance.

Improve this question
0

2 Answers 2

Reset to default
67

The first think to consider is this: do you really need to use SQS to start a Step Functions state machine? Can you use API gateway instead? Or could you write your messages to a S3 bucket and use the CloudWatch events to start a state machine?

If you must use SQS, then you will need to have a lambda function to act as a proxy. You will need to set up the queue as a lambda trigger, and you will need to write a lambda that can parse the SQS message and make the appropriate call to the Step Functions StartExecution API.

I’m on mobile, so I can’t type up the yaml right now, but if you need it, I can try to update with it later. For now, here is detailed walkthrough of how to invoke a Step Functions state machine from Lambda (including example yaml), and here is walkthrough of how to use CloudFormation to set up SQS to trigger a Lambda.

Improve this answer
9
  • 4
    With this approach, how you can handle errors on step functions? Because, when the message is received in the lambda function, the message will be deleted from SQS. And if an error is raised on Step Function, neither SQS nor Lambda Function will be notified. I am assuiming that in the most cases errors will appears on step function and not in the lambda that call it
    – Vladimir Venegas
    Jul 30, 2019 at 16:11
  • 2
    The SQS message is not deleted until the message is successfully processed (ie. Step Function started) by the Lambda Function. Inside your Step Function, you would use the same approach for error handling as in any other Step Function. docs.aws.amazon.com/step-functions/latest/dg/…
    – Matthew Pope
    Jul 30, 2019 at 16:17
  • 1
    @MatthewPope, why would it be an anti-pattern? Looks like a reasonable approach if you want the Step Function to acknowledge/retry the message in case of success/failure, in the same way a Lambda would do. In many cases, retrying individual steps in the SFN may be enough, but I can imagine some other cases where you would want to retry the entire SFN.
    – cahen
    Jul 12, 2021 at 12:54
  • 1
    Looking at the first question in this answer - do I really need SQS to start a step function, I wonder if rate limiting is a good reason, say you want to limit the number of concurrent sfn executions, e.g. a scraper that avoids bombarding the website, ... etc. Would be great to have some criteria to consider.
    – Shawn
    Jul 20, 2021 at 6:47
  • 2
    If your step function completes in reasobably fast times (<5 minutes) you can use an express step function as illustrated here: docs.aws.amazon.com/step-functions/latest/dg/… With this approach your jobs will be deleted from the queue only if the step function succeeds. Make sure to increase SQS VisibilityTimeout if the step function lasts more than 30secs: docs.aws.amazon.com/AWSSimpleQueueService/latest/… . This will help you to avoid triggering the same execution over and over
    – Luciano Mammino
    Apr 7, 2022 at 17:42
23

EventBridge Pipes (launched at re:Invent 2022) allows you to trigger Step Functions State Machines without need for a Lambda function.

https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes.html

You can find an example here:

https://github.com/aws-samples/aws-stepfunctions-examples/blob/main/sam/demo-trigger-stepfunctions-from-sqs/template.yaml

Improve this answer
AWS Employee
3

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.