AWS SWF vs Lambda + step functions? Simple answer is use Lambda for all new applications.

June 11, 2017

I do like my Lamb (daaaa)!

What's the difference between SWF and Lambda + step functions?

Well, for a start they can be used together.

Benefits and Limitations of using Lambda Tasks

There are a number of benefits of using Lambda tasks in place of a traditional Amazon SWF activity:

Lambda tasks don’t need to be registered or versioned like Amazon SWF activity types.
You can use any existing Lambda functions that you've already defined in your workflows.
Lambda functions are called directly by Amazon SWF; there is no need for you to implement a worker program to execute them as you must do with traditional activities.
Lambda provides you with metrics and logs for tracking and analyzing your function executions.

There are also a number of limitations regarding Lambda tasks that you should be aware of:

Lambda tasks can only be run in AWS regions that provide support for Lambda. See Lambda Regions and Endpoints in the Amazon Web Services General Reference for details about the currently-supported regions for Lambda.
Lambda tasks are currently supported only by the base SWF HTTP API and in the AWS Flow Framework for Java. There is currently no support for Lambda tasks in the AWS Flow Framework for Ruby.

One blog says:

SWF is much more complicated and is designed for longer processes, with multiple steps, that may take days to complete. For SWF I generally think of use cases like a customer placing an order on a website triggering a workflow that takes the order through all the steps of the process of billing, manufacturing, packaging, shipping, etc.

But doesn't Lambda have it's own workflow thingy? Yes, step functions.

There appear to be some significant limitations however, including throughput and constraints on event-driven behaviour.

With this blog suggesting step functions are better than SWF for:

SFN state machines have already demonstrated improvement over our old SWF solution in the following areas:

Latency. One of our workflows with several short deployment steps took thirty minutes to run under the old system. It takes — no kidding — thirty seconds to run to completion on SFN.
Cost. No SWF decider means no Lambda functions treated like persistent compute, which radically lowers cost. Moreover, the state machine structure of SFN means no calls to a DynamoDB state table.
Runtime State. The state machine seamlessly transfers the minimal amount of data needed between lambda functions, eliminating the need for other logs or database tables to hold runtime state.
Retries/Error Handling. SFN retries failed executions and easily reroutes errors.
Debugging. Visual workflows help quickly pinpoint solutions to state machine problems.
Code Maintainability. Our codebase has now been completely refactored to take advantage of SFN’s better abstractions and streamlined requirements. I think there’s about half as much code as there was before.

And this blog says:

Step Functions is a massive improvement over SWF. As the author said, there are major differences and areas for improvement. Lifecycle management of state machines is pretty poor, especially considering that deleting state machines is hit-or-miss. You'll need to version your state machines yourself--which is still less painful than SWF versioning. Also, because there are no "domains" as in SWF, development can be more of a pain. The last SWF feature that would let me drop SWF completely would be signaling ("sleeping" a workflow until an event occurs).

And the Amazon explanation. It looks like SWF has been deprecated:

Q: When should I use AWS Step Functions vs Amazon Simple Workflow Service (SWF)?

You should consider using AWS Step Functions for all your new applications, since it provides a more productive and agile approach to coordinating application components using visual workflows. If you require external signals to intervene in your processes, or you would like to launch child processes that return a result to a parent, then you should consider Amazon Simple Workflow Service (SWF). With Amazon SWF, instead of writing state machines in declarative JSON, you write a decider program to separate activity steps from decision steps. This provides you complete control over your orchestration logic, but increases the complexity of developing applications. You may write decider programs in the programming language of your choice, or you may use the Flow framework to use programming constructs that structure asynchronous interactions for you.

Q: What will happen to the existing Amazon Simple Workflow Service (SWF)?

AWS will continue to provide Amazon Simple Workflow (SWF) and to support all Amazon SWF customers.

Q: How does AWS Step Functions work with other AWS services?

AWS Step Functions works with several other AWS services, including AWS CloudFormation, AWS Lambda, Amazon EC2, Amazon EC2 Container Service (ECS), Amazon API Gateway, Amazon CloudWatch, AWS CloudTrail, and Auto Scaling.

The following are examples of some things you can do using various AWS services:

Launch multiple related resources required for AWS Step Functions state machines and activities using AWS CloudFormation.
Implement tasks using Lambda functions or programs running on EC2 or in ECS containers.
Associate your Step Functions APIs with methods in an API Gateway API.
Scale your EC2 task fleet for AWS Step Functions activities in response to CloudWatch metrics.

Why is SWF in the solution architecture certification? Looks like it SHOULD NOT BE given their advice to use Lambda Step Functions for new applications.

And the other missing major service is API Gateway which is a major new service:

Q: How does AWS Step Functions work with Amazon API Gateway?

Amazon API Gateway is a fully-managed service that makes it easy for developers to publish, maintain, monitor, and secure APIs at any scale. With just a few clicks in the AWS Management Console, you can associate your Step Functions APIs with methods in your API Gateway API, so that, when an HTTPS request is sent to an API method that you defined, API Gateway invokes your Step Functions API actions.

You can use an Amazon API Gateway API to start Step Functions state machines that coordinate the components of a distributed backend application. You can also integrate human activity tasks into the steps of your application, such as an approval requests and responses. You can even make serverless asynchronous calls to the APIs of services that your application uses.

Amazon API Gateway handles all of the tasks involved in accepting and processing hundreds of thousands of concurrent API calls, including traffic management, authorization and access control, monitoring, and API version management.