How to check if specific resource already exists in CloudFormation script

Question

I am using cloudformation to create a stack which inlcudes an autoscaled ec2 instance and an S3 bucket. For the S3 bucket I have DeletionPolicy set to Retain, which works fine, until I want to rerun my cloudformation script again. Since on previous runs, the script created the S3 bucket, it fails on subsequent runs saying my S3 bucket already exists. None of the other resources of course get created as well. My question is how do I check if my S3 bucket exists first inside the cloudformation script, and if it does, then skip creating that resources. I've looked in conditions in the AWS, but it seems all parameter based, I have yet to find a function which checks from existing resources.

Answer 1

There is no obvious way to do this, unless you create the template dynamically with an explicit check. Stacks created from the same template are independent entities, and if you create a stack that contains a bucket, delete the stack while retaining the bucket, and then create a new stack (even one with the same name), there is no connection between this new stack and the bucket created as part of the previous stack.

If you want to use the same S3 bucket for multiple stacks (even if only one of them exists at a time), that bucket does not really belong in the stack - it would make more sense to create the bucket in a separate stack, using a separate template (putting the bucket URL in the "Outputs" section), and then referencing it from your original stack using a parameter.

Update November 2019:

There is a possible alternative now. On Nov 13th AWS launched CloudFormation Resource Import. With that feature you can now creating a stack from existing resources. Currently not many resource types are supported by this feature, but S3 buckets are.

In your case you'd have to do it in two steps:

1. Create a template that only contains the preexisting S3 bucket using the "Create stack" > "With existing resources (import resources)" (this is the --change-set-type IMPORT flag in the CLI) (see docs)
2. Update the the template to include all resources that don't already exist.

As they note in their documentation; this feature is very versatile. So it opens up a lot of possibilities. See docs for more info.

Answer 2

Using cloudformation you can use Conditions I created an input parameter "ShouldCreateBucketInputParameter" and then using CLI you just need to set "true" or "false"

Cloudformation json file:

{
"AWSTemplateFormatVersion": "2010-09-09",
"Transform": "AWS::Serverless-2016-10-31",
"Description": "",
"Parameters": {
    "ShouldCreateBucketInputParameter": {
      "Type": "String",
      "AllowedValues": [
        "true",
        "false"
      ],
      "Description": "If true then the S3 bucket that will be proxied will be created with the CloudFormation stack."
    }
},
"Conditions": {
  "CreateS3Bucket": {
    "Fn::Equals": [
      {
        "Ref": "ShouldCreateBucketInputParameter"
      },
      "true"
    ]
  }
},
"Resources": {
    "SerialNumberBucketResource": {
        "Type": "AWS::S3::Bucket",
        "Condition": "CreateS3Bucket",
        "Properties": {
            "AccessControl": "Private"
        }
    }
},
"Outputs": {}
}

And then (I am using CLI do deploy the stack)

aws cloudformation deploy --template ./s3BucketWithCondition.json --stack-name bucket-stack --parameter-overrides ShouldCreateBucketInputParameter="true" S3BucketNameInputParameter="BucketName22211"

Answer 3

Just add an input parameter to the CloudFormation template to indicate that an existing bucket should be used.... unless you don't already know at the time when you are going to use the template? Then you can either add a new resource or not based on the parameter value.

Answer 4

If you do updates, (potentially of stacks within stacks aka Nested Stacks), the unchanged parts don't get updated. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-stack.html?icmpid=docs_cfn_console_designer

You can then set policies as mentioned to prevent deletion. [remember 'cancel update' permissions for rollbacks] https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/protect-stack-resources.html

There is also Cross-Stack Output to be aware of by adding Export Names to the Stack Outputs. http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/outputs-section-structure.html Walkthrough... http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/walkthrough-crossstackref.html

Then you need to use Fn::ImportValue ... http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-importvalue.html

It implies one could use a network stack name parameter.

Unfortunately you get an error like this when you try them in Conditions.

Template validation error: Template error: Cannot use Fn::ImportValue in Conditions.

Or in the Parameters?

Template validation error: Template format error: Every Default member must be a string.

Also this can happen while trying...

Template format error: Output ExportOut is malformed. The Name field of Export must not depend on any resources, imported values, or Fn::GetAZs.

So you can't stop it making the existing resource again from the same file. Only when putting it into another stack and using the export import reference.

But if you separate the two then there is a dependency that will stop and rollback for instance a dependency's deletion, thanks to the reference via the ImportValue function.

Example Given here is:

First Make a Group Template

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Metadata": {
    "AWS::CloudFormation::Designer": {
      "6927bf3d-85ec-449d-8ee1-f3e1804d78f7": {
        "size": {
          "width": 60,
          "height": 60
        },
        "position": {
          "x": -390,
          "y": 130
        },
        "z": 0,
        "embeds": []
      },
      "6fe3a2b8-16a1-4ce0-b412-4d4f87e9c54c": {
        "source": {
          "id": "ac295134-9e38-4425-8d20-2c50ef0d51b3"
        },
        "target": {
          "id": "6927bf3d-85ec-449d-8ee1-f3e1804d78f7"
        },
        "z": 1
      }
    }
  },
  "Resources": {
    "TestGroup": {
      "Type": "AWS::IAM::Group",
      "Properties": {},
      "Metadata": {
        "AWS::CloudFormation::Designer": {
          "id": "6927bf3d-85ec-449d-8ee1-f3e1804d78f7"
        }
      },
      "Condition": ""
    }
  },
  "Parameters": {},
  "Outputs": {
    "GroupNameOut": {
      "Description": "The Group Name",
      "Value": {
        "Ref": "TestGroup"
      },
      "Export": {
        "Name": "Exported-GroupName"
      }
    }
  }
}

Then make a User Template that needs the group.

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Metadata": {
    "AWS::CloudFormation::Designer": {
      "ac295134-9e38-4425-8d20-2c50ef0d51b3": {
        "size": {
          "width": 60,
          "height": 60
        },
        "position": {
          "x": -450,
          "y": 130
        },
        "z": 0,
        "embeds": [],
        "isrelatedto": [
          "6927bf3d-85ec-449d-8ee1-f3e1804d78f7"
        ]
      },
      "6fe3a2b8-16a1-4ce0-b412-4d4f87e9c54c": {
        "source": {
          "id": "ac295134-9e38-4425-8d20-2c50ef0d51b3"
        },
        "target": {
          "id": "6927bf3d-85ec-449d-8ee1-f3e1804d78f7"
        },
        "z": 1
      }
    }
  },
  "Resources": {
    "TestUser": {
      "Type": "AWS::IAM::User",
      "Properties": {
        "UserName": {
          "Ref": "UserNameParam"
        },
        "Groups": [
          {
            "Fn::ImportValue": "Exported-GroupName"
          }
        ]
      },
      "Metadata": {
        "AWS::CloudFormation::Designer": {
          "id": "ac295134-9e38-4425-8d20-2c50ef0d51b3"
        }
      }
    }
  },
  "Parameters": {
    "UserNameParam": {
      "Default": "testerUser",
      "Description": "Username For Test",
      "Type": "String",
      "MinLength": "1",
      "MaxLength": "16",
      "AllowedPattern": "[a-zA-Z][a-zA-Z0-9]*",
      "ConstraintDescription": "must begin with a letter and contain only alphanumeric characters."
    }
  },
  "Outputs": {
    "UserNameOut": {
      "Description": "The User Name",
      "Value": {
        "Ref": "TestUser"
      }
    }
  }
}

You will get

No export named Exported-GroupName found. Rollback requested by user.

if running User with no Group found Exported.

You could then use the Nested stack approach.

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Metadata": {
    "AWS::CloudFormation::Designer": {
      "66470873-b2bd-4a5a-af19-5d54b11f48ef": {
        "size": {
          "width": 60,
          "height": 60
        },
        "position": {
          "x": -815,
          "y": 169
        },
        "z": 0,
        "embeds": []
      },
      "ed1de011-f1bb-4788-b63e-dcf5494d10d1": {
        "size": {
          "width": 60,
          "height": 60
        },
        "position": {
          "x": -710,
          "y": 170
        },
        "z": 0,
        "dependson": [
          "66470873-b2bd-4a5a-af19-5d54b11f48ef"
        ]
      },
      "c978f2d9-3fb2-4420-b255-74941f10a28a": {
        "source": {
          "id": "ed1de011-f1bb-4788-b63e-dcf5494d10d1"
        },
        "target": {
          "id": "66470873-b2bd-4a5a-af19-5d54b11f48ef"
        },
        "z": 1
      }
    }
  },
  "Resources": {
    "GroupStack": {
      "Type": "AWS::CloudFormation::Stack",
      "Properties": {
        "TemplateURL": "https://s3-us-west-2.amazonaws.com/cf-templates-x-TestGroup.json"
      },
      "Metadata": {
        "AWS::CloudFormation::Designer": {
          "id": "66470873-b2bd-4a5a-af19-5d54b11f48ef"
        }
      }
    },
    "UserStack": {
      "Type": "AWS::CloudFormation::Stack",
      "Properties": {
        "TemplateURL": "https://s3-us-west-2.amazonaws.com/cf-templates-x-TestUserFindsGroup.json"
      },
      "Metadata": {
        "AWS::CloudFormation::Designer": {
          "id": "ed1de011-f1bb-4788-b63e-dcf5494d10d1"
        }
      },
      "DependsOn": [
        "GroupStack"
      ]
    }
  }
}

Unfortunately you can still delete the User stack even though it was made by MultiStack in this example but with deletion policies and other things it just might help.

Then you are only Updating the various stacks it creates, and you won't do the Multi Stack if you're for instance reusing a Bucket.

Otherwise you'll be looking at APIs and scripts in various flavors.

Answer 5

If you're trying to incorporate some existing resources into CF, it is unfortunately not possible. If you just want a set of resources to be part of your template or not depending on the value of some parameters, you can use Conditions. But they don't change the nature of CF itself, and only work to determine which resources are desired, not what actions will be taken, and cannot see whether a resource exists or not beforehand.

Answer 6

Something not explicitly stated. If your first deployment fails, resources will be deleted unless you have at retention policy. In this case it is safe to delete the resource in question manually. Next deployment will recreate it without generating error that resource already exists.