41

How to fix a drifted AWS CloudFormation stack? I modified a BackendECS Service and it is now "drifted" but there's no information on how to resolve this? This is driving me insane? Zero information on how to resolve this?

Improve this question
2
  • 7
    CloudFormation freaks out whenever there is a small change, and usually can't even detecting it. Advice: use Terraform. I wasted a lot of time on CloudFormation, finally I went back to Terraform.
    – forzagreen
    Nov 29, 2019 at 13:50
  • I'm honestly baffled. This is a NIGHTMARE to untangle. Like come on AWS... you want me to add "retain" flags, remove resources, and then reimport them? How the hell does that make any sense? Changing resources like that is risky. Why can't I just update the template on record in-place with the drifted values? I haven't even gotten to the production side yet, but this is a minefield.
    – mukunda
    Jul 21, 2023 at 20:14

4 Answers 4

Reset to default
17

Amazon AWS allows troubleshooting of a drifted stack via its "Drift Detection" feature.

Drift detection can be accessed by navigating to CloudFormation > Select Stack > Actions > Detect Drift for current stack

More on this here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/detect-drift-stack.html

Using this diagnostic tool you can view exactly what is different in your AWS configuration as compared to the state that cloudformation expects.

There are a few ways to potentially resolve this.

  1. If you did an automated update via an Infrastructure as code service, rollback your change

  2. If you manually updated the ECS Service, change the settings back to the expected state as shown in the drift detection. Once your system is back to the expected state your stack will behave normally

  3. Delete the current stack resource and re-create it - this is a dangerous way to resolve this as you will lose your update history and rollback states.

ADDING theimowski's fourth alternative from the comments section:

  1. First remove deleted resources from template and update the stack,the bringing it back exactly as it was before in the template and updating the stack again. – theimowski Dec 29, 2021 at 16:38
Improve this answer
7
  • 1
    I went thru this while trying to sort out if I could recover from someone manually tweaking settings in a S3 bucket and every template edit I made and after "Update Stack" was pressed it just kept saying "no updates performed". Maybe a future release will just let us undo right where the drift diff is?
    – Mark
    Jun 29, 2019 at 3:11
  • 44
    How can I recreate something that I have deleted? It will have different id. Why they make these useless features?
    – Matteo
    Apr 9, 2020 at 2:09
  • 1
    There is also forth option, to fix the drift through importing resources.
    – Marcin
    Sep 28, 2020 at 23:43
  • 4
    @Matteo in my case after I manually removed API Gateway, I managed to recreate it by a) removing it from template and updating stack, b) bringing it back exactly as it was before in the template and updating the stack again.
    – theimowski
    Dec 29, 2021 at 16:38
  • 1
    @theimowski 's solution for recreating a removed resource should be considered part of the answer because it's a relevant scenario.
    – Bart Swennenhuis
    Mar 18, 2022 at 10:09
12

I had a similar issue with cloudformation and likewise there is still a lack of documentation for how to "undrift" the instances back to what the template specified. I found that commenting out the drifted entity allowed cloudformation to "delete" it and then uncommenting it restores it to the desired state.

That said, I'll second the motion to just use terraform instead as enforcing the template is the default behavior whenever you do an apply.

Improve this answer
1
  • 2
    A great solution to fix things non-destructively, good thinking, and thanks for sharing...worked well.
    – TryTryAgain
    Jan 21, 2022 at 21:11
2

For those who for some reason can't switch to Terraform (preferred), I suspect each remaining case is going to be a bit different, so there could be a lot of valid and helpful answers as time goes on.

For example, five of my DRIFTED cases turned out to be insufficient IAM permissions to modify IAM Role tags, so that was a straightforward fix - get the IAM permissions and so ensure the tags were added, then redo the scan which gave the more pleasant IN_SYNC.

Another one resulted from a workaround for a known AWS CDK issue as the CDK uses CloudFormation under the hood, so I'm inclined to ignore it - if time allowed, I'd rather fix the CDK or switch to Terraform :)

P.S. The current AWS Management Console UI/UX isn't great either, I was fooled into clicking on the Physical ID link, which is basically a waste of time.

A better way to find the relevant issues is to sort by Drift status, then click on the relevant row's radio button, and click the View drift details button at the top right of the table.

Improve this answer
0

Further to this conversation - not all resources can be drift detected; endpoints amongst these, so CloudFormation will not even spot they have altered (took me ages to figure this out and in tests even deleting them did not trigger CF to notice).

Moral of the story - don't rely 100% on drift detection and as others have said, upgrade to Terraform and do it properly :(

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.