`aws_iam_role_policy_attachment.this` missing `lifecycle { create_before_destroy = true }` in eks-managed-node-group

[dsantanu]

Description

When kubernetes_groups on an aws_eks_access_entry is changed, it triggers assume_role_policy on the node IAM role to become known after apply. This cascades to policy_arn = (known after apply) on aws_iam_role_policy_attachment.this, forcing a +/- destroy+recreate. The destroy half completes successfully, but the recreate silently fails to complete, leaving node roles with no core policies attached (AmazonEKSWorkerNodePolicy, AmazonEC2ContainerRegistryReadOnly, AmazonEKS_CNI_IPv6_Policy). This breaks the entire cluster — nodes lose ECR access and aws-cni fails.

• ✋ I have searched the open/closed issues and my issue is not listed.

Cluster impact

Nodes lose ECR pull access, aws-cni gRPC connection fails, new pods cannot be scheduled

Versions

• Module version [Required]: 21.19.0
• Terraform version: v1.11.0
• Provider version(s): v6.23.0

Reproduction Code [Required]

1. Have an EKS cluster managed by this module with at least one managed node group
2. Have aws_eks_access_entry resources with kubernetes_groups configured, e.g.:

resource "aws_eks_access_entry" "this" {
  for_each          = var.eks_cluster_ops
  cluster_name      = module.cluster["eks"].cluster_name
  principal_arn     = each.value.principal_arn
  kubernetes_groups = each.value.k8s_groups
  type              = "STANDARD"
}

3. Change the value of kubernetes_groups on any access entry — e.g. rename a group from cluster:admins to cluster:eksadmins
4. Run terraform plan - observe the cascade:

• aws_eks_access_entry → in-place update (expected)
• module.nodegroups[*].aws_iam_role.this → assume_role_policy becomes known after apply
• module.nodegroups[*].aws_iam_role_policy_attachment.this[*] → +/- destroy+recreate because policy_arn = (known after apply)

5. Run terraform apply - apply reports success with no errors
6. Result: aws_iam_role_policy_attachment.this destroy half completes, recreate half silently does not. Node roles are left with no core policies attached.
7. Verify with:

aws iam list-attached-role-policies \
  --role-name <node-group-role-name> \
  --query 'AttachedPolicies[].PolicyName'
# Returns only additional policies e.g. AmazonEBSCSIDriverPolicy
# Core policies AmazonEKSWorkerNodePolicy, AmazonEC2ContainerRegistryReadOnly,
# AmazonEKS_CNI_IPv6_Policy are missing

Expected behavior

lifecycle { create_before_destroy = true } on aws_iam_role_policy_attachment.this (as is already done on aws_launch_template.this and aws_eks_node_group.this) would prevent the gap.

Actual behavior

Perviously attached core-policies are randomly missing

Additional context

Add this to fix:

resource "aws_iam_role_policy_attachment" "this" {
  ...
  lifecycle {
    create_before_destroy = true
  }
}

[bryantbiggs]

we'll need a reproduction to start

[dsantanu]

we'll need a reproduction to start

I assume that's something required from my side? if yes, how exactly can I do that? any pointer?

[bryantbiggs]

start with a simple cluster definition, can use your configs or one of our examples - and add the minimum configs and steps to reproduce the issue you are encountering

[dsantanu]

I can try..........
but looking into this again, the root cause seems to be the OIDC provider TLS certificate (data.tls_certificate.this) re-reads when the cluster changes, which makes aws_iam_openid_connect_provider.oidc_provider update in-place, which makes the nodegroup aws_iam_role.this assume_role_policy become known after apply, which cascades into the policy attachment +/- replacements. Any change to aws_eks_cluster.this will trigger this cascade - it's structural to how the module chains its dependencies.

Is providing a tf plan from my live systen would do any good at all?

[kreuzwerkermarc]

we have noticed something similar on our setup

[bryantbiggs]

most likely you all are adding an explicit depends_on = [...] in your module definition - don't do that, remove it

[sharansutrapu]

Hey @dsantanu and @kreuzwerkermarc, stepping in to provide some technical context on why Bryant is pointing to depends_on here, and why adding create_before_destroy will actually permanently break your node permissions.

1. The depends_on Cascade
   When you place an explicit depends_on on a module (or on a resource that a module passes in as a variable), Terraform defers reading the state of all attributes within that module until the apply phase. This causes the assume_role_policy to show up as (known after apply) during the plan, which forces Terraform to flag the downstream aws_iam_role_policy_attachment for replacement (+/-).

2. The create_before_destroy IAM Trap
   Adding lifecycle { create_before_destroy = true } to an IAM policy attachment is a known Terraform provider anti-pattern because of how the AWS API handles attachments. If you add it, here is exactly how Terraform executes the +/- replacement:

• Create Phase: Terraform asks AWS to attach the policy to the role. AWS returns success.
• Destroy Phase: Terraform executes the cleanup for the "old" state and asks AWS to detach the policy from the role. AWS returns success.
• Result: Your node role is left with 0 core policies attached. The recreate didn't "silently fail", it succeeded, but the subsequent destroy phase wiped it out.

The Fix:
As Bryant mentioned, you need to comb through your root module configuration and remove any explicit depends_on blocks that are tied to the EKS module, the VPC module, or the Access Entries. Terraform will naturally resolve the dependency graph via implicit variable references, which will stop the (known after apply) cascade and prevent the policy attachments from ever being flagged for replacement in the first place.