Troubleshooting¶

This guide covers common issues and their solutions when using the terraform-aws-website-pod module.

Deployment Issues¶

Certificate Validation Timeout¶

Symptoms: - Terraform hangs at aws_acm_certificate_validation.website - Error: "timeout while waiting for state to become 'ISSUED'"

Causes: - DNS propagation delay - Incorrect Route53 zone ID - Cross-account DNS misconfiguration

Solutions:

Verify the zone ID is correct:

aws route53 get-hosted-zone --id YOUR_ZONE_ID

Check if validation records were created:

aws route53 list-resource-record-sets --hosted-zone-id YOUR_ZONE_ID | grep CNAME

For cross-account DNS, verify the provider configuration:

provider "aws" {
  alias = "dns"
  assume_role {
    role_arn = "arn:aws:iam::DNS_ACCOUNT:role/route53-admin"
  }
}

Increase timeout (add to your configuration):

resource "aws_acm_certificate_validation" "website" {
  timeouts {
    create = "60m"
  }
}

Instances Not Becoming Healthy¶

Symptoms: - Terraform hangs at aws_autoscaling_group.website - Error: "timeout while waiting for state to become 'healthy'" - Instances keep getting replaced

Causes: - Health check path returns non-200 status - Application not starting properly - Security group blocking traffic - Instance failing to provision

Solutions:

Check instance status in AWS Console or CLI:

aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names YOUR_ASG_NAME

Connect to an instance and check logs:

# Check cloud-init
sudo cat /var/log/cloud-init-output.log

# Check application logs
sudo journalctl -u your-service

Test health check endpoint manually:
```
curl -v http://localhost/index.html
```

Verify security groups allow traffic from ALB:

aws ec2 describe-security-groups --group-ids YOUR_BACKEND_SG_ID

Temporarily increase timeouts:

module "website" {
  # ...
  health_check_grace_period   = 900   # 15 minutes
  wait_for_capacity_timeout   = "30m"
}

Provider Configuration Errors¶

Symptoms: - Error: "Provider configuration not present" - Error: "Configuration for provider 'aws.dns' is not present"

Solution:

Always pass both providers:

module "website" {
  providers = {
    aws     = aws
    aws.dns = aws  # Can be the same provider if same account/region
  }
  # ...
}

Runtime Issues¶

High Error Rate Alarms¶

Symptoms: - CloudWatch alarm: "Low Success Rate" - 5xx errors in ALB access logs

Diagnosis:

Check ALB access logs:
```
aws s3 ls s3://YOUR_ACCESS_LOG_BUCKET/
```

Look for error patterns:

# Download and analyze logs
zcat access_log.gz | grep " 5[0-9][0-9] "

Check target health:

aws elbv2 describe-target-health --target-group-arn YOUR_TG_ARN

Solutions:

If instances are unhealthy, check instance logs
If instances are healthy but returning errors, debug application
If specific instances are problematic, terminate and let ASG replace

High Latency Alarms¶

Symptoms: - CloudWatch alarm: "Target Response Time" - Slow page loads

Diagnosis:

Check CloudWatch metrics:
TargetResponseTime - Time to first byte from targets
RequestCount - Traffic volume
ActiveConnectionCount - Concurrent connections

Check instance CPU utilization:

aws cloudwatch get-metric-statistics \
  --namespace AWS/EC2 \
  --metric-name CPUUtilization \
  --dimensions Name=AutoScalingGroupName,Value=YOUR_ASG_NAME \
  --start-time 2024-01-01T00:00:00Z \
  --end-time 2024-01-01T01:00:00Z \
  --period 60 \
  --statistics Average

Solutions:

Scale up instance type if CPU is consistently high
Increase asg_max_size if hitting scaling limits
Optimize application code for slow endpoints

Consider using least_outstanding_requests algorithm:

load_balancing_algorithm_type = "least_outstanding_requests"

Unhealthy Host Alarms¶

Symptoms: - CloudWatch alarm: "Unhealthy Host Count" - Some instances marked unhealthy in target group

Diagnosis:

Check target group health:

aws elbv2 describe-target-health --target-group-arn YOUR_TG_ARN

Check instance status:

aws autoscaling describe-auto-scaling-instances

SSH to unhealthy instance and check:
Application is running
Health endpoint responds
No disk space issues
No memory issues

Solutions:

If transient (during deployments), adjust threshold:

alarm_unhealthy_host_threshold = 1  # Allow 1 unhealthy during updates

If persistent, investigate and fix root cause

No Email Notifications¶

Symptoms: - Alarms are firing but no emails received - SNS subscription shows "PendingConfirmation"

Solution:

Check for confirmation email in spam folder

Resend confirmation:

aws sns list-subscriptions-by-topic --topic-arn YOUR_SNS_TOPIC_ARN

Or recreate subscription via Terraform (destroy and apply)

Security Issues¶

Cannot SSH to Instances¶

Symptoms: - SSH connection timeout - "Connection refused"

Causes: - Security group blocking SSH - No route to instance (private subnet without bastion) - Wrong key pair

Solutions:

Check security group allows SSH:

aws ec2 describe-security-groups --group-ids YOUR_BACKEND_SG_ID

If in private subnet, use Session Manager:

aws ssm start-session --target INSTANCE_ID

Or deploy a bastion host in public subnet
Add ssh_cidr_block for your IP:
```
ssh_cidr_block = "YOUR_IP/32"
```

Certificate Not Working¶

Symptoms: - Browser shows "Certificate Invalid" - curl fails with SSL error

Diagnosis:

Check certificate status:

aws acm describe-certificate --certificate-arn YOUR_CERT_ARN

Verify DNS resolves to ALB:
```
dig +short your-domain.com
```

Test SSL:

openssl s_client -connect your-domain.com:443 -servername your-domain.com

Solutions:

If certificate is PENDING_VALIDATION, wait for DNS propagation
If certificate is FAILED, check validation records
If using wrong certificate, verify dns_a_records includes all hostnames

Cost Issues¶

Unexpected Charges¶

Common causes and solutions:

ALB charges: ALBs have hourly charges plus LCU charges
Review traffic patterns
Consider combining multiple services behind one ALB
Data transfer: Check CloudWatch for data transfer metrics
Enable compression in your application
Use CloudFront for static assets
Spot instance interruptions: Frequent replacements increase costs
Increase on_demand_base_capacity for stability
Use multiple instance types (requires custom configuration)
S3 access logs: Large log volumes increase storage costs
Set up lifecycle rules to delete old logs
Consider sampling in high-traffic scenarios

Getting Help¶

If you're still experiencing issues:

Check existing issues: GitHub Issues
Open a new issue with:
Terraform version
Module version
Relevant configuration (sanitized)
Error messages
Steps to reproduce