Troubleshooting

Setup Issues

400 Response when calling deployment API

Remove -fs in the curl -fs -d .... call to see the error message, there is likely missing required configuration. Ex. required configs for db setup is db_host , db_secret_arn , aws_region

Startup Issues

botocore.exceptions.NoCredentialsError

Assuming AWS RDS setup: Check that you can fetch aws secret manager via command line in the instance. If you can't, you need to add secretmanager permissions to your instance iam role.

If you can, increase the hop limit on your instance: aws ec2 modify-instance-metadata-options --instance-id i-XXXXXXXXXXXX --http-put-response-hop-limit 3

Pulling Logs

When things go wrong, cd to the folder with taas application docker-compose.yml file, and run below script to collect logs for investigation and restart the system.

#!/bin/bash

# Check if the script is running in a folder with docker-compose.yml containing 'taas_app'
if [ ! -f "docker-compose.yml" ]; then
    echo "docker-compose.yml file not found. Please cd to the appropriate folder with the latest taas_app docker-compose.yml file."
    exit 1
fi
if ! grep -q "name: taas_app" docker-compose.yml; then
    echo "The docker-compose.yml file does not contain an entry with 'name: taas_app'. Please check the file and try again."
    exit 1
fi

# Default output folder
OUTPUT_DIR="/tmp"
# Allow user to specify a different output folder
if [ "$1" ]; then
    OUTPUT_DIR="$1"
fi
# Ensure the output directory exists
mkdir -p "$OUTPUT_DIR"
# Print message about output folder
echo "Output will be saved to: $OUTPUT_DIR"
# Check available disk space in the output folder
AVAILABLE_SPACE=$(df --output=avail "$OUTPUT_DIR" | tail -1)
if [ "$AVAILABLE_SPACE" -lt 1000000 ]; then
    echo "Not enough disk space in $OUTPUT_DIR. At least 1G is required."
    exit 1
fi
# Check if gzip is available
if ! command -v gzip &> /dev/null; then
    echo "gzip could not be found. Please install it and try again."
    exit 1
fi
# Check if tar is available
if ! command -v tar &> /dev/null; then
    echo "tar could not be found. Please install it and try again."
    exit 1
fi

# Temp folder to hold individual logs
TEMP_DIR="$(mktemp -d)"
# Loop through containers starting with 'taas_app'
for CONTAINER in $(docker ps -a --filter "name=taas_app" --format "{{.Names}}"); do
    echo "Processing logs for container: $CONTAINER"
    # Dump logs to a file
    LOG_FILE="$TEMP_DIR/$CONTAINER.log"
    docker logs "$CONTAINER" >& "$LOG_FILE"
    # Compress the individual log file
    gzip -9 "$LOG_FILE"
done
# Add timestamp to the final gzip archive
TIMESTAMP=$(date +%Y%m%d%H%M%S)
FINAL_TAR="$OUTPUT_DIR/taas_app_logs_$TIMESTAMP.tar"
tar -cf "$FINAL_TAR" -C "$TEMP_DIR" .
# Clean up temporary files
rm -rf "$TEMP_DIR"
# Inform about the compressed log file
echo "Logs have been compressed and saved to: $FINAL_TAR"

# Prompt user to confirm log collection
echo ""
echo "IMPORTANT: Please ensure you have copied the log files from: $FINAL_TAR"
read -p "please confirm logs are copied and you want to continue to restart? (Y/y to continue): " LOG_COLLECTED

# Validate user input
case "$LOG_COLLECTED" in
    [yY])
        # Perform docker-compose down and up with --remove-orphans
        echo "Restarting docker-compose services..."
        docker compose down --remove-orphans
        docker compose up -d
        echo "Restart is complete."
        ;;
    *)
        echo "Exiting the script, and restart is skipped. Log files are located at: $FINAL_TAR"
        exit 1
        ;;
esac