Troubleshooting

Having trouble? We’re here to help.

First, contact support@fugue.co.

You may be instructed to install the Fugue Support Tool. If so, see the installation instructions, and then check out the Support Tool CLI reference.

Below, we’ve provided guidance for possible issues, along with some general troubleshooting tips.

Possible Issues

Here are some possible issues and error messages you may encounter when using Fugue.

Error: “Command is not supported” on any CLI command

Error message:

I'm sorry Dave. I'm afraid I can't do that.
[ ERROR ] This command is not supported by the Conductor currently installed.

Explanation: This error indicates a mismatched Fugue CLI and Fugue Conductor. The CLI and Conductor form a matched set, so if you upgraded the CLI to a newer version but did not upgrade the Conductor to the corresponding AMI ID, the CLI may support a command that the Conductor does not. In this case, attempting to run an unsupported command produces the above error message.

Solution: Visit the Fugue Download Portal to find the corresponding Conductor AMI ID for your version of the CLI, and then run fugue upgrade <ami_id> with that AMI to upgrade the Conductor. See upgrade for more details.

Error: “AWS CloudFormation stack creation failed” on Fugue installation

Error message:

Installing the Fugue Conductor into AWS account <user>/<account number>.

FugueInternetRoute                   Working...
FugueSubnet2                         Working...
FugueSubnet1RouteTableAssociation    Working...
FugueSubnet2RouteTableAssociation    Working...
FugueAutoScalingGroup                Working...
FugueVpcSecurityGroup                Working...
FugueIam                             Complete
FugueSubnet1                         Working...
FugueResourceEventsTopic             Complete
FugueHealthCheckDb                   Working...
FugueVpcGatewayAttachment            Working...
FugueVpc                             Complete
FugueRouteTable                      Complete
FugueLaunchConfiguration             Working...
FugueVpcGateway                      Complete
FugueInstanceProfile                 Working...
-----------------------------------------------
Overall Progress  [#######..................]   31%

[ HELP ] Exiting the install command while in progress (CTRL+C) will only stop progress tracking and *not* the install itself.

[ ERROR ] AWS CloudFormation stack creation failed

In this case, you may have encountered a faulty AZ availability indication – a limitation of the AWS API that can “fool” Fugue’s installation command. To confirm you have this problem, take a look at the fugue CloudFormation stack event log after the failed installation. You should see a message about an invalid AvailabilityZone parameter.

CloudFormation events typical of a Fugue installation where available AZs were incorrectly reported by AWS.

Example of erroneous events in this case.

To resolve this issue, follow the simple steps in this example. If you see a different error or are unsure if you are seeing this one, contact us at support@fugue.co.

Troubleshooting Tips

This section covers generalized troubleshooting in the Fugue system. Let’s take a look at a sample error message so we can discuss troubleshooting techniques. Suppose you get the following error when attempting init:

[ fugue init ] Initializing Fugue project with the following configuration:

Fugue Conductor AMI ID: ami-00000000
AWS Credentials: Environment variables

Checking your AWS Credentials for Fugue CLI use ...
[ OK ] Authorized.
Checking your AWS Credentials for Fugue Conductor installation ...
[ OK ] Authorized.

Validating Fugue Conductor AMI ID ...
[ ERROR ] ami-00000000 is not a valid AMI.

There are a few ways to start digging into the issue.

Check the CLI Log

A good place to start troubleshooting is the CLI log. There, you’ll find the full error message, along with the context surrounding it. In this case, we’ve given the CLI an AMI ID that does not exist, and we can confirm this by checking the very end of fuguecli.log:

2016-09-28T18:38:01.09 [ fugue.screen ] ERROR - ami-00000000 is not a valid AMI.
Traceback (most recent call last):
  File "site-packages/fugue_cli/utils.py", line 181, in validate_ami
  File "site-packages/botocore/client.py", line 159, in _api_call
  File "site-packages/botocore/client.py", line 494, in _make_api_call
botocore.exceptions.ClientError: An error occurred (InvalidAMIID.NotFound) when calling the DescribeImages operation: The image id '[ami-00000000]' does not exist

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "site-packages/fugue_cli/cli.py", line 157, in invoke
  File "site-packages/click/core.py", line 1060, in invoke
  File "site-packages/click/core.py", line 889, in invoke
  File "site-packages/click/core.py", line 534, in invoke
  File "site-packages/click/decorators.py", line 27, in new_func
  File "site-packages/fugue_cli/commands/init.py", line 96, in init
  File "site-packages/fugue_cli/utils.py", line 189, in validate_ami
fugue_cli.common.FugueError: ami-00000000 is not a valid AMI.

Here, we can see that the CLI encountered the error InvalidAMIID.NotFound while calling the DescribeImages operation. By cross-referencing AWS’s EC2 error code documentation, we can confirm that this error occurred because the AMI we specified during init does not exist. AWS provides some tips:

The specified AMI does not exist. Check the AMI ID, and ensure that you specify the region in which the AMI is located, if it’s not in the default region. This error may also occur if you specified an incorrect kernel ID when launching an instance.

Should this particular error happen to you, it’s worth checking the Fugue Download Portal to make sure you have the latest AMI ID. If the AMI ID is correct, check the region field in fugue.yaml. AMI IDs are only valid for one region, so if the Conductor is configured to run in a different region than your AMI, the same error is triggered.

Increase CLI Verbosity

The CLI log doesn’t append extra information for all errors, but there’s another way to find more information. Any Fugue command can be executed in verbose mode by using the global -v option. The option increases the verbosity incrementally, so -vvv shows more detail than -v. With increased verbosity, the CLI displays considerably more data that may be useful in troubleshooting, including the precise messages the CLI sends to the Conductor and the responses AWS returns.

Email Us

Finally, you can always send an email to support@fugue.co describing your issue, and we’ll be happy to help you.

The Fugue Support Tool

The Fugue Support Tool is a command-line interface that enables Conductor debug logging, collects diagnostic information, and sends the diagnostic data back to Fugue. It is packaged separately from the Fugue Client Tools and must be downloaded through the Fugue Download Portal.

Generally, you should only use the Fugue Support Tool if you’ve been instructed to do so. The tool is meant to aid the Fugue support team in assisting you, and it isn’t designed for self-service troubleshooting.

Installing the Fugue Support Tool

Log in to the Fugue Download Portal and click on the “Support Tool” button near the top right corner, then select your platform. Instructions based on package type follow.

Warning

These instructions use a wildcard filename for the package names. This works fine as long as you only have one matching file in the present directory.

For Linux RPM:

$ sudo rpm -ivh fugue-support*.rpm

For Linux DEB:

$ sudo dpkg -i fugue-support*.deb

For OSX PKG:

Double-click on the PKG file and follow the installation wizard.

Using the Fugue Support Tool

The Fugue Support Tool has two commands, debug and report.

  • debug enables and disables Conductor component debug-level logging. Debug-level logging is enabled by default.
  • report downloads Conductor component logs, generates a support report, zips it up, and uploads the zip file to an S3 bucket belonging to Fugue, Inc.

See the Fugue Support Tool CLI reference for more information.

Note: All start and end times are provided in UTC.

Warning

Remove any sensitive, private, or personal information from your Fugue project directory before executing fugue-support report. The command zips up the contents of the Fugue project directory and sends it to Fugue, Inc.

Uninstalling the Fugue Support Tool

For Linux RPM:

sudo yum remove fugue-support.x86_64

For Windows:

  1. Launch the Control Panel and locate the Uninstall feature.
  2. Locate Fugue Support Tool in the list of applications and select the option to uninstall the application.

For OS X:

for f in $(pkgutil --only-files --files co.fugue.fugue-support); do
  echo "Removing /$f"
  sudo rm "/$f"
done

for d in bin lib docs etc; do
  if [ "$(ls -A /opt/fugue/$d)" ]; then
    echo "/opt/fugue/$d not empty"
  else
    echo "/opt/fugue/$d empty, removing"
    sudo rmdir "/opt/fugue/$d"
  fi
done

if [ "$(ls -A /opt/fugue)" ]; then
  echo "/opt/fugue not empty"
else
  echo "/opt/fugue empty, removing"
  sudo rmdir /opt/fugue
fi

sudo pkgutil --forget co.fugue.fugue-support

For Ubuntu:

sudo apt-get remove fugue-support

Fugue Support Tool CLI Reference

Global Usage

$ fugue-support [GLOBAL ARGUMENTS] [debug|report] [COMMAND ARGUMENTS]

Global Arguments and Flags

--region=REGION
Required for debug: The AWS region the Conductor is in. Example: --region=us-east-1
-v | --verbose
Enable Fugue Support Tool verbose output mode
--version
Display Fugue Support Tool version
--help
Display Fugue Support Tool help message

Debug

debug enables or disables Conductor component debug-level logging. By default, Conductor component activity is logged at DEBUG level.

All logs are accessible from AWS CloudWatch.

Debug Usage

$ fugue-support --region=REGION debug [--enable LIST] [--disable LIST] [--help]

Debug Arguments and Flags

--region=REGION
Required: The AWS region the Conductor is in. Example: --region=us-east-1
--enable LIST
Enable debug-level logging for a specified Conductor component
--disable LIST
Disable debug-level logging for a specified Conductor component
LIST

Required: The specified Conductor component. Valid options:

  • reflector_query
  • scheduler_ticker
  • conductor_stats
  • broker
  • scheduler_signal_handler
  • all: Turns debug-level logging on or off for all Conductor components
--help
Display debug help message

Report

report downloads Conductor component logs, generates a support report, zips it up, and uploads the ZIP file to an S3 bucket belonging to Fugue, Inc.

Note: All start and end times are provided in UTC.

The ZIP contains the following files:

  • conductor_policies.json, containing the IAM policy document assigned to the Conductor role
  • a logs directory, containing individual log files for each Conductor component
  • a project directory, containing a copy of your Fugue project directory
  • vars.json, containing the Vars database

Warning

Remove any sensitive, private, or personal information from your Fugue project directory before executing fugue-support report. The command zips up the contents of the Fugue project directory and sends it to Fugue, Inc.

Report Usage

$ fugue-support report [--proj PATH] [-s|--start "mm/dd/YYYY HH:MM"] [-e|--end "mm/dd/YYYY HH:MM"] [--skip-logs] [--help]

Report Arguments and Flags

--proj PATH
Path to your Fugue project
-s | --start
Start time for logs. Format: “mm/dd/YYYY HH:MM” or “30m ago” or “2h ago” or “1d ago”
-e | --end
End time for logs. Format: format “mm/dd/YYYY HH:MM” or “30m ago” or “2h ago” or “1d ago”
--skip-logs
Generate and send a support report without downloading and sending logs
--help
Display report help message