Troubleshooting (General)

Having trouble? We’re here to help.

First, contact support@fugue.co.

You may be instructed to use the fugue support command. If so, check out the support documentation. If you’re looking for information about the actions Fugue is taking take a look at notifications.

Below, we’ve provided guidance for errors and known issues not tied to a specific fugue command. It includes info about troubleshooting Ludwig and Transcriber, and you’ll find some general troubleshooting tips. Troubleshooting (Fugue Commands) contains information about troubleshooting specific fugue commands.

You can also find known issues in our release notes.

Other Possible Issues

Here are some other possible issues and error messages you may encounter when using Fugue.

RBAC policy error after upgrade

Error message:

[ ERROR ] Due to an attached RBAC policy, command 'ops' is not allowed for user 'alice'.

Explanation: You may encounter RBAC errors if you attach a policy to the Fugue Conductor and then upgrade the Conductor. For example, if you attach a policy that enables alice to have access to allAccountActions, then upgrade to a Conductor AMI that supports newer actions (such as the ops command, above), alice will not have access to the new actions. The proper order for upgrading the Conductor is:

Solution: Re-attach the RBAC policy:

fugue policy rbac-attach <policy_file>

This recompiles and reapplies the RBAC permissions so that allAccountActions and other Fugue.System.Policy types include actions available on the new Conductor. As long as you attach the same policy as before, each user and secret will remain the same.

Error: “This client and the Conductor are incompatible” on any CLI command

Error message:

[ ERROR ] This client and the Conductor are incompatible: client API version 4.0.4 is incompatible with server API version 3.2.4.

Explanation: The Fugue CLI and the Fugue Conductor form a matched set. If you’ve installed a version of the CLI that is not compatible with your current version of the Conductor, or vice versa, you’ll see this error message upon executing any CLI command.

Solution: Visit the Download Portal to confirm that you have installed the correct version of the CLI and the correct AMI ID of the Conductor. Recall that to upgrade Fugue, the new CLI needs to be installed before you execute the upgrade command. If you encounter problems, reach out to support@fugue.co.

Error: “Fugue requires an English UTF-8 console environment” on any CLI command

Error message:

Fugue requires an English UTF-8 console environment.

Explanation: This error indicates that the character encoding for the shell in use is not U.S. English UTF-8. In order to execute any Fugue command, the encoding must be U.S. English UTF-8.

Solution: You can change the default character encoding in your shell by executing export LANG=en_US.UTF-8, which changes the environmental variable $LANG to use U.S. English UTF-8. You can also add the line export LANG=en_US.UTF-8 to your .bash_profile or .bashrc file, or wherever you keep your shell configuration. This will automatically set your shell’s character encoding to U.S. English UTF-8.

Error: “Command is not supported” on any CLI command

Error message:

I'm sorry Dave. I'm afraid I can't do that.
[ ERROR ] This command is not supported by the Conductor currently installed.

Explanation: This error indicates a mismatched Fugue CLI and Fugue Conductor. The CLI and Conductor form a matched set, so if you upgraded the CLI to a newer version but did not upgrade the Conductor to the corresponding AMI ID, the CLI may support a command that the Conductor does not. In this case, attempting to run an unsupported command produces the above error message.

Solution: Visit the Fugue Download Portal to find the corresponding Conductor AMI ID for your version of the CLI, and then run fugue upgrade --ami <ami_id> with that AMI to upgrade the Conductor. See upgrade for more details.

Error: “AWS CloudFormation stack creation failed” on Fugue installation

Error message:

Installing the Fugue Conductor into AWS account <user>/<account number>.

FugueInternetRoute                   Working...
FugueSubnet2                         Working...
FugueSubnet1RouteTableAssociation    Working...
FugueSubnet2RouteTableAssociation    Working...
FugueAutoScalingGroup                Working...
FugueVpcSecurityGroup                Working...
FugueIam                             Complete
FugueSubnet1                         Working...
FugueResourceEventsTopic             Complete
FugueHealthCheckDb                   Working...
FugueVpcGatewayAttachment            Working...
FugueVpc                             Complete
FugueRouteTable                      Complete
FugueLaunchConfiguration             Working...
FugueVpcGateway                      Complete
FugueInstanceProfile                 Working...
-----------------------------------------------
Overall Progress  [#######..................]   31%

[ HELP ] Exiting the install command while in progress (CTRL+C) will only stop progress tracking and *not* the install itself.
[ ERROR ] AWS CloudFormation stack creation failed

In this case, you may have encountered a faulty AZ availability indication – a limitation of the AWS API that can “fool” Fugue’s installation command. To confirm you have this problem, take a look at the fugue CloudFormation stack event log after the failed installation. You should see a message about an invalid AvailabilityZone parameter.

Note: While the CLI currently indicates that (CTRL+C) will not stop the installation, we do not recommend using this command as it may interrupt the successful creation of credentials. These recommendations will be updated in a future release.

CloudFormation events typical of a Fugue installation where available AZs were incorrectly reported by AWS.

Example of erroneous events in this case.

To resolve this issue, follow the simple steps in this example. If you see a different error or are unsure if you are seeing this one, contact us at support@fugue.co.

Error: “InvalidParameter” while modifying SNS subscription

Error message

Invalid parameter: SubscriptionArn Reason: An ARN must have at least 6 elements, not 1

Explanation

You might see this error in the “Status” column of fugue ops output or the “Description” column of fugue history output when attempting to create an SNS topic, create an SNS subscription, and set the subscription’s attributes in the same composition. This applies to the subscription protocols HTTP, HTTPS, email, and email-JSON.

In AWS, subscriptions to SNS topics are not fully created for HTTP, HTTPS, email, or email-JSON protocols until the endpoint confirms the subscription request. During this time, Fugue cannot modify the subscription attributes. Once the subscription is confirmed, Fugue issues the modify operation.

However, in a certain case Fugue will try to modify the subscription while it is pending, resulting in a job error. The following job will not include the modification operation and will result in a success. Fugue will still correctly reissue the modify call when the subscription is confirmed.

Normal Behavior

Generally, subscription modify requests are not attempted during the pending phase because they would result in failure. Since a job can contain multiple operations, if the modify operation fails, any remaining operations in the same job will fail. By temporarily ignoring requests to modify the pending subscription, Fugue is able to safely carry out other operations as usual, including enforcing other resources.

For example, if a user runs a composition that declares an SNS topic and SNS subscription, and then updates it to set the subscription attributes, Fugue doesn’t issue the modify call until the subscription is confirmed.

As a result, fugue history reports that the update was successful. And if you execute fugue ops <job_id> with the update job ID, you’ll see the message “No operations have been recorded for this process.” This shows that Fugue has not issued the modify call yet, which would otherwise appear in the output as aws.sns.set_subscription_attributes.

Known Issue: Simultaneous Create/Modify Operations

However, if the user declares the subscription and sets its attributes at the same time, whether on run or update, the operation will fail with an InvalidParameter error.

Because Fugue handles both operations simultaneously, the subscription creation and subscription modification API calls are contained in the same job, and that job will fail with an InvalidParameter error. However, fugue status reports success at the process level.

Fugue ignores subsequent modify attempts to modify the subscription, which allows Fugue to continue enforcing other resources without risking job failure, as discussed earlier. So even if you try to update the subscription attributes multiple times, you’ll only see InvalidParameter once.

The error message is visible with fugue ops <FID|alias>:

fugue ops snsProcess
Job Id      Instruction                          Started    Last Updated    Status
----------  -----------------------------------  ---------  --------------  ----------------
1533336266  aws.sns.set_topic_attributes         6:44pm     6:44pm          Success
1533336236  aws.sns.set_subscription_attributes  6:43pm     6:43pm          InvalidParameter
1533336236  aws.sns.subscribe                    6:43pm     6:43pm          Success
1533336236  aws.sns.create_topic                 6:43pm     6:43pm          Success

Above, aws.sns.set_subscription_attributes indicates that Fugue issued the request to modify the subscription but it was not successful.

Note how fugue status reports SUCCEEDED:

Fugue Status Report for main-user/012345678912 - Fri Aug 3 2018 6:45pm

State    Updated    Created    Account    FID/Alias    Flags    Last Message
-------  ---------  ---------  ---------  -----------  -------  --------------
Running  6:43pm     6:43pm     fugue      snsProcess   -e       SUCCEEDED

To see more information about the InvalidParameter error, you can grab the job ID from the “Job Id” column in the fugue ops output, then execute fugue ops <FID|alias> [job_id] –verbose:

fugue ops snsProcess 1533336236 --verbose

Here’s the relevant snippet of output:

- message: Invalid parameter: SubscriptionArn Reason: An ARN must have at least 6 elements, not 1
  code: InvalidParameter
  started: 1533336238
  fid: ea008ec2-dbd9-4ee2-b370-9dc31f03b6b9
  job_id: 1533336236
  status: InvalidParameter
  resource_name: ea008ec2-dbd9-4ee2-b370-9dc31f03b6b9.d930eb28-9034-5c0e-a99d-d0fe555b0749
  operation: aws.sns.set_subscription_attributes
  attempts: 1
  http_code: 400
  running: False
  last_updated: 1533336238

You can see that the full error message is “Invalid parameter: SubscriptionArn Reason: An ARN must have at least 6 elements, not 1.” This just means that AWS has not given the subscription an ARN yet because it’s pending confirmation.

Solution

To resolve the error, confirm the subscription. Precise steps will vary depending on the protocol, but you might wish to use the AWS CLI’s aws sns confirm-subscription command. Once the subscription is confirmed, AWS gives it an ARN, which allows Fugue to automatically retry the modification request. You’ll see a Success status for aws.sns.set_subscription_attributes in the list of fugue ops output. This means Fugue has successfully set the subscription attributes as declared in the composition.

Note: Subscriptions to the fugue-notification-* event notification topics are unaffected by the behavior described in this section because the topics are created during install and exist outside of any process.

Troubleshooting Ludwig errors

Error: “Specify either rootBlockDevice, volumes, and instance stores or blockDeviceMappings, but not both”

Error message:

ludwig (validation error):
  "/opt/fugue/lib/Fugue/AWS/EC2/Instance.lw" (line 284, column 17):
  Validations failed:

    284|   let resource: EC2.Instance {
    285|     keyName: keyName,
    286|     instanceType: instanceType,
    287|     monitoring: monitoring,
    288|     blockDeviceMappings: blockDeviceMappings,
    289|     tags: tags,
    290|     iamInstanceProfile: iamInstanceProfile,
    291|     ebsOptimized: ebsOptimized,
    292|     image: image,
    293|     disableApiTermination: disableApiTermination,
    294|     instanceInitiatedShutdownBehavior: instanceInitiatedShutdownBehavior,
    295|     userData: encodedUserData,
    296|     primaryNetworkInterface: primaryNetworkInterface,
    297|     secondaryNetworkInterfaces: secondaryNetworkInterfaces,
    298|     rootBlockDevice: rootBlockDevice,
    299|     volumes: volumes,
    300|     instanceStores: instanceStores,
    301|   }

    - Specify either rootBlockDevice, volumes, and instance stores or blockDeviceMappings, but not both. It is recommended to avoid using blockDeviceMappings as that field will be deprecated in a future release
      (from Fugue.AWS.EC2.Instance.newOrOldVolumeSupport)

  Stack trace:
    In call to newWithNetworkInterfaces at "/opt/fugue/lib/Fugue/AWS/EC2/Instance.lw" (line 160, column 3)
    In call to new at "MixedEBS.lw" (line 16, column 21)

Explanation:

This validation error indicates that a composition mixes the old style of specifying EBS volumes of EC2 instances (EC2.InstanceBlockDeviceMapping) with the new style (EC2.RootBlockDevice, EC2.InstanceStore, EC2.Volume) in the same binding. A composition may contain EBS specifications in the new style or the old style; it cannot contain both styles in separate bindings, and it cannot contain both styles within the same binding. The latter produces this error on compilation.

This is the binding that caused the error:

myNewStyleInstance: EC2.Instance.new {
    instanceType: EC2.T2_micro,
    subnet: mySubnet,
    securityGroups: [mySecurityGroup],
    image: "ami-e689729e", # Amazon Linux
    rootBlockDevice: myNewStyleRootVolume,
    volumes: [EC2.VolumeAttachment.new{volume: myNewStyleVolume, deviceName: "/dev/xvdb"}],
    instanceStores: [EC2.InstanceStore.new{deviceName: "/dev/xvdc", virtualName: "ephemeral0"}],
    blockDeviceMappings: [
        EC2.InstanceBlockDeviceMapping {
            deviceName: "/dev/xvda",
            ebs: EC2.EbsInstanceBlockDevice{volume: myOldStyleRootVolume},
        },
    ],
}

Note the use of the rootBlockDevice, volumes, instanceStores, and blockDeviceMappings fields.

Solution:

Select an EBS specification style and use it consistently within the composition. The new style is recommended because the old style is deprecated. However, Fugue honors the old style for backwards compatibility.

In this example, the user could remove the blockDeviceMappings field (old style), leaving rootBlockDevice, volumes, and instanceStores (new style) to ensure the composition compiles and passes validation.

For more information, and to see code examples of the old style and new style, see Writing Ludwig.

Troubleshooting Transcriber

For more information about Transcriber, including notes on service coverage, see Fugue Transcriber.

Error: “Invalid database name” when compiling transcribed composition

Error message:

ludwig (validation error):
  "RDSMaria.lw" (line 19, column 19):
  Validation failed:

    19| rds-dbinstance-1: RDS.DBInstance.new {
    ...
    28|   dbName: "my_db",
    ...
    45| }

  Invalid database name for MariaDB. Must contain 1 to 64 letters or numbers. Cannot be a word reserved by the specified database engine

  (from Fugue.AWS.RDS.Validation.MariaDB.validateDatabaseName)

Explanation:

Validations for RDS database names follow the constraints in AWS documentation. However, in certain cases the documented constraints are overly strict, and a transcribed RDS database with a valid name in the dbName field may trigger a validation error.

In the above case, the MariaDB instance name my_db was correctly transcribed but the validateDatabaseName validation in the Fugue Standard Library incorrectly produced an error message.

Solution:

Contact support@fugue.co for assistance.

Troubleshooting Tips

This section covers generalized troubleshooting in the Fugue system.

Check the CLI Log

A good place to start troubleshooting is the CLI log. There, you’ll find the full error message, along with the context surrounding it. In this case, during fugue init we’ve given the CLI an AMI ID that does not exist, and we can confirm this by checking the very end of fuguecli.log:

2018-03-16T18:22:07.03 [ fugue.screen ] INFO - Fugue Conductor AMI ID: ami-00000000.
2018-03-16T18:22:07.03 [ fugue.screen ] INFO - AWS Credentials: Environment variables
2018-03-16T18:22:07.03 [ fugue.screen ] INFO - Region: us-east-1
2018-03-16T18:22:07.03 [ fugue.screen ] INFO - Validating Fugue Conductor AMI ID ...
2018-03-16T18:22:07.03 [ fugue ] DEBUG - attempting lookup for 'ami' by environment 'FUGUE_CONDUCTOR_AMI'
2018-03-16T18:22:07.03 [ fugue ] DEBUG - environment lookup failed 'FUGUE_CONDUCTOR_AMI', attempting config lookup
2018-03-16T18:22:07.03 [ fugue.screen ] ERROR - You do not have access to the ami-00000000 conductor in region 'us-east-1'.
Traceback (most recent call last):
  File "site-packages/fugue_cli/cli.py", line 242, in invoke
  File "site-packages/click/core.py", line 1060, in invoke
  File "site-packages/fugue_cli/cli.py", line 214, in invoke
  File "site-packages/click/core.py", line 889, in invoke
  File "site-packages/click/core.py", line 534, in invoke
  File "site-packages/click/decorators.py", line 27, in new_func
  File "site-packages/fugue_cli/commands/init.py", line 65, in init
  File "site-packages/fugue_cli/commands/init.py", line 121, in base_init
  File "site-packages/fugue_cli/provider/aws.py", line 375, in validate_image
fugue_cli.amis.AMINotFound: You do not have access to the ami-00000000 conductor in region 'us-east-1'.

Should this particular error happen to you, it’s worth checking the Fugue Download Portal to make sure you have the latest AMI ID. If the AMI ID is correct, make sure you entered the region correctly. AMI IDs are only valid for one region, so if the Conductor is configured to run in a different region than your AMI, the same error is triggered.

Increase CLI Verbosity

The CLI log doesn’t append extra information for all errors, but there’s another way to find more information. Any Fugue command can be executed in verbose mode by using the global -v option. The option increases the verbosity incrementally, so -vvv shows more detail than -v. With increased verbosity, the CLI displays considerably more data that may be useful in troubleshooting, including the precise messages the CLI sends to the Conductor and the responses AWS returns.

Check CloudWatch Logs

Fugue’s CloudWatch logs may also provide useful context for troubleshooting. The /fugue/conductor log group contains all Fugue’s log streams, including three of particular note. Audit logs keep track of commands that are run in response to a user action or as part of an enforcement action. They’re more human-readable than other logs, and they’re located in the audit log stream. Broker logs show Fugue’s AWS API calls and are in fugue-broker. Manager logs show detailed plans of action and are in manager. For more information, see Logging & Notifications.

Check Ludwig Compiler Messages

When a composition fails compilation, the Ludwig compiler (lwc) produces a helpful error message. For example, if you were to use the erroneous region AWS.Us-west-3 in a composition, you’d see the following error:

ludwig (scope error):
  "HelloWorld.lw" (line 13, column 11):
  Not in scope:

    13|   region: AWS.Us-west-3,
                  ^^^^^^^^^^^^^

  Constructor not in scope: AWS.Us-west-3

  Hint:
    perhaps you mean one of:
      AWS.Us-west-2 (from Fugue.Core.AWS.Common)
      AWS.Us-west-1 (from Fugue.Core.AWS.Common)

Along with the useful error message “Constructor not in scope: AWS.Us-west-3,” the compiler suggests two types that you might have meant, AWS.Us-west-2 and AWS.Us-west-1.

If you come across an error message that seems unclear, reach out to support@fugue.co.

To learn more about Ludwig, see Ludwig Guide. Or watch our short video about lwc, Compiler Errors Are a Feature.

Email Us

Finally, you can always send an email to support@fugue.co describing your issue, and we’ll be happy to help you.

Recovery From Provider Downtimes

In the rare event that provider platforms we support experience downtimes, we will place instructions on recovery from those downtimes here. We designed Fugue to be resilient and fail gracefully in the event of dependent service outages. In short, the system is designed to keep running if possible, and save state and “wait out the storm” if it is not. In the latter case, some human intervention may be required to judge when it is safe to resume normal operation.

Entries here should be helpful not only for a particular incident, but also similar incidents that might occur on a smaller scale.

February 28th, 2017: AWS S3 Increased Error Rates

AWS experienced high error rates for the S3 service in the us-east-1 region, which impacted several other services as well.

We found that, as designed, Fugue Conductors put all processes into the Suspended state during this outage. This state means that the Conductor retains all process information, but remains hands-off. This is the safest response to a provider control plane failure, since we cannot be sure of infrastructure state or the consistency of changes we might try to make. You can read more about Fugue’s process model here.

Recovery from this state is simple. You can just use the resume command like this:

$ fugue resume -y (<alias> | <FID>)

This will resume the processes one at time. If you wish, it is safe to resume all processes at once with the following command:

$ for fid in $(fugue status --json | jq -r .[].fid); do fugue resume -y $fid; done

As S3 service recovered, some processes we resumed re-entered suspension due to continued AWS API errors. If this happens in any future incidents, we recommend you wait at least a few minutes — or until there is some more news about the incident — and try again.