Troubleshooting (Fugue Commands)

Having trouble? We’re here to help.

First, contact support@fugue.co.

You may be instructed to use the fugue support command. If so, check out the support documentation. If you’re looking for information about the actions Fugue is taking take a look at notifications.

Below, we’ve provided guidance for errors and known issues related to specific fugue commands. Troubleshooting (General) contains info about Ludwig, Transcriber, and other issues not directly tied to specific fugue commands. It also includes some general troubleshooting tips.

You can also find known issues in our release notes.

Troubleshooting fugue init

Error: Cannot access Conductor AMI

In this case the fugue init command generates the error “You do not have access to the ami-xxxxxxxx conductor.”

Error message:

$ fugue init --ami ami-xxxxxxxx us-east-1

[ fugue init ] Initializing Fugue project with the following configuration:

Fugue Conductor AMI ID: ami-xxxxxxxx.
AWS Credentials: Profile (default)
Region: us-east-1

Validating Fugue Conductor AMI ID ...
[ ERROR ] You do not have access to the ami-xxxxxxxx conductor in region 'us-east-1'.

Explanation:

  • The account associated with the current AWS profile has not been whitelisted for access to the Fugue Conductor AMI in the specified region.
    • You may be using a non-whitelisted AWS account.
    • You may have specified an unsupported region for Conductor installation. (See list of supported regions below.)
    • You may have specified a nonexistent AMI, or a Conductor AMI from a different region.

Solution(s):

  • Verify that your AWS account ID appears correctly in your list of Conductors in the Download Portal. If it does not, select the “Edit” button to update the account ID.
  • Ensure whatever profile you use is associated with a whitelisted AWS account. The AWS profile may be set via environment variables FUGUE_AWS_CREDENTIALPROFILE (which takes precedence) or AWS_DEFAULT_PROFILE. It also may be set via config file fugue.yaml by using the field credentialProfile in the aws block.
  • Double-check the AMI ID you entered. You can find the AMI ID for your account and region by logging in to the Download Portal.

You can also check out Hello World, Part 1: Fugue Quick Setup for additional information on setting up your account and the associated environmental variables.

Important Note:

The Fugue Conductor may only be installed in the following regions:

  • us-east-1
  • us-east-2
  • us-west-2
  • eu-west-1
  • us-gov-west-1

Troubleshooting fugue install

Error: `fugue.yaml` doesn’t exist

In this case the fugue install command generates the error “fugue.yaml doesn’t exist.” (Applies to Fugue v2017.11.28 and earlier)

Error message:

$> fugue install

[ ERROR ] config file /Users/test/fugue.yaml does not exist.
You can run "fugue init" to initialize your project.

Explanation:

The fugue.yaml was not created because you haven’t yet run fugue init to specify the profile and Fugue Conductor AMI you wanted to use.

Solution:

Run fugue init to specify your profile and AMI, and then run fugue install.

You can also upgrade to a more recent version of Fugue. The init command and fugue.yaml file are now optional. You can install with your default AWS profile and configuration, or use environment variables to set non-default values. For more details, see Configure Fugue Settings in the Quick Setup.

Error: `CloudFormation stack` access

In this case the fugue install command generates the errors “User: <arn> is not authorized to perform: cloudformation:DescribeStacks on resource: <arn>” and “AWS CloudFormation stack creation failed.”

Error message:

$ fugue install

[ ERROR ] There was a problem executing this command.

   Reason: An error occurred (AccessDenied) when calling the DescribeStacks operation: User: arn:aws:iam::[AWS_ACCOUNT_NUMBER]:user/test_user is not authorized to perform: cloudformation:DescribeStacks on resource: arn:aws:cloudformation:us-east-1:[AWS_ACCOUNT_NUMBER]:stack/fugue/*

Error message:

$ fugue install

[ fugue install ] Installing Fugue Conductor

Install Details:
   Conductor AMI ID: ami-xxxxxxxx
   AWS Account: test_user/[ACCOUNT NUMBER]
   Region: us-east-1

[ WARN ] Would you like to proceed with installing? [y/N]: y

Installing the Fugue Conductor into AWS account test_user/[ACCOUNT NUMBER].

FugueSubnet1                         Working...
FugueVpc                             Working...
FugueSubnet2RouteTableAssociation    Working...
...
FugueVpcGatewayAttachment            Working...
FugueAutoScalingGroup                Working...

-----------------------------------------------
Overall Progress  [#........................]    6%
[ HELP ] Exiting the install command while in progress (CTRL+C) will only stop progress tracking and *not* the install itself.
[ ERROR ] AWS CloudFormation stack creation failed.

Note: While the CLI currently indicates that (CTRL+C) will not stop the installation, we do not recommend using this command as it will interrupt the successful creation of credentials. In the event (CTRL+C) is used you can manually create your credentials using fugue support reset-secret. These recommendations will be updated in a future release.

Explanation:

  • In the first example, the profile you are using does not have the appropriate access to CloudFormation. Fugue requires CloudFormation access to create the infrastructure needed to run the Fugue Conductor.
  • In the second example, the AWS account you are using does not have sufficient permissions to install the infrastructure needed for the Fugue Conductor. In addition to CloudFormation, the account you use to install Fugue must have permissions to create, delete, and update all the infrastructure needed for the Fugue Conductor which includes any selected services and infrastructure you want Fugue to manage.

Solution:

Update the profile you are using to install Fugue with the appropriate permissions for the services you are attempting to access or create (e.g. for CloudFormation you need to include: Describe, Create, Delete and Update)

Note: Details about Fugue’s policy on AWS Permissions IAM are available here.

Error: `A previous Conductor installation failed to uninstall`

In this case the error when using the fugue install command is “A previous Conductor installation failed to uninstall.”

Error message:

$ fugue install -y

[ ERROR ] There was a problem executing this command.

   Reason: A previous Conductor installation failed to uninstall.

   Details: The following resource(s) failed to delete: [FugueResourceEventsTopic].

Explanation:

Sometimes CloudFormation doesn’t delete correctly on uninstall, leaving the stack running in us-east-1.

Solution:

Access the AWS console and go to CloudFormation in us-east-1 and delete the stack named FUGUE.

Troubleshooting fugue run

Error: `(403) Forbidden`

In this case the fugue run command generates a “(403) Forbidden” error.

Error message:

$ fugue run alarm.lw -a alarm
[ fugue run ] Running alarm.lw
Run Details:
    Alias: alarm

Compiling Ludwig file /Users/test/fugue/alarm.lw

[ OK ] Successfully compiled. No errors.

Uploading compiled Ludwig composition to S3...

[ ERROR ] There was a problem executing this command.

   Reason: An error occurred (403) when calling the HeadBucket operation: Forbidden

Error message:

fugue run HelloWorld.lw -a hello
[ fugue run ] Running /Users/test/fugue/HelloWorld.lw

Run Details:
    Account: default
    Alias: hello

Compiling Ludwig file /Users/test/fugue/HelloWorld.lw
[ OK ] Successfully compiled. No errors.

Uploading compiled Ludwig composition to S3...
[ ERROR ] unable to create S3 bucket fugue-xxxxxxxxxxxx-us-east-1 in region us-east-1: error 403

Explanation:

  • This error occurs when the account you are using doesn’t have access to the S3 bucket configured under compositionBucket in the fugue.yaml file or in the FUGUE_CONDUCTOR_COMPOSITIONBUCKET environment variable.
    • This can occur for a number of reasons. It can occur when the fugue.yaml file or FUGUE_CONDUCTOR_COMPOSITIONBUCKET environment variable is configured to use your default AWS profile instead of a named profile. If you switch the default profile in ~/.aws/credentials and then use the same folder and fugue.yaml file or FUGUE_CONDUCTOR_COMPOSITIONBUCKET value to install Fugue in the new default account it will not generate a new bucket name and will attempt to use the old bucket.
    • Another less common scenario is one where the bucket name auto-generated by Fugue already exists and is owned by a different account.

Solution:

Create a new S3 bucket in the account you are using and then change the compositionBucket field in the fugue.yaml file or the value of the FUGUE_CONDUCTOR_COMPOSITIONBUCKET environment variable to point to that S3 bucket.

Error: `Fugue has timed out`

In this case the fugue status command indicates a “Fugue has timed out” error.

Error message:

$ fugue status

[ ERROR ] Fugue has timed out waiting for a response from the server.

Explanation:

  • Fugue Conductor and client versions are matched pairs which is why we release them together. If your Conductor and client are from different releases you may experience this error.
  • This also may be caused by delays in AWS queues and signifies nothing.

Solution:

Use the fugue --version command to determine your versions of the client and CLI, along with the version of your Fugue Conductor AMI. This will allow you to determine if you are working with a matched pair. You can verify the pair via the Download Portal, or for additional assistance reach out to support@fugue.co.

Troubleshooting fugue runtime

The Fugue runtime command has the potential to trigger CloudWatch alarms that may terminate a Conductor when using a wildcard with fugue runtime set and fugue runtime unset.

Error `CloudWatch alarms`

You may see alarms in the AWS Console similar to the following example:

CloudWatch Alarms

CloudWatch Alarms.

Explanation

In some instances, issuing a fugue runtime set or fugue runtime unset command using a wildcard can set off CloudWatch alarms that terminate the Conductor. This is not an issue in all cases and is actively under review. If you encounter this issue please reach out to us via support@fugue.co.

Solution

Fugue support may advise you to work around this scenario by executing this script to turn on instance termination protection, applying your desired settings, waiting for the CloudWatch/Conductor alarms to clear (if these are triggered, in some instances they may not be triggered), and then executing the script again to turn instance protection off. Read more from AWS about enabling termination protection here.

For example, you can run the following script to enable instance protection. When it’s completed, the EC2 Management Console will show that termination protection is set to TRUE):

Note: This is a bash script and will only work with Linux and OSX.

./instance_termination.sh

You should see the following:

Found Fugue Conductor with instance-id "i-09454d39bf22e59e8"

Please select the appropriate option to enable or disable instance termination protection:
1) enable
2) disable
3) quit
Please enter your choice:

Enter 1 to enable instance termination protection. You’ll see this output:

Updating Fugue Conductor to enable termination protection...
Done
Verifying Fugue Conductor instance termination protection...
i-09454d39bf22e59e8
DISABLEAPITERMINATION    True

Apply your desired Fugue settings, for example:

fugue runtime set "**log-level" INFO

Requesting the Conductor set runtime configuration ...

Setting                                                 Value   Default
-------------------------------------------------------  -------  ---------
aws.fugue-capture.log-level                             INFO    WARNING
aws.fugue-planner-emit-instructions-go.log-level        INFO    WARNING
aws.fugue-reflector-descriptor-autoscaling.log-level    INFO    WARNING
aws.fugue-reflector-descriptor-cloudformation.log-level   INFO  WARNING
aws.fugue-reflector-descriptor-cloudfront.log-level     INFO    WARNING
[output adjusted for length]

Applying Conductor settings, this may take a couple minutes...

Wait until CloudWatch no longer contains alarms.

CloudWatch Alarms are clear

CloudWatch Alarms are clear.

Once you have verified that the alarms have cleared, run the script again to turn the instance protection off like so:

./instance_termination.sh

You should see the following output:

Found Fugue Conductor with instance-id "i-09454d39bf22e59e8"

Please select the appropriate option to enable or disable instance termination protection:
1) enable
2) disable
3) quit
Please enter your choice:

Enter 2 to disable instance termination protection. You’ll see this output:

Updating Fugue Conductor to disable termination protection...
Done
Verifying Fugue Conductor instance termination protection...
i-09454d39bf22e59e8
DISABLEAPITERMINATION    False

Note that you can verify the script’s results in the EC2 Dashboard. Refer to this page for more information on fugue runtime.

Troubleshooting fugue update

Error: “MapperFailedException: MapperFailed with code 2”

Sometimes updating a process with a composition containing typed IAM can fail with a “MapperFailedException” error.

Error message:

[ fugue update ] Updating process with Alias: iam with /Users/main-user/projects/ServiceRolePolicy.lw

Compiling Ludwig file /Users/main-user/projects/ServiceRolePolicy.lw ...
[ OK ] Successfully compiled. No errors.

Uploading compiled Ludwig composition to S3 ...
[ OK ] Successfully uploaded.

Requesting the Conductor to update process with composition ...
[ ERROR ] MapperFailedException: MapperFailed with code 2: /opt/fugue/bin/fugue-id-mapper --serialization protobuf --old-node-stream s3://fugue-123456789012-us-east-1/compositions/282db168f98510c1bfcb6aaaf9398cf66d51b168.compiled --new-node-stream s3://fugue-123456789012-us-east-1/compositions/c154131ebc31db786dd7b4c41f7feac14fdda59f.compiled --mapped-node-stream s3://fugue-123456789012-us-east-1/mapped-node-streams/3a1a2b60-0baf-4fc0-ad6c-b87a88788418/1535568950-compositions/c154131ebc31db786dd7b4c41f7feac14fdda59f.compiled --log-level WARNING --region us-east-1

Explanation:

When Ludwig is compiled, every value of a sum type becomes a node. In this case, the Conductor has encountered an internal error related to its handling of typed IAM nodes. This can sometimes occur on fugue update.

Solution:

Contact support@fugue.co for assistance.

To prevent this issue from occurring again, you can put the typed IAM policy in a separate module. Then, import the typed policy into the composition containing the rest of your infrastructure.

So, you might have the typed IAM policy myBucketPolicy in a module named BucketPolicy.lw:

import Fugue.AWS.IAM.Typed as Typed

myBucketPolicy: Typed.Policy.new {
  version: Typed.Policy.V2012-10-17,
  statements: [
    Typed.Policy.allow {
      actions: [ "s3:ListBucket" ],
      resources: [ "arn:aws:s3:::test" ],
      principals: [ Typed.Policy.aws ["123456789012"] ]
    }
  ]
}

And you’d add the line import BucketPolicy as . to the composition containing the bucket, BucketComposition.lw:

composition

import BucketPolicy as .
import Fugue.AWS as AWS
import Fugue.AWS.S3 as S3
import Fugue.AWS.IAM.Typed as Typed

bucketRepo: S3.Bucket.new {
  name: "test",
  region: AWS.Us-east-1,
  policy: Typed.Policy.toString(myBucketPolicy)
}

When you fugue update BucketComposition.lw, the Conductor will update the infrastructure without returning the “MapperFailedException” error.

Error: “ECS Services cannot be modified in-place”

The following error message shows in the output of fugue status [alias|FID]. The fugue status output will show a “Last Message” of FAILED.

Error message:

LastMessage: "400 Encountered an error while planning [There were multiple errors:\n\
  \tError processing ECS Service [my-service]: ECS Services cannot be modified in-place.\
  \ Please remove the Service first, by removing it from the composition and updating\
  \ the Fugue process [cd553a6a-793f-4344-98f6-8217dc17c26f], then create it again.]:\
  \ Planner returned an error"

Explanation:

This error message is encountered after using fugue update to add an ELB to an existing ECS service. With the exception of the mutable fields taskDefinition and deploymentConfiguration, ECS service properties cannot be changed once the service has been created, so an ECS service’s load balancer must be declared at service creation time. Attempts to update any process with an ECS service by adding an ELB will fail with the above error.

Solution:

If you want to preserve the name of the ECS service:

  • Remove the service from the composition
  • Run fugue update on the process
  • Add the service back into the composition
  • Run fugue update on the process again

This will delete the ECS service and recreate it with the load balancer attached.

Note: You can comment out lines of Ludwig by prepending a # to each line.

If you don’t need to preserve the ECS service’s name:

  • Change the name of the service in the composition
  • Run fugue update on the process

This will delete the ECS service and recreate it with the new name, with the load balancer attached.

Troubleshooting fugue upgrade

Error: “The Conductor is in the process of installing”

In this case, running fugue upgrade produces the error “The Conductor is in the process of installing.” (Applies to Fugue v2015.05.23 and earlier)

Error message:

$ fugue upgrade ami-xxxxxxxx
[ fugue upgrade ] Upgrading Conductor

[ ERROR ] There was a problem executing this command.
   Reason: The Conductor is in the process of installing.

Explanation: The Conductor has not completed its installation process, and its CloudFormation stack is still in the CREATING or CREATED stage. This can also happen if the Conductor instance has been terminated and its Auto Scaling group is in the process of launching a new Conductor.

One other possibility is that a previous Conductor was not uninstalled properly, leaving behind some artifacts.

Solution: The Conductor takes 5-15 minutes to boot once its instance has been created. To find out whether the Conductor is up and running, follow the instructions here. Once the Conductor is finished booting, you may execute upgrade.

If the Conductor still hasn’t booted after 15 minutes and you are still receiving this error message, you may need to remove artifacts left behind from a previous Conductor. To effectively remove any such artifacts, execute the following command:

fugue uninstall --force

Warning

If you use the --force option with the uninstall command while processes are active/running you will receive a warning that any existing processes and infrastructure will be orphaned and will no longer be managed by Fugue.

In addition, if you suspect that the previous Conductor installation has left infrastructure running, please email Fugue Support (support@fugue.co) before taking any further action. It is important to address these running workloads before attempting to uninstall a previous installation.