Project Setup Options

Overview

Part of the process for installing and configuring Fugue includes initializing and configuring your project. If you’re interested in a complete walkthrough for installing Fugue, check out the Fugue Quick Setup. Otherwise, if you’ve installed Fugue and want to explore options for initializing your project, or review optional configuration details we have you covered.

Fugue includes two options for initializing a project. The Fugue CLI includes an optional init command that initializes a project in the current directory. It also creates a fugue.yaml file containing your configuration details. You can review the init command here.

In addition, Fugue can also initialize a project without using the init command or creating a fugue.yaml file. In this scenario a project is initialized simply by configuring your region and running fugue install. In this scenario all of the configuration options can be set as environment variables.

Either option is suitable for running Fugue. If you’re driving automation, or managing one or more configuration files, you may choose to use environment variables as a way to simplify the process of modifying your configuration. You can review additional details on Configuration File vs. Environment Variables or if you have questions or concerns reach out to us at support@fugue.co.

Fugue Proxy Support

Fugue allows you to configure the Conductor so it connects to the internet through an HTTP CONNECT proxy. This functionality can be configured as part of fugue install or fugue upgrade. The variable can be specified in the fugue.yaml or as an environmental variable (details are included below).

Note: Proxy functionality is only recommended for advanced users. If you have questions contact support@fugue.co.

Support for Installing the Conductor in an Existing VPC

Fugue supports installing the Conductor in an existing VPC instead of creating a new VPC. This functionality can be configured as part of fugue install or fugue upgrade. The variables can be specified in the fugue.yaml or as environment variables (details are included below).

See the System Architecture page for full details.

Note: Only advanced users should attempt to install the Conductor in an existing VPC. If you have questions contact support@fugue.co.

Configuring fugue.yaml and Environment Variables

Depending on your setup, you can initialize a project using the fugue init command and a fugue.yaml file, or manually by configuring environment variables. This section details the fugue.yaml file line-by-line, and provides examples of the corresponding environment variable for each value.

Note: This section details each value but if you want to skip to a simple quick reference, take a look at the table here.

Region is the only prerequisite for running Fugue commands, and it may be set via `fugue.yaml`, environment variable, command line argument, or AWS CLI config file. All other settings listed below are completely optional.

If you ran fugue init -p <profile>, then the aws block contains four fields, related to the AWS credential profile. However, if you’re using the default profile, the aws block is optional and does not appear. Visit the credentials page for more details.

aws Lines
aws:
  credentialProfile: corporate-profile
  accessKeyId: AKIAIOSFODNN7EXAMPLE
  secretKeyId: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
  sessionToken: AQoDYXdzEPT//////////wEXAMPLEtc764bNrC9SAPBSM22wDOk4x4HIZ8j4FZTwdQWLWsKWHGBuFqwAeMicRXmxfpSPfIeoIYRqTflfKD8YUuwthAx7mSEI/qkPpKPi/kMcGd [cut here for length]

The AWS credential block manages settings for communicating with AWS.

credentialProfile

Specifies the AWS profile to use for communicating with AWS.

  • Use the environment variable FUGUE_AWS_CREDENTIALPROFILE.
accessKeyId

Specifies the access key to use when communicating with AWS.

  • Use the environment variable FUGUE_AWS_ACCESSKEYID.
secretKeyId

Specifies the secret key to use when communicating with AWS.

  • Use the environment variable FUGUE_AWS_SECRETKEYID.
sessionToken

Specifies the session token to use when communicating with AWS.

  • Use the environment variable FUGUE_AWS_SESSIONTOKEN.

The conductor block governs how the CLI installs Fugue in an AWS account and interacts with it.

conductor Lines
conductor:
  region: us-east-1
  ami: ami-f00ba5
  instanceType: m4.large
  keyName: MyAwsKey
  proxyUrl: https://myproxy.com:1234
  secretsKeyId: alias/fugue/rbac/secrets
  compositionBucket: fugue-xxxxxxxxxxxx-us-east-1
  largeValueBucket: fugue-large-value-xxxxxxxxxxxx-us-east-1
  installAZ:
    - us-east-1c
    - us-east-1d
  vpcId: vpc-d78997ac
  subnetIds:
    - subnet-3ff72563
    - subnet-e0875b87
  securityGroupId: sg-4be92900
region

Specifies the AWS region in which the fugue binary operates for this project. All calls to the AWS API that the CLI makes are directed to this region. This influences all commands, including install. Example value: us-east-1

  • Note: The AWS region where the Fugue Conductor is installed must be specified prior to or during the installation of Fugue.
  • To specify the region you can either: set the region per command using fugue --region <region> <command> with the desired region; or use the environment variable FUGUE_CONDUCTOR_REGION.
ami

Specifies the Amazon Machine Image that the EC2 Fugue Conductor will launch with. Different AMIs may have different versions of the Fugue Conductor installed. This is set with the AMI ID argument during fugue install, e.g. fugue install --ami id <ami>. The fugue binary version and Conductor version are generally a matched set, and compatibility between different versions is not guaranteed, so this setting should only be modified with caution. Make sure if you do change this value the CLI and Conductor are a matched set. If you have questions reach out to support@fugue.co. Example value: ami-f00ba5.

  • Use the environment variable FUGUE_CONDUCTOR_AMI.
instanceType

Specifies the instance type that the EC2 Fugue Conductor launches with. Currently, the default size is m4.large. This value should not be changed unless specifically directed by Fugue support. Example value: m4.large

  • Use the environment variable FUGUE_CONDUCTOR_INSTANCETYPE.
keyName

Allows you to specify an SSH keypair to associate with the Fugue Conductor EC2 instance at launch. Fugue launches the Conductor with no support for inbound TCP connections of any kind by default, so to log in to the instance via SSH, you can specify a keypair with this setting, and the CLI will add a security group ingress rule allowing access to the Conductor on port 22 from the IP address where fugue install is run. Example value: MyAwsKey

  • Note: The default SSH username is: admin
  • Use the environment variable FUGUE_CONDUCTOR_KEYNAME
proxyUrl

Specifies the details of a user-defined proxy to use to connect to the cloud. Note: This functionality should only be implemented by advanced users. Contact support@fugue.co for details.

  • Use the environment variable FUGUE_CONDUCTOR_PROXYURL.
secretsKeyId

Sets the KMS key used to encrypt user secrets when saving to S3. You can specify a key by its alias or key ID. If using an alias, prepend alias/ to the name. The default value is alias/fugue/rbac/secrets. Example key ID value: 96b8bb05-42e8-49e5-aae9-d69fbc57a940.

  • Use the environment variable FUGUE_CONDUCTOR_SECRETSKEYID.
installAZ

A list of Availability Zones (AZs) to use for the Fugue Conductor installation in your account. The Fugue Conductor installation uses an Autoscaling Group that straddles two AZs to ensure availability. Normally, the fugue install command will figure out two suitable AZs for the installation on its own. However, in some narrow cases this detection mechanism can get false positives from the AWS API (causing the Fugue CloudFormation stack to fail), or you may need to specifically control the AZs that Fugue runs in. In such cases, provide a list in this field with one or two AZs for Fugue to use (you may put more in the list, but only the first two will be used). Note that if you supply only one AZ, you will see a warning in the CLI log file from the fugue install command. Example value: us-east-1c.

  • Use the environment variable FUGUE_CONDUCTOR_INSTALLAZ (e.g. FUGUE_CONDUCTOR_INSTALLAZ=us-east-1c,us-east-1d).
  • In the fugue.yaml for example:
installAZ Line
conductor:
  installAZ:
    - us-east-1c
    - us-east-1d
vpcId

Specifies the VPC where the Conductor should be installed. By default, Fugue creates a new VPC during install and upgrade. To have Fugue install the Conductor in an existing VPC, provide the VPC ID. If vpcId is used, you must also use subnetIds; securityGroupId is optional. Example value: vpc-d78997ac.

  • Use the environment variable FUGUE_CONDUCTOR_VPCID.
subnetIds

Specifies the subnets Fugue should use when installing the Conductor into an existing network (see vpcId). If subnetIds is used, you must also use vpcId; securityGroupId is optional. Example value: subnet-3ff72563,subnet-e0875b87.

  • Use the environment variable FUGUE_CONDUCTOR_SUBNETIDS (e.g., FUGUE_CONDUCTOR_SUBNETIDS=subnet-3ff72563,subnet-e0875b87).
  • In the fugue.yaml for example:
subnetIds Line
conductor:
  subnetIds:
    - subnet-3ff72563
    - subnet-e0875b87
securityGroupId

Specifies the security group Fugue should use when installing the Conductor into an existing network (see vpcId, subnetIds). Optional. Example value: sg-4be92900.

  • Use the environment variable FUGUE_CONDUCTOR_SECURITYGROUPID.

The requestTimeoutSecs field governs how long the CLI waits for a response from the Conductor for requests. Example value: 5

  • Use the environment variable FUGUE_REQUESTTIMEOUTSECS.
requestTimeoutSecs Line
requestTimeoutSecs: 5
lwcOptions

The lwcOptions field is for internal or support use only.

Warning

lwcOptions is deprecated. Instead, pass lwc options directly to these Fugue CLI commands:

The FUGUE_LWC_OPTIONS environment variable is no longer required for passing Ludwig options to the Fugue CLI.

Note: In previous versions of Fugue, the user block (outlined below) contained information about the current Fugue user. This block was added to fugue.yaml during the install process. Current versions of Fugue store this information in the credentials file, and the fields operate as follows:

user

Field is generated during fugue install. Specifies the current Fugue user (root by default). You can use the user command to change the user, or you can change it in credentials manually. Just remember to also change the corresponding secret. Example value: alice

  • Use the environment variables FUGUE_USER_USERID and FUGUE_USER_NAME.
secret

Field is generated during fugue install. A symmetric access token that the CLI uses to sign requests to the Conductor, authenticating the user. Each user must have a corresponding secret to use the CLI. To generate a secret, use the policy generate-secret <user_id> command (see policy). Note: As with any secret, the secret should be guarded carefully. Do not check your credentials file into source control. Example value: 12345abdcef

  • Use the environment variables FUGUE_USER_USERSECRET and FUGUE_USER_SECRET.
profile

The named profile associated with the active Fugue user. Refer to the details on the credentials page here. Example value: default-xxxxxxxxxxxx-us-east-1

  • Use the environment variable FUGUE_USER_PROFILE.

Warning

Do not lose your root credentials. They cannot be replaced with the policy generate-secret command. If you have lost your root credentials, see the reset-secret command or contact support@fugue.co.

The experimental block contains information about experimental user settings. These settings generally will only take effect if the FUGUE_CLI_PREVIEW environment variable is set to yes.

  • Note: In general, do not use experimental user settings unless instructed to do so. Changing these settings can have unintended consequences.
experimental Lines
experimental:
  varsSystemTable: fugue-vars-headless-store-2
varsSystemTable

Specifies the name of the Conductor Vars DynamoDB table, which Vars uses for key/value storage. Overrides the default table name, fugue-vars-headless-store. For more information, see Changing the System Vars DDB Table Prior to Installation and Changing the System Vars DDB Table During Upgrade. Example value: fugue-vars-headless-store-2

  • Use the environment variable FUGUE_EXPERIMENTAL_VARSSYSTEMTABLE.

Warning

Changing your Vars table can have severe consequences, including dropping your entire dataset. Do not change your Vars table unless you have been instructed to do so by Fugue Support.

Most of the remaining settings should not be modified during normal operation. However, for troubleshooting or experimentation, there are several settings you can change that govern how the CLI finds and interacts with the Fugue Conductor:

compositionBucket

Is the name of the bucket used to store loaded and running Fugue compositions. By default, it is calculated as a composite of the string fugue-, your AWS account number, and the value of the region field. This allows the bucket name to be globally unique, which is important since all S3 buckets across AWS share one namespace. Example value: fugue-xxxxxxxxxxxx-us-east-1.

  • Use the environment variable FUGUE_CONDUCTOR_COMPOSITIONBUCKET.
largeValueBucket

Is the name of the bucket used to store large values entered into Vars. Vars values greater than 150KB are kept in this bucket rather than in the Vars DynamoDB table. By default, the bucket name is calculated as a composite of the string fugue-large-value-, your AWS account number, and the value of the region field. Example value: fugue-large-value-xxxxxxxxxxxx-us-east-1.

  • Use the environment variable FUGUE_CONDUCTOR_LARGEVALUEBUCKET.

Quick Reference Table

The table below offers an at-a-glance overview identifying the fugue.yaml value, a brief definition, and the corresponding environment variable. Refer to the detailed section above for complete definitions, example values, and information about configuration and usage.

fugue.yaml Value Definition Environment Variable Default Value
region AWS region in which Fugue operates. FUGUE_CONDUCTOR_REGION None
ami Specifies the Amazon Machine Image for the Conductor. FUGUE_CONDUCTOR_AMI None (looked up if no AMI is specified)
instanceType Specifies the instance type of the Conductor. FUGUE_CONDUCTOR_INSTANCETYPE M4_LARGE
keyName Specify an SSH keypair at launch FUGUE_CONDUCTOR_KEYNAME None
proxyURL Specify a proxy URL for the Conductor to use to connect. FUGUE_CONDUCTOR_PROXYURL None
secretskeyID Sets the KMS key used to encrypt user secrets when saving to S3. FUGUE_CONDUCTOR_SECRETSKEYID alias/fugue/rbac/secrets
installAZ A list of Availability Zones to use for the Conductor. FUGUE_CONDUCTOR_INSTALLAZ None
vpcId Specifies the VPC where the Conductor should be installed. FUGUE_CONDUCTOR_VPCID None
subnetIds Specifies the private subnets for installing Fugue in an existing network. FUGUE_CONDUCTOR_SUBNETIDS None
securityGroupId Specifies the security group for installing Fugue in an existing network. FUGUE_CONDUCTOR_SECURITYGROUPID None
requestTimeoutSecs Governs how long the CLI waits for a Conductor response. FUGUE_REQUESTTIMEOUTSECS 900
user Specifies the current Fugue user. FUGUE_USER_USERID and FUGUE_USER_NAME None
secret Access token the CLI uses to sign requests to the Conductor for authentication. FUGUE_USER_USERSECRET and FUGUE_USER_SECRET None
profile The named profile associated with the active Fugue user. FUGUE_USER_PROFILE default-<account_id>-<region>
varsSystemTable Specifies the name of the Conductor Vars DynamoDB table, used for key/value. FUGUE_EXPERIMENTAL_VARSSYSTEMTABLE None
compositionBucket The name of the bucket used to store compositions. FUGUE_CONDUCTOR_COMPOSITIONBUCKET fugue-<account_id>-<region>
largeValueBucket The name of the bucket used to store large values entered into Vars. FUGUE_CONDUCTOR_LARGEVALUEBUCKET fugue-large-value-<account_id>-<region>
credentialProfile The AWS profile to use for communicating with AWS. FUGUE_AWS_CREDENTIALPROFILE default
accessKeyId The access key to use for communicating with AWS. FUGUE_AWS_ACCESSKEYID None
secretKeyId The secret key to use for communicating with AWS. FUGUE_AWS_SECRETKEYID None
sessionToken The session token to use for communicating with AWS. FUGUE_AWS_SESSIONTOKEN None