Managing Processes with the Fugue API

Overview

This example shows you how to use the Fugue API to create a simple process and manage it programmatically over HTTP.

Prerequisites

You’ll need to have the Fugue CLI set up and the Conductor installed before you can run this example. If you haven’t done so yet, it just takes a few quick steps.

You’ll also need an HTTP or REST client. This example uses curl and Postman, but feel free to substitute your favorite client.

What We’ll Do In This Example

We’ll cover how to start the Fugue API server and send it HTTP requests to run a simple composition, check its status, and kill it. We’ll also demonstrate how to apply an RBAC policy to the Conductor and retrieve it.

What We’ll Have When We’re Done

We’ll be using the HelloWorld.lw composition from Hello World, Part 2: Fugue Basics, so we’ll create the same infrastructure: a Virtual Private Cloud (VPC) in AWS, complete with tags.

How Long It Will Take

About 30 minutes.

Download

You can download the source code for this example from Github:

Save it to the directory in which you ran fugue init during the Quick Setup.

Get editor plug-ins here.

Let’s Go!

Choosing an Authentication Method

Before we can get started, we need to decide how to handle authentication. API requests are granted or denied according to credential owner’s permissions in the Conductor’s RBAC policy. (For more on RBAC, or role-based access control, see Adding Users (RBAC).)

There are two authentication options:

  • By default, the Fugue API server requires the HTTP client to send Fugue credentials in each request header using Basic Authentication.
  • Or, we can launch the server with a Fugue user profile. This way, we don’t need to send credentials in the headers; in fact, authentication passed in HTTP headers is ignored.

We’ll show it both ways for the first step, but for simplicity’s sake, the rest of the walkthrough assumes that you’ve launched the API server with a profile.

The profile we’re going to use for this example is the default Fugue profile. This profile is associated with the root user, which has full administrative permissions. Since APIs are generally designed for automation and programmatic access, it’s logical to use the default Fugue profile when starting a server with --profile.

Starting the Fugue API Server

Before we can start the Fugue API server, we need to export a special environment variable to access the fugue server command:

export FUGUE_BETA_API=true

Now, let’s start up the API server. We’re going to run it with the default profile, which is named default-<AWS account number>-<region>. You can execute cat credentials to find the precise name. (The credentials file is typically located in the same directory as your fugue.yaml file, but Fugue also looks for it in the locations listed here.)

To start the Fugue API server with a profile, run the following command, replacing the profile name with your own:

fugue server run --profile default-123456789012-us-east-1

To start the Fugue API server without a profile, run the following command:

fugue server run

Note: Remember that with this authentication method, all API requests must include Basic Authentication in the header. We’ll show you an example in the section below.

Whichever method you’ve chosen, you’ll see this output indicating that the server has started on the default port, 8080:

Fugue server starting on port 8080

Once the server is up and running, you won’t see any other output. fugue server run blocks at the command line, so open up a new terminal window to continue.

Note: If you want to run the server in the background, you can execute fugue server run & or use nohup.

Running a Composition

We’ve started the server! We’re about to run a composition. But first, let’s take a look at what happens behind the scenes when you execute fugue run from the Fugue CLI (as opposed to hitting the API).

  1. The Fugue CLI calls the Ludwig compiler (lwc).
  2. lwc compiles the composition as a snapshot.
  3. The CLI then uploads the snapshot to S3.
  4. Finally, the CLI sends a request to the Conductor to create a process.

A snapshot is an archive file that includes a composition, all of the required files to compile it, and any associated environment variables. This is the compilation output that the Conductor needs in order to create a new process.

Since we’re not using the Fugue CLI to execute run, we need to take the extra step to call lwc and compile the snapshot ourselves before we can send a run request to the API.

Creating a Snapshot

Open up a new terminal window if you haven’t already. Make sure you’ve downloaded HelloWorld.lw to the directory containing your fugue.yaml file, then cd into that directory.

To compile HelloWorld.lw as a snapshot, we need to call lwc and set two options:

  • The -s snapshot option selects the compiler’s snapshot semantic backend.
  • The -o HelloWorld.tar.gz option sets the name of the output file. This name must end in .tar.gz.
lwc HelloWorld.lw -s snapshot -o HelloWorld.tar.gz

Note: For more information about lwc options, see The Ludwig Compiler.

If the composition compiles successfully, you won’t see any output on the command line and you should have a shiny, new HelloWorld.tar.gz file in your directory.

Sending the Run API Call With Curl

As you can tell from the API Reference, the API call corresponding to fugue run uses the POST method, consumes the application/x-tar media type, and accepts a query parameter, alias, for specifying the alias for a process.

If you started the server with a profile, run the following curl command:

curl -X POST http://127.0.0.1:8080/api/v0/processes?alias=HelloWorld -H 'content-type: application/x-tar' --data-binary @HelloWorld.tar.gz

This command does the following:

  • Passes the HelloWorld.tar.gz snapshot in the body of the API request.
  • Sets the content-type to application/x-tar.
  • Gives the process the alias HelloWorld.
  • Sets the HTTP method to POST.
  • Sends the request to the path http://127.0.0.1:8080/api/v0/processes.

If you started the server without a profile, run this command with your own credentials in user:secret format:

curl -X POST http://127.0.0.1:8080/api/v0/processes?alias=HelloWorld -H 'content-type: application/x-tar' --data-binary @HelloWorld.tar.gz -u root:0ioA/S1coUbEm9pAlwp9716Trrux5EXAMPLEEXAMPLE=

This adds the -u curl option, but otherwise is identical to the previous command. (We’ll just show this once; the rest of the walkthrough assumes you’ve started the server with a profile.)

Note: The credentials file is typically located in the same directory as your fugue.yaml file, but Fugue also looks for it in the locations listed here.

In either case, you’ll see output like this:

{
  "location": "/api/v0/processes/2b56d491-0c4f-485f-9e14-479381687b08",
  "process": {
    "fid": "2b56d491-0c4f-485f-9e14-479381687b08"
  }
}

Success! This means the API request was accepted and the Conductor has launched the HelloWorld process.

Sending the API Run Call With Postman

To use Postman to send the API call corresponding to run, select POST from the method dropdown menu and enter http://127.0.0.1:8080/api/v0/processes?alias=HelloWorld into the request URL field.

On the Header tab:

  • Enter the key Content-Type
  • Enter the value application/x-tar

On the Body tab:

  • Select the binary radio button
  • Select the “Choose Files” button and locate the HelloWorld.tar.gz snapshot file

On the Authorization tab (if you’ve started the API server without a Fugue user profile):

  • Set Type to Basic Auth
  • Set Username to your Fugue user (in this case, root)
  • Set Password to your user secret

Finally, hit “Send.”

You’ll see the same JSON output as you’d see with curl:

{
    "location": "/api/v0/processes/7a204e2a-5c8c-45e8-938f-8064f7fd4ec3",
    "process": {
        "fid": "7a204e2a-5c8c-45e8-938f-8064f7fd4ec3"
    }
}
A successful 'run' API call in Postman.

A successful run API call in Postman.

Checking Status

Now that we’ve successfully launched a process, let’s check its status. The API call that corresponds to status is documented here.

Assuming you’ve started the server with a profile, run the following curl command:

curl http://127.0.0.1:8080/api/v0/processes

This uses the default GET method to return the status for all running processes. Your output might look like this:

[
  {
    "account_id": "fugue-1504045452320",
    "alias": "HelloWorld",
    "created": 1504214165,
    "current_job_status": {
      "current_time": 1504217631,
      "dry_run": false,
      "job_id": 1504217617,
      "start_time": 1504217617
    },
    "fid": "7a204e2a-5c8c-45e8-938f-8064f7fd4ec3",
    "last_job_id": 1504217587,
    "last_job_status": {
      "description": "JobSucceeded",
      "dry_run": false,
      "finish_time": 1504217590,
      "job_id": 1504217587,
      "job_status": "SUCCEEDED",
      "message": "SUCCESS",
      "start_time": 1504217587,
      "status_code": 200
    },
    "last_message": "SUCCESS",
    "state": "BUSY",
    "updated": 1504214165
  }
]

You’ll receive the same response in Postman.

The output of a 'status' API call in Postman.

The output of a status API call in Postman.

Applying and Downloading an RBAC Policy

To demonstrate how to send an API request that returns application/x-tar data, we’re going to attach a dummy role-based access control (RBAC) policy to the Conductor and then retrieve it as a .tar.gz file.

Attaching the RBAC Policy

Here’s the dummy policy we’re using, SamplePolicy.lw:

composition

import Fugue.System.Policy as Policy

user1: Policy.User(userId:"user1")
user2: Policy.User(userId:"user2")

This simply creates two users, user1 and user2. Neither user has any permissions. For more information about Fugue’s RBAC feature, see Adding Users (RBAC). For a detailed walkthrough, see Using the RBAC Feature.

The API request corresponding to fugue policy rbac-attach is documented in the API Reference. We’ll need to create a snapshot for SamplePolicy.lw, just like we did for HelloWorld.lw:

lwc SamplePolicy.lw -s snapshot -o SamplePolicy.tar.gz

If you’re curious, feel free to unzip the file with tar -xzvf SamplePolicy.tar.gz and poke around. You’ll notice multiple files and folders, and your policy is located at /Sample/src/SamplePolicy.lw.

Next, we’ll use the following curl command to attach the policy:

curl -X POST http://127.0.0.1:8080/api/v0/rbac -H 'content-type: application/x-tar' --data-binary @SamplePolicy.tar.gz

This is similar to the run API call we made earlier, but the path is different, and so is the name of the snapshot.

You’ll see output like this:

{
    "result": "OK"
}

In Postman, you’ll see the same response.

The output of a 'policy rbac-attach' API call in Postman.

The output of a policy rbac-attach API call in Postman.

Downloading the Policy as a .tar.gz

To retrieve the policy, we’ll use the API request that corresponds with fugue policy rbac-get, which is documented in the API Reference.

Because this API call returns application/x-tar data, we need to use the redirect operator > with our curl command to write the output to a file. The filename must end in .tar.gz.

We’re going to name the output SamplePolicy.tar.gz and overwrite the snapshot we saved earlier and sent to the Conductor over API. (The contents are exactly the same, since the Conductor is just sending the snapshot back!)

curl http://127.0.0.1:8080/api/v0/rbac/archive > SamplePolicy.tar.gz

You’ll see output like this:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 23742  100 23742    0     0   5832      0  0:00:04  0:00:04 --:--:--  5833

Once it’s been downloaded, you can use the tar -xzvf SamplePolicy.tar.gz command to unzip the snapshot. As before, you’ll find your original policy in /SamplePolicy/src/SamplePolicy.lw.

In Postman, you’ll need to select “Send and Download” from the “Send” dropdown menu. A dialog box will prompt you to choose a name and directory for the file. Make sure you add .tar.gz to the file suffix.

The 'Send and Download' button in Postman.

The Send and Download button in Postman.

Killing the Process

We’re ready to send one final API call. This time, we are going to kill the process. We’ll use the API request that corresponds with kill, which is documented in the API Reference.

There are a couple things to note about this particular API call.

  1. The default action for the path /api/v0/processes/{process} and the method DELETE is release. This is not the same as kill. You must add a query parameter, ?kill=true, to the path if you want to terminate your infrastructure (kill) instead of just ending Fugue management of it (release).
  2. As with the Fugue CLI, you can use a Fugue ID (FID) or alias to target a process. Replace the {process} in the path with your FID or alias.
curl -X DELETE http://127.0.0.1:8080/api/v0/processes/HelloWorld?kill=true

You’ll see output like this:

{
    "correlation_id": "bdd7d6d1-0137-484b-a3b9-bf9da0c77a7e",
    "fid": "7a204e2a-5c8c-45e8-938f-8064f7fd4ec3"
}

In Postman, you’ll notice a similar response. Don’t forget to add the query parameter in the request URL field!

A successful 'kill' API call in Postman.

A successful kill API call in Postman.

Feel free to send an API call to check status again, in order to confirm that your process has been killed:

curl http://127.0.0.1:8080/api/v0/processes

If there are no processes running, the output will look like this:

[]

Shutting Down the Fugue API Server

In the course of this walkthrough, you’ve completed the following Fugue actions via API requests:

  • Created a process (equivalent to fugue run)
  • Checked its status (equivalent to fugue status)
  • Attached an RBAC policy (equivalent to fugue policy rbac-attach)
  • Retrieved an RBAC policy (equivalent to fugue policy rbac-get)
  • Killed the process (equivalent to fugue kill)

Way to go! Now it’s time to wrap up.

We’re done using the API server, so go back to the terminal window running the server and press <CTRL>-C to shut it down.

If you’ve used the & operator or nohup to launch the server in the background, you can execute pkill -f fugue to terminate the server process.

Next Steps

Want to learn more about the Fugue API? Check out Fugue REST API. If you want to dive right into making API requests, see the API Reference. For more information about RBAC, we recommend reading our walkthrough, Using the RBAC Feature. And as always, reach out to support@fugue.co with any questions.