Skip to main content

CLI documentation

caution

The Octavia CLI is an alpha, unofficial CLI that won't be maintained.

Recommendation

We recommend all users leverage the official Airbyte Terraform Provider, instead of this CLI.

What is octavia CLI?

Octavia CLI is a tool to manage Airbyte configurations in YAML. It has the following features:

  • Scaffolding of a readable directory architecture that will host the YAML configs (octavia init).
  • Auto-generation of YAML config file that matches the resources' schemas (octavia generate).
  • Manage Airbyte resources with YAML config files.
  • Safe resources update through diff display and validation (octavia apply).
  • Simple secret management to avoid versioning credentials.

Why should I use octavia CLI?

A CLI provides freedom to users to use the tool in whatever context and use case they have. These are non-exhaustive use cases octavia can be convenient for:

  • Managing Airbyte configurations with a CLI instead of a web UI.
  • Versioning Airbyte configurations in Git.
  • Updating of Airbyte configurations in an automated deployment pipeline.
  • Integrating the Airbyte configuration deployment in a dev ops tooling stack: Helm, Ansible etc.
  • Streamlining the deployment of Airbyte configurations to multiple Airbyte instance.

Readers can refer to our opened GitHub issues to check the ongoing work on this project.

Table of content

Workflow

1. Generate local YAML files for sources or destinations

  1. Retrieve the definition id of the connector you want to use using octavia list command.
  2. Generate YAML configuration running octavia generate source <DEFINITION_ID> <SOURCE_NAME> or octavia generate destination <DEFINITION_ID> <DESTINATION_NAME>.

2. Edit your local YAML configurations

  1. Edit the generated YAML configurations according to your need.
  2. Use the secret management feature feature to avoid storing credentials in the YAML files.

3. Create the declared sources or destinations on your Airbyte instance

  1. Run octavia apply to create the sources and destinations

4. Generate connections

  1. Run octavia octavia generate connection --source <PATH_TO_SOURCE_CONFIG> --destination <PATH_TO_DESTINATION_CONFIG> <CONNECTION_NAME> to create a YAML configuration for a new connection.
  2. Edit the created configuration file according to your need: change the scheduling or the replicated streams list.

5. Create the declared connections

  1. Run octavia apply to create the newly declared connection on your Airbyte instance.

6. Update your configurations

Changes in your local configurations can be propagated to your Airbyte instance using octavia apply. You will be prompted for validation of changes. You can bypass the validation step using the --force flag.

Secret management

Sources and destinations configurations have credential fields that you do not want to store as plain text in your VCS. octavia offers secret management through environment variables expansion:

configuration:
password: ${MY_PASSWORD}

If you have set a MY_PASSWORD environment variable, octavia apply will load its value into the password field.

Install

Requirements

We decided to package the CLI in a docker image with portability in mind. Please install and run Docker if you are not.

As a command available in your bash profile

curl -s -o- https://raw.githubusercontent.com/airbytehq/airbyte/master/octavia-cli/install.sh | bash

This script:

  1. Pulls the octavia-cli image from our Docker registry.
  2. Creates an octavia alias in your profile.
  3. Creates a ~/.octavia file whose values are mapped to the octavia container's environment variables.

Using docker run

touch ~/.octavia # Create a file to store env variables that will be mapped the octavia-cli container
mkdir my_octavia_project_directory # Create your octavia project directory where YAML configurations will be stored.
docker run --name octavia-cli -i --rm -v my_octavia_project_directory:/home/octavia-project --network host --user $(id -u):$(id -g) --env-file ~/.octavia airbyte/octavia-cli:0.40.32

Using docker-compose

Using octavia in docker-compose could be convenient for automatic apply on start-up.

Add another entry in the services key of your Airbyte docker-compose.yml

services:
# . . .
octavia-cli:
image: airbyte/octavia-cli:latest
command: apply --force
env_file:
- ~/.octavia # Use a local env file to store variables that will be mapped the octavia-cli container
volumes:
- <path_to_your_local_octavia_project_directory>:/home/octavia-project
depends_on:
- webapp

Other commands besides apply can be run like so:

docker compose run octavia-cli <command>`

Commands reference

octavia command flags

FlagDescriptionEnv VariableDefault
--airbyte-urlAirbyte instance URL.AIRBYTE_URLhttp://localhost:8000
--airbyte-usernameAirbyte instance username (basic auth).AIRBYTE_USERNAMEairbyte
--airbyte-passwordAirbyte instance password (basic auth).AIRBYTE_PASSWORDpassword
--workspace-idAirbyte workspace id.AIRBYTE_WORKSPACE_IDThe first workspace id found on your Airbyte instance.
--enable-telemetry/--disable-telemetryEnable or disable the sending of telemetry data.OCTAVIA_ENABLE_TELEMETRYTrue
--api-http-headerHTTP Header value pairs passed while calling Airbyte's APINoneNone
--api-http-headers-file-pathPath to the YAML file that contains custom HTTP Headers to send to Airbyte's API.NoneNone

Using custom HTTP headers

You can set custom HTTP headers to send to Airbyte's API with options:

octavia --api-http-header Header-Name Header-Value --api-http-header Header-Name-2 Header-Value-2 list connectors sources

You can also use a custom YAML file (one is already created on init in api_http_headers.yaml) to declare the HTTP headers to send to the API:

headers:
Authorization: Bearer my-secret-token
User-Agent: octavia-cli/0.0.0

Environment variable expansion is available in this Yaml file

headers:
Authorization: Bearer ${MY_API_TOKEN}

Options based headers are overriding file based headers if a header is declared in both.

octavia subcommands

CommandUsage
octavia initInitialize required directories for the project.
octavia list connectors sourcesList all sources connectors available on the remote Airbyte instance.
octavia list connectors destinationList all destinations connectors available on the remote Airbyte instance.
octavia list workspace sourcesList existing sources in current the Airbyte workspace.
octavia list workspace destinationsList existing destinations in the current Airbyte workspace.
octavia list workspace connectionsList existing connections in the current Airbyte workspace.
octavia get sourceGet the JSON representation of an existing source in current the Airbyte workspace.
octavia get destinationGet the JSON representation of an existing destination in the current Airbyte workspace.
octavia get connectionGet the JSON representation of an existing connection in the current Airbyte workspace.
octavia import allImport all existing sources, destinations and connections to manage them with octavia-cli.
octavia import sourceImport an existing source to manage it with octavia-cli.
octavia import destinationImport an existing destination to manage it with octavia-cli.
octavia import connectionImport an existing connection to manage it with octavia-cli.
octavia generate sourceGenerate a local YAML configuration for a new source.
octavia generate destinationGenerate a local YAML configuration for a new destination.
octavia generate connectionGenerate a local YAML configuration for a new connection.
octavia applyCreate or update Airbyte remote resources according to local YAML configurations.

octavia init

The octavia init commands scaffolds the required directory architecture for running octavia generate and octavia apply commands.

Example:

$ mkdir my_octavia_project && cd my_octavia_project
$ octavia init
🐙 - Octavia is targetting your Airbyte instance running at http://localhost:8000 on workspace e1f46f7d-5354-4200-aed6-7816015ca54b.
🐙 - Project is not yet initialized.
🔨 - Initializing the project.
✅ - Created the following directories: sources, destinations, connections.
$ ls
connections destinations sources

octavia list connectors sources

List all the source connectors currently available on your Airbyte instance.

Example:

$ octavia list connectors sources
NAME DOCKER REPOSITORY DOCKER IMAGE TAG SOURCE DEFINITION ID
Airtable airbyte/source-airtable 0.1.1 14c6e7ea-97ed-4f5e-a7b5-25e9a80b8212
AWS CloudTrail airbyte/source-aws-cloudtrail 0.1.4 6ff047c0-f5d5-4ce5-8c81-204a830fa7e1
Amazon Ads airbyte/source-amazon-ads 0.1.3 c6b0a29e-1da9-4512-9002-7bfd0cba2246
Amazon Seller Partner airbyte/source-amazon-seller-partner 0.2.16 e55879a8-0ef8-4557-abcf-ab34c53ec460

octavia list connectors destinations

List all the destinations connectors currently available on your Airbyte instance.

Example:

$ octavia list connectors destinations
NAME DOCKER REPOSITORY DOCKER IMAGE TAG DESTINATION DEFINITION ID
Azure Blob Storage airbyte/destination-azure-blob-storage 0.1.3 b4c5d105-31fd-4817-96b6-cb923bfc04cb
Amazon SQS airbyte/destination-amazon-sqs 0.1.0 0eeee7fb-518f-4045-bacc-9619e31c43ea
BigQuery airbyte/destination-bigquery 0.6.11 22f6c74f-5699-40ff-833c-4a879ea40133
BigQuery (denormalized typed struct) airbyte/destination-bigquery-denormalized 0.2.10 079d5540-f236-4294-ba7c-ade8fd918496

octavia list workspace sources

List all the sources existing on your targeted Airbyte instance.

Example:

$ octavia list workspace sources
NAME SOURCE NAME SOURCE ID
weather OpenWeather c4aa8550-2122-4a33-9a21-adbfaa638544

octavia list workspace destinations

List all the destinations existing on your targeted Airbyte instance.

Example:

$ octavia list workspace destinations
NAME DESTINATION NAME DESTINATION ID
my_db Postgres c0c977c2-48e7-46fe-9f57-576285c26d42

octavia list workspace connections

List all the connections existing on your targeted Airbyte instance.

Example:

$ octavia list workspace connections
NAME CONNECTION ID STATUS SOURCE ID DESTINATION ID
weather_to_pg a4491317-153e-436f-b646-0b39338f9aab active c4aa8550-2122-4a33-9a21-adbfaa638544 c0c977c2-48e7-46fe-9f57-576285c26d42

octavia get source <SOURCE_ID> or <SOURCE_NAME>

Get an existing source in current the Airbyte workspace. You can use a source ID or name.

ArgumentDescription
SOURCE_IDThe source id.
SOURCE_NAMEThe source name.

Examples:

$ octavia get source c0c977c2-48e7-46fe-9f57-576285c26d42
{'connection_configuration': {'key': '**********',
'start_date': '2010-01-01T00:00:00.000Z',
'token': '**********'},
'name': 'Pokemon',
'source_definition_id': 'b08e4776-d1de-4e80-ab5c-1e51dad934a2',
'source_id': 'c0c977c2-48e7-46fe-9f57-576285c26d42',
'source_name': 'My Poke',
'workspace_id': 'c4aa8550-2122-4a33-9a21-adbfaa638544'}
$ octavia get source "My Poke"
{'connection_configuration': {'key': '**********',
'start_date': '2010-01-01T00:00:00.000Z',
'token': '**********'},
'name': 'Pokemon',
'source_definition_id': 'b08e4776-d1de-4e80-ab5c-1e51dad934a2',
'source_id': 'c0c977c2-48e7-46fe-9f57-576285c26d42',
'source_name': 'My Poke',
'workspace_id': 'c4aa8550-2122-4a33-9a21-adbfaa638544'}

octavia get destination <DESTINATION_ID> or <DESTINATION_NAME>

Get an existing destination in current the Airbyte workspace. You can use a destination ID or name.

ArgumentDescription
DESTINATION_IDThe destination id.
DESTINATION_NAMEThe destination name.

Examples:

$ octavia get destination c0c977c2-48e7-46fe-9f57-576285c26d42
{
"destinationDefinitionId": "c0c977c2-48e7-46fe-9f57-576285c26d42",
"destinationId": "18102e7c-5160-4000-841b-15e8ec48c301",
"workspaceId": "18102e7c-5160-4000-883a-30bc7cd65601",
"connectionConfiguration": {
"user": "charles"
},
"name": "pg",
"destinationName": "Postgres"
}
$ octavia get destination pg
{
"destinationDefinitionId": "18102e7c-5160-4000-821f-4d7cfdf87201",
"destinationId": "18102e7c-5160-4000-841b-15e8ec48c301",
"workspaceId": "18102e7c-5160-4000-883a-30bc7cd65601",
"connectionConfiguration": {
"user": "charles"
},
"name": "string",
"destinationName": "string"
}

octavia get connection <CONNECTION_ID> or <CONNECTION_NAME>

Get an existing connection in current the Airbyte workspace. You can use a connection ID or name.

ArgumentDescription
CONNECTION_IDThe connection id.
CONNECTION_NAMEThe connection name.

Example:

$ octavia get connection c0c977c2-48e7-46fe-9f57-576285c26d42
{
"connectionId": "c0c977c2-48e7-46fe-9f57-576285c26d42",
"name": "Poke To PG",
"namespaceDefinition": "source",
"namespaceFormat": "${SOURCE_NAMESPACE}",
"prefix": "string",
"sourceId": "18102e7c-5340-4000-8eaa-4a86f844b101",
"destinationId": "18102e7c-5340-4000-8e58-6bed49c24b01",
"operationIds": [
"18102e7c-5340-4000-8ef0-f35c05a49a01"
],
"syncCatalog": {
"streams": [
{
"stream": {
"name": "string",
"jsonSchema": {},
"supportedSyncModes": [
"full_refresh"
],
"sourceDefinedCursor": false,
"defaultCursorField": [
"string"
],
"sourceDefinedPrimaryKey": [
[
"string"
]
],
"namespace": "string"
},
"config": {
"syncMode": "full_refresh",
"cursorField": [
"string"
],
"destinationSyncMode": "append",
"primaryKey": [
[
"string"
]
],
"aliasName": "string",
"selected": false
}
}
]
},
"schedule": {
"units": 0,
"timeUnit": "minutes"
},
"status": "active",
"resourceRequirements": {
"cpu_request": "string",
"cpu_limit": "string",
"memory_request": "string",
"memory_limit": "string"
},
"sourceCatalogId": "18102e7c-5340-4000-85f3-204ab7715801"
}
$ octavia get connection "Poke To PG"
{
"connectionId": "c0c977c2-48e7-46fe-9f57-576285c26d42",
"name": "Poke To PG",
"namespaceDefinition": "source",
"namespaceFormat": "${SOURCE_NAMESPACE}",
"prefix": "string",
"sourceId": "18102e7c-5340-4000-8eaa-4a86f844b101",
"destinationId": "18102e7c-5340-4000-8e58-6bed49c24b01",
"operationIds": [
"18102e7c-5340-4000-8ef0-f35c05a49a01"
],
"syncCatalog": {
"streams": [
{
"stream": {
"name": "string",
"jsonSchema": {},
"supportedSyncModes": [
"full_refresh"
],
"sourceDefinedCursor": false,
"defaultCursorField": [
"string"
],
"sourceDefinedPrimaryKey": [
[
"string"
]
],
"namespace": "string"
},
"config": {
"syncMode": "full_refresh",
"cursorField": [
"string"
],
"destinationSyncMode": "append",
"primaryKey": [
[
"string"
]
],
"aliasName": "string",
"selected": false
}
}
]
},
"schedule": {
"units": 0,
"timeUnit": "minutes"
},
"status": "active",
"resourceRequirements": {
"cpu_request": "string",
"cpu_limit": "string",
"memory_request": "string",
"memory_limit": "string"
},
"sourceCatalogId": "18102e7c-5340-4000-85f3-204ab7715801"
}

octavia import all

Import all existing resources (sources, destinations, connections) on your Airbyte instance to manage them with octavia-cli.

Examples:

$ octavia import all
🐙 - Octavia is targetting your Airbyte instance running at http://localhost:8000 on workspace b06c6fbb-cadd-4c5c-bdbb-710add7dedb9.
✅ - Imported source poke in sources/poke/configuration.yaml. State stored in sources/poke/state_b06c6fbb-cadd-4c5c-bdbb-710add7dedb9.yaml
⚠️ - Please update any secrets stored in sources/poke/configuration.yaml
✅ - Imported destination Postgres in destinations/postgres/configuration.yaml. State stored in destinations/postgres/state_b06c6fbb-cadd-4c5c-bdbb-710add7dedb9.yaml
⚠️ - Please update any secrets stored in destinations/postgres/configuration.yaml
✅ - Imported connection poke-to-pg in connections/poke_to_pg/configuration.yaml. State stored in connections/poke_to_pg/state_b06c6fbb-cadd-4c5c-bdbb-710add7dedb9.yaml

You know have local configuration files for all Airbyte resources that were already existing. You need to edit any secret values that exist in these configuration files as secrets are not imported. You can edit the configuration files and run octavia apply to continue managing them with octavia-cli.

octavia import destination <DESTINATION_ID> or <DESTINATION_NAME>

Import an existing destination to manage it with octavia-cli. You can use a destination ID or name.

ArgumentDescription
DESTINATION_IDThe destination id.
DESTINATION_NAMEThe destination name.

octavia import source <SOURCE_ID> or <SOURCE_NAME>

Import an existing source to manage it with octavia-cli. You can use a source ID or name.

ArgumentDescription
SOURCE_IDThe source id.
SOURCE_NAMEThe source name.

Examples:

$ octavia import source poke
🐙 - Octavia is targetting your Airbyte instance running at http://localhost:8000 on workspace 75658e4f-e5f0-4e35-be0c-bdad33226c94.
✅ - Imported source poke in sources/poke/configuration.yaml. State stored in sources/poke/state_75658e4f-e5f0-4e35-be0c-bdad33226c94.yaml
⚠️ - Please update any secrets stored in sources/poke/configuration.yaml

You know have local configuration file for an Airbyte source that was already existing. You need to edit any secret value that exist in this configuration as secrets are not imported. You can edit the configuration and run octavia apply to continue managing it with octavia-cli.

octavia import destination <DESTINATION_ID> or <DESTINATION_NAME>

Import an existing destination to manage it with octavia-cli. You can use a destination ID or name.

ArgumentDescription
DESTINATION_IDThe destination id.
DESTINATION_NAMEThe destination name.

Examples:

$ octavia import destination pg
🐙 - Octavia is targetting your Airbyte instance running at http://localhost:8000 on workspace 75658e4f-e5f0-4e35-be0c-bdad33226c94.
✅ - Imported destination pg in destinations/pg/configuration.yaml. State stored in destinations/pg/state_75658e4f-e5f0-4e35-be0c-bdad33226c94.yaml
⚠️ - Please update any secrets stored in destinations/pg/configuration.yaml

You know have local configuration file for an Airbyte destination that was already existing. You need to edit any secret value that exist in this configuration as secrets are not imported. You can edit the configuration and run octavia apply to continue managing it with octavia-cli.

octavia import connection <CONNECTION_ID> or <CONNECTION_NAME>

Import an existing connection to manage it with octavia-cli. You can use a connection ID or name.

ArgumentDescription
CONNECTION_IDThe connection id.
CONNECTION_NAMEThe connection name.

Examples:

$ octavia import connection poke-to-pg
🐙 - Octavia is targetting your Airbyte instance running at http://localhost:8000 on workspace 75658e4f-e5f0-4e35-be0c-bdad33226c94.
✅ - Imported connection poke-to-pg in connections/poke-to-pg/configuration.yaml. State stored in connections/poke-to-pg/state_75658e4f-e5f0-4e35-be0c-bdad33226c94.yaml
⚠️ - Please update any secrets stored in connections/poke-to-pg/configuration.yaml

You know have local configuration file for an Airbyte connection that was already existing. N.B.: You first need to import the source and destination used by the connection. You can edit the configuration and run octavia apply to continue managing it with octavia-cli.

octavia generate source <DEFINITION_ID> <SOURCE_NAME>

Generate a YAML configuration for a source. The YAML file will be stored at ./sources/<resource_name>/configuration.yaml.

ArgumentDescription
DEFINITION_IDThe source connector definition id. Can be retrieved using octavia list connectors sources.
SOURCE_NAMEThe name you want to give to this source in Airbyte.

Example:

$ octavia generate source d8540a80-6120-485d-b7d6-272bca477d9b weather
✅ - Created the source template for weather in ./sources/weather/configuration.yaml.

octavia generate destination <DEFINITION_ID> <DESTINATION_NAME>

Generate a YAML configuration for a destination. The YAML file will be stored at ./destinations/<destination_name>/configuration.yaml.

ArgumentDescription
DEFINITION_IDThe destination connector definition id. Can be retrieved using octavia list connectors destinations.
DESTINATION_NAMEThe name you want to give to this destination in Airbyte.

Example:

$ octavia generate destination 25c5221d-dce2-4163-ade9-739ef790f503 my_db
✅ - Created the destination template for my_db in ./destinations/my_db/configuration.yaml.

octavia generate connection --source <path-to-source-configuration.yaml> --destination <path-to-destination-configuration.yaml> <CONNECTION_NAME>

Generate a YAML configuration for a connection. The YAML file will be stored at ./connections/<connection_name>/configuration.yaml.

OptionRequiredDescription
--sourceYesPath to the YAML configuration file of the source you want to create a connection from.
--destinationYesPath to the YAML configuration file of the destination you want to create a connection to.
ArgumentDescription
CONNECTION_NAMEThe name you want to give to this connection in Airbyte.

Example:

$ octavia generate connection --source sources/weather/configuration.yaml --destination destinations/my_db/configuration.yaml weather_to_pg
✅ - Created the connection template for weather_to_pg in ./connections/weather_to_pg/configuration.yaml.

octavia apply

Create or update the resource on your Airbyte instance according to local configurations found in your octavia project directory. If the resource was not found on your Airbyte instance, apply will create the remote resource. If the resource was found on your Airbyte instance, apply will prompt you for validation of the changes and will run an update of your resource. Please note that if a secret field was updated on your configuration, apply will run this change without prompt.

OptionRequiredDescription
--fileNoPath to the YAML configuration files you want to create or update.
--forceNoRun update without prompting for changes validation.

Example:

$ octavia apply
🐙 - weather exists on your Airbyte instance, let's check if we need to update it!
👀 - Here's the computed diff (🚨 remind that diff on secret fields are not displayed):
E - Value of root['lat'] changed from "46.7603" to "45.7603".
❓ - Do you want to update weather? [y/N]: y
✍️ - Running update because a diff was detected between local and remote resource.
🎉 - Successfully updated weather on your Airbyte instance!
💾 - New state for weather stored at ./sources/weather/state_<workspace_id>.yaml.
🐙 - my_db exists on your Airbyte instance, let's check if we need to update it!
😴 - Did not update because no change detected.
🐙 - weather_to_pg exists on your Airbyte instance, let's check if we need to update it!
👀 - Here's the computed diff (🚨 remind that diff on secret fields are not displayed):
E - Value of root['schedule']['timeUnit'] changed from "days" to "hours".
❓ - Do you want to update weather_to_pg? [y/N]: y
✍️ - Running update because a diff was detected between local and remote resource.
🎉 - Successfully updated weather_to_pg on your Airbyte instance!
💾 - New state for weather_to_pg stored at ./connections/weather_to_pg/state_<workspace_id>.yaml.

Contributing

  1. Please sign up to Airbyte's Slack workspace and join the #octavia-cli. We'll sync up community efforts in this channel.
  2. Pick an existing GitHub issues or open a new one to explain what you'd like to implement.
  3. Assign the GitHub issue to yourself.
  4. Fork Airbyte's repo, code and test thoroughly.
  5. Open a PR on our Airbyte repo from your fork.

Developing locally

  1. Build the project locally (from the root of Airbyte's repo): SUB_BUILD=OCTAVIA_CLI ./gradlew build # from the root directory of the repo.
  2. Install Python 3.8.12. We suggest doing it through pyenv.
  3. Create a virtualenv: python -m venv .venv.
  4. Activate the virtualenv: source .venv/bin/activate.
  5. Install dev dependencies: pip install -e .\[tests\].
  6. Install pre-commit hooks: pre-commit install.
  7. Run the unittest suite: pytest --cov=octavia_cli. Note, a local version of airbyte needs to be running (e.g. docker compose up from the root directory of the project)
  8. Make sure the build passes (step 0) before opening a PR.

Telemetry

This CLI has some telemetry tooling to send Airbyte some data about the usage of this tool. We will use this data to improve the CLI and measure its adoption. The telemetry sends data about:

  • Which command was run (not the arguments or options used).
  • Success or failure of the command run and the error type (not the error payload).
  • The current Airbyte workspace id if the user has not set the anonymous data collection on their Airbyte instance.

You can disable telemetry by setting the OCTAVIA_ENABLE_TELEMETRY environment variable to False or using the --disable-telemetry flag.