Full CLI reference#
Trel data pipeline automation software.
usage: trel [-h] [--output_json]
{dataset_classes,dataset_class_add,dataset_class_update,datasets,dataset_report,dataset_add,dataset_do_action,register,jobs,job_update,show_key,execute,tasks,task_report,task_update,attempts,schema,schema_add,schema_update,schema_delete,repositories,sensors,sensor_update,sensor_register,sensor_delete,credentials,credential_set,credential_delete,sla_report,sla_central_rules,sla_central_rules_update,sla_rules,sla_rule_set,sla_rule_delete,lifecycle_rules,lifecycle_rules_update,tos,published,published_add,published_delete,publisher_set,subscribed,subscribed_add,subscribed_delete,explore,user_info,tags,tag_set,templates,template_set,template_delete,incidents,incident_ignore,tunnel}
...
Positional Arguments#
- command
Possible choices: dataset_classes, dataset_class_add, dataset_class_update, datasets, dataset_report, dataset_add, dataset_do_action, register, jobs, job_update, show_key, execute, tasks, task_report, task_update, attempts, schema, schema_add, schema_update, schema_delete, repositories, sensors, sensor_update, sensor_register, sensor_delete, credentials, credential_set, credential_delete, sla_report, sla_central_rules, sla_central_rules_update, sla_rules, sla_rule_set, sla_rule_delete, lifecycle_rules, lifecycle_rules_update, tos, published, published_add, published_delete, publisher_set, subscribed, subscribed_add, subscribed_delete, explore, user_info, tags, tag_set, templates, template_set, template_delete, incidents, incident_ignore, tunnel
Named Arguments#
- --output_json
Instead of tabular format, outputs will be newline delimited JSON
Default: False
Sub-commands#
dataset_classes#
Show all dataset classes
trel dataset_classes [-h] [--hidden]
Named Arguments#
- --hidden
Show the hidden dataset classes
Default: False
dataset_class_add#
Add a dataset class
trel dataset_class_add [-h] dataset_class
Positional Arguments#
- dataset_class
This is a string that represents the type of this dataset
dataset_class_update#
Show all dataset classes
trel dataset_class_update [-h] [--description DESCRIPTION]
[--add_tag TAG | --delete_tag TAG]
[--hide | --show]
dataset_class
Positional Arguments#
- dataset_class
This is a string that represents the type of this dataset
Named Arguments#
- --description
The new description of this dataset_class
- --add_tag
Add this tag to this dataset_class
- --delete_tag
Add this tag to this dataset_class
- --hide
Hide this dataset class
Default: False
- --show
Show this dataset class in the catalog
Default: False
datasets#
Show datasets for a dataset_class
trel datasets [-h] [--all_versions_of ALL_VERSIONS_OF]
[--limit LIMIT] [--offset OFFSET] [--label LABEL]
[--instance_ts INSTANCE_TS]
[--repository REPOSITORY]
[dataset_class]
Positional Arguments#
- dataset_class
This is a string that represents the type of this dataset
Named Arguments#
- --all_versions_of
Show all versions of this dataset
- --limit
Number of entries to return
Default: 40
- --offset
Number of entries to return
Default: 0
- --label
Restrict to this label
- --instance_ts
Restrict to this instance_ts. E.g, ‘2023-04-02 00:00:00’
- --repository
Restrict to this repository
dataset_report#
Show details of a specific dataset
trel dataset_report [-h] dataset_id
Positional Arguments#
- dataset_id
dataset_add#
Insert dataset into Trel’s catalog
trel dataset_add [-h] [-s SCHEMA_VERSION] [--comment COMMENT]
[--copy_source_id COPY_SOURCE_ID | --data_source_id DATA_SOURCE_ID]
dataset_class instance_prefix instance_ts_str
label repository uri
Positional Arguments#
- dataset_class
This is a string that represents the type of this dataset.
- instance_prefix
This allows you to specify the origin of a dataset that is not time based. E.g., upload id, partner name etc.
- instance_ts_str
A datetime, with format yyyymmDD_HHMMSS, where the hours, minutes and seconds are optional. Represents the date that the data corresponds to.
- label
Represents the code that generated the data. In practice, can be the environment, like prod, qa etc.
- repository
Repositories are entities in Trel. You should provide the repository that corresponds to the location of this dataset. (See Repositories)
- uri
The URI of this dataset. Folders should end with a /. Files can be registered with a full path.
Named Arguments#
- -s, --schema_version
The schema version for this dataset. If missing, default schema will be selected. Provide empty string to force schema to NULL.
- --comment
Provide an explanation in 150 words.
- --copy_source_id
If the dataset you are entering is an exact copy of another dataset, and if the new way you are registering it is identical except the repository, then provide the original dataset id here. Only repository and uri has to be specified additionally.
- --data_source_id
If the dataset you are entering depends on another dataset, provide the ID here. Given this, dimentional parameters are optional. However if provided, all must be provided.
dataset_do_action#
Execute an action on the dataset
trel dataset_do_action [-h] [--params PARAMS] id [action]
Positional Arguments#
- id
The dataset_id that you want to operate on.
- action
The action that you wish to execute. To see a list of possibile actions based on the current state of the dataset, drop this argument.
Named Arguments#
- --params
A json representation of the parameters that accompany this action.
register#
Register a job from the YAML file
trel register [-h] registration_file
Positional Arguments#
- registration_file
The path to the registration file.
jobs#
List the jobs registered with Trel
trel jobs [-h] [--list_versions] [job_name] [reg_version]
Positional Arguments#
- job_name
The name of the job you want to see in detail. If missing, some info about all jobs are shown
- reg_version
The version of the registration you want to see
Named Arguments#
- --list_versions
Show all registrations for <job_name>
Default: False
job_update#
List the jobs registered with Trel
trel job_update [-h] [--activate | --deactivate]
[--enable_scheduler | --disable_scheduler]
[--trigger_scheduler] [--favorite] [--unfavorite]
job_name
Positional Arguments#
- job_name
The name of the job you want to update.
Named Arguments#
- --activate
Activate this job. This job can now be executed.
- --deactivate
Deactivate this job. It can no longer be executed manually or by the scheduler.
- --enable_scheduler
Once the scheduler is enabled, a new dataset added is examined and a task for this job will be created if possible. (See Scheduling Classes)
- --disable_scheduler
Disable the scheduling. New datasets will not be monitored.
- --trigger_scheduler
All the existing datasets in the catalog will be examined to create all possible tasks for this job {’ (See Scheduling Classes)’ if sphinx else ‘’}.
Default: False
- --favorite
Adds this job to the user’s favorites.
Default: False
- --unfavorite
Removes this job from the user’s favorites.
Default: True
show_key#
Shows your auth key from .trel.user.config
trel show_key [-h]
execute#
Execute a job. This will create a Task.
trel execute [-h] [-i INPUT [INPUT ...]] [-s SCHEDULE_ID] [-v]
[-w] [-f FILE_MAP [FILE_MAP ...]]
[-a ARGUMENTS_FOR_CODE [ARGUMENTS_FOR_CODE ...]]
[-l OUTPUT_LABEL] [-b BRANCH]
[-p PARAMETERS [PARAMETERS ...]]
[--allow_non_canonical_inputs] [-c COMMENT]
job_name
Positional Arguments#
- job_name
The name of the job to execute.
Named Arguments#
- -i, --input
The input datasets in format <dataset_class>,<instance_prefix>,<instance_ts>,<label> separated by space
- -s, --schedule_id
The schedule id that you wish to execute.
- -v, --verbose
Default: False
- -w, --wait_for_completion
If this flag is provided, this command will block until the Task is completed and will show a task report.
Default: False
- -f, --file_map
If your job requires additional files to execute, please provide them here. You can also override existing file specifications.
Default: []
- -a, --arguments_for_code
These are provided as it is to the user code. Only relevant for profiles where such an action is possible. lstrip() is applied for convenience.
Default: []
- -l, --output_label
All the labels in the outputs will be set to this.
- -b, --branch
The branches of all the source code other than frozen ones will be changed to the provided value.
- -p, --parameters
Parameters can be overridden here if they are allowed. See Scheduling Classes for details.
Default: []
- --allow_non_canonical_inputs
If the inputs you are providing are inconsistent with the schedule, use this parameter to force Trel to accept.
Default: False
- -c, --comment
Let others know why you are running this. This comment will also be inherited by outputs.
tasks#
List tasks
trel tasks [-h] [-j JOB_NAME] [--limit LIMIT] [--offset OFFSET]
[-t TASK_ID] [-s SCHEDULE_ID]
[status]
Positional Arguments#
- status
One of (all, pending, finished, unsuccessful, success, failed, cancelled). Default all. If you provide an invalid status, it will be treated as a job name.
Default: “all”
Named Arguments#
- -j, --job_name
Show only tasks from this job.
- --limit
How many most recent tasks to show. Default: 70.
Default: 70
- --offset
How many tasks to skip from the beginning.
Default: 0
- -t, --task_id
Show only this specific task.
- -s, --schedule_id
Show only tasks for this schedule_id.
task_report#
Show details of a task.
trel task_report [-h] task_id
Positional Arguments#
- task_id
The task id that you want the report for.
task_update#
Update a task’s internal settings.
trel task_update [-h] [--cancel]
[--hide_from_scheduler | --nohide_from_scheduler]
task_id
Positional Arguments#
- task_id
Named Arguments#
- --cancel
Default: False
- --hide_from_scheduler
- --nohide_from_scheduler
attempts#
List attempts of executing a task.
trel attempts [-h] [-j JOB_NAME] [--limit LIMIT] [-t TASK_ID]
[-s SCHEDULE_ID]
[status]
Positional Arguments#
- status
One of (all, pending, running, finished, success, failed). Default all.
Default: “all”
Named Arguments#
- -j, --job_name
Show only attempts from this job.
- --limit
How many most recent attempts to show. Default: 70.
Default: 70
- -t, --task_id
Show only for this specific task.
- -s, --schedule_id
Show only attempts for this schedule_id.
schema#
Show schema for a dataset_class
trel schema [-h] [-v SHOW_CONTENT_OF_VERSION] dataset_class
Positional Arguments#
- dataset_class
Show schema for this dataset_class
Named Arguments#
- -v, --version
Show the entire schema content for this schema version.
schema_add#
Add a schema
trel schema_add [-h] [--set_default]
dataset_class schema_version content
Positional Arguments#
- dataset_class
The dataset_class whose schema you want to add.
- schema_version
The version of the schema.
- content
The content of the schema.
Named Arguments#
- --set_default
Makes this schema version the default for the dataset_class.
Default: False
schema_update#
Update a schema
trel schema_update [-h] [--content CONTENT] [--set_default]
dataset_class schema_version
Positional Arguments#
- dataset_class
The dataset_class whose schema you want to update.
- schema_version
The version of the schema.
Named Arguments#
- --content
The content of the schema.
- --set_default
Makes this schema version the default for the dataset_class.
Default: False
schema_delete#
Delete a schema
trel schema_delete [-h] dataset_class schema_version
Positional Arguments#
- dataset_class
The dataset_class whose schema you want to delete.
- schema_version
The version of the schema to delete.
repositories#
List the repositories registered with Trel
trel repositories [-h]
sensors#
List the sensors registered with Trel
trel sensors [-h] [sensor_id]
Positional Arguments#
- sensor_id
The sensor you want to see in more detail.
sensor_update#
List the sensors registered with Trel
trel sensor_update [-h] [--enable | --disable] sensor_id
Positional Arguments#
- sensor_id
The sensor you want to see in more detail.
Named Arguments#
- --enable
Activate the sensor.
Default: False
- --disable
Activate the sensor.
Default: True
sensor_register#
Register a new sensor
trel sensor_register [-h] registration_file_name
Positional Arguments#
- registration_file_name
Path to the sensor registration file.
sensor_delete#
Delete a sensor
trel sensor_delete [-h] sensor_id
Positional Arguments#
- sensor_id
The sensor to delete
credentials#
Show your credentials
trel credentials [-h] [--download]
Named Arguments#
- --download
Download the credentials to credentials.yml
Default: False
credential_set#
Add or set a credential for yourself
trel credential_set [-h] key value
Positional Arguments#
- key
The key of the credential. See documentation for possible values.
- value
The value of the credential.
credential_delete#
Delete one of your credentials
trel credential_delete [-h] key
Positional Arguments#
- key
The credential key to delete
sla_report#
Show the SLA report for yourself or a watcher.
trel sla_report [-h] [watcher]
Positional Arguments#
- watcher
Show SLA for this watcher. Defaults to you.
sla_central_rules#
Show centrally managed SLA rules.
trel sla_central_rules [-h]
sla_central_rules_update#
Update the centrally managed SLA rules. Requires /governance permission.
trel sla_central_rules_update [-h] file_name
Positional Arguments#
- file_name
Path to file containing SLA rules.
sla_rules#
Show the individual SLA rules. By default, show the current user’s
trel sla_rules [-h] [--dataset_class [DATASET_CLASS]]
[--repository [REPOSITORY]] [--user [USER]]
Named Arguments#
- --dataset_class
Show where the dataset_class is published.
- --repository
Show all dataset_classes that are published from this repository.
- --user
Show all that this user has published
sla_rule_set#
Add an SLA rules for a dataset_class in a repository.
trel sla_rule_set [-h] dataset_class repository sla
Positional Arguments#
- dataset_class
Show where the dataset_class is published.
- repository
Show all dataset_classes that are published from this repository.
- sla
The SLA rules for this dataset.
sla_rule_delete#
Remove the SLA rule for a dataset_class in a repository.
trel sla_rule_delete [-h] dataset_class repository
Positional Arguments#
- dataset_class
Show where the dataset_class is published.
- repository
Show all dataset_classes that are published from this repository.
lifecycle_rules#
List the lifecycle rules registered with Trel
trel lifecycle_rules [-h]
lifecycle_rules_update#
Update the lifecycle rules
trel lifecycle_rules_update [-h] file_name
Positional Arguments#
- file_name
Path to file containing Lifecycle rules.
tos#
Show and sign Terms of Service
trel tos [-h] [--sign]
Named Arguments#
- --sign
Sign the terms of service.
Default: False
published#
Shows published datasets classes
trel published [-h] [--dataset_class [DATASET_CLASS]]
[--repository [REPOSITORY]] [--user [USER]]
Named Arguments#
- --dataset_class
Show where the dataset_class is published.
- --repository
Show all dataset_classes that are published from this repository.
- --user
Show all that this user has published
published_add#
Publishes a dataset_class/repo with a pre-existing SLA rule.
trel published_add [-h] dataset_class repository
Positional Arguments#
- dataset_class
Show where the dataset_class is published.
- repository
Show all dataset_classes that are published from this repository.
published_delete#
Un-publishes a dataset
trel published_delete [-h] dataset_class repository
Positional Arguments#
- dataset_class
Show where the dataset_class is published.
- repository
Show all dataset_classes that are published from this repository.
publisher_set#
Changes the publisher of a published dataset or the owner of an SLA.
trel publisher_set [-h] dataset_class repository user
Positional Arguments#
- dataset_class
Change the publisher or SLA owner for this dataset_class.
- repository
In this repository.
- user
To this user.
subscribed#
Shows subscribed datasets classes
trel subscribed [-h] [--dataset_class [DATASET_CLASS]]
[--repository [REPOSITORY]] [--user [USER]]
Named Arguments#
- --dataset_class
Show where the dataset_class is subscribed.
- --repository
Show all dataset_classes that are subscribed from this repository.
- --user
Show all that this user has subscribed
subscribed_add#
Subscribes a dataset
trel subscribed_add [-h] dataset_class repository
Positional Arguments#
- dataset_class
The dataset_class to subscribe to.
- repository
The repository where the dataset_class will be located.
subscribed_delete#
Un-subscribes a dataset
trel subscribed_delete [-h] dataset_class repository
Positional Arguments#
- dataset_class
The dataset_class to un-subscribe.
- repository
The repository where the dataset_class is located.
explore#
ChatGPT answers your queries regarding the catalog.
trel explore [-h] [--short] [--long] query
Positional Arguments#
- query
Your query. E.g., how to find unique devices seen per day?
Named Arguments#
- --short
Provide an explanation in 150 words.
Default: False
- --long
Provide an explanation in 150 words.
Default: False
user_info#
Gives the details of the user.
trel user_info [-h]
tag_set#
Add or change a tag.
trel tag_set [-h] [--display_name [DISPLAY_NAME]]
[--color [COLOR]] [--effects [EFFECTS]]
[--priority [PRIORITY]] [--hidden]
id
Positional Arguments#
- id
The id of the tag. E.g., bronze
Named Arguments#
- --display_name, -d
How to show this tag in the UI
- --color, -c
Color of this tag in #RRGGBB format
- --effects, -e
Effects to apply to the tag. E.g., metallic
- --priority, -p
Priority of this tag. E.g., 10.
Higher priority tags show up earlier in the list.
- --hidden
Hide this tag in the UI.
Default: True
templates#
Show all Job Registration Templates
trel templates [-h] [-t TEMPLATE_ID]
Named Arguments#
- -t, --template_id
The id of the template to show the content of
template_set#
Add or Update a Job Registration Template
trel template_set [-h]
template_id template_type description
content_file
Positional Arguments#
- template_id
The id of the template
- template_type
Type of template (job or sensor)
- description
Description of the template
- content_file
The file containing the template
template_delete#
Delete a Job Registration Template
trel template_delete [-h] template_id
Positional Arguments#
- template_id
The id of the template
incidents#
Show latest incidents
trel incidents [-h] [incident_id]
Positional Arguments#
- incident_id
The id of the incident to be removed
incident_ignore#
Ignore an Incident
trel incident_ignore [-h] incident_id
Positional Arguments#
- incident_id
The id of the incident to be removed
tunnel#
Set up the tunneling
trel tunnel [-h] ssh_key_path
Positional Arguments#
- ssh_key_path
Path of ssh key