Welcome to trel’s documentation!

Contents:

• First Time Use
  • Starting Trel platform (Adminstrator only)
  • Welcome e-mail for users
  • CLI Installation & Setup
  • Logging in to Web-UI
  • Trel Interfaces
  • Initial Status of Trel
  • Troubleshooting and Support
• Administration
  • Starting Trel platform
  • Initial Status of Trel
  • Adding data repositories
  • Configure the execution profile plugins
  • Important files
    • trel_contrib
  • Adding users
  • Adding templates
  • Backup and Restore Mechanisms
    • Backup Process
    • Restore Process
    • Configuration Parameters
    • Backup and Restore Summary
• Tutorial
  • Tutorial scenario
  • Register a server log as a dataset within Trel
    • Interpretation of the dimensions
  • Register the first job: format_log
  • Run format_log
    • Manual Run using inputs
    • Manual Run using schedule info
    • Automatic run using scheduler
  • The concept of schedule_id
  • Fixing a failed job
  • Running a pyspark job
  • Deploying fix for format_log
  • What we have learned
• Concepts
  • Datasets
  • Repositories
    • Default Output URI
  • Jobs
    • Registration file
    • Executing a Job: Tasks and Attempts
  • Execution Profiles
    • Business Logic Contract
    • Naming Datasets
    • Return Codes
    • Retry Strategies
  • Scheduling Classes
    • schedule_info
    • Prefix label selection
    • Other datasets
  • Sensors
    • Sensor creation
    • Clock based sensor
    • Registering a sensor
    • Running a sensor
  • Plugins
  • Schema Management
  • Data Quality Checks
  • Temp Table Creation and Configuration
  • Parameters
    • UI/CLI parameters
    • Platform parameters
    • Job parameters mechanism
    • Job parameters: .trel.config
    • Job parameters: Registration
    • Job parameters: User config
    • Job parameters: Execute CLI
    • Execution profile parameters
    • Parameters on the command line
  • Resource
    • Virtual Resource pool
    • Resource Specifications
  • Credential management
    • Known Credentials
    • Credential use
    • Approval rule
  • Permissions Management
  • SLA - service-level agreement
  • Lifecycle Management
    • Setting up
    • Lifecycle rules
    • Supported predicates
    • Supported actions
    • Manual operations
• Design guidelines
  • Design goals
    • Importance of execution time invariance
  • Recommendations
  • Example 1: Show tweet and stock price analytics in a dashboard
    • Bringing the data in
    • Transforming the data
    • Exporting the data
    • Summary
  • Implementation guidelines
• Plugins
  • Execution Profile Plugins
    • python
    • emr_pyspark
    • bigquery
    • athena
    • databricks_notebookjob
    • ec2_single
  • Execution APIs
    • Azure Synapse
  • Repository Plugins
    • athena: AWS Athena (Hive metastore)
    • trino: In-house Trino cluster
    • s3: AWS S3
    • bq: BigQuery
    • gs: Google Storage
  • Scheduling Class Plugins
    • BaseSchedule
    • single_instance
    • multi_period
    • periodic
  • Resource Plugins
    • ec2: EC2 Instance
  • Retry Strategy Plugins
    • default
  • Output Generator Plugins
    • default
  • Multiple repository configurations and fronzen branch
  • Source Code Downloader Plugins
    • github
    • s3
  • Resource Plugins
    • emr_spark
• Trel SDK
  • treldev
  • treldev.gcputils
  • treldev.awsutils
• How to…
  • Fix bad data due to Sensor bug
  • Fix bad data due to bad inputs
  • Fix bad data due to transformation bug
  • Use a sensor from trel_contrib
  • Delete data created by a sensor without having the sensor replace it
  • Keep storage costs low
  • Safely add a new job to production
  • Safely modify an existing job
• Full CLI reference
  • Positional Arguments
  • Named Arguments
  • Sub-commands
    • dataset_classes
    • dataset_class_add
    • dataset_class_update
    • datasets
    • dataset_report
    • dataset_add
    • dataset_do_action
    • register
    • jobs
    • job_update
    • show_key
    • execute
    • tasks
    • task_report
    • task_update
    • attempts
    • schema
    • schema_add
    • schema_update
    • schema_delete
    • repositories
    • sensors
    • sensor_update
    • sensor_register
    • sensor_delete
    • credentials
    • credential_set
    • credential_delete
    • sla_report
    • sla_central_rules
    • sla_central_rules_update
    • sla_rules
    • sla_rule_set
    • sla_rule_delete
    • lifecycle_rules
    • lifecycle_rules_update
    • tos
    • published
    • published_add
    • published_delete
    • publisher_set
    • subscribed
    • subscribed_add
    • subscribed_delete
    • explore
    • user_info
    • tags
    • tag_set
    • templates
    • template_set
    • template_delete
    • incidents
    • incident_ignore
    • tunnel

Indices and tables

• Index

• Module Index

• Search Page