TrimbleCloud.Processing

Processing Framework SDK for Python

The Processing Framework is the Trimble Cloud Platform system for constructing operations, procedures, and executions for processing.

Getting Started

Installation

This SDK is available as a NuGet package from Trimble Artifactory. To install, use the following command:

pip install TrimbleCloud.Processing

Processing Endpoints

Environment	Base URL Endpoint
Stage	https://cloud.stage.api.trimblecloud.com/Processing/api/1/api/
Production	https://cloud.api.trimble.com/Processing/api/1/api/

Create Processing Client

To create a client object to access the Processing framework, you will need a processing endpoint and a token provider. The Processing client can use token providers from the trimble-id sdk to authenticate and access the processing endpoint.

For more details on token providers, refer to the Trimble Identity Python SDK Documentation.

from trimble.id import ClientCredentialTokenProvider, OpenIdEndpointProvider, BearerTokenHttpClientProvider

from TrimbleCloud.Processing import ProcessingClient

# We first setup a TID provider for communications with our authorization server TID
endpoint_provider = OpenIdEndpointProvider("https://stage.id.trimblecloud.com/.well-known/openid-configuration")

# We then set up a token provider to handle the token exchange with TID
token_provider = ClientCredentialTokenProvider(endpoint_provider=endpoint_provider, client_id="<CLIENT_ID>", client_secret="<CLIENT_SECRET>").with_scopes(["<SCOPES>"])

# Create processing client
processing_client = ProcessingClient(token_provider=token_provider, base_url="<PROCESSING_BASE_URL>")

Operations

• Operations are the basic building blocks of processing.
• An Operation exposes a single function of a backend Processing Engine such as reprojection, writing a file to data ocean, or transforming one type of data into another.

List

Gets Operation objects that the user has access to view.

query_params = { "per_page": 5, "page": 1 } # Optional
list = await processing_client.operations.list(query_params)

List (Sync)

List all operations that the user has access to view.

NOTE: This method abstracts the complexity of pagination and provides an iterable interface to retrieve all items.

operations = client.operations.list_sync()
for operation in operations:
    print(operation)

Create

Create a new operation.

operation = await processing_client.operations.create(
    identifier="dataocean_download_path",
    version=10,
    deployment_identifier="dataoceanoperations",
    name="Data Ocean Download by Path",
    description="Download a file or directory of files from Data Ocean.",
    shared_with=[ { "account_id": "f0e1d2c3-b4a5-6789-0fed-cba987654321", "client_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890" } ],
    output_parameters=[
        {
            "output_parameter_identifier": {
                "type": "string",
                "description": "a sample output parameter",
                "name": "sample output parameter",
                "optional": False
            }
        }
    ],
    parameters=[
        {
            "parameter_identifier": {
                "type": "string",
                "description": "The Dropbox access token used to download the file",
                "name": "Dropbox Access Token",
                "optional": True,
                "encrypted": False
            }
        }
    ],
    inputs=[
        {
            "input_identifier": {
                "data_types": [
                "*"
                ],
                "description": "Anything goes",
                "name": "first input",
                "optional": True
            }
        }
    ],
    outputs=[
        {
            "output_identifier": {
                "data_type": "*",
                "description": "this output is x",
                "name": "first output port"
            }
        }
    ]
)

Create Operation Parameters

Parameter	Description	Type	Required
identifier	Unique identifier string for the Operation. Must be between 8 to 256 characters, start with a letter, and can contain only lowercase letters, numbers, underscores, and hyphens.	string	Required
version	The version of the Operation. Operations can be versioned to facilitate lifecycle events of engines and Procedures.	integer	Required
deployment_identifier	The identifier of the Deployment that supports this Operation. The Deployment is the abstraction layer that accommodates the ability to create a new Engine and expose it through a Deployment so there is no need to change existing Operations and Procedures.	string	Required
regions	Regions where this Operation can run. Inherited from the linked Deployment. See Regions for region identifier formats and availability by API environment.	List of strings
public	The state of the Operation’s publication. If true, the Operation is available for public use.	boolean
name	Human-readable name of the Operation.	string
description	A description of an Operation and its function.	string
engine_name	The name of the backend engine that supports this Operation. LEGACY is used for Engine’s that have already been established in the system prior to the creation of the Engines APIs.	string
shared_with	List of identities with whom this Operation has been shared. Each entry must contain an account_id and optionally a client_id. An entry with only account_id grants access to all clients in that account. Sending an array replaces the existing list.	List of SharedWithIdentity
parameters	A list of parameter objects attached to this Operation.	dict
output_parameters	A list of parameter objects that define values to be output from an Operation.	dict
inputs	Named inputs for the Operation.	dict
outputs	Named inputs for the Operation.	dict
dynamic_output	Indicates whether or not the Output content is considered dynamic or not. In all cases except for the Switcher Operation, this will be false.	boolean
deprecation_message	Optional string parameter.	string
retired_after	Optional string parameter.	string

Refer this link for more details on parameters.

Get

Get an operation by ID.

operation = await processing_client.operations.get(id="<OPERATION_ID>")

Update

Update an operation.

NOTE: An Operation can only be updated when it is in a MUTABLE state (prior to approval).

operation = await processing_client.operations.update(id="<OPERATION_ID>", name="New name", description="A new description for the operation.")

Delete

Delete an operation.

await processing_client.operations.delete(id="<OPERATION_ID>")

Approve

Approve an operation.

operation = await processing_client.operations.approve(id="<OPERATION_ID>")

Clone

Clone an operation.

operation = await processing_client.operations.clone(id="<OPERATION_ID>")

Publish

Publish an operation.

operation = await processing_client.operations.publish(id="<OPERATION_ID>")

Retire

Retire an operation.

operation = await processing_client.operations.retire(id="<OPERATION_ID>")

Procedures

• A Procedure defines a processing workflow in the Processing Framework.
• It encapsulates a set of Operations (one or multiple), describes the workflow’s parameters, and defines parameter and input/output mappings between Operations.

List

Get a list of procedures.

query_params = { "per_page": 5, "page": 1 } # Optional
list = await processing_client.procedures.list(query_params)

List (Sync)

List all procedures that the user has access to view.

NOTE: This method abstracts the complexity of pagination and provides an iterable interface to retrieve all items.

procedures = client.procedures.list_sync()
for procedure in procedures:
    print(procedure)

Create

Create a new procedure.

procedure = await processing_client.procedures.create(
        identifier="pdf_proc_test",
        version=1,
        default_region="aws-us1",
        operations={
            "download_file": {
                "identifier": "dataocean_read",
                "version": 2,
                "name": "Download file",
                "description": "download a file from Trimble Data Ocean",
                "parameters": {
                    "target": {
                        "source": "INPUT_FILE_ID"
                    }
                }
            }
        },
        name="PDF Proc Example",
        description="Example procedure to take DO image, convert to PDF, load to DO",
        parameters={
            "INPUT_FILE_ID": {
                "name": "Input file id",
                "description": "uuid of the input file"
            }
        }
    )

Create Procedure Parameters

Parameter	Description	Type	Required Fields
identifier	Unique identifier string for the Procedure. Must be between 8 to 256 characters, start with a letter, and can contain only lowercase letters, numbers, underscores, and hyphens.	string	Required
version	The version of the Procedure.	integer	Required
default_region	Region name the Procedure should execute in by default.	string	Required
operations	Mapping of Procedure names and definitions.This is where the user specifies which Procedures make up the Procedure and how those Operations chain together. 1 or many Procedures can be combined in to a procedure. All Procedures in a Procedure must share at least one common Region.	dict	Required
name	Human-readable name of the Procedure.	string
description	A description of an Procedure and its function.	string
tags	List of tags (as strings) associated with the Procedure.	string
documentation_url	Optional URL to reference external documentation for the Procedure.	string
shared_with	List of identities with whom this Procedure has been shared. Each entry must contain an account_id and optionally a client_id. An entry with only account_id grants access to all clients in that account. Sending an array replaces the existing list.	List of SharedWithIdentity
parameters	Parameter mappings for the Procedures contained within the Procedure.	dict

Refer this link for more details on parameters.

Get

Get a procedure by ID.

procedure = await processing_client.procedures.get(id="<PROCEDURE_ID>")

Update

Update a Procedure object.

A Procedure can only be updated after its creation but before its approval. i.e. It must have a status of MUTABLE and an execution_status of NOT_READY. Once a Procedure has been approved, only its shared_with, documentation_url, and default_region attributes can be updated.

procedure = await processing_client.procedures.update(id="<PROCEDURE_ID>", description="Description Updated Partial Update - This Procedure tests the tzf_to_las operation.")

Delete

Delete a procedure.

await processing_client.procedures.delete(id="<PROCEDURE_ID>")

Approve

Approve a procedure.

procedure = await processing_client.procedures.approve(id="<PROCEDURE_ID>")

Publish

Publish a procedure.

procedure = await processing_client.procedures.publish(id="<PROCEDURE_ID>")

Retire

Retire a procedure.

procedure = await processing_client.procedures.retire(id="<PROCEDURE_ID>")

Executions

• An Execution, or ‘Procedure Execution’, represents a single request for processing data against a Procedure.
• Executions specify the source data and parameter values used in processing of a desired Procedure.

List

Get a list of executions.

query_params = { "per_page" : 5 , "page" : 1 } # Optional
list = await processing_client.executions.list(query_params)

List (Sync)

List all executions that the user has access to view.

NOTE: This method abstracts the complexity of pagination and provides an iterable interface to retrieve all items.

executions = client.executions.list_sync()
for execution in executions:
    print(execution)

Create

Creates and starts a new execution in the system.

execution = await processing_client.executions.create(
    procedure_id="<PROCEDURE_ID>",
    data_export=False,
    region="aws-us1",
)

Create Execution Parameters

Parameter	Description	Type	Required Fields
procedure_id	The unique identifier of the target Procedure.	string	Required
data_export	Whether or not the execution’s data will be available for export. Default to false.	boolean
region	The location(s) where this Execution can run. Must be a name seen in the Procedure’s regions list. If not provided, then default_region of the Procedure is used. See Regions for region identifier formats and availability by API environment.	string
parameters	Mapping of Procedure-specified parameter names to parameter values. This is where the user specifies Procedure specific parameter field names and values for the Execution.	dict
procedure_identifier	The identifier of the target Procedure for Execution.	string
procedure_version	The version of the target Procedure for Execution.	integer
metadata	User-specified metadata to be associated with the Execution.	string
bound_parameters	Mapping of Operation-specified output parameter names to output parameter values. This is where Operation output parameters are written once an Operation within the Execution completes.	dict

Refer this link for more details on parameters.

Get

Get an execution by ID.

execution = await processing_client.executions.get(id="<EXECUTION_ID>")

List Activities

Get the list of activities tied to an execution.

query_params = { "per_page" : 10 , "page" : 1 } # Optional
activities_list = await processing_client.executions.list_activities(id="<EXECUTION_ID>", query_params=query_params)

List Activities (Sync)

List all activities tied to an execution.

NOTE: This method abstracts the complexity of pagination and provides an iterable interface to retrieve all items.

activities = client.executions.list_activities_sync(execution_id="<EXECUTION_ID>")
for activity in activities:
    print(activity)

Get Data

Get execution data.

execution = await processing_client.executions.get_data(id="<EXECUTION_ID>", output_path="<OUTPUT_PATH>")

Get Logs

Get execution logs via DataOcean.

execution = await processing_client.executions.get_logs(id="<EXECUTION_ID>", output_path="<OUTPUT_PATH>")

List Events

Get the list of events tied to an execution.

query_params = { "per_page" : 10 , "page" : 1 } # Optional
events_list = await processing_client.executions.list_events(id="<EXECUTION_ID>", query_params=query_params)

List Events (Sync)

List all events tied to an execution.

NOTE: This method abstracts the complexity of pagination and provides an iterable interface to retrieve all items.

events = client.executions.list_events_sync(execution_id="<EXECUTION_ID>")
for event in events:
    print(event)

Engines

• Engines are low-level resource in the Processing Framework that represent a container image stored in the service.
• They provide the interface for how to define and configure a container with direct, optional, relation to how the code in the container is written.

List

Get a list of engines.

query_params = { "per_page" : 5 , "page" : 1 } # Optional
list = await processing_client.engines.list(query_params)

List (Sync)

List all engines available to the requesting user.

NOTE: This method abstracts the complexity of pagination and provides an iterable interface to retrieve all items.

engines = client.engines.list_sync()
for engine in engines:
    print(engine)

Create

Create a new engine.

engine = await processing_client.engines.create(
    identifier="example-engine",
    tag="v1",
    ingestion_type="acr_token",
    os="linux",
    regions=[ "aws-us1" ],
    name="Example Engine",
    description="A description the of the engine."
)

Create Engine Parameters

Parameter	Description	Type	Required Fields
identifier	Engine identifier, unique within the system. It must 32 characters or less, start with a-z, end with an alphanumeric value, and can only contain the characters a-z, 0-9, and -.	string	Required
tag	An Engine tag must start with [a-z], end with an alphanumeric value, and can only contain [a-z, 0-9, ‘-’].	string	Required
ingestion_type	The method of uploading a container image into our system. Currently the only option is a temporary ACR token upload. Allowed: acr_token	enum	Required
os	Which OS for the Engine to use. Allowed: linux | windows	enum	Required
regions	The location(s) where this Engine can run. Regions are defined on the Engine. See Regions for region identifier formats and availability by API environment.	List of strings	Required
name	Engine name. Metadata only. Constraints: Max 64 chars	string
description	A description of the Engine and its function.	string
os_build	Specify which OS build to use. This is a required parameter for Windows Engines only. At this time, the Engines API directly supports Windows Server 2019 (10.0.17763), and can optionally support Windows Server 2022 (10.0.20348). Allowed: 10.0.17763 | 10.0.20348	enum
env_vars	Environment variables that are stored and ingested into the Engine’s container as literal plain-text values.	list
secret_variables	Secret variables	list
secret_files	Secret files	list
public	The state of the Engine’s publication. Publishing Engines is not currently supported.	boolean

Refer this link for more details on parameters.

Get

Get an engine by ID.

engine = await processing_client.engines.get(id="<ENGINE_ID>")

Update

Update an engine. Only the name and description can be updated.

engine = await processing_client.engines.update(
    id="<ENGINE_ID>",
    name="A New Name",
    description="A new description."
)

Retire

Retire an engine.

engine = await processing_client.engines.retire(id="<ENGINE_ID>")

Push Image

Push your source image to Processing AWS Container Registry for deployment.

The source image is a local image referenced by name. It must be the full image name in the format of SOURCE_IMAGE[:TAG]. If the tag is not specified, it will use latest by default.

response = await client.engines.push_image(
    id="<ENGINE_ID>",
    source_image_name="source_image_name:tag"
)

for line in response:
    print(line)

Deployments

• The concept of Deployments provides the configurable interface for the following topics:
  • Operation to Engine Image relationship
  • Scaling
  • Compute resource definition

List

Get a list of deployments.

query_params = { "per_page" : 5 , "page" : 1 } # Optional
list = await processing_client.deployments.list(query_params)

List (Sync)

List all deployments available to the requesting user.

NOTE: This method abstracts the complexity of pagination and provides an iterable interface to retrieve all items.

deployments = client.deployments.list_sync()
for deployment in deployments:
    print(deployment)

Create

Create a new deployment.

deployment = await processing_client.deployments.create(
    identifier="example-deployment",
    regions=[ "aws-us1" ],
    scaling={ "min": 0, "max": 2 },
    computing={ "cpu": 0.5, "memory": 512 },
    name="Example Deployment",
    description="A description the of the Deployment."
)

Create Deployment Parameters

Parameter	Description	Type	Required Fields
identifier	A Deployment Identifier is unique within the system. The value must start with [a-z], end with an alphanumeric value, and can only contain [a-z, 0-9, ‘-’]. Constraints: 1 to 32 chars	string	Required
regions	The location(s) where this Deployment runs. Must be a subset of the attached Engine’s regions. See Regions for region identifier formats and availability by API environment.	List of strings	Required
scaling	The configuration object to define minimum and maximum number of concurrent Engines that can be running simultaneously.	dict	Required
computing	The configuration object to define the Engine-compute resources for memory and CPU allocation.	dict	Required
name	Deployment name. Metadata only.	string
description	A description of the Deployment and its function.	string
active_engine	The Engine ID that is connected to this Deployment.	string
public	The state of the Deployment’s publication. Publishing Deployments is not currenty supported.	boolean

Refer this link for more details on parameters.

Get

Get a deployment by ID.

deployment = await processing_client.deployments.get(id="<DEPLOYMENT_ID>")

Update

Update a deployment. Only the name and description can be updated.

deployment = await processing_client.deployments.update(
    id="<DEPLOYMENT_ID>",
    name="A New Name",
    description="A new description."
)

Retire

Retire a deployment.

deployment = await processing_client.deployments.retire(id="<DEPLOYMENT_ID>")

Deployment Secrets

• Deployment Secrets represent sensitive information that is mounted on an Engine container.
• Secrets are an optional capability that are used to provide a means of keeping sensitive information out of built container images.

List

Get a list of secrets belonging to the specified deployment.

query_params = { "per_page" : 5 , "page" : 1 } # Optional
list = await processing_client.deployment_secrets.list(deployment_id="<DEPLOYMENT_ID>", query_params=query_params)

List (Sync)

List all secrets belonging to the specified deployment.

NOTE: This method abstracts the complexity of pagination and provides an iterable interface to retrieve all items.

secrets = client.deployment_secrets.list_sync(deployment_id="<DEPLOYMENT_ID>")
for secret in secrets:
    print(secret)

Create

Create a new secret.

deployment_secret = await processing_client.deployment_secrets.create(
    deployment_id="<DEPLOYMENT_ID>",
    identifier="my-api-key",
    value="c29tZSBzZWNyZXQgdGV4dA==",
    name="My API Key",
    description="A description the Owner wrote."
)

Create Deployment Secret Parameters

Parameter	Description	Type	Required Fields
identifier	A Deployment Secret identifier is unique within the system. They must start and end with an alphanumeric value and can only contain: [a-z, A-Z, 0-9, ‘-’, ‘.’]. Constraints: 1 to 64 chars	string	Required
value	A Base64 encoded string of the Secret value. The Base64 encoded string will be displayed in the response body after the Deployment Secret has been processed into the service.	string	Required
name	Name of the Secret. This is metadata only.	string
description	A description of the Secret.	string

Refer this link for more details on parameters.

Get

Get a secret by ID.

deployment_secret = await processing_client.deployment_secrets.get(id="<DEPLOYMENT_SECRET_ID>", deployment_id="<DEPLOYMENT_ID>")

Update

Update a secret. Only name, description and value can be updated.

deployment_secret = await processing_client.deployment_secrets.update(
    id="<DEPLOYMENT_SECRET_ID>",
    deployment_id="<DEPLOYMENT_ID>",
    name="A New Name",
    description="A new description."
)

Delete

Delete a deployment secret.

await processing_client.deployment_secrets.delete(id="<DEPLOYMENT_SECRET_ID>", deployment_id="<DEPLOYMENT_ID>")

Retire

Retire a deployment secret.

deployment_secret = await processing_client.deployment_secrets.retire(id="<DEPLOYMENT_SECRET_ID>", deployment_id="<DEPLOYMENT_ID>")