Skip to content
File Service

Azure Synapse Workspace Setup

Steps to configure

Section titled “Steps to configure”

To use the Trimble Cloud File Service (TCFS) on Azure Synapse, follow these steps:

  1. Download the TCFS JAR: The latest JAR is available as a Maven package in Trimble Artifactory.
  2. Workspace creation: In Microsoft Azure portal, create a Synapse workspace.
  3. Spark Pool Creation: Create a medium Apache Spark pool. Make sure that the Spark pool has access to the TCFS JAR from Azure Blob Storage.
  4. Set the spark properties: Make sure that these properties are set in your EMR Serverless application configuration or in spark job test class itself set these spark properties:
    • spark.jars : abfss://{container-name}@{storage-account-name}.dfs.core.windows.net/{path/to/jar}
    • spark.hadoop.fs.tcfs.impl : com.trimble.hadoop.fs.FileServiceFileSystem
    • spark.hadoop.fs.AbstractFileSystem.tcfs.impl : com.trimble.hadoop.fs.Tcfs
    • spark.hadoop.fs.defaultFS : tcfs://{space id}@{virtual domain} eg: tcfs://3841cdee-9c67-411a-a6f7-6056d37e057a@stage.us.fs.trimblecloud.com
    • spark.hadoop.fs.tcfs.client.id : {CLIENT-ID}
    • spark.hadoop.fs.tcfs.client.secret : {CLIENT-SECRET}
  5. Add the jars to your Spark job submission: If you are using the Azure CLI to submit your Spark job, include the –jars option with the ABFS path to the TCFS JAR.
  6. Example command to submit spark job
Terminal window
az synapse spark job submit --name laz_processor_test \
--workspace-name ${workspaceName} --spark-pool-name ${sparkPoolName} \
--main-definition-file "abfss://${blobName}@${storageAccountName}.dfs.core.windows.net/synapse/artifacts/tests.jar" \
--reference-files "abfss://${blobName}@${storageAccountName}.dfs.core.windows.net/synapse/artifacts/driver.jar" \
--main-class-name com.trimble.seurat.LazFileProcessorTest --executors 2 --executor-size Small