Spark Configurations Supported by HDFS Driver
The Trimble Cloud Fileservice HDFS Driver supports the following Spark configurations to enable seamless integration with Trimble Cloud File Service:
| Configuration |
Value |
Description |
|---|---|---|
spark.hadoop.fs.tcfs.impl |
com.trimble.hadoop.fs.FileServiceFileSystem |
Specifies the implementation class for the TCFS FileSystem. Required. |
spark.hadoop.fs.AbstractFileSystem.tcfs.impl |
com.trimble.hadoop.fs.Tcfs |
Specifies the abstract file system implementation for TCFS. Required. |
spark.hadoop.fs.defaultFS |
tcfs://{space id}@{virtual domain} |
Sets the default file system URI to TCFS. Required. |
spark.hadoop.fs.tcfs.client.id |
{CLIENT-ID} |
Client ID for authenticating with Trimble Cloud File Service. Required. |
spark.hadoop.fs.tcfs.client.secret |
{CLIENT_SECRET} |
Client secret for authenticating with Trimble Cloud File Service. Required. |
spark.jars |
s3://{BUCKET}/{PATH}/trimble-hadoop-fs-tcfs-{VERSION}-all.jar |
Path to the TCFS JAR file. Required. |
spark.hadoop.fs.tcfs.http.requestTimeout |
Timeout in seconds (e.g., 60) |
HTTP request timeout for TCFS operations. Default: 30 seconds. |
spark.hadoop.fs.tcfs.http.retryTimeout |
Timeout in seconds (e.g., 120) |
HTTP retry timeout for TCFS operations. Default: 30 seconds. |
spark.hadoop.fs.tcfs.tid.token.refresh |
Interval in minutes (e.g., 50) |
Token refresh interval. Default: 50 minutes. |
spark.hadoop.fs.tcfs.io.readbuffer.size |
Size in bytes (e.g., 16384 for 16 KB) |
Buffer size for read operations. Default: 256 MB. Min: 16 KB, Max: 1 GB. |
spark.hadoop.fs.tcfs.io.writebuffer.size |
Size in bytes (e.g., 16384 for 16 KB) |
Buffer size for write operations. Default: 4 MB. Min: 16 KB, Max: 1 GB. |
spark.hadoop.fs.tcfs.io.multipart.size |
Size in bytes (e.g., 5242880 for 5 MB) |
Multipart block size for upload operations. Default: 8 MB. Min: 5 MB, Max: 1 GB. |
spark.hadoop.fs.tcfs.io.activationTimeout |
Timeout in seconds (e.g., 60) |
File activation timeout. Default: 60 seconds. |
spark.hadoop.fs.tcfs.io.activationRetryDelay |
Delay in seconds (e.g., .1f for 100 ms) |
Delay between activation retries. Default: 0.1 seconds. |
spark.hadoop.fs.tcfs.io.writeThreads.max |
Number of threads (e.g., 8) |
Maximum number of threads for write operations. Default: 8. Min: 1, Max: 8. |
spark.hadoop.fs.tcfs.io.queuedWrites.max |
Number of queued writes (e.g., 20) |
Maximum number of queued write operations. Default: 20. Should be at least 2x write threads. |
spark.hadoop.fs.tcfs.groupAuth.enabled |
true or false |
Enables/disables IAM group authentication. Default: false. |