Create Cloud Storage Connection File (Data Management)

Summary

Creates a connection file for ArcGIS-supported cloud storage. This tool allows existing raster geoprocessing tools to write cloud raster format (CRF) datasets into the cloud storage bucket or read raster datasets (not limited to CRF) stored in the cloud storage as input.

Usage

• You must provide the necessary information to make a cloud storage connection—such as Access Key, Secret Access Key, and Bucket Name—to run this tool.

• The tool outputs a binary cloud storage connection file (.icsd) in ArcGIS Cloud Storage format.

• The raster dataset stored in the cloud storage can be referenced through a file path such as c:/temp/amazons3.acs/someraster.

• This tool supports connections to Amazon Simple Storage Service (S3) buckets, Microsoft Azure Blob storage containers, Alibaba Cloud Object Storage Service (OSS) buckets, and Google Cloud Storage.

• If the dataset is stored in a folder in the bucket, the folder name must be included in the path, for example, c:/temp/amazons3.acs/foldername/someraster.

• The tool will validate the credentials provided at run time. If the connection cannot be made, the tool will fail.

• To use this tool with the Amazon Web Services (AWS) Identity and Access Management (IAM) role, leave the Access Key ID (Account Name) and Secret Access Key (Account Key) parameters blank. If the IAM role is properly configured when running this tool on an Amazon Elastic Compute Cloud (EC2) machine, a successful connection will occur. This connection file can then be used in the same environment.

Parameters

Dialog

Label	Explanation	Data Type
Connection File Location	The folder path where the connection file will be created.	Folder
Connection File Name	The name of the cloud storage connection file.	String
Service Provider	Specifies the cloud storage service provider. Azure —The service provider will be Microsoft Azure. Amazon —The service provider will be Amazon S3. Google —The service provider will be Google Cloud Storage. Alibaba —The service provider will be Alibaba Cloud Storage. WebHDFS —The service provider will be WebHDFS. MinIO —The service provider will be MinIO. Azure Data Lake —The service provider will be Azure Data Lake.	String
Bucket (Container) Name	The name of the cloud storage container where the raster dataset will be stored. Many cloud providers also call it a bucket.	String
Access Key ID (Account Name) (Optional)	The access key ID string for the specific cloud storage type. It can also be the account name, as is the case with Azure.	String
Secret Access Key (Account Key) (Optional)	The secret access key string to authenticate the connection to cloud storage.	Encrypted String
Region (Environment) (Optional)	The region string for the cloud storage. If provided, the value must use the format defined by the cloud storage choice. The default is the selected cloud provider's default account.	String
Service End Point (Optional)	The service endpoint (uris) of the cloud storage, such as oss-us-west-1.aliyuncs.com. If a value is not provided, the default endpoint for the selected cloud storage type will be used. The CNAME redirected endpoint can also be used if needed.	String
Provider Options (Optional)	The configuration options pertaining to the specific type of cloud service. Some services offer options, some do not. You only need to set the option if you want to turn them on. Role-based access control (RBAC) is available for both Amazon and Azurecloud providers. Keeping all authentication parameters empty while using an EC2 or Azure virtual machine will enable ArcGIS Pro to access Blob storage using IAM roles or Azure RBAC. For Amazon, IMDSv1 and IMDSv2 are supported. Microsoft Azure Data Lake Storage Gen2 follows the same options as Azure but provides true directory support and atomic operations using a DFS endpoint. Some network errors during cloud operations are retried following exponential backoff. For performance considerations and additional information, see the GDAL virtual file systems documentation. Azure / Microsoft Azure Data Lake AZURE_SAS—Specify an shared access signature. Ensure that its value is URL encoded and does not contain leading '?' or '&' characters. When using this option, the Secret Access Key (Account Key) parameter must be empty. AZURE_NO_SIGN_REQUEST (default: FALSE)—Anonymously connect to buckets (containers) that don't require authenticated access. When using this option, the Secret Access Key (Account Key) parameter must be empty. AZURE_STORAGE_CONNECTION_STRING—Specify an Azure Storage connection string. This string embeds the account name, key, and endpoint. When using this option, the Access Key ID (Account Name) and Secret Access Key (Account Key) parameter must be empty. CPL_AZURE_USE_HTTPS (default: TRUE): Set FALSE to use http requests. Note that some servers might be configured to only support https requests. Amazon / MinIO AWS_NO_SIGN_REQUEST (default: FALSE)—Anonymously connect to buckets (containers) that don't require authenticated access. AWS_SESSION_TOKEN—Specify temporary credentials. AWS_DEFAULT_PROFILE—AWS credential profiles are automatically used when Access Key/ID is missing. This option can be used to specify the profile to use. AWS_REQUEST_PAYER—Requester Pays buckets can be accessed by setting this option to requester. AWS_Virtual_Hosting (default: TRUE)—Amazon S3 and S3 compatible cloud providers supporting only path-style requests must set this option to TRUE. It is recommended that you use virtual hosting if it's supported CPL_VSIS3_USE_BASE_RMDIR_RECURSIVE (default: TRUE)— Some older S3 compatible implementations do not support the Bulk Delete operation. Set this option to FALSE for these providers. AWS_HTTPS (default: TRUE)—Set FALSE to use http requests. Note that some servers might be configured to only support https requests Alibaba OSS_Virtual_Hosting (default: TRUE)—Alibaba and S3 compatible cloud providers supporting only path-style requests must set this option to TRUE. It is recommended that you use virtual hosting if it's supported. OSS_HTTPS (default: TRUE)—Set FALSE to use http requests. Note that some servers might be configured to only support https requests. WebHDFS WEBHDFS_REPLICATION (integer)—The replication value is used when creating a file WEBHDFS_PERMISSION (decimal)—A permission mask is used when creating a file. If multiple authentication parameters are provided, precedence is as follows: Azure—AZURE_STORAGE_CONNECTION_STRING, account name / key, AZURE_SAS, AZURE_NO_SIGN_REQUEST, RBAC. Amazon—AWS_NO_SIGN_REQUEST, access id / key and/or AWS_SESSION_TOKEN, AWS Credential Profile, IAM Role. Other than the provider options listed above, this option ARC_DEEP_CRAWL (default:TRUE), can be used with all the service providers. If True, this option is used to identify CRFs with no extension in the cloud; this is an expensive operation and it is highly recommended to set this option to FALSE for faster catalog browsing experience and crawling.	Value Table
Folder (Optional)	The folder in the Bucket (Container) Name parameter where the raster dataset will be stored.	String

Derived Output

Label	Explanation	Data Type
Output Connection File	The output cloud storage connection file path.	File

Python

arcpy.management.CreateCloudStorageConnectionFile(out_folder_path, out_name, service_provider, bucket_name, {access_key_id}, {secret_access_key}, {region}, {end_point}, {config_options}, {folder})

Name	Explanation	Data Type
out_folder_path	The folder path where the connection file will be created.	Folder
out_name	The name of the cloud storage connection file.	String
service_provider	Specifies the cloud storage service provider. AZURE —The service provider will be Microsoft Azure. AMAZON —The service provider will be Amazon S3. GOOGLE —The service provider will be Google Cloud Storage. ALIBABA —The service provider will be Alibaba Cloud Storage. WEBHDFS —The service provider will be WebHDFS. MINIO —The service provider will be MinIO. AZUREDATALAKE —The service provider will be Azure Data Lake.	String
bucket_name	The name of the cloud storage container where the raster dataset will be stored. Many cloud providers also call it a bucket.	String
access_key_id (Optional)	The access key ID string for the specific cloud storage type. It can also be the account name, as is the case with Azure.	String
secret_access_key (Optional)	The secret access key string to authenticate the connection to cloud storage.	Encrypted String
region (Optional)	The region string for the cloud storage. If provided, the value must use the format defined by the cloud storage choice. The default is the selected cloud provider's default account.	String
end_point (Optional)	The service endpoint (uris) of the cloud storage, such as oss-us-west-1.aliyuncs.com. If a value is not provided, the default endpoint for the selected cloud storage type will be used. The CNAME redirected endpoint can also be used if needed.	String
config_options [config_options,...] (Optional)	The configuration options pertaining to the specific type of cloud service. Some services offer options, some do not. You only need to set the option if you want to turn them on. Role-based access control (RBAC) is available for both Amazon and Azurecloud providers. Keeping all authentication parameters empty while using an EC2 or Azure virtual machine will enable ArcGIS Pro to access Blob storage using IAM roles or Azure RBAC. For Amazon, IMDSv1 and IMDSv2 are supported. Microsoft Azure Data Lake Storage Gen2 follows the same options as Azure but provides true directory support and atomic operations using a DFS endpoint. Some network errors during cloud operations are retried following exponential backoff. For performance considerations and additional information, see the GDAL virtual file systems documentation. Azure / Microsoft Azure Data Lake AZURE_SAS—Specify an shared access signature. Ensure that its value is URL encoded and does not contain leading '?' or '&' characters. When using this option, the Secret Access Key (Account Key) parameter must be empty. AZURE_NO_SIGN_REQUEST (default: FALSE)—Anonymously connect to buckets (containers) that don't require authenticated access. When using this option, the Secret Access Key (Account Key) parameter must be empty. AZURE_STORAGE_CONNECTION_STRING—Specify an Azure Storage connection string. This string embeds the account name, key, and endpoint. When using this option, the Access Key ID (Account Name) and Secret Access Key (Account Key) parameter must be empty. CPL_AZURE_USE_HTTPS (default: TRUE): Set FALSE to use http requests. Note that some servers might be configured to only support https requests. Amazon / MinIO AWS_NO_SIGN_REQUEST (default: FALSE)—Anonymously connect to buckets (containers) that don't require authenticated access. AWS_SESSION_TOKEN—Specify temporary credentials. AWS_DEFAULT_PROFILE—AWS credential profiles are automatically used when Access Key/ID is missing. This option can be used to specify the profile to use. AWS_REQUEST_PAYER—Requester Pays buckets can be accessed by setting this option to requester. AWS_Virtual_Hosting (default: TRUE)—Amazon S3 and S3 compatible cloud providers supporting only path-style requests must set this option to TRUE. It is recommended that you use virtual hosting if it's supported CPL_VSIS3_USE_BASE_RMDIR_RECURSIVE (default: TRUE)— Some older S3 compatible implementations do not support the Bulk Delete operation. Set this option to FALSE for these providers. AWS_HTTPS (default: TRUE)—Set FALSE to use http requests. Note that some servers might be configured to only support https requests Alibaba OSS_Virtual_Hosting (default: TRUE)—Alibaba and S3 compatible cloud providers supporting only path-style requests must set this option to TRUE. It is recommended that you use virtual hosting if it's supported. OSS_HTTPS (default: TRUE)—Set FALSE to use http requests. Note that some servers might be configured to only support https requests. WebHDFS WEBHDFS_REPLICATION (integer)—The replication value is used when creating a file WEBHDFS_PERMISSION (decimal)—A permission mask is used when creating a file. If multiple authentication parameters are provided, precedence is as follows: Azure—AZURE_STORAGE_CONNECTION_STRING, account name / key, AZURE_SAS, AZURE_NO_SIGN_REQUEST, RBAC. Amazon—AWS_NO_SIGN_REQUEST, access id / key and/or AWS_SESSION_TOKEN, AWS Credential Profile, IAM Role. Other than the provider options listed above, this option ARC_DEEP_CRAWL (default:TRUE), can be used with all the service providers. If True, this option is used to identify CRFs with no extension in the cloud; this is an expensive operation and it is highly recommended to set this option to FALSE for faster catalog browsing experience and crawling.	Value Table
folder (Optional)	The folder in the bucket_name parameter where the raster dataset will be stored.	String

Derived Output

Name	Explanation	Data Type
out_connection	The output cloud storage connection file path.	File

Code sample

CreateCloudStorageConnectionFile example 1 (Python window)

This is a Python sample for the CreateCloudStorageConnectionFile function.

#====================================
# CreateCloudStorageConnectionFile
# Usage:
# arcpy.management.CreateCloudStorageConnectionFile(
#     out_folder_path, out_name, AZURE | AMAZON | GOOGLE | ALIBABA, bucket_name,
#     {access_key_id}, {secret_access_key}, {region}, {end_point},
#     { {Name} {Value}; {Name} {Value}...})
# arcpy.management.CreateCloudStorageConnectionFile(
#     out_folder_path, out_name, AZURE | AMAZON | GOOGLE | ALIBABA, bucket_name,
#     {access_key_id}, {secret_access_key}, {region}, {end_point},
#     {config_options})

import arcpy

# Create connection to open public bucket with requester pay option
arcpy.management.CreateCloudStorageConnectionFile(
    "C:/Workspace/connections", "awss3storage.acs", "AMAZON", "publicrasterstore",
    config_options="AWS_REQUEST_PAYER requester")

# Create connection to secured Azure bucket
arcpy.management.CreateCloudStorageConnectionFile(
    "C:/Workspace/connections", "azurestorage.acs", "AZURE", "rasterstore", "imageaccount",
    "NOGEOU1238987OUOUNOQEWQWEIO")

# Create Alibaba connection with end points
arcpy.management.CreateCloudStorageConnectionFile(
    "C:/Workspace/connections", "aliyun.acs", "ALIBABA", "rasterstore", "AYOUER9273PJJNY",
"NOGEOU1238987OUOUNOQEWQWEIO", end_point="rasterstore.oss-us-west-1.aliyuncs.com")

CreateCloudStorageConnectionFile example 2 (stand-alone script)

This is a Python sample for the CreateCloudStorageConnectionFile function.

#====================================
# CreateCloudStorageConnectionFile
# Usage:
# arcpy.management.CreateCloudStorageConnectionFile(
#     out_folder_path, out_name, AZURE | AMAZON | GOOGLE | ALIBABA, bucket_name,
#     {access_key_id}, {secret_access_key}, {region}, {end_point},
#     { {Name} {Value}; {Name} {Value}...})
# arcpy.management.CreateCloudStorageConnectionFile(
#     out_folder_path, out_name, AZURE | AMAZON | GOOGLE | ALIBABA, bucket_name,
#     {access_key_id}, {secret_access_key}, {region}, {end_point},
#     {config_options})

import arcpy

outfolder = "C:/Workspace/connections"
connectname = "googlecloudos.acs"
provider = "GOOGLE"
accesskey = "AYOUER9273PJJNY"
secretkey = "NOGEOU1238987OUOUNOQEWQWEIO"
bucketname = "rasterstore"

# Create connection to Google cloud object storage
arcpy.management.CreateCloudStorageConnectionFile(
outfolder, connectname, provider, bucketname, accesskey, secretkey)

Environments

This tool does not use any geoprocessing environments.

Licensing information

• Basic: Yes
• Standard: Yes
• Advanced: Yes