Creating a Custom Connector Using Enable S3 Pull

System admins can configure a custom inbound connector to source data directly from the data vendor’s S3 data lake to the Nitro data lake. This replaces the need for customers to stage their data in costly SFTP locations.

Nitro has customer-specific IAM roles to provide better data governance and security in the data lake. For data exchange between a source and Nitro, this role requires list and get permissions to the specific data folder in the source S3 bucket. If you are unsure of how to manage S3 bucket permissions, see the AWS documentation for managing bucket policies.

To allow users to upload large files using S3 without the system timing out, the Nitro AWS role timeout value is 12 hours.

To create an inbound connector:

  1. Log into the NAC.

  2. Select Connectors > Inbound Connectors from the side menu.

  3. Select the New Connector button.

  4. Complete the Inbound Connector Details fields.

  5. Select Custom as the Connector Type.

  6. Select the Enable S3 Pull check box.

  7. Select Include Folder Name in Copy from S3 to append folder names to the file names when only net new files are pulled into Nitro. This step is optional.

    If "Include Folder Name in Copy from S3" is selected, you must use the FTP/Data Lake Connector Load job. Do not use the FTP/Data Lake Intelligent Load job.

  8. Complete the Source S3 data key and Source S3 Region fields with your system's information. Optionally, admins can specify the file extension or regex pattern they want to use. The following folder designations are available:

    Source S3 Data Key Designation 

    Description

    s3://veevaQA-myBucket

    The entire contents of the bucket are processed.

    s3://veevaQA-myBucket/myFolder

    All files in the specified folder are copied. The folder name can be specified with or without a forward slash (/).

    s3://veevaQA-myBucket/myFolder/*.csv

    Only uncompressed or compressed files in the specified folder with file-type extension of .csv are copied.

    s3://veevaQA-myBucket/myFolder/myfiles*.csv;*.json

    Only uncompressed and compressed files in the specified folder with file-type extension of .csv and/or .json are copied. 

    s3://veevaQA-myBucket/myFolder/regex("^(.*)(?=_PA_)")

    Only files in the specified folder with filenames matching the regular expression are copied.

    s3://veevaQA-myBucket/myFolder/regex("^(.*)(?=_PA_)");*.json

    Only files in the specified folder with filenames matching the regular expression and/or with a file-type extension of .json are copied.

  9. Select Save.

  10. Select the Key icon.

  11. Select the Copy icon.

  12. Paste this information into an existing or new S3 bucket policy to modify the trust relationship configuration of your AWS S3 bucket.

  13. Return to the Inbound Connector page in Nitro and select the Verify Connection icon. This step is optional.

Detect and Pull New Files by Object Using Enable S3 Pull

To configure a scheduled job to detect what was copied and successfully loaded for a given table object and only copy new files, create an upload file in the upload/T/I/C/lastProcessed folder named pullfromS3config.yml.

Existing connectors using Enable S3 Pull without the upload file continue to operate as before without new file detection logic.

The upload file must contain the following:

Key

Value Description

Example

tableObjectName

Staging table name

symphony_reassign_stg__c

fileNamePattern

Can be text, wildcard, or regex

SW_P_CSREASSIGN_*.txt

batchSize
  • Use a positive number to batch
  • Use -1 for loading all files matching the filename pattern

-1

originalStartBatchAfter

File name after which files are pulled in ascending order

SW_P_CSREASSIGN_20220204.txt.gz

The originalStartBatchAfter key is only used the first time the S3 pull is executed with the new version.