Nitro Workspace SDK (Beta)
To avoid performance issues when moving very large data sets, data scientists can now use the Nitro Workspace SDK to gain full access to their personal workspace data lake location. They can then write their analyzed result sets back to Nitro workspaces directly into the Nitro data lake instead of the data warehouse.
The Nitro Workspace SDK provides the following functionality:
- Validates Nitro user for workspace access
- Grants the user all necessary permissions to perform S3 operations for the duration of the session
- Supported file operations include csv, parquet, and json
- Available operations:
- List directories/files under exact workspace location in S3
- Delete file from exact workspace location in S3
- Upload files from local file system to exact workspace location in S3
- Download files from exact workspace location in S3 to local filesystem
- Read files from exact workspace location in S3 into Pandas dataframe
- Write Pandas dataframe as a file to exact workspace location in S3
- Unload Redshift table to files in exact workspace location in S3
- Copy files in exact workspace location in S3 into Redshift table
- A profile in the .cdw configuration file for the specific tenant/instance. The profile should have the following attributes:
- An API key generated from NAC (after logging into the specific tenant instance)
To use the Nitro Workspace SDK (Beta):
Submit a support ticket requesting access to the SDK (python wheel) file. Describe the specific use-case for which it is being used. Once the use-case is evaluated, the Nitro Product Team sends an Egnyte link of the installable file location.
Download the installable wheel file (for example, nitro_sdk-1.0.0.py3-none-any.whl) to your local computer.
Open a Command Prompt and run the following commands, in this order:Copy
# create new venv
python3 -m venv /path/to/new/virtual/environment
# Activate new venv
# pip install the wheel file (installs in venv with all the dependencies)
pip3 install /path/to/wheel/nitro_sdk-1.0.0-py3-none-any.whl
# pip force-re-install the wheel file (if you are upgrading the version)
pip3 install --force-reinstall nitro_sdk-1.0.0-py3-none-any.whl
- Open a Python IDE in the same virtual environment to begin writing code. Access the jupyter notebook template or readme.md file for more information on the methods.