An introduction to the Globus command line interface and the SDK for accessing Globus platform services. This tutorial was presented at the GlobusWorld 2021 conference in Chicago, IL by Greg Nawrocki.

1. The Globus
CLI, Platform
and SDK
Greg Nawrocki
GlobusWorld - May 13, 2021

2. 2
Custom portals? Science Gateways? Unique workflows? Our
Command Line Interface, open REST APIs and Python SDK
empower you to create an integrated ecosystem of research data
services and applications.

3. Globus Command Line Interface
Open source, uses
the Python SDK
Because of this
correspondence the CLI is
an excellent tool for getting
the gist of how he SDK
functions.

4. PaaS Security Challenges – Globus Auth
• How to provide:
– Login to apps
o Web apps (Jupyter Notebook, Portals), Mobile, Desktop, Command line
– Protect all REST API communications
o App à Globus service (Jupyter Notebook, Portals)
o App à non-Globus service (Portals)
o Service à service (Portals)
• While:
– Not introducing even more identities
o Providing a platform to consolidate those identities
– Providing least privileges security model (consents)
– Being agnostic to programming language and framework
– Being web friendly
– Making it easy for users and developers
4

5. Globus Auth: Native apps
• Client that cannot keep a secret, e.g…
– Command line, desktop apps – The Globus CLI
– Mobile apps
– Jupyter notebooks
• Native app is registered with Globus Auth
– Not a confidential client, the task on behalf of an authenticated user
– Redirect to give consent
• Globus SDK:
– To get tokens: NativeAppAuthClient
– To use tokens: AccessTokenAuthorizer
5

6. Browser
Native App grant
6
Native App
(Client)
1. Run
application
2. URL to
authenticate
3. Authenticate and
consent
4. Auth code
5. Register
auth code
6. Exchange
code
7. Access tokens
8. Authenticate with access
tokens to invoke transfer
service as user App/Service
(Resource Server)
Globus Auth
(Authorization Server)

7. Refresh tokens
• Common use cases
– Portal checking transfer status when user is not logged in
– Running command line app from script
o The CLI gets access and refresh tokens upon ”globus login”
• Refresh tokens issued to client, in particular scope
• Client uses refresh token to get access token
– Confidential client: client_id and client_secret required
– Native app: client_secret not required
• Refresh token good for 6 months after last use
• Consent rescindment revokes resource token
7

8. Refresh tokens
8
Native App
(Client)
App/Service
(Resource Server)
Globus Auth
(Authorization Server)
1. Run
application
2. URL to
authenticate
Browser
3. Authenticate and consent
4. Auth code
5. Register
auth code
6. Exchange code,
request refresh tokens
7. Access
tokens and refresh tokens
9. Exchange refresh token
for new access tokens
8. Store refresh tokens
10. Access tokens
11. Authenticate with access
tokens to invoke service as user

9. Globus CLI
• It’s a native application distributed by Globus
– https://docs.globus.org/cli/
– https://github.com/globus/globus-cli
• Easy install and updates
• Command “globus login” gets access tokens and refresh
tokens
– Stores the token locally (~/.globus.cfg )
• All interactions with the service use the tokens
– Tokens for Globus Auth and Transfer services
• Command “globus logout” deletes those

10. CLI Basics
• Getting help / list of commands
– globus –help
– globus list-commands
• UUIDs for endpoint, task, user identity, groups…
– Use search/list options
• get-identities for identity username to UUID
$ globus endpoint search 'Globus Tutorial'
$ globus task list
$ globus get-identities vas@globus.org 2867d9fb-
d5b5-4f21-95e7-312b288b8d11 --verbose

11. The Globus CLI – Simple tasks
• Find endpoints
– globus endpoint search 'Globus Tutorial'
– globus endpoint search Midway
– globus endpoint search --filter-scope=recently-used
• Find endpoint contents
– globus ls af7bda53-6d04-11e5-ba46-22000b92c6ec
– globus ls ddb59aef-6d04-11e5-ba46-22000b92c6ec:/share/godata/
• Transfer a file
– From Globus Tutorial Endpoint 1 to Globus Tutorial Endpoint 2
– Note the specific paths
– globus transfer ddb59aef-6d04-11e5-ba46-22000b92c6ec:/share/godata/file3.txt ddb59af0-6d04-
11e5-ba46-22000b92c6ec:/~/file3.txt
• Transfer a directory
– From Globus Tutorial Endpoint 1 to Globus Tutorial Endpoint 2
– globus transfer --recursive ddb59aef-6d04-11e5-ba46-22000b92c6ec:/share/godata/ ddb59af0-
6d04-11e5-ba46-22000b92c6ec:/~/syncDemo
• https://docs.globus.org/cli/examples/

12. Batch Transfers
• Transfer tasks have one source/destination, but can have
any number of files
• Provide input source-dest pairs via local file
• e.g. move files listed in files.txt from $ep1 to $ep2
$ ep1=ddb59aef-6d04-11e5-ba46-22000b92c6ec
$ ep2=ddb59af0-6d04-11e5-ba46-22000b92c6ec
$ globus transfer $ep1:/share/godata/ $ep2:/~/ --
batch --label 'CLI Batch' < files.txt

13. Parsing CLI output
• Default output is text; for JSON output use --format json
$ globus endpoint search --filter-scope my-endpoints
$ globus endpoint search --filter-scope my-endpoints --
format json
• Extract specific attributes using --jmespath <expression>
$ globus endpoint search --filter-scope my-endpoints --
jmespath 'DATA[].[id, display_name]'

14. Managing notifications
• Turn off emails sent for tasks
• Useful when an application manages tasks for a user
• Disable notifications with the --notify option
--notify off (all notifications)
--notify succeeded|failed|inactive (select notifications)

15. Permission management
• Set and manage permissions on shared endpoint
• Requires access manager role
$ share=<shared_endpoint_UUID>
$ globus endpoint permission create --permissions r --
identity greg@nawrockinet.com $share:/nawrockipersonal/
$ globus endpoint permission list $share
$ globus endpoint permission delete $share <perm_UUID>

16. Automation with the CLI
• Interactions are as user: both for data access and to
Globus services
– Globus login to get tokens
• Collection access
– Mapped Collections
o Use the –skip-activation-check to submit the task even if endpoint is not
activated at submit time
– Guest Collections
o Guest Collection / Shared Endpoints auto-activate
o Use Guest Collections whenever possible
• Reference
– Basic Data Automation with the Globus Command Line Interface
(CLI)
o https://www.youtube.com/watch?v=qIQTC6YOvrE

17. Data centric applications leveraging Globus
17

18. JupyterHub
• Multi-user hub
• Manages multiple instances
of Jupyter notebook server
– Python, R, etc.
• Use it to serve notebooks to
research team, class, etc.
• Configurable HTTP proxy
jupyterhub.readthedocs.io/en/stable

19. Securing JupyterHub with Globus Auth plugin
• Existing OAuth framework
• Can restrict IdP
• Custom scopes
• Tokens passed into
notebook environment
• Documentation covers app
registration and config
github.com/jupyterhub/oauthenticator#globus-setup

20. https://www.globus.org/blog/using-globus-jupyter-notebooks
Securing JupyterHub with Globus Auth

21. Client
(Web Portal,
Application)
Globus Transfer
(Resource Server)
Globus Auth
(Authorization
Server)
5. Authenticate using client id
and secret, send authorization
code
Authorization Code Grant
Browser (User)
1. Access
portal
2.
Redirects
user
3. User authenticates and
consents
4. Authorization
code
6. Access token(s)
7. Authenticate with access
token(s) to give the client
the authority invoke the
transfer service
Identity
Provider

22. Tokens and Jupyter Hub
login
REST APIs
{ “tokens”:…
{“tokens”:…
REST APIs
REST APIs
Bearer a45cd…

23. Globus Transfer API
• Globus Web App consumes public Transfer API
• Resource named by URL (standard REST approach)
– Query params allow refinement (e.g., subset of fields)
• Globus APIs use JSON for documents and resource
representations
• Requests authorized via OAuth2 access token
– Authorization: Bearer asdflkqhafsdafeawk
docs.globus.org/api/transfer
23

24. Globus Python SDK
• Python client library for the Globus Auth and Transfer
REST APIs
• globus_sdk.TransferClient class handles
connection management, security, framing,
marshaling
from globus_sdk import TransferClient
tc = TransferClient()
globus.github.io/globus-sdk-python
24

25. TransferClient higher-level calls
• One method for each API resource and HTTP verb
• Largely direct mapping to REST API
endpoint_search(filter_fulltext=None,
filter_scope=None,
num_results=25,
**params)
25

26. Endpoint Search
• Plain text search for endpoint
– Searches owner, display name, keywords, description,
organization, department
– Full word and prefix match
• Limit search to pre-defined scopes
– all, my-endpoints, recently-used, in-use, shared-
by-me, shared-with-me
• Returns: List of endpoint documents
26

27. Endpoint Activation
• Activating endpoint means binding a credential to an
endpoint for login
• Mapped Collections require login via web app
• Auto-activate
– Globus Connect Personal and Guest Collections use
Globus-provided credential
– Must auto-activate before any API calls to endpoints
27

28. File operations
• List directory contents (ls)
• Make directory (mkdir)
• Rename
• Note:
– Path encoding & UTF gotchas
– Don’t forget to auto-activate first
28

29. Task submission
• Asynchronous operations
– Transfer
o Sync level option
– Delete
• Get submission_id, followed by submit
– Once and only once submission
29

30. Walkthrough API with our Jupyter Hub
• https://jupyter.demo.globus.org
– Sign in with Globus
– Verify the consents
– Start My Server (this will take about a minute)
– Open folder: globus-jupyter-notebooks
– Run Platform_Introduction_JupyterHub_Auth.ipynb
• If you mess it up and want to “go back to the beginning”
– Just stop and restart the notebook
• If you want to use the notebook outside of our hub
– https://github.com/globus/globus-jupyter-notebooks
– Authentication is a manual cut and paste of exchanging the
authorization code for an access token – Native App
30

31. Automation Examples
• Simple code examples for various use cases using
Globus
– https://github.com/globus/automation-examples
– Syncing a directory
o Bash script that calls the Globus CLI and a Python module that can
be run as a script or imported as a module.
– Staging data in a shared directory
o Bash / Python
– Removing directories after files are transferred
o Python script
31

32. Resources
• Documentation: docs.globus.org
• YouTube channel: youtube.com/GlobusOnline
• Helpdesk and issue escalation: support@globus.org
• Mailing list: discuss@globus.org
• Globus customer team: outreach@globus.org
– We’d love to be part of your events
– GlobusWorld Tours
– Office Hours