Skip to content
 

How to set up Jupyter Lab on Shannon for Remote Access

- Posted by: Dr. Lander Ver Hoef - June 16, 2023 Back to CIRA Wiki Home


Topic Catagories: , ,
Tags: , , , ,

Steps 1 and 2 are one-time-only steps, while steps 3 through 6 should be completed every time you use Jupyter Lab. Throughout this tutorial, we will use the convention that variables you should replace with your own values are all in capital letters, like MY_ENV_NAME, TMUX_NAME, etc. Other commands should be run verbatim. These commands should work independent of which operating system you are using — so long as you have the ability to ssh from your terminal, everything should work.

Initial Setup

1. Install miniforge on Shannon and set up an environment

Mamba-forge is a virtual environment manager that lets you work with python packages in compartmentalized environments to simplify dependencies. Log into Shannon via SSH and install it by running the following:

curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
bash Miniforge3-$(uname)-$(uname -m).sh

Accept the license agreement, and respond “yes” when it asks if you would like to initialize conda. After these complete, run the command bash and you should be in the new (base) environment.

Now, create a new conda environment to install Jupyter Lab into:

conda create -n MY_ENV_NAME

and activate it by running

conda activate MY_ENV_NAME

2. Set up the Jupyter Lab environment

Jupyter Lab is an interactive notebook interface for python which runs in a browser. With MY_NEW_ENV activated, run the following command to install Jupyter Lab and the Git extension:

conda install jupyterlab jupyterlab-git jupyter-resource-usage ipympl

To make Jupyter Lab always aware of your conda environment, run the following command

python -m ipykernel install --user --name=MY_ENV_NAME

Note that this command may bring up an error saying that your python installation has been compiled incorrectly, as it uses relative paths. In this instance, that warning is safe to ignore and the remainder of these steps should complete successfully.

Finally, set up a password for your notebook – this doesn’t have to be particularly secure, as most of the security will be handled by authenticating on Shannon, but having a password means we don’t have to copy a long token to access our notebook. You can even leave the password blank to use “password” authentication without having to remember anything. To set the password, run

jupyter server password

and create a password (possibly blank) when prompted.

2.5 Creating additional Conda environments for coding (Optional)

The main benefit of using Conda to create virtual environments is that each project can have its own set of packages which will not interfere with one another, to simplify dependency trees and make managing compatible package versions easier. The recommended way to set up Jupyter Lab here is to have Jupyter Lab in its own environment with just itself and any plugins, as we can make that single Jupyter Lab installation aware of all your other Conda environments.

To create a new Conda environment for coding, you would run

conda create -n MY_CODING_CONDA_ENV

then activate your environment using conda activate MY_CODING_CONDA_ENV. With that environment activated, install the “ipykernel” package by running

conda install ipykernel

and run

python -m ipykernel install --user --name=MY_CODING_ENV_NAME

to register that environment with your Jupyter Lab installation, even if they are in different environments. This way, you can have a single Jupyter Lab installation that can utilize multiple conda environments.

Starting and Connecting to Jupyter Lab

3. Start your instance of Jupyter Lab

We will start Jupyter Lab in a tmux screen so that if our connection gets dropped, the Jupyter session will continue running. To set this up, create a new tmux instance with

tmux new -s TMUX_NAME

where TMUX_NAME is some readable name for what you’re doing — one suggestion is “jlab_serv”. You should see a green bar appear at the bottom of your screen. You’ll need to activate your conda environment again, then execute the following command:

jupyter lab --no-browser --port=PORT_NUMBER

where PORT_NUMBER is some number between 2000 and 65000. This must be unique amongst Jupyter Lab instances running on Shannon, and the default is 8888, so choose some random number in that interval avoiding 8888 — if the port you try is already in use, Jupyter Lab will make note of that in the setup text and automatically try another port, usually the next largest port number. If that happens, use the port Jupyter Lab is actually running on as PORT_NUMBER for the steps below.

To detach this tmux screen, press CTRL-B then D, which should return you to your regular SSH session. At this point, you can detach your tmux session, close your SSH connection, and continue on to step 4 below. If you’d like a little more info on using tmux, see the remainder of this section.

To view a list of currently active tmux sessions, run

tmux ls

To view an active tmux session, run

tmux attach -t TMUX_NAME

where TMUX_NAME is the name of the session you want to join. To close a tmux session, run

tmux kill-session -t TMUX_NAME

In general, it is good practice to shut down your tmux session while you are not using Jupyter Lab.

4. Set up SSH Port Forwarding

Now, we want to use SSH to create a tunnel that will let us access the port on Shannon that our Jupyter Lab instance is running on via our local web browser. To do this, from your local machine, run the following command in a command window (Terminal/Bash/Windows Powershell, etc.):

ssh -N -L PORT_NUMBER:localhost:PORT_NUMBER USER@shannon.cira.colostate.edu

where PORT_NUMBER is the port number you chose in step 3, and USER is your username on Shannon. The -L flag is what makes the port forwarding work, and the -N flag makes this be a non-interactive SSH session — after it executes and you authenticate on Shannon, you won’t get the usual command entry prompt, just a blank line. If you’d prefer to have an SSH session that you can use to navigate/run commands in, you can remove the -L flag, just remember to keep that session open for as long as you want to use Jupyter Lab.

5. Connect via local browser

To actually start using Jupyter Lab, open a browser on your local machine and in the address bar type

localhost:PORT_NUMBER

where PORT_NUMBER is the port number you chose in step 3. You should be prompted to enter the password you chose in step 2 — if you chose to have an empty password, simply click through. You should have a Jupyter Lab instance pop up that is running on Shannon.

As a note, if you need to install further packages or otherwise use a terminal on Shannon, the “Terminal” option in Jupyter Lab is an easy way to do this without having to initiate another SSH session.

Shutting Down

6. Shutting down Jupyter Lab

When you’re done using Jupyter Lab and want to shut down your instance, follow these steps:

  1. In the Jupyter Lab browser window, go to File > Shut Down. Close the Jupyter Lab tab.
  2. In your local terminal window that is maintaining the SSH Port Forwarding, press CTRL-C to shut down the SSH connection (if you used the -N flag) or logout of the SSH connection (if you didn’t use the -N flag). Close that
  3. SSH back into Shannon, then type tmux kill-session -t TMUX_NAME to shut down the tmux session that was running your server. (Using File > Shut Down should close the tmux session, but it’s good to double check.)