Setup to process FSD data on neutrino01

After the collection of data a few steps are needed to easily use them. One can flow the data to convert them in the hdf5 format, easy to handle. Otherwise one can also directly look at the raw data using the afi viewer. We will see how to set this up on the neutrino01 server. Some information concerning errors can be found on the Debug page.

ndlar_flow

To install ndlar_flow the README on the git repo contains probably the most up-to-date information. This page is based on the following commit.

Installation

A good practice is to work in a virtual envirnment, venv will be used in this tutorial. The ndlar_flow repo is based on the h5flow framework, therefore it will be installed first and finally ndlar_flow could be installed. If one wants to work with a jupyter notebook, an additional step would be needed due to the version of python installed on the server.

virtual environment

Create and activate a virtual environment:

[user@neutrino01 ~]$ python -m venv ndlar_flow.venv
[user@neutrino01 ~]$ source ndlar_flow.venv/bin/activate

(Note that the name of the environment can be changed) To be able to start a jupyter notebook later, an additional step is needer: On neutrino01 the default python version is too old to install the latest Jupyter version. A solution is to install pyenv in your virtual environment.

pyenv [optional]

The default python version on neutrino01 is is

[user@neutrino01 ~]$ python --version
Python 3.9.21

which is unfortunately too old support the latest Jupyter version. A lot of programmes depend on python which make it quite an hassle to upgrade it and additionally the admin right are needed to do it. A solution is to install a new python via the package pyenv and to do it in the virtual environment previously created. First clone the pyenv repository

(ndlar_flow.venv) [user@neutrino01 ~]$ git clone https://github.com/pyenv/pyenv.git ~/.pyenv

Then you will need to set up the environment variables and evaluate the initialisation script

(ndlar_flow.venv) [user@neutrino01 ~]$ export PYENV_ROOT="$HOME/.pyenv"
(ndlar_flow.venv) [user@neutrino01 ~]$ [[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
(ndlar_flow.venv) [user@neutrino01 ~]$ eval "$(pyenv init - bash)"

To avoid to run the initialisation every time the virtual environment is started, you can add it to the activate script. I would propose to create a setup script, e.g. ~/setup_venv.sh and add

#!/bin/bash
echo "Running setup script..."

# Setup pyenv
export PYENV_ROOT="$HOME/.pyenv"
[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init - bash)"
</code>
Then you can add the following commands to the <code>ndlar_flow.venv/bin/activate</code> script:
<pre>
# Run a setup script if it exist
SETUP_SCRIPT="~/setup_venv.sh"
if [ -f "$SETUP_SCRIPT" ]; then
    source "$SETUP_SCRIPT"
fi

Then every time you activate the virtual environment the setup script will be run and pyenv configured.

Finally to install a new python version, e.g. 3.11.9, activate your environment and run the following command:

(ndlar_flow.venv) [user@neutrino01 ~]$ pyenv install 3.10.4

To list the installed version run

(ndlar_flow.venv) [user@neutrino01 ~]$ pyenv versions

, the star indicating the currently used one. Then you can switch to the desired version (e.g. 3.11.9) with

(ndlar_flow.venv) [user@neutrino01 ~]$ pyenv global 3.11.9

(it is also possible to switch only the local version if you replace the global parameter). Now if we check again which python version is used we get

(ndlar_flow.venv) [user@neutrino01 ~]$ python --version
Python 3.11.9

There is also an automatic installer if preferred, instruction and further documentation can be found in the git repo README.

h5flow framework

Now we can install the h5flow framework

(ndlar_flow.venv) [user@neutrino01 ~]$ git clone https://github.com/larpix/h5flow
(ndlar_flow.venv) [user@neutrino01 ~]$ cd h5flow
(ndlar_flow.venv) [user@neutrino01 h5flow]$ pip install -e .
(ndlar_flow.venv) [user@neutrino01 h5flow]$ cd ..
(ndlar_flow.venv) [user@neutrino01 ~]$

ndlar_flow

And finally we can install ndlar_flow

(ndlar_flow.venv) [user@neutrino01 ~]$ git clone git@github.com:DUNE/ndlar_flow.git
(ndlar_flow.venv) [user@neutrino01 ~]$ cd ndlar_flow
(ndlar_flow.venv) [user@neutrino01 ndlar_flow]$ pip install -e .

Note that the -e option installs the packages in "editable" mode, meaning any changes you make will take effect without the need to re-run pip install.

How to use

A good starting point is the README of the git repo, it describes how to run simple workflow such as light event building and also propose some tutorials. If you want to develop new script or update current one, a good practice is to create a new branch in the git project and work there. To edit the scripts traditional editors like vim are available on the server, but there is also the possibility to open the repository on the local machine. A very interactive solution is to start a jupyter server on the remote neutrino01 server and then open a jupyter notebook from your local machine through an ssh tunnel. A second practical possibility is to ssh to the server through Visual Studio Code (VS Code), which provides a nice coding environment. In both case you also have access to the remote terminal inside the programme. In my opinion the two options are very complementary, VS Code to write code easily and jupyter notebook to use it and play around with it.

Visual Studio Code

To use this program you will simply have to download VS Code and the ssh extension. Detailed information can be found here.

Jupyter Notebook

The Jupyter Notebook option is slightly more demanding but also accessible. First you will have to download jupyter in your virtual environment on the server

(ndlar_flow.venv) [user@neutrino01 ~]$ pip install jupyter

Then once the installation success full you can start a jupyter server without openning a browser by running

(ndlar_flow.venv) [user@neutrino01 ~]$ jupyter notebook --no-browser --port=8080

, you can choose the desired port number. Once the server is running you can open a new terminal on your local machine and open an ssh tunnel to the previously chosen port on the remote server

user_local ~/$ ssh -L 8080:localhost:8080 user@neutrino01.lhep.unibe.ch

Once this is done, you can go back to the first terminal where the jupyter server is running and look for the following line

[C 2025-06-18 14:27:48.107 ServerApp] 
    
    To access the server, open this file in a browser:
        file:///terabig/user/.local/share/jupyter/runtime/jpserver-664549-open.html
    Or copy and paste one of these URLs:
        http://localhost:8080/tree?token=<some number and letter>
        http://127.0.0.1:8080/tree?token=<some number and letter>

and copy the address of the server, e.g. here http://localhost:8080/tree?token=<some number and letter> in your favorite browser on your local machine. Note that some browser have difficulties with jupyter notebook, download an other one if your is not compatible. Now you can select a file and start work on it, good luck :).

AFI viewer

Ask Jan.