Setup to process FSD data on neutrino01

After the collection of data a few steps are needed to easily use them. One can flow the data to convert them in the hdf5 format, easy to handle. Otherwise one can also directly look at the raw data using the afi viewer. We will see how to set this up on the neutrino01 server. Some information concerning errors can be found on the Debug page.

ndlar_flow

To install ndlar_flow the 'README' on the git repo contains probably the most up-to-date information. This page is based on the following commit.

Installation

A good practice is to work in a virtual envirnment, venv will be used in this tutorial. The ndlar_flow repo is based on the h5flow framework, therefore it will be installed first and finally ndlar_flow could be installed. If one wants to work with a jupyter notebook, an additional step would be needed due to the version of python installed on the server.

virtual environment

Create and activate a virtual environment:

[user@neutrino01 ~]$ python -m venv ndlar_flow.venv
[user@neutrino01 ~]$ source ndlar_flow.venv/bin/activate

(Note that the name of the environment can be changed) To be able to start a jupyter notebook later, an additional step is needer: On neutrino01 the default python version is

[user@neutrino01 ~]$ python --version
Python 3.9.21

which is too old to install the latest jupyter version. A solution is to install pyenv in your virtual environment.

pyenv [optional]

Clone the repository

[user@neutrino01 ~]$ git clone https://github.com/pyenv/pyenv.git ~/.pyenv

Then you will need to set up the environment variables and evaluate the initialisation script

[user@neutrino01 ~]$ export PYENV_ROOT="$HOME/.pyenv"
[user@neutrino01 ~]$ [[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
[user@neutrino01 ~]$ eval "$(pyenv init - bash)"

To avoid to run the initialisation every time the virtual environment is started, you can add it to the activate script. I would propose to create a setup script, e.g. ~/setup_venv.sh and add

#!/bin/bash
echo "Running setup script..."

# Setup pyenv
export PYENV_ROOT="$HOME/.pyenv"
[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init - bash)"
</code>
Then you can add the following commands to the <code>ndlar_flow.venv/bin/activate</code> script:
<pre>
# Run a setup script if it exist
SETUP_SCRIPT="~/setup_venv.sh"
if [ -f "$SETUP_SCRIPT" ]; then
    source "$SETUP_SCRIPT"
fi

Then every time you activate the virtual environment the setup script will be run and pyenv configured.

Finally to install a new python version, e.g. 3.11.9, activate your environment and run the following command:

(ndlar_flow.venv) [user@neutrino01 ~]$ pyenv install 3.10.4

To list the installed version run

(ndlar_flow.venv) [user@neutrino01 ~]$ pyenv versions

Then you can switch to the desired version (e.g. 3.11.9) with

(ndlar_flow.venv) [user@neutrino01 ~]$ pyenv global 3.11.9

Now if we check again which python version is used we get

(ndlar_flow.venv) [user@neutrino01 ~]$ python --version
Python 3.11.9

h5flow framework

Install the h5flow framework

[user@neutrino01 ~]$ git clone https://github.com/larpix/h5flow
[user@neutrino01 ~]$ cd h5flow
[user@neutrino01 h5flow]$ pip install -e .
[user@neutrino01 h5flow]$ cd ..
[user@neutrino01 ~]$