Setup to process FSD data on neutrino01 - Revision history

Nsallin at 16:40, 10 December 2025

2025-12-10T16:40:48Z

← Older revision		Revision as of 18:40, 10 December 2025
Line 135:		Line 135:
	On neutrino01:		On neutrino01:
	<pre>		<pre>
	cd afi-viewer/adc64-tlv/AFIViewer/build2/		cd ~/../jkunzmann/afi-viewer/adc64-tlv/AFIViewer/build2/
	./ADC64Viewer		./ADC64Viewer
	</pre>		</pre>
	Ask Jan for more details.		Ask Jan for more details.

Nsallin at 16:39, 10 December 2025

2025-12-10T16:39:29Z

← Older revision		Revision as of 18:39, 10 December 2025
Line 133:		Line 133:

	= AFI viewer =		= AFI viewer =
	Ask Jan.		On neutrino01:
			<pre>
			cd afi-viewer/adc64-tlv/AFIViewer/build2/
			./ADC64Viewer
			</pre>
			Ask Jan for more details.

Nsallin at 10:17, 27 November 2025

2025-11-27T10:17:33Z

← Older revision		Revision as of 12:17, 27 November 2025
Line 3:		Line 3:
	Recall, to connect to neutrino01:		Recall, to connect to neutrino01:
	<pre>		<pre>
	[user@computer ~]$ssh user@neutrino01.lhep.unibe.ch		[user@computer ~]$ ssh user@neutrino01.lhep.unibe.ch
	</pre>		</pre>

Nsallin: add neutrino01 address

2025-11-27T10:17:20Z

add neutrino01 address

← Older revision		Revision as of 12:17, 27 November 2025
Line 1:		Line 1:
	After the collection of data a few steps are needed to easily use them. One can flow the data to convert them in the ''hdf5'' format, easy to handle. Otherwise one can also directly look at the raw data using the afi viewer. We will see how to set this up on the <code>neutrino01</code> server. Some information concerning errors can be found on the [[Debug]] page.		After the collection of data a few steps are needed to easily use them. One can flow the data to convert them in the ''hdf5'' format, easy to handle. Otherwise one can also directly look at the raw data using the afi viewer. We will see how to set this up on the <code>neutrino01</code> server. Some information concerning errors can be found on the [[Debug]] page.

			Recall, to connect to neutrino01:
			<pre>
			[user@computer ~]$ssh user@neutrino01.lhep.unibe.ch
			</pre>

	= ndlar_flow =		= ndlar_flow =

Nsallin at 09:32, 19 June 2025

2025-06-19T09:32:33Z

← Older revision		Revision as of 11:32, 19 June 2025
Line 96:		Line 96:

	== How to use ==		== How to use ==
	A good starting point is the [https://github.com/DUNE/ndlar_flow# ''README''] of the git repo, it describes how to run simple workflow such as light event building and also propose some tutorials. If you want to develop new script or update current one, a good practice is to create a new branch in the git project and work there. To edit the scripts traditional editors like ''vim'' are available on the server, but there is also ~~the possibility~~ to open the repository on ~~the~~ local machine. ~~A very~~ interactive solution is to start a ~~jupyter~~ server on the remote neutrino01 server and then open a ~~jupyter notebook~~ from your local machine through an ssh tunnel. A second ~~practical~~ possibility is to ssh to the server through Visual Studio Code (VS Code), which provides a nice coding environment. In both case you also have access to the remote terminal inside the programme. In my opinion the two options are very complementary, VS Code to write code easily and jupyter notebook to use it and play around with it.		A good starting point is the [https://github.com/DUNE/ndlar_flow# ''README''] of the git repo, it describes how to run simple workflow such as light event building and also propose some tutorials. If you want to develop new script or update current one, a good practice is to create a new branch in the git project and work there. To edit the scripts, traditional editors like ''vim'' are available on the server, but there are also possibilities to open the repository on your local machine. An interactive solution is to start a Jupyter server on the remote neutrino01 server and then open a Jupyter Notebook from your local machine through a ssh tunnel. A second possibility is to ssh to the server through Visual Studio Code (VS Code), which provides a nice coding environment. In both case you also have access to the remote terminal inside the programme. In my opinion the two options are very complementary, VS Code to write code easily and jupyter notebook to use it and play around with it.

	=== Visual Studio Code ===		=== Visual Studio Code ===
Line 102:		Line 102:

	=== Jupyter Notebook ===		=== Jupyter Notebook ===
	~~The Jupyter Notebook option is slightly more demanding but also accessible.~~ First you will have to ~~download jupyter~~ in your virtual environment on the server		First you will have to install Jupyter in your virtual environment on the server
	<pre>		<pre>
	(ndlar_flow.venv) [user@neutrino01 ~]$ pip install jupyter		(ndlar_flow.venv) [user@neutrino01 ~]$ pip install jupyter
	</pre>		</pre>
	Then once the ~~installation success full~~ you can start a jupyter server without ~~openning~~ a browser by running		Then, once the installed successfully, you can start a jupyter server without opening a browser by running
	<pre>		<pre>
	(ndlar_flow.venv) [user@neutrino01 ~]$ jupyter notebook --no-browser --port=8080		(ndlar_flow.venv) [user@neutrino01 ~]$ jupyter notebook --no-browser --port=8080
	</pre>		</pre>
	, you can choose the desired port number. ~~Once~~ the server is running ~~you can open~~ a new terminal on your local machine ~~and~~ open an ssh tunnel to the previously chosen port on the remote server		,the no-browser open is necessary but you can choose the desired port number. When the server is running, from a new terminal on your local machine you can open an ssh tunnel to the previously chosen port on the remote server
	<pre>		<pre>
	user_local ~/$ ssh -L 8080:localhost:8080 user@neutrino01.lhep.unibe.ch		user_local ~/$ ssh -L 8080:localhost:8080 user@neutrino01.lhep.unibe.ch
	</pre>		</pre>
	~~Once this~~ is ~~done~~, you can go back to the first terminal where the jupyter server is running and look for the following line		When the connexion is established, you can go back to the first terminal where the jupyter server is running and look for the following line
	<pre>		<pre>
	[C 2025-06-18 14:27:48.107 ServerApp]		[C 2025-06-18 14:27:48.107 ServerApp]
Line 124:		Line 124:
	http://127.0.0.1:8080/tree?token=<some number and letter>		http://127.0.0.1:8080/tree?token=<some number and letter>
	</pre>		</pre>
	~~and copy~~ the address of the server, e.g. here <code>http://localhost:8080/tree?token=<some number and letter></code> in your ~~favorite~~ browser on your local machine. Note that some browser have difficulties with ~~jupyter notebook~~, download an other one if ~~your~~ is not compatible. Now you can select a file and start work on it, good luck :).		. Copy the address of the server, e.g. here <code>http://localhost:8080/tree?token=<some number and letter></code> in your favourite browser on your local machine, it will allow you to access the Jupyter server. Note that some browser have difficulties with Jupyter notebooks, download an other one if yours is not compatible. Now you can select a file and start to work on it, good luck :).


	= AFI viewer =		= AFI viewer =
	Ask Jan.		Ask Jan.

Nsallin at 12:42, 18 June 2025

2025-06-18T12:42:50Z

← Older revision		Revision as of 14:42, 18 June 2025
Line 60:		Line 60:
	Finally to install a new python version, e.g. 3.11.9, activate your environment and run the following command:		Finally to install a new python version, e.g. 3.11.9, activate your environment and run the following command:
	<pre>		<pre>
	(ndlar_flow.venv) [user@neutrino01 ~]$ pyenv install 3.10.4		(ndlar_flow.venv) [user@neutrino01 ~]$ pyenv install 3.11.9
	</pre>		</pre>
	To list the installed version run		To list the installed version run

Nsallin at 12:41, 18 June 2025

2025-06-18T12:41:55Z

← Older revision		Revision as of 14:41, 18 June 2025
Line 17:		Line 17:

	=== ''pyenv'' [optional] ===		=== ''pyenv'' [optional] ===
	The default python version on ''neutrino01'' is		The default python version on ''neutrino01''is
	is <!-- <syntaxhighlight lang="cpp"> [user@neutrino01 ~]$ python --version		<!--
	Python 3.9.21 </syntaxhighlight> -->		<syntaxhighlight lang="cpp"> [user@neutrino01 ~]$ python --version
			Python 3.9.21 </syntaxhighlight>
			-->
	<pre>		<pre>
	[user@neutrino01 ~]$ python --version		[user@neutrino01 ~]$ python --version

Nsallin at 12:41, 18 June 2025

2025-06-18T12:41:18Z

← Older revision		Revision as of 14:41, 18 June 2025
Line 13:		Line 13:
	[user@neutrino01 ~]$ source ndlar_flow.venv/bin/activate		[user@neutrino01 ~]$ source ndlar_flow.venv/bin/activate
	</pre>		</pre>
	(Note that the name of the environment can be changed)		(Note that the name of the environment can be changed).
	To be able to start a jupyter notebook later, an additional step is needer: On ''neutrino01'' the default python version is too old to install the latest Jupyter version. A solution is to install ''pyenv'' in your virtual environment.		To be able to start a jupyter notebook later, an additional step is needer: On ''neutrino01'' the default python version is too old to install the latest Jupyter version. A solution is to install ''pyenv'' in your virtual environment.

Nsallin at 12:40, 18 June 2025

2025-06-18T12:40:59Z

← Older revision		Revision as of 14:40, 18 June 2025
Line 14:		Line 14:
	</pre>		</pre>
	(Note that the name of the environment can be changed)		(Note that the name of the environment can be changed)
	To be able to start a jupyter notebook later, an additional step is needer: On ''neutrino01'' the default python version is <!-- <syntaxhighlight lang="cpp"> [user@neutrino01 ~]$ python --version		To be able to start a jupyter notebook later, an additional step is needer: On ''neutrino01'' the default python version is too old to install the latest Jupyter version. A solution is to install ''pyenv'' in your virtual environment.

			=== ''pyenv'' [optional] ===
			The default python version on ''neutrino01'' is
			is <!-- <syntaxhighlight lang="cpp"> [user@neutrino01 ~]$ python --version
	Python 3.9.21 </syntaxhighlight> -->		Python 3.9.21 </syntaxhighlight> -->
	<pre>		<pre>
Line 20:		Line 24:
	Python 3.9.21		Python 3.9.21
	</pre>		</pre>
	which is too old ~~to install~~ the latest ~~jupyter~~ version. A solution is to install ''pyenv'' in ~~your~~ virtual environment.		which is unfortunately too old support the latest Jupyter version. A lot of programmes depend on python which make it quite an hassle to upgrade it and additionally the admin right are needed to do it. A solution is to install a new python via the package ''pyenv'' and to do it in the virtual environment previously created.
			First clone the ''pyenv'' repository
	~~===~~ ''pyenv'' ~~[optional] ===~~
	~~Clone the~~ repository
	<pre>		<pre>
	(ndlar_flow.venv) [user@neutrino01 ~]$ git clone https://github.com/pyenv/pyenv.git ~/.pyenv		(ndlar_flow.venv) [user@neutrino01 ~]$ git clone https://github.com/pyenv/pyenv.git ~/.pyenv

Nsallin at 12:35, 18 June 2025

2025-06-18T12:35:12Z

← Older revision		Revision as of 14:35, 18 June 2025
Line 25:		Line 25:
	Clone the repository		Clone the repository
	<pre>		<pre>
	[user@neutrino01 ~]$ git clone https://github.com/pyenv/pyenv.git ~/.pyenv		(ndlar_flow.venv) [user@neutrino01 ~]$ git clone https://github.com/pyenv/pyenv.git ~/.pyenv
	</pre>		</pre>
	Then you will need to set up the environment variables and evaluate the initialisation script		Then you will need to set up the environment variables and evaluate the initialisation script
	<pre>		<pre>
	[user@neutrino01 ~]$ export PYENV_ROOT="$HOME/.pyenv"		(ndlar_flow.venv) [user@neutrino01 ~]$ export PYENV_ROOT="$HOME/.pyenv"
	[user@neutrino01 ~]$ [[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"		(ndlar_flow.venv) [user@neutrino01 ~]$ [[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
	[user@neutrino01 ~]$ eval "$(pyenv init - bash)"		(ndlar_flow.venv) [user@neutrino01 ~]$ eval "$(pyenv init - bash)"
	</pre>		</pre>

Line 62:		Line 62:
	(ndlar_flow.venv) [user@neutrino01 ~]$ pyenv versions		(ndlar_flow.venv) [user@neutrino01 ~]$ pyenv versions
	</pre>		</pre>
	Then you can switch to the desired version (e.g. 3.11.9) with		, the star indicating the currently used one. Then you can switch to the desired version (e.g. 3.11.9) with
	<pre>		<pre>
	(ndlar_flow.venv) [user@neutrino01 ~]$ pyenv global 3.11.9		(ndlar_flow.venv) [user@neutrino01 ~]$ pyenv global 3.11.9
Line 75:		Line 75:
	Now we can install the ''h5flow'' framework		Now we can install the ''h5flow'' framework
	<pre>		<pre>
	[user@neutrino01 ~]$ git clone https://github.com/larpix/h5flow		(ndlar_flow.venv) [user@neutrino01 ~]$ git clone https://github.com/larpix/h5flow
	[user@neutrino01 ~]$ cd h5flow		(ndlar_flow.venv) [user@neutrino01 ~]$ cd h5flow
	[user@neutrino01 h5flow]$ pip install -e .		(ndlar_flow.venv) [user@neutrino01 h5flow]$ pip install -e .
	[user@neutrino01 h5flow]$ cd ..		(ndlar_flow.venv) [user@neutrino01 h5flow]$ cd ..
	[user@neutrino01 ~]$		(ndlar_flow.venv) [user@neutrino01 ~]$
	</pre>		</pre>

Line 85:		Line 85:
	And finally we can install ''ndlar_flow''		And finally we can install ''ndlar_flow''
	<pre>		<pre>
	[user@neutrino01 ~]$ git clone git@github.com:DUNE/ndlar_flow.git		(ndlar_flow.venv) [user@neutrino01 ~]$ git clone git@github.com:DUNE/ndlar_flow.git
	[user@neutrino01 ~]$ cd ndlar_flow		(ndlar_flow.venv) [user@neutrino01 ~]$ cd ndlar_flow
	[user@neutrino01 ndlar_flow]$ pip install -e .		(ndlar_flow.venv) [user@neutrino01 ndlar_flow]$ pip install -e .
	</pre>		</pre>
	Note that the <code>-e</code> option installs the packages in "editable" mode, meaning any changes you make will take effect without the need to re-run <code>pip install</code>.		Note that the <code>-e</code> option installs the packages in "editable" mode, meaning any changes you make will take effect without the need to re-run <code>pip install</code>.

	=== jupyter notebook ===		== How to use ==
			A good starting point is the [https://github.com/DUNE/ndlar_flow# ''README''] of the git repo, it describes how to run simple workflow such as light event building and also propose some tutorials. If you want to develop new script or update current one, a good practice is to create a new branch in the git project and work there. To edit the scripts traditional editors like ''vim'' are available on the server, but there is also the possibility to open the repository on the local machine. A very interactive solution is to start a jupyter server on the remote neutrino01 server and then open a jupyter notebook from your local machine through an ssh tunnel. A second practical possibility is to ssh to the server through Visual Studio Code (VS Code), which provides a nice coding environment. In both case you also have access to the remote terminal inside the programme. In my opinion the two options are very complementary, VS Code to write code easily and jupyter notebook to use it and play around with it.

			=== Visual Studio Code ===
			To use this program you will simply have to download VS Code and the ssh extension. Detailed information can be found [https://code.visualstudio.com/docs/remote/ssh here].

			=== Jupyter Notebook ===
			The Jupyter Notebook option is slightly more demanding but also accessible. First you will have to download jupyter in your virtual environment on the server
			<pre>
			(ndlar_flow.venv) [user@neutrino01 ~]$ pip install jupyter
			</pre>
			Then once the installation success full you can start a jupyter server without openning a browser by running
			<pre>
			(ndlar_flow.venv) [user@neutrino01 ~]$ jupyter notebook --no-browser --port=8080
			</pre>
			, you can choose the desired port number. Once the server is running you can open a new terminal on your local machine and open an ssh tunnel to the previously chosen port on the remote server
			<pre>
			user_local ~/$ ssh -L 8080:localhost:8080 user@neutrino01.lhep.unibe.ch
			</pre>
			Once this is done, you can go back to the first terminal where the jupyter server is running and look for the following line
			<pre>
			[C 2025-06-18 14:27:48.107 ServerApp]

			To access the server, open this file in a browser:
			file:///terabig/user/.local/share/jupyter/runtime/jpserver-664549-open.html
			Or copy and paste one of these URLs:
			http://localhost:8080/tree?token=<some number and letter>
			http://127.0.0.1:8080/tree?token=<some number and letter>
			</pre>
			and copy the address of the server, e.g. here <code>http://localhost:8080/tree?token=<some number and letter></code> in your favorite browser on your local machine. Note that some browser have difficulties with jupyter notebook, download an other one if your is not compatible. Now you can select a file and start work on it, good luck :).


	= AFI viewer =		= AFI viewer =
			Ask Jan.