Use IPC for local calib connections

[davidrohr]

@chiarazampolli : Haven't tested this yet, but this is what I had in mind for the proxy channels:
So for testing locally, but in 2 separate shells, i.e. without CALIB_LOCAL_INTEGRATED_AGGREGATOR, one has to set both CALIB_PROXIES and CALIB_LOCAL_AGGREGATOR. Then it will use SHM transport with a socket file based on the channel name.
If CALIB_PROXIES is set, but CALIB_LOCAL_AGGREGATOR is not, i.e. when we run online on the EPN, we use zeromq transport and leave the address empty, to be filled by FairMQ automatically, in the same way we are doing for the calib workflows at P2 right now already.

[chiarazampolli]

Ciao @davidrohr ,

Seems fine to me, thanks!

EPN:

1. To run locally:

source setenv_extra.sh; OVERRIDE_SESSION=default IS_SIMULATED_DATA=0 WORKFLOWMODE=print DISABLE_ROOT_OUTPUT="" TFDELAY=5 NTIMEFRAMES=100 SHMSIZE=16000000000 DDSHMSIZE=32000 WORKFLOW_PARAMETERS=CALIB,CALIB_LOCAL_INTEGRATED_AGGREGATOR IGNORE_EXISTING_SHMFILES=1 $O2_ROOT/prodtests/full-system-test/run-workflow-on-inputlist.sh CTF list_epn1.list

adds the calib processes to the workflow:

...
o2-secondary-vertexing-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 --disable-root-input  --vertexing-sources ITS-TPC,TPC-TRD,ITS-TPC-TRD,TPC-TOF,ITS-TPC-TOF,TPC-TRD-TOF,ITS-TPC-TRD-TOF,ITS,MFT,TPC,TOF,FT0,MID,FDD,FV0,TRD --pipeline secondary-vertexing:1  --configKeyValues "NameConf.mDirGeom=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirMatLUT=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirCollContext=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirGRP=/home/zampolli/O2/CalibWithBeams/epn1;keyval.input_dir=/home/zampolli/O2/CalibWithBeams/epn1;keyval.output_dir=/dev/null;;" | \
o2-tpc-scdcalib-interpolation-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0  --disable-root-input --pipeline tpc-track-interpolation:1  --configKeyValues "NameConf.mDirGeom=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirMatLUT=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirCollContext=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirGRP=/home/zampolli/O2/CalibWithBeams/epn1;keyval.input_dir=/home/zampolli/O2/CalibWithBeams/epn1;keyval.output_dir=/dev/null;;" | \
o2-calibration-mean-vertex-calibration-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 | \
o2-calibration-tof-calib-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 --do-lhc-phase --tf-per-slot 10 | \
o2-calibration-tof-calib-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 --do-channel-offset --update-at-end-of-run-only --min-entries 8 --range 100000 | \
o2-calibration-tof-diagnostic-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 --tf-per-slot 26400 | \
o2-calibration-trd-vdrift-exb --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 | \
o2-dpl-run --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0

2. Aggregator on the same node, transport with a socket file based on the channel name.

source setenv_extra.sh; OVERRIDE_SESSION=default IS_SIMULATED_DATA=0 WORKFLOWMODE=print DISABLE_ROOT_OUTPUT="" TFDELAY=5 NTIMEFRAMES=100 SHMSIZE=16000000000 DDSHMSIZE=32000 WORKFLOW_PARAMETERS=CALIB,CALIB_PROXIES,CALIB_LOCAL_AGGREGATOR IGNORE_EXISTING_SHMFILES=1 $O2_ROOT/prodtests/full-system-test/run-workflow-on-inputlist.sh CTF list_epn1.list

produces:
CALIBDATASPEC_BARREL = pvtx:GLO/PVTX/0;calibTOF:TOF/CALIBDATA/0;diagWords:TOF/DIAFREQ/0;angResHistoTRD:TRD/ANGRESHISTS/0
PROXY_CONN = --proxy-channel-name aggregator-proxy-barrel --channel-config "name=aggregator-proxy-barrel,method=connect,type=push,transport=shmem,address=ipc://@aggregator-shm-barrel,rateLogging=1"

and then in the workflow:

...
o2-secondary-vertexing-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 --disable-root-input  --vertexing-sources ITS-TPC,TPC-TRD,ITS-TPC-TRD,TPC-TOF,ITS-TPC-TOF,TPC-TRD-TOF,ITS-TPC-TRD-TOF,ITS,MFT,TPC,TOF,FT0,MID,FDD,FV0,TRD --pipeline secondary-vertexing:1  --configKeyValues "NameConf.mDirGeom=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirMatLUT=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirCollContext=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirGRP=/home/zampolli/O2/CalibWithBeams/epn1;keyval.input_dir=/home/zampolli/O2/CalibWithBeams/epn1;keyval.output_dir=/dev/null;;" | \
o2-tpc-scdcalib-interpolation-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0  --disable-root-input --pipeline tpc-track-interpolation:1  --configKeyValues "NameConf.mDirGeom=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirMatLUT=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirCollContext=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirGRP=/home/zampolli/O2/CalibWithBeams/epn1;keyval.input_dir=/home/zampolli/O2/CalibWithBeams/epn1;keyval.output_dir=/dev/null;;" | \
o2-dpl-output-proxy --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 --dataspec "pvtx:GLO/PVTX/0;calibTOF:TOF/CALIBDATA/0;diagWords:TOF/DIAFREQ/0;angResHistoTRD:TRD/ANGRESHISTS/0" --proxy-channel-name aggregator-proxy-barrel --channel-config "name=aggregator-proxy-barrel,method=connect,type=push,transport=shmem,address=ipc://@aggregator-shm-barrel,rateLogging=1" | \
o2-dpl-run --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 

2. Aggregator on a different node, with ZMQ:

source setenv_extra.sh; OVERRIDE_SESSION=default IS_SIMULATED_DATA=0 WORKFLOWMODE=print DISABLE_ROOT_OUTPUT="" TFDELAY=5 NTIMEFRAMES=100 SHMSIZE=16000000000 DDSHMSIZE=32000 WORKFLOW_PARAMETERS=CALIB,CALIB_PROXIES IGNORE_EXISTING_SHMFILES=1 $O2_ROOT/prodtests/full-system-test/run-workflow-on-inputlist.sh CTF list_epn1.list

Produces:

CALIBDATASPEC_BARREL = pvtx:GLO/PVTX/0;calibTOF:TOF/CALIBDATA/0;diagWords:TOF/DIAFREQ/0;angResHistoTRD:TRD/ANGRESHISTS/0

and

PROXY_CONN = --proxy-channel-name aggregator-proxy-barrel --channel-config "name=aggregator-proxy-barrel,method=connect,type=push,transport=zeromq,rateLogging=1"

in the workflow:

...
o2-secondary-vertexing-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 --disable-root-input  --vertexing-sources ITS-TPC,TPC-TRD,ITS-TPC-TRD,TPC-TOF,ITS-TPC-TOF,TPC-TRD-TOF,ITS-TPC-TRD-TOF,ITS,MFT,TPC,TOF,FT0,MID,FDD,FV0,TRD --pipeline secondary-vertexing:1  --configKeyValues "NameConf.mDirGeom=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirMatLUT=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirCollContext=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirGRP=/home/zampolli/O2/CalibWithBeams/epn1;keyval.input_dir=/home/zampolli/O2/CalibWithBeams/epn1;keyval.output_dir=/dev/null;;" | \
o2-tpc-scdcalib-interpolation-workflow --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0  --disable-root-input --pipeline tpc-track-interpolation:1  --configKeyValues "NameConf.mDirGeom=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirMatLUT=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirCollContext=/home/zampolli/O2/CalibWithBeams/epn1;NameConf.mDirGRP=/home/zampolli/O2/CalibWithBeams/epn1;keyval.input_dir=/home/zampolli/O2/CalibWithBeams/epn1;keyval.output_dir=/dev/null;;" | \
o2-dpl-output-proxy --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 --dataspec "pvtx:GLO/PVTX/0;calibTOF:TOF/CALIBDATA/0;diagWords:TOF/DIAFREQ/0;angResHistoTRD:TRD/ANGRESHISTS/0" --proxy-channel-name aggregator-proxy-barrel --channel-config "name=aggregator-proxy-barrel,method=connect,type=push,transport=zeromq,rateLogging=1" | \
o2-dpl-run --session default --severity info --shm-segment-id 0 --shm-segment-size 16000000000  --early-forward-policy noraw --monitoring-backend no-op:// --fairmq-rate-logging 0 

For the aggregator:

1. when on the same node:

WORKFLOWMODE=print WORKFLOW_PARAMETERS=CALIB_PROXIES,CALIB_LOCAL_AGGREGATOR WORKFLOW_DETECTORS=ITS,TPC,TOF,FV0,FT0,FDD,MID,MFT CCDB_POPULATOR_UPLOAD_PATH=http://ccdb-test.cern.ch:8080 /home/zampolli/SOFT/alibuild/ali-o2-dev/O2/prodtests/full-system-test/aggregator-workflow.sh

produces:

CALIBDATASPEC_BARREL = pvtx:GLO/PVTX/0;calibTOF:TOF/CALIBDATA/0;diagWords:TOF/DIAFREQ/0
PROXY_CONN = --proxy-channel-name aggregator-proxy-barrel --channel-config "name=aggregator-proxy-barrel,method=bind,type=pull,transport=shmem,address=ipc://@aggregator-shm-barrel,rateLogging=1"
Workflow command adding aggregator:
o2-dpl-raw-proxy --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 --dataspec "pvtx:GLO/PVTX/0;calibTOF:TOF/CALIBDATA/0;diagWords:TOF/DIAFREQ/0" --proxy-channel-name aggregator-proxy-barrel --channel-config "name=aggregator-proxy-barrel,method=bind,type=pull,transport=shmem,address=ipc://@aggregator-shm-barrel,rateLogging=1" |
o2-calibration-mean-vertex-calibration-workflow --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 |
o2-calibration-tof-calib-workflow --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 --do-lhc-phase --tf-per-slot 10 |
o2-calibration-tof-calib-workflow --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 --do-channel-offset --update-at-end-of-run-only --min-entries 8 --range 100000 |
o2-calibration-tof-diagnostic-workflow --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 --tf-per-slot 26400 |
o2-calibration-ccdb-populator-workflow --ccdb-path http://ccdb-test.cern.ch:8080 --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 |
o2-dpl-run --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 -b

2. on a different node:

WORKFLOWMODE=print WORKFLOW_PARAMETERS=CALIB_PROXIES WORKFLOW_DETECTORS=ITS,TPC,TOF,FV0,FT0,FDD,MID,MFT CCDB_POPULATOR_UPLOAD_PATH=http://ccdb-test.cern.ch:8080 /home/zampolli/SOFT/alibuild/ali-o2-dev/O2/prodtests/full-system-test/aggregator-workflow.sh

produces:

CALIBDATASPEC_BARREL = pvtx:GLO/PVTX/0;calibTOF:TOF/CALIBDATA/0;diagWords:TOF/DIAFREQ/0
PROXY_CONN = --proxy-channel-name aggregator-proxy-barrel --channel-config "name=aggregator-proxy-barrel,method=bind,type=pull,transport=zeromq,rateLogging=1"

o2-dpl-raw-proxy --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 --dataspec "pvtx:GLO/PVTX/0;calibTOF:TOF/CALIBDATA/0;diagWords:TOF/DIAFREQ/0" --proxy-channel-name aggregator-proxy-barrel --channel-config "name=aggregator-proxy-barrel,method=bind,type=pull,transport=zeromq,rateLogging=1" |
o2-calibration-mean-vertex-calibration-workflow --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 |
o2-calibration-tof-calib-workflow --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 --do-lhc-phase --tf-per-slot 10 |
o2-calibration-tof-calib-workflow --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 --do-channel-offset --update-at-end-of-run-only --min-entries 8 --range 100000 |
o2-calibration-tof-diagnostic-workflow --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 --tf-per-slot 26400 |
o2-calibration-ccdb-populator-workflow --ccdb-path http://ccdb-test.cern.ch:8080 --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 |
o2-dpl-run --session default --severity detail --shm-segment-id 0 --shm-segment-size 8589934592 --early-forward-policy noraw --fairmq-rate-logging 0 -b

Chiara

[chiarazampolli]

Before merging, let me test it in run mode.

[chiarazampolli]

We need this one:
davidrohr#1

With that, I am getting errors on the aggregator:

[65268:aggregator-proxy-barrel]: [23:09:46][INFO] aggregator-proxy-barrel[0]: in: 1 (0.110372 MB) out: 0 (0 MB)
[65268:aggregator-proxy-barrel]: [23:09:47][INFO] aggregator-proxy-barrel[0]: in: 0 (0 MB) out: 0 (0 MB)
[65268:aggregator-proxy-barrel]: [23:09:48][INFO] aggregator-proxy-barrel[0]: in: 0 (0 MB) out: 0 (0 MB)
[65268:aggregator-proxy-barrel]: [23:09:49][INFO] aggregator-proxy-barrel[0]: in: 0 (0 MB) out: 0 (0 MB)
[65268:aggregator-proxy-barrel]: [23:09:49][ERROR] data on input 0 does not follow the O2 data model, DataHeader missing
[65268:aggregator-proxy-barrel]: [23:09:49][ERROR] data on input 2 does not follow the O2 data model, DataHeader missing
[65268:aggregator-proxy-barrel]: [23:09:49][ERROR] data on input 3 does not follow the O2 data model, DataHeader missing
[65268:aggregator-proxy-barrel]: [23:09:49][ERROR] data on input 4 does not follow the O2 data model, DataHeader missing
[65268:aggregator-proxy-barrel]: [23:09:49][ERROR] data on input 5 does not follow the O2 data model, DataHeader missing
[65268:aggregator-proxy-barrel]: [23:09:49][ERROR] data on input 6 does not follow the O2 data model, DataHeader missing
[65268:aggregator-proxy-barrel]: [23:09:49][ERROR] data on input 7 does not follow the O2 data model, DataHeader missing
[65269:internal-dpl-ccdb-backend]: [23:09:49][ERROR] Dummy creation time is not supported for CCDB objects. Setting creation to last one used.

I need to investigate.

[chiarazampolli]

Must have been some shm file left hanging, because cleaning up did the trick.
Now everything works fine.
There is one error left, but not related to the setup but to TOF (@noferini - I will take a look):

97447:calib-lhcclock-calibration]: Error in <TFile::TFile>: file /archive/O2/CalibWithBeams/aggregator/localTimeSlewing.root does not exist

But it is harmless.
With 2 EPNs (local) and the aggregator, 100 TFs per EPN were processed correctly. CCDB entries also were uploaded (e.g. GLO/Calib/MeanVertex). There are other small issues with the TOF diagnostic calib (validity of the object). The other calibrations did not get enough statistics.

So I would merge.

[chiarazampolli]

Tests ok, merging.