John Harvey, Marco Cattaneo, Jean-Jacques Blaising, Andreas Pfeiffer, Dirk Duellmann, Lucia Silvestris, Lassi Tuura, Fabiola Gianotti, Andrea DellAcqua, David Quarrie, Alberto Aimar, Pere Mato, Fons Rademakers, Gabriele Cosmo, Federico Carminati
Applications Area plans discussion
The Following message had been circulated to the AF by Pere prior to the meeting in order to guide the discussion :
The goal should be to clarify the issues raised by the experiments to the AA Plans document and identify the necessary changes or additions to the existing draft such that can be endorsed by the experiments. We need to stick to a purely technical discussion (requirements, needs, practical problems, architectural issues, etc.).
1) Preserve SEAL functionality. SEAL software packages will be discontinued in the future ( < 1 year ) unless the new core libraries project (ROOT) assumes its long term maintenance, which I don't think is reasonable. Therefore the experiments need to make sure that what they are using/relying on in the SEAL (and PI) packages appears in one form or another in the software provided by the new ROOT project. Some of the libraries are being taken on board (Mathlib, Reflex, Plug-in manager) but others probably need to be added in the plans. Topics the experiments have already mentioned in their feedback include: component model classes, dictionaries generated with gccxml, AIDA implementations, etc.
2) Help from the ROOT project to integrate the changes into the experiment/POOL software. The experiments rely on POOL for their data persistency, therefore any changes in ROOT and the discontinuation of SEAL need to be integrated in the existing POOL framework and experiments’ frameworks. We need to discuss what of this work needs to be stated in the plans.
3) Partial releases and mixing ROOT versions. ROOT packages are being (or
going to be) used as the foundation for the experiments’ frameworks, to provide
the I/O for the their experiment-wide applications, and at the same are being
used by the end-physicists doing analysis.
In this kind of usage you want on the one hand stability implying that changes
of versions will typically undertake a long period of validation, and on the
other hand you want to have at the disposal of the end-users the latest bug fix
in the analysis packages. We need to discuss how this can be achieved with the
current structure of the ROOT packages.
4) Lightweight linear algebra package. ATLAS, CMS and LHCb are requesting to
in have math functionality as much decoupled as possible from the rest of the
ROOT framework. We should discuss the technical advantages and disadvantages of
doing this.
Federico would like to have the following statement added here: "ALICE has as
a very strong requirement to maintain the tight integration of the math package
code and functionality with ROOT. In particular, the removal of the TObject
inheritance would not be acceptable for ALICE, as it would create a problem for
all the object which are now stored in ROOT polymorphic containers, which are at
the foundation of the AliRoot infrastructure."
Pere started with the goal for the meeting, which is to discuss the plan in view of clarifying the various points raised by the experiments [ATLAS, CMS and LHCb] in the area of the SEAL+ROOT merge and to propose changes to the work plan that reflect more completely the needs of the experiments. The other projects in the plan (SPI, POOL) can be sorted out outside the meeting. Architectural issues should also be included as technical issues.
It was agreed to go through the 4 topics listed above to guide the discussion. No additional points were suggested. Jean-Jacques reminded us that the meeting should stick to technical issues.
Pere said that the starting assumption is that SEAL and ROOT should merge, with the understanding that the functionality of both systems would be preserved. However the plan as it stands today lacks content on how the merging will preserve the existing functionality, including the interface offered to the client programs. The interface needs to be similar but not identical. The assumption is that the SEAL packages will be discontinued in the future, unless the new core project maintains them. What will be done to preserve the functionality should be in the plan. For example, experiments would like to see preserved the component model, AIDA interfaces, etc.
Lucia said that not only functionality must be preserved but also modularity in terms of the architecture. As in the past, CMS does not want SEAL to be discontinued in one year. CMS is based on it and the transition will take more time, unless LCG gives manpower to the experiments. Marco adds that modularity allows to migrate the various modules step by step and not all in one go.
Lucia noted that the plug-in manager proposal presented in the AA meeting starts from the current ROOT and not from what 3 out of 4 experiments have asked. Pere says that what we need to do is to provide a similar interface and similar functionality from both sides: SEAL and ROOT. For example, the SEAL feature of looking into a list of directories for plug-in modules should be preserved and the ROOT user/developer would also like to have a system that resembles ROOT for the reason of backward compatibility. Pere thinks that the easy way today would be to follow Fons’ proposal in order to do it quickly, implying a dependency on CINT, but that the dependency would be removed later. This should be written in the plan. Marco says that LHCb is worried that all depends on the new CINT [the one that will be based on Reflex] and this is somewhere in the future. So, all depends on it, and if it does not come all will be delayed by 6 months. In that line, David says that ROOT IO also depends on Reflex API in the future too, but the development should be separated and not have to wait for the new CINT to be available.
David sees that the time scale of the phase-out of SEAL should be an outcome of the planning. Marco sees the end of the year as a hard date, to avoid being too late next year. Andrea claims that the first 2 years will be difficult enough without changes in the software. So, the next opportunity to have a big change is in 2008 or 2009. Fabiola reminds us that ATLAS will be in cosmics mode from Spring 2006 until 2007 startup and during this period major changes in the software will not be possible. So, SEAL will still be in use if the migration is not done before the end of 2005?. If a component is not usable after the merge the consequence is that both versions will have to be supported. Therefore the plan will have to be modified to reflect that both need to be maintained.
ATLAS and CMS will need the SEAL components as a stable reference platform and for production. They will not be ready to accept major changes, only simple bug-fixes (major changes are expected not to be stable enough). Both experiments are happy with the existing functionality in SEAL. The merger of SEAL and ROOT does not prevent users from using what's existing today. Pere reminds us that the idea was that not all packages will change in one go, this will be done gradually one at a time.
David said that the ATLAS level 2 trigger group have stipulated that only essential dependencies should be implemented in order to provide adequate real-time performance. As of today the Level2 code does not depend on SEAL, nor ROOT, nor Python, nor…. Rene does not see there is a real problem to link with libCore, libCINT. He thinks there is a misunderstanding that inheriting from TObject implies a new framework, which is not the case.
Rene proposes two possibilities:
1) continue to run with SEAL, put people on this.
This would be perfectly fine for him.
2) full merge as in the plan
and Fabiola adds a third:
3) adiabatically move from one to the other. This needs a planning which does
not imply that the bulk of changes come in a couple of months just before the
software is going to be deployed.
Rene says the third possibility is unclear for him and needs some examples. Andrea used the vector library (MathCore) to illustrate the point. This represents an adiabatic move from CLHEP and the way it has been done does not require MathCore be integrated into anything.
Marco asked when the new pluginMgr would be based directly on Reflex (i.e. and independent of CINT), and whether this change would be completely transparent to the users?
Rene answered that CINT is an integral part of ROOT. There is no question of changing this. This position was strongly supported by Fons. Following discussion it was decided that there is no intention to remove the dependency on CINT from the plug-in manager in the foreseeable future.
John reminded everyone that the goal was to merge the SEAL and ROOT project teams, so that better coherency could be achieved by ‘natural forces’. This would be a gradual process which may require in the short term that both SEAL and ROOT packages are maintained side by side However the limited manpower available implies that in the longer term it is highly desirable that there be a convergence on a single maintainable set of components. This could be managed both by implementation changes or by relaxing requirements. It is clear from the discussion that in the short term the experiments would like to see continuation of support for SEAL components used in their software and they would like to see this reflected in the AA workplan.
Rene proposed that his team could support the SEAL libraries independently as before. Pere said that the experiments should therefore make a list of the SEAL components that should be maintained. In practice, this proposal from Rene means that from now on the ROOT project will take the responsibility for the former SEAL project software: fixing bugs (through savannah), making releases, evolving with platforms, etc. The proposal is generally accepted.
There were a number of clarifications concerning whether libraries such as Reflex, MathCore, etc. need to be maintained in both SEAL and ROOT CVS repositories. The rule of thumb should be that if the libraries are identical and the SEAL libraries are not in use, then they can be abandoned. This requires agreement with the experiments. It should be no problem to have the two sets of libraries since they have different names and the classes are in different namespaces.
Pere said that somehow the situation with the Linear Algebra is different with respect to preserving existing functionality. It was in the SEAL program of work to have a L.A. package but this didn't happen before the merge. It does not make sense to have two versions of the package, so we need to use the one developed in the context of ROOT. Rene said that this work was done by a Russian colleague and Eddie Offerman, and Eddie keeps working on it.
ATLAS, CMS and LHCb would like to see this package be independent from the rest of ROOT. Pere said that there are packages with a well-defined mathematical functionality that don't need the framework functions such as storage, plotting, manipulating. So the L.A. package should be (like in the case of MathCore) releasable independently. Rene supports TObject inheritance by saying that TObject gives some methods to all objects, that people are used to it and therefore that it is needed for backward compatibility reasons. So, one would need to have this functionality preserved by providing the old classes double inheriting from the L.A. classes without inheriting from TObject and TObject. Federico states that ALICE does not need it, and Fons that the work would involve changing /adding 58 classes.
There was a discussion on how this work could be done and the relative priorities with respect other ongoing work. There was no final agreement on how this work could be done.
Pere explained why partial releases are necessary. There are many layers in the experiment software, and each of them needs validation every time a major release of ROOT is integrated, which may take a long time (several months in same cases). At the same time there are people in the experiments using ROOT for their final analysis that require updates as soon as possible. Usually fixes in ROOT are done in the HEAD revision within 24 hours. However, questions remain as to how these can be integrated into the "frozen" versions of ROOT and how the user can get the latest fixes/version of a package (e.g., neural networks) into the "validated" version of the framework for the experiments. There is a need for components/packages that can be released independently.
Federico said that what one needs is a set of stable interfaces and stable core libraries. Marco asked how he should update? Does he need the whole source of ROOT or can he just download the part he needs?
Rene says that some things can be done at the level of the single user, other things need the work/agreement of the experiment coordinators/librarian. Marco encouraged the ROOT team to look into the way experiments build and distribute their software and make sure that changes can be introduced without changing this process. Pere would like to see an item in the work plan on devising a strategy for handling this problem.
The following conclusions were reached in the final wrap-up:
1) The ROOT team will take the maintenance of existing SEAL packages (this
will be reflected in the plan i.e. the creation of a work package ‘SEAL’ with a
named coordinator)
- Planning for the other items as proposed in the merge are presently deferred
to after the ROOT workshop in September.
[clarification added afterwards: There are items of the
merge that were already in the plan (dictionary, python bindings, math
libraries), these will continue to be developed, only the plan for the next
batch will be done when we will know more on issues like the feasibility of the
adaptation of CINT to Reflex, which is expected by the ROOT workshop in
September.]
- The list of what functionality needs to be maintained needs be provided by
the experiments.
2) The ROOT project should help the experiments and POOL on how to adopt the new merged components.
3) Linear Algebra. Pere proposed that there be an evaluation of an
independent (from TObject) version of the linear algebra package.
Rene didn't want to take a commitment on this during the meeting.
4) Study the implications of packaging to understand how the experiments can mix latest releases from certain packages to be used in conjunction with the usually older, base/core packages.