Computational Accelerator Physics Grand Challenge: Need for Grand Challenge Scale Modeling

The design of accelerators for next-generation accelerator applications and spallation sources requires computations that are at present beyond the state-of-the-art in the accelerator community. Advanced high performance computing techniques will be needed to carry out a number of critical modeling tasks. These tasks and the issues associated with them are listed below. An overview of present-day modeling capability is also included.

Beam Halo

Accelerator applications such as ATW, ABC, APT and ADEP utilize high average power linacs that accelerate protons, while next-generation spallation sources are pulsed (lower average power) machines that accelerate H- ions. These accelerators have energies of 1-2 GeV and average currents of 10-200 mA. For comparison, the highest average power proton linac at present, located at the Los Alamos Neutron Science Center (LANSCE), has an energy of 800 MeV and an average current of 1 mA. This large increase in the average current makes controlling beam loss a major issue for the success of these projects, since reliability, availability and maintainability are crucial. If the beam loss is too high, it will cause component degradation and, more importantly, will cause radioactivation which hinders or prevents hands-on maintenance. At an energy of 1 GeV one would like to limit beam loss to 0.1 nA per meter, or 100 nA for the entire linac, in order to permit hands-on maintenance shortly after shutdown. For a CW system, such as APT, which has an average current of 100 mA, this corresponds to a fractional beam loss of one part per million.

Beam loss is known to be associated with the formation of an extremely low-density halo far from the beam core. Understanding the physics of beam halo formation and propagation has become a key issue. Since only very simple models can be treated analytically, this also presents a formidable computational challenge: accurately simulating the dynamics of an intense charged particle beam propagating through kilometers of complex accelerating structures and predicting the resulting halo formation and beam loss at extremely small levels. For an end-to-end simulation to predict extremely small losses with confidence one needs on the order of 100 particles in the halo to obtain good statistics; thus to observe beam loss of a part per million will require 100 million particles. Compared with present linac simulations, which typically utilize 10,000 to 100,000 particles, this corresponds to an increase of 3 to 4 orders of magnitude. The increase necessitates an extremely large amount of memory -- storing an array of six coordinates and momenta for 100 million particles in double precision requires almost 10 Gbytes. Considering the need for additional copies, overhead, and additional arrays for the space charge calculation, the required storage for such a simulation is well over 50 Gbytes.

The ability to predict beam halo can have profound consequences on the cost of a project. In the now-defunct superconducting super collider, lack of confidence in the initial design to meet dynamic aperture requirements led to an increase in the the superconducting magnet aperture. This resulted in a cost increase of approximately $1 Billion.

Next-generation spallation sources have an additional modeling challenge. Though they are pulsed low duty factor machines, they have accumulators associated with them where of order a few thousand pulses are compressed before final delivery to the targets. This accumulation is a complicated process in which H- ions are stripped and converted to H+ ions using a thin foil at the point of injection. This process increases the chances for particles to be scattered into the halo, so very high resolution modeling is essential. Considering an accumulator with a circumference of roughly a tenth of a kilometer, this means that the particles that enter the ring at the beginning of the compression process travel a few hundred kilometers inside the ring before extraction and transport to the targets. Even with TFLOPS-performance hardware it will be necessary to develop special techniques beyond that used for high current linacs in order to have any hope of modeling an intense charged particle beam over a distance of a few hundred kilometers in a reasonable amount of computer time.

Three-Dimensional Electromagnetics Modeling of Components

Three-dimensional electromagnetics modeling is the main tool for the detailed design and evaluation of RFQ's and beam funneling devices. In the case of RFQ's, the interior of the structure can be modeled with 2D codes but a 3D computation is required for the end regions and for the coupling regions that are present in new, segmented high-energy RFQ designs. In the case of beam funnels, the entire structure is 3D in nature. Besides using 3D computations to predict the electromagnetic properties of components, the results of such calculations are also used as input to thermal/stress codes. Most thermal/stress codes use irregular grids, whereas the electromagnetics codes most often used by the accelerator community use regular or quasi-regular grids. Because of these grids, large errors in electromagnetic heating calculations have been observed. In general, modeling with an unstructured grid is needed to follow complicated boundaries and accurately compute the magnetic fields near these boundaries so that adequate cooling channels can be provided. This is extremely important with regard to components in high average power linacs, since components will melt without proper cooling.

Electron Bunch, Radiation Field, and Undulator Models for LCLS

Gain growth in the LCLS amplifier depends on the nonlinear interaction of the electron bunch with the co-propagating radiation field, induced by passage through a long undulator. The efficiency of this process is a sensitive function of many factors, among them the quality of the undulator field, spectral-angular and temporal mode structures of the radiation field, electron beam emittance, homogeneous and non-homogeneous energy spreads, and details of the phase-space distributions within the electron bunch. Present simulations use very simplified models for the undulator, which is a long (~50m), high-field device with exceptionally strong focusing fields. Typical designs include permeable materials, permanent magnets, and currents, each with characteristic errors. In order to evaluate designs for the LCLS it is important to include more realistic models of the undulator, with the field calculated or re-interpolated for each successive location of the electron beam. Present simulation codes also employ idealized models of the electron bunch and radiation field, including: (1) simulation of gain within only one wavelength region of the bunch under the assumption of perfect periodicity from region to region; (2) representation of the several billion electrons within a typical bunch by a few hundred to a few thousand macroparticles; (3) decomposition of the radiation field into modes, with iterative (serial) computing to simulate the relative weight of each mode; (4) assumption of analytically simple electron distributions (e.g., Gaussian, rectangular, top-hat, etc.); (5) representation of the self and external fields by simplified analytical models; and (6) starting the gain process with an injected ``seed light'' signal, rather than from spontaneous noise. By utilizing HPC platforms, these approximations, as well as those associated with the undulator, can be avoided. This would greatly increase design reliability and improve radiation-modeling support for experimental applications of the LCLS.

Wakefield Suppression and Dark Current Capture in the NLC

The performance of the NLC can be adversely affected by undesirable beam-environment interactions. Foremost among them are beam emittance growth and dark current capture in the main accelerating linac. Much theoretical and experimental work has contributed to the understanding of what causes these phenomena, and technological measures have been incorporated in the NLC design to alleviate their effects. However, the theoretical studies have been based on approximate models and verification by realistic simulation will provide increased confidence in the design.

Trains of closely spaced bunches are needed in many NLC designs to obtain the luminosity required for physics studies. The stability of these multiple bunch trains during their passage through the accelerator is of primary concern because the retarded electromagnetic fields (wakefields) left by each particle can disrupt the trajectories of particles that follow it. The accelerator design is dominated by the need to control the size of the wakefields to prevent beam emittance growth. At SLAC, a complex 3D structure called the Damped and Detuned Structure (DDS) has been specifically designed to suppress wakefields. The DDS consists of 206 cells connected via slot openings to four pumping manifolds that run the length of an accelerator section to terminate in matched loads. The dimensions of the cells are chosen such that the deflecting modes are tuned in a prescribed manner so that the wakefields are decohered. In order to design a structure with these properties, very high resolution modeling with on the order of 100 million mesh points is required.

Another modeling challenge for the NLC presents itself in the form of dark current capture. These are electrons drawn off the surfaces in the high field region of the accelerator structures and captured on the accelerating RF wave. Deleterious effects due to dark current include parasitic beam loading and wall heating, background increase at the interaction region, interference with instrumentation, and possible deflection of the primary beam. All of the theoretical studies on dark currents to date have assumed cylindrically symmetric structures for simplicity, although it is recognized that 3D field enhancement in the RF input and output cells contributes to higher dark current generation. Furthermore, trajectory calculations on the field-emitted electrons have been carried out in constant impedance (uniform geometry) rather than constant gradient structures. With these simplifications the agreement between simulations and experiments have been found to be qualitative at best. It is crucial for the NLC design to quantify the effect of dark current capture by performing realistic simulations that include 3D actual structure geometry, RF power transmission through input to output coupler, and secondary electron emission. This will require computational resources even in excess of that required in the DDS problem described above, since in addition to field calculations, multi-million particle tracking is also involved.

Present Modeling Tools and Limitations

Beam Dynamics

Beam dynamics codes vary greatly depending on their regime of applicability. In the low current regime, one is concerned mainly with the dynamics of charged particles moving in externally applied fields (bending magnets, quadrupole focusing magnets, high order multipoles, rf gaps, etc). The firstcode to treat this (magnetic optics) problem and become widely used was the code TRANSPORT. It is used extensively in the United States for the design of beam transport systems and circular accelerators. The analogous code in Europe is called MAD (for Methodical Accelerator Design). When using this type of code one is often interested in the effects of high order nonlinearities, and the methods used in TRANSPORT quickly become unwieldy. This problem was addressed first by the Lie algebraic beam transport codes, notably MARYLIE, which uses sophisticated techniques that allow one to perform high order perturbation theory around a design orbit in the most ``natural'' mathematical framework for studying Hamiltonian systems. Slightly later, automatic differentiation techniques were adopted to perform calculations to arbitrarily high order.

The codes mentioned above are excellent for designing and evaluating beamlines and circular machine lattices, and for computing the trajectories of particles. However, they have only a very simplified treatment of space charge or no treatment at all. Space charge effects are becoming increasingly important as the intensity of beams is pushed to higher values. Examples of codes that treat these effects are PARMILA and PARMELA, which are used for analyzing ion linacs and electron linacs, respectively. But even these codes make unrealistic approximations, e.g., 2D (azimuthally symmetric) instead of fully 3D space charge calculations. Furthermore, the algorithms on which these codes are based (e.g., serial charge deposition and field interpolation) do not transfer directly onto parallel machines. This prevents simple scale-ups of these codes to handle the very large number of particles required for high-resolution modeling tasks.

Finally, space charge effects are also important in linear and circular colliders at the point where the beams collide. In such cases it would be necessary to couple a TRANSPORT-like code with a code that can accurately compute the space charge fields of the colliding beams.

Electromagnetics

Electromagnetics codes have very many uses in the field of accelerator physics. Examples include eigenmode solvers to design rf cavities, magnetostatic codes to design magnets, and wakefield codes to study the effect of beam interaction with the accelerator environment. Prior to the mid 1980's most such codes were two dimensional in nature. These included SUPERFISH and URMEL (for rf cavity design), POISSON (for magnetic design), and TBCI (for studying beam-cavity interactions). These codes are available in the public domain, and are widely used by the accelerator community.

As accelerator structures and components have become more complex, 3D codes have become increasingly important. At the present time, for rf cavity design and wakefield analysis, the most widely used 3D electromagnetics package is MAFIA (for ``solution of Maxwell's Equations by the Finite Integration Algorithm''). It was originally developed during a collaboration between DESY and LANL in the 1980's. Though an early version of the code was made available to US laboratories and Universities in 1988, it has since been commercialized by T. Weiland of THD, Germany, who now controls its development. Currently, essentially all US accelerator laboratories and many U.S. companies use MAFIA to meet their 3D modeling needs. Since MAFIA runs only on workstations it cannot be used for Grand Challenge scale modeling problems. For example, the design of new accelerating structures to minimize wakefields in the Next Linear Collider will require the modeling of 1-meter sections containing 206 different cells. This will require of order 100 million mesh points, a task that can only be performed using HPCC technology. Similarly, accelerator physicists have for years relied on commercial software like TOSCA from Vector Fields, England, to solve their 3D magnet design problems. TOSCA is not likely to be implemented on HPCC platforms, so it is also unsuitable for very large scale magnet design calculations. In summary, a parallel 3D electromagnetics modeling capability must be developed to meet advanced simulation needs and to take advantage of HPCC technologies and resources.

Back to Background Information.

Back to main page.