Parallel Performance of Linear Solvers and Preconditioners

The need to effectively and efficiently solve linear systems of equationjavascript:void(0);s is an important computational challenge affecting a wide range of applications in scientific computing from solid mechanics and quantum mechanics to climate modeling and computational geometry. A linear system of equations can be represented as
AijXj=bi
where Aij is the coefficient matrix, bi is the right-hand side vector, and xi is the vector of unknowns to be solved for. In this report, we solve the linear system of equations related to the three-dimensional (3-D) linear elastostatic boundary value problem (BVP). The focus on 3-D linear elasticity is motivated by our effort to accurately capture free-surface effects in discrete dislocation dynamics (DDD) simulations. This can be achieved by coupling a DDD simulator for bulk material (Arsenlis et al., 2007) to a finite element method (FEM) code that computes the image stress field resulting from the presence of free surfaces at each timestep. We have developed a massively parallel FE code, FED3, to perform the free-surface computations (Crone et al., 2013). A typical DDD simulation in bulk will require many timesteps (on the order of 1e5 to 1e6) to reach the desired loading conditions. To achieve similar simulation time scales with free surfaces and highly refined FEM meshes requires extremely efficient and scalable linear solvers to compute image stresses with a wall-clock time of a few seconds (s) or less. The keys to reducing solve time is to minimize parallel communication, computational expense per iteration, and number of iterations. In this discrete dislocation dynamics-finite element method (DDD-FEM) coupling application, the coefficient matrix (stiffness matrix) remains constant from one timestep to the next and only the right-hand side vector (due to changing boundary conditions) needs to be updated. Therefore, the setup time for the linear solver and preconditioner only occur on the first solve. For this reason, we restrict our solver performance study to the time spent solving the system of equation and ignore the setup costs—except in the case of direct solvers where the setup cost can become prohibitively expensive for large systems. In this report, we present a brief description of the problem we are solving as well as the linear solvers and preconditioners employed for this study. We go on to present the performance results for each of the solver – preconditioner combinations for various system sizes in terms of both serial performance and parallel scalability. We conclude by comparing the relative advantages and drawbacks of the solver – preconditioner options. We emphasize that while we restrict our evaluation to solving 3-D linear elasticity in this report, the results from this study can be applied to a wide range of applications where the solution of a linear system of equations is required.