SpMV and BiCG-Stab sparse solver on Multi-GPUs for reservoir simulation

This paper is on a multi-GPU simulation of a petroleum reservoir using a 3D structured grid, where each point is represented by its state. Using the Darcy model for porous media, each grid point is related to its six neighbors by a linear relation. The system equation is a sparse linear system AX = b, where A is a hepta-diagonal matrix, b is model parameters, and X is a state vector. The simulation repeatedly computes (1) A and b given X and (2) X given A and b, which is the focus of this paper. The BiCG-Stab is an iterative procedure for solving AX = b for X. This work focuses on developing a scalable multi-GPU approach for solving large sparse systems Ax = b using BiCG-Stab. We extend a previously developed storage scheme for blocked hepta-diagonal matrices to minimize storage and computing overheads of the sparse matrix–vector multiply (SpMV) used in BiCG-Stab. To honor data dependencies in BiCG-Stab tasks we propose a hierarchical multi-GPUs synchronization scheme that reduces the polling, combines barriers, termination BiCG-Stab iteration, and assembles the solution X. We present a multi-GPU implementation of BiCG-Stab by distribution operations overall units within a GPU and overall GPUs. Reduce-add operations are orchestrated by assembling partial results across all units and all GPUs. Since the cuSparse library works only on a single GPU, we present an SpMV for multi-GPU. In the evaluation, we present the testing of the pinned/paged memory to implement multi-GPU synchronization and communication. The scalability of the multi-GPU implementation of BiCG-Stab is presented showing a smooth increase in the computational load when the problem size grows to a billion, which is useful for developing scalable petroleum reservoir simulation on a clusterof GPUs.