Compilation of CalculiX Extras with NVIDIA CUDA Support and the Cholesky Method

Kazancev July 27, 2017 at 13:14

Compilation of CalculiX Extras with NVIDIA CUDA Support and the Cholesky Method

CalculiX is a fairly well-known narrow-minded pre / postprocessor and solver for problems in the mechanics of a deformable solid and problems in the mechanics of liquid and gas. The main code is completely written by a team of two people - their names are “in free Russian transcription” Guido Dont (solver), Klaus Wittig (pre / post). In addition to the main code, you can find a convenient functional graphical Launcher for preparing the calculation file. A little more detailed information is in the Russian Wiki , a few years ago driven by me and a couple of concerned users (ETCartman and Prul, hello! By the way, I can suspect that ETCartman wrote CalculiX Launcher, but this is a story covered in secrets of nicknames).

From the point of view of an ordinary Russian engineer, CalculiX is not so important and necessary in the daily work of software to pay attention to it. Scientists who previously conducted computational experiments at Abaqus may look at CalculiX quite differently - CalculiX is actually an open clone of Abaqus, since they have only one founder, Guido Dont.

The big two pluses of CalculiX are cross-platform and open source. The big two minuses are almost complete unknown among engineers in the CIS and somewhat less functionality compared to Abaqus. Nevertheless, I decided to make a short note on how to get the CalculiX binary with the support of the solver on CUDA, in the faint hope that this information will be useful to someone in the CIS.

I mean that this note will be read by a person (or cyber, LOL), who has managed to get acquainted with the structure of a typical settlement * .inp file and the principle of organization of calculations in CalculiX and familiar with Linux OS at least at the level of “went to the console and I know about apt-get” , besides, the GIC and interested in working with a bunch of CalculiX and CUDA, well, or compiling a project with Cholesky decomposition (cholmod).
To get started, take a look at what exactly is suggested by the author of CalculiX Extras on the project page , do not miss the link to compile the project in Ubuntu . The attentive reader on the author’s page will understand that to compile the project without ~~beer and days off~~detailed manuals will not work. Therefore, I took the liberty of deciphering all the spaces that the author of CalculiX Extras kindly placed between the lines.

1. Required Libraries

I don’t know how things are on Linux distributions other than Mint, and in Mint 18 you will need to install approximately the following necessary minimum libraries for the project:

binutils
cpp-5
gcc-5
gfortran-5
libstdc ++ 6
libstdc ++ 6: i386
autoconf
autoconf2.64
g ++
g ++ - 5
libarpack ++ 2-dev
libarpack ++ 2c2a
libbtf1.2.1
libcr0
libcsparse3.1.4
libcxsparse3.1.4
libhdf5- mpi-dev
libhdf5-mpich-10
libhdf5-openmpi-10
libhdf5-openmpi-dev
libldl2.2.1
metis
libmetis-dev
libmetis5
libmpich12
netcdf-bin
libnetcdf-c ++ 4
libnetcdf-c ++ 4-1
libnetcdf-c ++ 4 -dev
libopenblas-base
libopenblas-dev
libparpack2-dev
libstdc ++ - 5-dev
libsuitesparse-dev
libexodusii5
libexodusii-dev
libnemesis3.

2. Video card drivers

It is best to install a fresh driver through the Settings-> Driver Manager utility (on Linux Mint) and reboot.

The second option is to download NVIDIA-driver with CUDA support from your official NVIDIA website and install it:

i) `dpkg -i nvidia-diag-driver-local-repo-ubuntu1604_375.66-1_amd64.deb 'for Ubuntu
ii)` apt-get update`
iii) `apt-get install cuda-drivers`
iv)` reboot`

It is also possible to install the driver through the Synaptic package manager. Search for packages named NVIDIA.

3. Installing the CUDA-Toolkit

Download the 8th version from the official site (7.5 I get errors when compiling cudacusp.cu), start the installation in the console with the command:

sudo ./cuda_8.0.61_375.26_linux.run --override

Do not install the driver from the Toolkit. Answers to installer questions:

Do you accept the previously read EULA?
accept / decline / quit: accept

You are attempting to install on an unsupported configuration. Do you wish to continue?
(y) es / (n) o [default is no]: y

Install NVIDIA Accelerated Graphics Driver for Linux-x86_64 375.26?
(y) es / (n) o / (q) uit: n

Install the CUDA 8.0 Toolkit?
(y) es / (n) o / (q) uit: y

Enter Toolkit Location
[default is /usr/local/cuda-8.0]:

Do you want to install a symbolic link at / usr / local / cuda?
(y) es / (n) o / (q) uit: y

Install the CUDA 8.0 Samples?
(y) es / (n) o / (q) uit: y

Enter CUDA Samples Location
[default is / home / usr]:

The second installation option is through the Synaptic package manager after updating the system to a fresh one.

The third option is what NVIDIA developers write:

Update the CUDA network repo keys using the following command
# sudo apt-key adv --fetch-keys developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
Add the CUDA network repo and update the package lists on your system to get new versions of the software and their dependencies.
# sudo sh -c 'echo "deb developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 /"> /etc/apt/sources.list.d/cuda.list'
# sudo apt-get update
If you already have CUDA 8 installed on your instance and only need to update the NVIDIA driver, install the cuda-drivers meta-package. Then reboot the instance to complete the installation of the 375.66 NVIDIA driver.
# sudo apt-get -y --no-install-recommends install cuda-drivers
# sudo reboot
If you also need to install the CUDA toolkit, then install the cuda-toolkit-8-0 meta-package to download and install CUDA 8 .
# sudo apt-get -y install cuda-toolkit-8-0

Refer to the Linux Installation Guide for CUDA Toolkit for more information on using runfiles or local installers to install CUDA on various Linux distributions. The guide is located at the following URL: (http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html)

4. You need to download the sources CalculiX, ARPACK, CUDA cusp

The assembly process of ARPACK is described in detail in the article . Yes, it is not small, but you are adults too) You will understand.

I put the cusp folder from the CUDA cusp archive into the hamster (/ home / usr in my case), version 0.4.0 itself! (this is important) took here .

At this point, you can check the compurability of CalculiX without CUDA (see article).

5. Change system variables

I added CUDA library paths and CalculiX source paths in /home/usr/.bashrc:

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/usr/CalculiX/ccx/src
PATH=$PATH:/usr/local/cuda/bin
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/cuda/lib

6. Connecting libraries

The only problem for beginners can arise with the indication of the necessary dynamic libraries that the ccx_2.12.a object file will require when linking the cudacusp.a library. In order to find out the required libraries, enter the CUDA cusp examples directory (/ home / usr / cusp / examples in my case) and compile some binary with the command:

nvcc -o example example.cu -I/home/usr

Next, find out the list of required libraries:

ldd example

In my case, the list of lib data in the Makefile will be like this:

CUDACUSPLDFLAGS = -L/lib64 -l:libcufft_static.a -lstdc++ -lcuda -lcudart -lm -lgcc_s -lc -l:ld-linux-x86-64.so.2 -ldl -lpthread -lrt #-llinux-vdso

7. CCX-Extras patch and cudacusp.a assembly

We install the patches as it is written on the page of the author CCX-Extras . You will have modified sources, including the files cudacusp.h, cudacusp.cu, cudacusp.thrustassembly.cu. Delete cudacusp.cu (or rename), copy cudacusp.thrustassembly.cu -> cudacusp.cu. Open cudacusp.cu in a text editor and remove the word “thrustassembly” from the function name. Next, compile the lib:

nvcc -O3 -lib -o cudacusp.a -c cudacusp.cu -arch=sm_20 -I. -I/home/usr -L/usr/local/cuda/lib64 -L/usr/local/cuda/lib -L. -lstdc++ -lcuda -lcudart -DCUDACUSP

8. Build ccx

Change the Makefile (see mine):

## This file is heavily modified from the Makefile distributed by with
## ccx. 
## pkg-config is used to pull all flags from local enviroment.  You
## must set this up.  Typically this would be defining
## PKG_CONFIG_PATH=$PKG_CONFIG_PATH:~/local/lib64/pkgconfig or
## similar.  You will also have to add .pc file for each library you
## use.  I did this because I have several machines with different
## library enviroments.  You could alternatively use compiler flags
## such as -L/path/to/lib -I/path/to/include -llib.  Examples of this
## on my distribution are in the comments below
################################################################################
################################################################################
##                                                                            ##
## Flags related to CPU, optimizations, etc                                   ##
## GPU based optimization later in the make file                              ##
##                                                                            ##
################################################################################
################################################################################
CC=gcc
FC=gfortran
## CFLAGS=-march=native -g -O0 ## debugging  -pg
## FFLAGS=-march=native -g -O0 ## debugging  -pg
## LDFLAGS += -g
## CFLAGS=-march=native -O2 -Wall ## conservative
## FFLAGS=-march=native -O2 -Wall ## conservative
CFLAGS=-march=native -O3 -Wall ## -flto ## aggressive -fprofile-generate -fprofile-use 
FFLAGS=-march=native -O3 -Wall ## -flto ## aggressive -fprofile-generate -fprofile-use 
## Integer8.  Note at least Arpack and Pardiso need to be compiled for
## Integer8 as well.  Spooles does not appear to be int8 according to Guido
## LONGLONG = -DLONGLONG
## CFLAGS  += $(LONGLONG)
## FFLAGS  += -fdefault-integer-8
## INTEXT = 64
## Other CCX Options
CFLAGS  += -DARCH="Linux" -DMATRIXSTORAGE 
LDFLAGS += 
## Multi Threaded and MPI
CFLAGS += -DUSE_MT
#CFLAGS += -DCALCULIX_MPI -fopenmp
## This is now default for calculix and relates to CFD
CFLAGS += -DNETWORKOUT
################################################################################
################################################################################
##                                                                            ##
## Flags related to CPU based solvers                                         ##
##                                                                            ##
################################################################################
################################################################################
## SPOOLES
CFLAGS  += -I/usr/include/spooles -I/usr/include/spooles/MT -DSPOOLES
LDFLAGS += -lspooles -lpthread
## CFLAGS  += `pkg-config --cflags spooles` -DSPOOLES
## LDFLAGS += `pkg-config --libs spooles`
## ARPACK
CFLAGS  += -DARPACK
LDFLAGS += -L/home/usr/ARPACK/ -l:libarpack_linux.a
## CFLAGS  += `pkg-config --cflags arpack$(INTEXT)` -DARPACK
## LDFLAGS += `pkg-config --libs arpack$(INTEXT)`
## TAUCS 
## CFLAGS  += -DTAUCS
## LDFLAGS += -ltaucs -lmetis
## LAPACK
## CFLAGS  += -I/usr/include/openblas
## LDFLAGS += -lreflapack -lopenblas
CFLAGS  += `pkg-config --cflags lapack$(INTEXT)`
LDFLAGS += `pkg-config --libs lapack$(INTEXT)`
## BLAS
## CFLAGS  += -I/usr/include/openblas
## LDFLAGS += -lopenblas
CFLAGS  += `pkg-config --cflags blas$(INTEXT)`
LDFLAGS += `pkg-config --libs blas$(INTEXT)`
## PARDISO
## CFLAGS  += -DPARDISO
## LDFLAGS += -L/home/pete/local/lib64/ -lpardiso -lgfortran -lpthread -lm -fopenmp
## CFLAGS  += `pkg-config --cflags pardiso` -DPARDISO
## LDFLAGS += `pkg-config --libs pardiso`
################################################################################
################################################################################
##                                                                            ##
## Flags related to GPU based solvers                                         ##
##                                                                            ##
################################################################################
################################################################################
# these libraries you can see when compile examples in cusp folder and see results of command "ldd "
CUDACUSPLDFLAGS = -L/lib64 -l:libcufft_static.a -lstdc++ -lcuda -lcudart -lm -lgcc_s -lc -l:ld-linux-x86-64.so.2 -ldl -lpthread -lrt #-llinux-vdso
CUDACUSPCFLAGS = -I/usr/include -I/usr/local/include -I/usr/local/cuda-8.0/include -I/usr/local/cuda-8.0/include/crt
## Flags for the gpu compiler
NVCCCFLAGS = $(CUDACUSPCFLAGS) -arch=sm_20 -I. -I/home/usr
NVCCLDFLAGS = -lib -L. -L/usr/local/cuda/lib64 -L/usr/local/cuda/lib -DCUDACUSP
NVCCLDFLAGS += $(CUDACUSPLDFLAGS)
## 
#NVCC=nvcc -O3 $(LONGLONG) `pkg-config --cflags cusp` `pkg-config --libs cusp` $(NVCCCFLAGS) # -Xcompiler -fopenmp
NVCC=nvcc -O3 $(LONGLONG) -o cudacusp.a -c cudacusp.cu $(NVCCCFLAGS) $(NVCCLDFLAGS) # -Xcompiler -fopenmp
# wrong         nvcc -O3 --compiler-options '-fPIC' -dc cudacusp.cu -arch=sm_20 -I. -I/home/usr -L/usr/local/cuda/lib64 -L/usr/local/cuda/lib -L. -lstdc++ -lcuda -lcudart -DCUDACUSP
# wrong        nvcc -O3 -lib -o cudacusp.a cudacusp.a -arch=sm_20 -I. -I/home/usr-L/usr/local/cuda/lib64 -L/usr/local/cuda/lib -L. -lstdc++ -L. -lcuda -lcudart -DCUDACUSP
# wrong        cd /home/usr/cusp/examples/Solvers/
# right for compilation cudacusp.a:
# nvcc -O3 -lib -o cudacusp.a -c cudacusp.cu -arch=sm_20 -I. -I/home/usr -L/usr/local/cuda/lib64 -L/usr/local/cuda/lib -L. -lstdc++ -lcuda -lcudart -DCUDACUSP
#nvcc -O3 -lib -o cudacusp.o -c cudacusp.cu -arch=sm_20 -I. -I/home/usr -L/usr/local/cuda/lib64 -L/usr/local/cuda/lib -L. -lstdc++ -lcuda -lcudart -DCUDACUSP
## CUDACUSP
## This is unique because it a template library rather than binary library
CFLAGS  += -I/home/usr/local/include -DCUDACUSP -I/home/usr/cusp -I.
LDFLAGS += -L. -L/usr/local/cuda-8.0/lib64/stubs -L/opt/cuda/lib64 -l:cudacusp.a
CFLAGS  += $(CUDACUSPCFLAGS)
LDFLAGS += $(CUDACUSPLDFLAGS)
## CFLAGS  += `pkg-config --cflags cusp` `pkg-config --libs cusp` -DCUDACUSP
## LDFLAGS += `pkg-config --libs cusp`
## CHOLDMOD
## This is unique because it can be CPU or GPU based, depending on how
## SuiteSparse was compiled.  Here it is assumed that SuiteSparse also
## uses CUDA
CFLAGS  += -DSUITESPARSE
LDFLAGS += -L/usr/local/cuda-8.0/lib64 -lcublas
LDFLAGS += -lcholmod -lmetis -lcolamd -lccolamd -lamd -lcamd -ldl -lcxsparse -lbtf
## LDFLAGS += `pkg-config --libs cublas$(INTEXT)`
## LDFLAGS += `pkg-config --libs cholmod$(INTEXT)`
## LDFLAGS += `pkg-config --libs metis$(INTEXT)`
## LDFLAGS += `pkg-config --libs colamd$(INTEXT)`
## LDFLAGS += `pkg-config --libs ccolamd$(INTEXT)`
## LDFLAGS += `pkg-config --libs amd$(INTEXT)`
## LDFLAGS += `pkg-config --libs camd$(INTEXT)`
## LDFLAGS += `pkg-config --libs ldl$(INTEXT)`
## LDFLAGS += `pkg-config --libs cxsparse$(INTEXT)`
## LDFLAGS += `pkg-config --libs btf$(INTEXT)`
################################################################################
################################################################################
##                                                                            ##
## Flags related to ExodusII output                                           ##
##                                                                            ##
################################################################################
################################################################################
## EXODUSII
CFLAGS  += -DEXODUSII
LDFLAGS += -lexoIIv2c -lnetcdf
## CFLAGS  += `pkg-config --cflags exodusii` -DEXODUSII
## LDFLAGS += `pkg-config --libs exodusii`
################################################################################
################################################################################
##                                                                            ##
## Recipes                                                                    ##
##                                                                            ##
################################################################################
################################################################################
## .cu file so not have a default implicit rule.  Define all implicit rules used.
.SUFFIXES: .o .c .cu
.c.o :
	$(CC) $(CFLAGS) -c $<
.f.o :
	$(FC) $(FFLAGS) -c $<
.cu.o:
	$(NVCC) -DCUDACUSP -c $<
include Makefile.inc
SCCXMAIN = ccx_2.12.c
## Define all the object file rules to identify dependencies
OCCXCU = $(SCCXCU:.cu=.o)
OCCXF = $(SCCXF:.f=.o)
OCCXC = $(SCCXC:.c=.o)
OCCXMAIN = $(SCCXMAIN:.c=.o)
## Link to math and standard c
CFLAGS += -lm -lc
ccx_2.12: $(OCCXMAIN) ccx_2.12.a
	./date.pl; 
	$(CC) $(CFLAGS) -c ccx_2.12.c $(LDFLAGS); $(FC) -Wall $(FFLAGS) -o $@ $(OCCXMAIN) ccx_2.12.a $(LDFLAGS)
ccx_2.12.a: $(OCCXF) $(OCCXC) # $(OCCXCU)
	ar vr $@ $?
clean:
	rm *.a *.o

We compile:


make
./ccx_2.12 --help

9. Testing ccx with CUDA support

We take some example with static analysis and check the solvers:

* STEP
* STATIC ** - working (fast) (this is SPOOLES)
*** STATIC, solver = CUDACUSP - working (slow)
*** STATIC, solver = CHOLMOD - working (fast)
*** STATIC, solver = SUITESPARSEQR - working (slow)
*** STATIC, solver = ITERATIVESCALING - working (very slow)
*** STATIC, solver = ITERATIVECHOLESKY - working (very slow)
*** STATIC, solver = SPOOLES - working (fast)

Conclusion

I didn’t like to solve static test problems using the CalculiX + CUDA bundle - it’s too slow, because the CUSP library is designed to solve equations of the form A * X = B with sparse matrices of large dimension. It may be that the bunch works out more efficiently in the tasks of MZHG. But I do not dare to try this sphere of numerical aeronautics - maybe the reader will dare?

PSChecking the performance of the CalculiX + CUDA binder was my personal initiative and aimed at analyzing the possibility of using CalculiX Extras for the tasks of metal forming and stamping of sheet products. What can I say from the results of this test? Bummer-s, as in the joke with Bash, “the flower wasn’t for you”, which, incidentally, was warned by Peter A. Gustafson (author of CalculiX Extras) in a personal letter,

I note your doing metal forming. Fyi if you plan an explicit model, cuda is not implemented. Also, iterative solvers don't tend to perform better for this type of structure. This includes the static cuda solver implemented so far.

for which he is deeply grateful. Although, for the sake of sports interest (and contrary to common sense), I still compiled the project, having received a storm of incendiary emotions from the process (in particular, from a translation from Japanese).

With respect to the authors of CalculiX, CalculiX Extras, CalculiX Launcher and other add-ons,
as well as to the residents of Habrahabr and Giktayms, with thanks to the OpenSource community, AlexKaz.

Tags:

Compilation of CalculiX Extras with NVIDIA CUDA Support and the Cholesky Method