Tutorial 26.

Parallel Processing

Introduction
This tutorial illustrates the setup and solution of a simple 3D problem using the parallel
processing capabilities of ANSYS FLUENT. In order to be run in parallel, the mesh must
be divided into smaller, evenly sized partitions. Each ANSYS FLUENT process, called a
compute node, will solve on a single partition, and information will be passed back and
forth across all partition interfaces. The solver of ANSYS FLUENTallows parallel pro-
cessing on a dedicated parallel machine, or a network of workstations running Windows,
UNIX, or Linux.
The tutorial assumes that both ANSYS FLUENT and network communication software
have been correctly installed (see the separate installation instructions and related infor-
mation for details). The case chosen is the mixing elbow problem you solved in Tutorial 1.
This tutorial demonstrates how to do the following:

Start the parallel version of ANSYS FLUENTusing either Windows or Linux/UNIX.

Partition a mesh for parallel processing.

Use a parallel network of workstations.

Check the performance of the parallel solver.

Prerequisites
This tutorial is written with the assumption that you have completed Tutorial 1, and
that you are familiar with the ANSYS FLUENT navigation pane and menu structure.
Some steps in the setup and solution procedure will not be shown explicitly.

Problem Description
The problem to be considered is shown schematically in Figure 26.1. A cold fluid at
20 C flows into the pipe through a large inlet, and mixes with a warmer fluid at 40 C
that enters through a smaller inlet located at the elbow. The pipe dimensions are in
inches, and the fluid properties and boundary conditions are given in SI units. The
Reynolds number for the flow at the larger inlet is 50,800, so a turbulent flow model will
be required.

Release 12.0
c ANSYS, Inc. March 12, 2009 26-1
Parallel Processing

Density: = 1000 kg/m3

Viscosity: = 8 x 10 4 Pas
Conductivity: k = 0.677 W/mK
Specific Heat: Cp = 4216 J/kgK 8"

4"

Ux = 0.4 m/s
1"
T = 20oC 4" Dia.
I = 5%
3"
1" Dia.
8"
Uy = 1.2 m/s
T = 40oC
I = 5%

Figure 26.1: Problem Specification

Setup and Solution

Preparation
1. Download parallel_process.zip from the User Services Center to your working
folder (as described in Tutorial 1).

2. Unzip parallel_process.zip.
The case file elbow3.cas.gz can be found in the parallel process folder created
after unzipping the file.
You can partition the mesh before or after you set up the problem (define models,
boundary conditions, etc.). It is best to partition after the problem is set up, since
partitioning has some model dependencies (e.g., sliding-mesh and shell-conduction
encapsulation). Since you have already followed the procedure for setting up the
mixing elbow in Tutorial 1, elbow3.cas.gz is provided to save you the effort of
redefining the models and boundary conditions.

26-2 Release 12.0

c ANSYS, Inc. March 12, 2009
 Parallel Processing

Step 1: Starting the Parallel Version of ANSYS FLUENT

Since the procedure for starting the parallel version of ANSYS FLUENT is dependent upon
the type of machine(s) you are using, two versions of this step are provided here.

Step 1A: Multiprocessor Machine

Step 1B: Network of Computers

Step 1A: Multiprocessor Machine

Use FLUENT Launcher to start the 3D parallel version of ANSYS FLUENT on a Windows,
Linux, or UNIX machine using 2 processes.

1. Specify 3D for Dimension.

2. Select Parallel (Local Machine) under Processing Options.

3. Set Number of Processes to 2.

To show details of the parallel settings, click Show More >>, then go to the Parallel
Settings tab. Note that your Run Types will be Shared Memory on Local Machine.

4. Click OK.

Release 12.0
c ANSYS, Inc. March 12, 2009 26-3
Parallel Processing

To start ANSYS FLUENT on a Linux or UNIX machine, type at the command prompt

fluent 3d -t2

If you type fluent at the command prompt, then FLUENT Launcher will appear.
For additional information about parallel command line options, see Chapter 32 in the
separate Users Guide.

26-4 Release 12.0

c ANSYS, Inc. March 12, 2009
 Parallel Processing

Step 1B: Network of Computers

You can start the 3D parallel version of ANSYS FLUENT on a network of Windows, Linux,
or UNIX machines using 2 processes and check the network connectivity by performing
the following steps:

1. In FLUENT Launcher, restore the default settings by clicking the Default button.

2. Specify 3D for Dimension.

3. Select Parallel (Local Machine) under Processing Options.

4. Set the Number of Processes to 2.

5. Click the Show More >> button and select the Parallel Settings tab.
Retain the selection of default in the Interconnects and MPI Types drop-down
lists.
Select Distributed Memory on a Cluster.
Make sure that File Containing Machine Names is selected to specify the file.
Type the name and location of the hosts text file in the text box below File
Containing Machine Names, or browse and select it using the Browsing Machine
File dialog box.
Alternatively, you can select Machine Names and type the names of the ma-
chines in the text box.

6. Click OK.

Release 12.0
c ANSYS, Inc. March 12, 2009 26-5
Parallel Processing

You can also start parallel ANSYS FLUENT by typing the following at the command
prompt:
fluent 3d -t2 -cnf=fluent.hosts
where -cnf indicates the location of the hosts text file. The hosts file is a text file
that contains a list of the computers on which you want to run the parallel job.
If the hosts file is not located in the directory where you are typing the startup
command, you will need to supply the full pathname to the file.
For example, the fluent.hosts file may look like the following:

my_computer
another_computer

For additional information about hosts files and parallel command line options, see
Chapter 32 in the separate Users Guide.

26-6 Release 12.0

c ANSYS, Inc. March 12, 2009
 Parallel Processing

7. Check the network connectivity information.

Although ANSYS FLUENT displays a message confirming the connection to each
new compute node and summarizing the host and node processes defined, you may
find it useful to review the same information at some time during your session,
especially if more compute nodes are spawned to several different machines.
Parallel Network Show Connectivity...

(a) Set Compute Node to 0.

For information about all defined compute nodes, you will select node 0, since
this is the node from which all other nodes are spawned.
(b) Click Print.

------------------------------------------------------------------------------
ID Comm. Hostname O.S. PID Mach ID HW ID Name
------------------------------------------------------------------------------
n1 mpich2 another_computer Windows-32 21240 1 1 Fluent Node
host net my_computer Windows-32 1204 0 3 Fluent Host
n0* mpich2 my_computer Windows-32 1372 0 0 Fluent Node

------------------------------------------------------------------------------

ID is the sequential denomination of each compute node (the host process is

always host), Comm. is the communication library (i.e., MPI type), Hostname
is the name of the machine hosting the compute node (or the host process),
O.S. is the architecture, PID is the process ID number, Mach ID is the compute
node ID, and HW ID is an identifier specific to the communicator used.
(c) Close the Parallel Connectivity dialog box.

Release 12.0
c ANSYS, Inc. March 12, 2009 26-7
Parallel Processing

Step 2: Reading and Partitioning the Mesh

When you use the parallel solver, you need to subdivide (or partition) the mesh into
groups of cells that can be solved on separate processors. If you read an unpartitioned
mesh into the parallel solver, ANSYS FLUENT will automatically partition it using the
default partition settings. You can then check the partitions to see if you need to modify
the settings and repartition the mesh.

1. Inspect the automatic partitioning settings.

Parallel Auto Partition...

If the Case File option is enabled (the default setting), and there exists a valid par-
tition section in the case file (i.e., one where the number of partitions in the case
file divides evenly into the number of compute nodes), then that partition informa-
tion will be used rather than repartitioning the mesh. You need to disable the Case
File option only if you want to change other parameters in the Auto Partition Mesh
dialog box.
(a) Retain the Case File option.
When the Case File option is enabled, ANSYS FLUENT will automatically select
a partitioning method for you. This is the preferred initial approach for most
problems. In the next step, you will inspect the partitions created and be able
to change them, if required.
(b) Click OK to close the Auto Partition Mesh dialog box.

2. Read the case file elbow3.cas.gz.

File Read Case...

3. Examine the front view of the symmetry mesh zone (Figure 26.2).
Note: Since the Display Options were enabled by default in the launcher, the mesh
was displayed in the embedded graphics window after reading in the case.

26-8 Release 12.0

c ANSYS, Inc. March 12, 2009
 Parallel Processing

Figure 26.2: Mesh Along the Symmetry Plane for the Mixing Elbow

4. Check the partition information.

Parallel Partitioning and Load Balancing...

(a) Click Print Active Partitions.

Release 12.0
c ANSYS, Inc. March 12, 2009 26-9
Parallel Processing

ANSYS FLUENT will print the active partition statistics in the console.

>> 2 Active Partitions:

P Cells I-Cells Cell Ratio Faces I-Faces Face Ratio Neighbors Load
0 10414 177 0.017 34000 209 0.006 1 1
1 10417 173 0.017 34646 209 0.006 1 1

----------------------------------------------------------------------
Collective Partition Statistics: Minimum Maximum Total
----------------------------------------------------------------------
Cell count 10414 10417 20831
Mean cell count deviation -0.0% 0.0%
Partition boundary cell count 173 177 350
Partition boundary cell count ratio 1.7% 1.7% 1.7%

Face count 34000 34646 68437

Mean face count deviation -0.9% 0.9%
Partition boundary face count 209 209 209
Partition boundary face count ratio 0.6% 0.6% 0.3%

Partition neighbor count 1 1

----------------------------------------------------------------------
Partition Method Metis
Stored Partition Count 2
Done.

Note: ANSYS FLUENT distinguishes between two cell partition schemes within
a parallel problemthe active cell partition, and the stored cell partition.
Here, both are set to the cell partition that was created upon reading the
case file. If you repartition the mesh using the Partition Mesh dialog box,
the new partition will be referred to as the stored cell partition. To make it
the active cell partition, you need to click the Use Stored Partitions button
in the Partition Mesh dialog box. The active cell partition is used for the
current calculation, while the stored cell partition (the last partition per-
formed) is used when you save a case file. This distinction is made mainly
to allow you to partition a case on one machine or network of machines
and solve it on a different one.
For details, see Chapter 32 in the separate Users Guide.

26-10 Release 12.0

c ANSYS, Inc. March 12, 2009
 Parallel Processing

(b) Review the partition statistics.

An optimal partition should produce an equal number of cells in each parti-
tion for load balancing, a minimum number of partition interfaces to reduce
interpartition communication bandwidth, and a minimum number of partition
neighbors to reduce the startup time for communication. Here, you will be
looking for relatively small values of mean cell and face count deviation, and
total partition boundary cell and face count ratio.
(c) Close the Partitioning and Load Balancing dialog box.

5. Examine the partitions graphically.

(a) Initialize the solution using the default values.
Solution Initialization Initialize
In order to use the Contours dialog box to inspect the partition you just created,
you have to initialize the solution, even though you are not going to solve the
problem at this point. The default values are sufficient for this initialization.
(b) Display the cell partitions (Figure 26.3).
Graphics and Animations Contours Set Up...

i. Make sure Filled is enabled in the Options group box.

ii. Select Cell Info... and Active Cell Partition from the Contours of drop-down
lists.
iii. Select symmetry from the Surfaces selection list.

Release 12.0
c ANSYS, Inc. March 12, 2009 26-11
Parallel Processing

iv. Set Levels to 2, which is the number of compute nodes.

v. Click Display and close the Contours dialog box.

Figure 26.3: Cell Partitions

As shown in Figure 26.3, the cell partitions are acceptable for this problem.
The position of the interface reveals that the criteria mentioned earlier will be
matched. If you are dissatisfied with the partitions, you can use the Partition
Mesh dialog box to repartition the mesh. Recall that, if you wish to use the
modified partitions for a calculation, you will need to make the Stored Cell
Partition the Active Cell Partition by either clicking the Use Stored Partitions
button in the Partition Mesh dialog box, or saving the case file and reading it
back into ANSYS FLUENT.
For details about the procedure and options for manually partitioning a mesh,
see Section 32.5.4 in the separate Users Guide.

6. Save the case file with the partitioned mesh (elbow4.cas.gz).

File Write Case...

26-12 Release 12.0

c ANSYS, Inc. March 12, 2009
 Parallel Processing

Step 3: Solution
1. Initialize the flow field using the boundary conditions set at velocity-inlet-5.
Solution Initialization

(a) Select velocity-inlet-5 from the Compute from drop-down list.

(b) Click Initialize.
A Warning dialog box will open, asking if you want to discard the data generated
during the first initialization, which was used to inspect the cell partitions.
(c) Click OK in the Warning dialog box to discard the data.

2. Enable the plotting of residuals during the calculation.

Monitors Residuals Edit...

3. Start the calculation by requesting 200 iterations.

Run Calculation
The solution will converge in approximately 180 iterations.

Release 12.0
c ANSYS, Inc. March 12, 2009 26-13
Parallel Processing

4. Save the data file (elbow4.dat.gz).

File Write Data...

Step 4: Checking Parallel Performance

Generally, you will use the parallel solver for large, computationally intensive problems,
and you will want to check the parallel performance to determine if any optimization is
required. Although the example in this tutorial is a simple 3D case, you will check the
parallel performance as an exercise.
For details, see Chapter 32 in the separate Users Guide.
Parallel Timer Usage

Performance Timer for 176 iterations on 2 compute nodes

Average wall-clock time per iteration: 0.141 sec
Global reductions per iteration: 147 ops
Global reductions time per iteration: 0.000 sec (0.0)
Message count per iteration: 383 messages
Data transfer per iteration: 0.217 MB
LE solves per iteration: 7 solves
LE wall-clock time per iteration: 0.030 sec (21.2)
LE global solves per iteration: 2 solves
LE global wall-clock time per iteration: 0.000 sec (0.0)
LE global matrix maximum size: 11
AMG cycles per iteration: 12.506 cycles
Relaxation sweeps per iteration: 314 sweeps
Relaxation exchanges per iteration: 146 exchanges

Total wall-clock time: 24.866 sec

Total CPU time: 49.813 sec

The most accurate way to evaluate parallel performance is by running the same par-
allel problem on 1 CPU and on n CPUs, and comparing the Total wall-clock time
(elapsed time for the iterations) in both cases. Ideally you would want to have the Total
wall-clock time with n CPUs be 1/n times the Total wall-clock time with 1 CPU.
In practice, this improvement will be reduced by the performance of the communication
subsystem of your hardware, and the overhead of the parallel process itself. As a rough
estimate of parallel performance, you can compare the Total wall-clock time with
the Total CPU time. In this case, the CPU time was approximately twice the Total
wall-clock time. For a parallel process run on two compute nodes, this reveals very
good parallel performance, even though the advantage over a serial calculation is small,
as expected for this simple 3D problem.

26-14 Release 12.0

c ANSYS, Inc. March 12, 2009
 Parallel Processing

Note: The wall clock time, the CPU time, and the ratio of iterations to convergence time
may differ depending on the type of computer you are running (e.g., Windows32,
Linux 64, etc.).

Step 5: Postprocessing
See Tutorial 1 for complete postprocessing exercises for this example. Here, two plots are
generated so that you can confirm that the results obtained with the parallel solver are the
same as those obtained with the serial solver.

1. Display an XY plot of temperature across the exit (Figure 26.4).

Plots XY Plot Set Up...

(a) Select Temperature... and Static Temperature from the Y Axis Function drop-
down lists.
(b) Select pressure-outlet-7 from the Surfaces selection list.
(c) Click Plot and close the Solution XY Plot dialog box.

Release 12.0
c ANSYS, Inc. March 12, 2009 26-15
Parallel Processing

Figure 26.4: Temperature Distribution at the Outlet

2. Display filled contours of the custom field function dynamic-head (Figure 26.5).
Graphics and Animations Contours Set Up...

26-16 Release 12.0

c ANSYS, Inc. March 12, 2009
 Parallel Processing

(a) Select Custom Field Functions... from the Contours of drop-down list.
The custom field function you created in Tutorial 1 (dynamic-head) will be
selected in the lower drop-down list.
(b) Enter 80 for Levels.
(c) Select symmetry from the Surfaces selection list.
(d) Click Display and close the Contours dialog box.

Figure 26.5: Contours of the Custom Field Function, Dynamic Head

Summary
This tutorial demonstrated how to solve a simple 3D problem using the parallel solver of
ANSYS FLUENT. Here, the automatic mesh partitioning performed by ANSYS FLUENT
when you read the mesh into the parallel version was found to be acceptable. You also
learned how to check the performance of the parallel solver to determine if optimizations
are required.
For additional details about using the parallel solver, see Section 32.7 in the separate
Users Guide.

Release 12.0
c ANSYS, Inc. March 12, 2009 26-17
Parallel Processing

26-18 Release 12.0

c ANSYS, Inc. March 12, 2009