Trans Les Dns PDF

Intro. CFD DNS Capacity Ex.
LML DNS Managing Databases LES models
International Master on Turbulence

Master Calcul Scientifique
Unsteady Simulation of turbulent flows
(DNS & LES)
J.-P. Laval
[email protected]
Laboratoire de Mécanique de Lille (LML), CNRS

Blv Paul Langevin
59655 Vileneuve d’Ascq, FRANCE
(Office #2)
Intro. CFD DNS Capacity Ex. LML DNS Managing Databases LES models
Objectives of the course
Problematic of numerical simulations of (turbulent) flows

Research context (simple flows, high fidelity)
Industrial context (complex flows, large number of configurations)
Why do we always need turbulence modeling ?
Requirements for Direct Numerical Simulation (DNS)
Limitations of DNS (Research context)
Different concepts of turbulent models (RANS, URANS, LES, DES, ...)
Focus on LES and Subgrid Scale (SGS) models
Energy transfer between scales
Introduction on some concept of SGS models
Algorithm adapted for High Performance Computing
Implication of parallelization
Maintenance of large databases
Outline
Introduction to CFD
Capability of DNS
Homogeneous Isotropic Turbulence
Wall bounded turbulence
DNS of converging-diverging channel flow
numerical procedure
parallelization
what next ?
Saving a Post-processing large databases
LES models
derivation of the equations (physical & spectral space)
concept of turbulent viscosity
turbulent viscosity models (Smagorinsky, WALE)
dynamic procedure of Germano
scale similarity models
models based on approximated deconvolution
N levels models
Conclusions
Industrial framework
Need to simulate complex flows

Multi-components problems (airplane, car)
Non-homogeneous flows
High Reynolds numbers (aeronautics)
Multi-phase or reacting flows
Need a result in 1 hour to maximum 1 day
Research and development tool
Large number of parameters to investigate
Limited computing resources
The cost of numerical simulations is important
Experienced people but with a limited knowledge on CFD
⇒ Need fast and robust methodologies.

⇒ Does not have the time to understand the physics.
Research framework
Restricted to simple (or simplified) geometries

Isotropic turbulence (period boundary conditions)
Turbulent boundary layer flows
Flows around a reference airfoil (NACA 12)
...
Not limited in time: can afford a one year simulation (or even more !)
Access to the largest computers in the world (> 1 Peta Flops)
The cost is not so important
The cost is focused in some large simulations
The data are accessible to the research community (shared)
Cost needs to be related to the scientific outcome
Highly qualified people (on specific subject and simple flows)
⇒ Numerical simulation is a tool to understand the physics.

⇒ Need optimized algorithm and accurate results
Industrial and Research practices
Industrial practices
Most of simulations in steady states (RANS models)
Unsteady simulations restricted to specific parts of the flow (URANS,
DES, LES)
Use of robust numerical techniques (usually not accurate)
Research practices
Interest on unsteady flows (turbulence) to understand the physics
Increasing popularity and use of DNS and LES
Intensive research on turbulence models
Increasing number of LES models since 1980’s
New development in RANS models (more than 100 models !)
Research in DNS and LES
From J. ISI Web of Science

Top500 List of supercomputers
From J. Dongarra, “On the Future of High Performance Computing: How to Think for Peta and Exascale Computing” (2012)
Top500 List of supercomputers
From J. Dongarra, “On the Future of High Performance Computing: How to Think for Peta and Exascale Computing” (2012)
Massively Parallel Supercomputer
Extreme parallelism and hybrid

design
Tightening memory/bandwidth
bottleneck
Limits on power/clock speed
implication on multi-cores
Reducing communication will
become much more intense
Byte-to-flop ratio will change
Necessary Fault Tolerance
(Checkpoint/restart has
limitations)
Incompressible Navier Stokes equations
∂ui ∂ ∂p ∂
+ (ui uj ) = − + 2ν (Sij ) , (1)
∂t ∂xj ∂xi ∂xj
∂ui
= 0 (2)
∂xi
where the (strain rate tensor Sij is defined by:

1 ∂ui ∂uj
Sij = + (3)
2 ∂xj ∂xi
and where
p = P/ρ is the static pressure
ν = µ/ρ is the cinematic viscosity
Energy spectrum: definition
The energy spectra E(k) defined the way the energy is distributed among
the scales.
By defining the Fourier Transform of the velocity field u by:
Z Z Z
1
ûi (k) = ui (x)e −ik·x d 3 x (4)
(2π)3
For isotropic homogeneous turbulence, one can define the 3D isotropic

spectral function by:
x
E(k) = ûi (k)ûi∗ (k)dS(k) (5)
dS(k) is the surface element of the sphere

“*” means ”conjugate complex number”
√
k = |k| = k2 is the modulus of the wavenumber k
Energy spectrum of turbulent flows
⇒ km is the inverse of the energy injection scale

Why do we need models ?
Direct Numerical Simulation (DNS) truncated simulation

(without model)
energy pill−up
E(k) E(k)
e
to ner
sm g y
al ca
l sc
sc ad
al e
es
Energy Energy
Injection Injection
energy
dissipation
k k
ki kd ki kmax < kd
Modeling levels
RANS modeling
⇒ Simulation of the statistical average

⇒ All physical scales are modeled
LES modeling
⇒ Unsteady simulation of large scales

⇒ Modeling of sub-grid scales only
URANS modeling
⇒ Unsteady simulation of the largest scales

⇒ All physical scales are affected by the model
How to choose a model ? Before to choose a model, one must ask himself:
What is the Reynolds number ?
Is the flow turbulent ?
Can I perform a DNS of this flow at reasonable cost ?
What is the maximum cost (money and time) I can afford ?
Is it important to do unsteady simulations (noise emission, ...) ?
Do I need statistics at small scales (chemical reactions, ...) ?
Then, you must do some compromises between the accuracy and the cost of
your simulation.
Navier Stokes equation in Fourier Space
Applying a Fourier Transform to the Navier Stokes equations and using the
fact that the incompressibility condition reduces to k · û(k) = 0:
∂ûi (k)
= Ti (k) + νk 2 ûi (k) (6)
∂t
where
Z Z
i
Ti (k)=− (km Pij (k) + kj Pim (k)) ûj (p)ûm (q)δ(k − p + q)d 3 p d 3 q
2
is the Fourier transform of the nonlinear term (convolution product) and

ki kj
Pij (k)= δij − 2
k
is the projector operator onto the plan perpendicular to the vector k
Energy equation in spectral space
The equation for Fi,i (k) = ui (k)ui∗ (k) is obtained by multiplying the Navier
Stokes equation (6) by ui∗ (k):
∂Fi,i (k)
= Γi,i (k) − 2νk 2 Fi,i (k) (7)
| ∂t
{z } | {z } | {z }
transfer between scales dissipation
time decay
By integrating over a sphere of radius k = |k|

x ∂F (k) x x
i,i
dS(k) = Γi,i (k)dS(k) − 2νk 2 Fi,i (k)dS(k) (8)
∂t
we obtain the equation for the 3-dimensional spectral function E(k):
∂E(k)
= T(k) − 2νk 2 E(k) (9)
∂t
T(k) is the energy being removed (or added) at the wavelenght k due to
triadic interactions of term Γi,i (convolution product)
(large scale) + (large scale) ⇒ (large scale)

(large scale) + (small scale) ⇒ (small scale)
(small scale) + (small scale) ⇒ (small scale) or (large scale)
Characteristic scales (isotropic turbulence)

• Energy containing scale (“integral scale”) L can be estimated by:
R
E(k)/k dk
L= R (10)
E(k)dk
• Dissipative scale (“Kolmogorov scale”) η is defined by

1/4
ν3

η= (11)

• Taylor micro-scale λ is defined such that

2
u2

∂u
=2 (12)
∂t λ
and, for isotropic turbulence, λ can be estimated by

R
2 E(k)dk
λ =5 R (13)
k 2 E(k)dk
Reynolds numbers
Reynold based on integral scale (V rms = [ 13 u 2 ]1/2 ):

V rms L
ReL = (14)
ν
Reynolds based on Taylor micro-scale:
V rms λ
Reλ = (15)
ν
However, Reλ ∝ (ReL )1/2 and we have:
3/2
L/η ∝ Re 3/4 ∝ Reλ (16)
DNS at Reτ = 150
Pao’s spectrum
4/3
E (k) = (1 − e −k/kL )α2/3 k −5/3 e −2α(kη) /4/3
DNS at Reτ = 150

DNS at Reτ = 1500

DNS at Reτ = 1500

Resolution for a DNS at Reτ = 1500
For a DNS to be correctly resolved, one need to resolved scales

which are 10 times larger than the integral scale L
(to avoid side or periodic effects)
down to the smallest scales such that most of the dissipation peak is
resolved (' η)
This leads to the following estimation:

3/2 9/2
N ' 1.7Reλ ⇒ N 3 ' 5Reλ (17)
This means 1015 mesh points for Reλ = 1500

DNS of Isotropic turbulence: history
From Y. Kaneda, “Computing Turbulence in the 40963 Range”

Largest DNS of Isotropic turbulence
Simulation of Forced Isotropic Turbulence performed on “Earth

simulator”(Kaneda, 2002) (' 7. 1010 grid points):
N Reλ kmax ν hi L λ η
4096 1201 1930 0.173 10−4 0.0668 1.17 0.0360 0.528 10−3
BUT :
L/η = 2017 > kmax and kL << 10 (18)
Largest DNS of Isotropic turbulence
DNS 40963 gris points Kaneda et al.[?]

DNS of Isotropic turbulence
Memory requirement (time integration, Poisson equation):
M ' 40N 3 (bytes) (19)

Number of Operation per time step (in Flop or [Flops s]):
Q ' 10N 3 log2 N (20)

Cost to run one time step in a Computer of memory M (Bytes) and Speed S
(Flops):
Q/S = 4log2 N[M/S](s) ⇒ Q/S ' [M/S](min) for N ' 104

For the last 30 years, the ration [M/S] has been almost constant and of the
order of 1 (M ' 4Gb for S ' 10GFlops) but this could change (high speed
graphics: M ' 1Gb and S ' 100GFlops) ⇒ Dangerous
DNS of Isotropic turbulence
Number of time step required: ⇒ need converged statistics

depends on the integral scale (time for two record to be uncorrelated)
for Isotropic turbulence, τ ' L/u
depends on the type of statistics (second order, third order, ...)
depends on the required accuracy (estimation, validation of theory, ...)
Let’s suppose that
one need to simulate a time corresponding to 100 integral scales with a
CFL=0.5
the computation box size corresponds to 10 integral scales
⇒ need to simulate 10 flow through times
⇒ corresponds to 20N time steps
DNS of Isotropic turbulence: Future Possibilities
From J. Jimenez [2]
Objective: 1 EFlops in 2020 (from PRACE)
⇒ Need to wait 2020 to 2030 to be able to simulate the first High Reynolds
DNS of isotropic turbulence (in which L >> η)
3D FFT parallelization
For Isotropic Homogeneous Turbulence, more than 90% of CPU are due to
FFT:
Use efficient FFT parallelization algorithms

ALL TO ALL transfer are very expensive (Np2 messages)
limit the total volume of transfers
Take care of number of operations per memory call
Vectorization may also be useful (GPU)
For Isotropic Homogeneous Turbulence, more than 90% of CPU are due to
FFT:
From Bodart et al.[5]

3D FFT routines adapted for a large number of processes:
From Bodart et al.[5]

Wall turbulent flows
Particularity of wall turbulent flows:

Less results than for Isotropic Turbulence
More complicated energy transfer
Production of turbulence kinetic energy located in buffer layer
Energy transfer from (near wall) small scales to large eddies
Intermediate scales are not “neutral” in energy transfer
anisotropy in viscous sublayer ans buffer layers
small scales not only important for dissipation
Challenges:
Improved knowledge on energy Production and energy Transfer
among scales
Universality of Log layer (not necessary Log ?)
Effect of pressure gradient ?
⇒ improve turbulence models (theoretical & numerical)
Flat plate Turbulent boundary layers

Turbulent boundary layers: definitions

∂U
Shear stress τw = µ
∂y y =0
τ
Friction coefficient Cf = 2
1/2ρU∞
!1/2 1/2
∂U τ
Friction velocity uτ = ν =
∂y y =0
ρ
Boundary layer thickness δ = {y such as U(y ) = 0.99U∞ }
Z δ
∗ U
Displacement Thickness δ = 1− dy
0 U∞
Z δ
u U
Momentum Thickness θ= 1− dy
0 U U∞
yuτ
Wall unit y+ =
ν
Wall turbulent flows: difficulties
Intensive Research activities (numerical & experimental)

More challenging in term of simulation (boundary conditions)
Need for high Reynolds databases (Reθ > 50000)
Two main configurations:
Channel flows: simplest configuration
periodic boundary condition in streamwise and spanwise direction
wall boundary condition in normal (Neumann or Dirichlet)
but constant pressure gradient
Boundary Layers:
normal boundary conditions ?
inlet boundary conditions ? (not periodic)
Turbulent boundary layer: statistics
From Jimenez (2009) [3]

Channel flow: Reynolds effect
From Jimenez (2007)

Channel flow: Reynolds effect
From Jimenez (2007)

Wall turbulent flows: coherent structures [4]

Wall turbulent flows: resolution
Anisotropic structures near the wall (streaks, vortices)

up to 20 δ long ⇒ long simulation box
down to 20 wall unit width ⇒ spanwise grid resolution
Large structures (size δ) ⇒ height of simulation box
Minimum simulation domain for turbulent boundary layer:
(10 − 300)δmax × (3 − 10)δmax × (5 − 10)δmax (21)
Maximum spacial resolution (with spectral method):
8+ × (1+ − 8+ ) × 4+ (22)
Wall turbulent flows: resolution
For a simulation box size of Lx δ × δ × Lz δ
N 3 = 0.012 Lx Lz Reτ3 (23)
Integration time: 10 wash out (100 Integral Scales)
Nt = 0.4 Lx Reτ (24)
Using the same estimate than for isotropic turbulence for the computational
cost per grid point:
Qt = 40L2x Lz Nw Reτ4 Flop (25)
For a computational cost of 1 min per time step
History of channel flow DNS
Reτ Lx Lz Points Year Ref.

180 12 6 5M 1987 Kim, 1987
590 6 3 40 M 1997 Moser, 1999
550 25 12 600 M 2001 Alamo, 2003
950 25 9 4G 2003 Alamo, 2004
2000 25 9 16 G 2006 Hoyas, 2006
10000 12 6 900 G 2013-2015 ?
Reτ = 10000 ⇒ 100 − 200Tb of memory required

Turbulent Boundary Layer
Challenges:
“Verify” the law of the wall and its bound in y

unicity of van Karmann constant (d(log (y + ))/d(U + )) κ = 0.38 − 0.44
minimum y + the law can be used
minimum Reynolds number required to define κ (δ + > 6000?)
⇒ not accessible by DNS for at least 20 to 50 years
Large scale motions :
large scales / small scales interactions
large scales / small scales correlations
⇒ easier to investigate by DNS than by experiments
Turbulent Boundary Layer: inlet conditions
Generated realistic inflow Boundary Conditions is very challenging.
Different possibilities for Turbulent Boundary Layer ([4]):
Starting from laminar flow and resolve the transition to turbulence

→ The only perfect solution
→ Very expensive in computational resources (CPU & Memory)
Precursor simulation to generate inlet flows
→ Very accurate solution
→ Expensive in computational resources (CPU & Memory)
→ Expensive in storage (all time steps)
→ For Turbulent Boundary layer: identical to first solution
Turbulent Boundary Layer: inlet conditions
Recycling turbulent velocity component from a downstream plane ([4])

→ Commonly used method
→ Cheap (but computation of mean statistics on the fly)
→ Tricky as scalings are unknown
→ Possibility to let some degree of freedom ([2]
Mean velocity profile with additional random noise ([4])
→ Quite easy to perform
→ Need a much longer domain to recover correct statistics
→ Can be improved by adding a forcing
Mean velocity profile with turbulent structures (vortices) ([1])
→ Complicated
→ A little bit better than random noise
From Schlatter (2010) [5]

Turbulent Boundary Layer: most recent large DNS

Computed on 8192 nodes of 4 cores (Blue Gene P)
From Sillero (2011) [5]

Turbulent Boundary Layer: Jimenez’s team code

Parallelization using OpenMP/MPI
From Sillero (2011) [5]

Schlatter’s DNS of Turbulent Boundary Layer

Turbulent Boundary Layer: Reynolds effect
Evolution of the peak of normal Reynolds stress components



Possible reasons for discrepancy between DNS of TBL:
grid resolution and numerics

inflow Reynolds number and turbulence generation
sufficient settling length to reach final turbulent state
box dimensions and boundary conditions (e.g. pressure gradients)
or the results are maybe not as universal as expected
⇒ need a lot off care

⇒ not compatible with ‘Reynolds number braking Record”
⇒ need to admit possible problems when detected
Immersed Boundary technique
Motivation:
need for complex geometries
need for high quality grids
need for specific user skills (meshing)
Principle:
The (moving) body is not physically present in the computational
domain, only the effect of boundaries is accounted for
Starting from the description of the body, a module tags the cells as
internal, external or interface
A reconstruction (interpolation) scheme is used at interface to enforce
the boundary condition and the body surface (by an additional forcing
term in the equations)
DNS of Grid Turbulence with Incompact3D
From Laizet et al.(2011) [6]

Incompact3D : Domain decomposition

Incompact3D : Domain decomposition

DNS of complex geometries

Flow around a golf ball by Squires et al.[8]
Immersed boundary approach
Staggered grid in cylindrical coordinates
Second-order central differences discretization in space
Third order minimal storage Runge-Kutta integration scheme
Reynolds number of Re = 1.1 105
1.2 109 grid points
From Smith et al.(2010) [8]

DNS of complex geometries
Iso-value of Q criterion (vortices visualization)
From Smith et al.(2010) [8]

Objectives of the converging-diverging channel DNS
Investigation of wall turbulence with adverse pressure gradient

Generation of Databases
Investigation of flow separation
Development of an optimized code for DNS/LES of flows with adverse
pressure gradient
Investigation of turbulence
Development of control strategy
Development of turbulence models (RANS/LES)
Framework: WALLTURB Project
A European Synergy for the assessment of wall turbulence

4 years European project (2005-2009) funded by the EC.
Consortium of 16 partners from 10 different countries.
Objectives: to assess and better model near wall turbulence.
generating and analyzing new data on near wall turbulence,
extracting physical understanding from these data,
putting more physics in the near wall RANS models,
developing better LES models near the wall,
investigating alternative models based on Low Order Dynamical Systems
(POD).
Framework: WALLTURB Project
WALLTURB Database on Adverse Pressure Gradient Flows

→ Experiments on flat wall (Czestochowa)
→ Experiments on curved wall (Surrey & LML)
→ DNS of APG boundary layer on flat wall (Madrid)
→ DNS of APG in channel flow: flat wall & curved wall (LML)
Experimental setup at the LML
Characteristics :
20m x 1m x 2m
bump : 33cm
u = 3 − 10m/s
Reθ = 7000 − 21000
Why a new CFD codes ?
Several ”Industrial” codes available

Fluent, StarCD, CFX, Numeca, ...
Saturne (EDF), elsA (ONERA), AVBP, NSMB (CERFACS)
Usually Low Order Finite volume codes
General solver (adapted for several equations)
Not necessary optimized for DNS and Fine LES
Why a new CFD codes
Development of a “Research” code

Incompressible
Structured grid
High order spatial discretization
High order time integration scheme
Optimized for simple geometries
Possibility to easily implement not conventional turbulent models
Easy to access for students and Post-docs
Collaboration between several Laboratories (IRPHE)
Equations and discretization
Navier-Stokes equations
∂~ū ~¯ + 1 ∆
~¯ ~ū = −∇p̄ ¯ ~ū
+ (~ū .∇)
∂ t̄ Re
~¯ ~ū = 0
∇.
Spatial discretization :
4th order finite differences (streamwise)
Collocation-Chebyshev (normal)
Fourier (spanwise)
Temporal discretization :
2nd order backward Euler
2nd Adams-Bashforth
Boundary conditions
1
y
0
0 1 2 3 4 5 6 7 8 9 10 11 12
x
Periodic in spanwise direction (Fourier)

No slip boundary conditions at the walls
Convective boundary condition at outflow : ∂~
u ∂~
u
∂t + Uc ∂x = 0
Inlet conditions : DNS/LES of plane channel flow at the same Reynolds
number
Complex geometry → mapping
1
y
0
0 1 2 3 4 5 6 7 8 9 10 11 12
x
1 L + η̄(x̄)
y= (1 − γ(x̄)) ȳ + γ(x̄) , γ(x̄) =
L η̄(x̄) − L
2
1
y
0
0 1 2 3 4 5 6 7 8 9 10 11 12
x
η̄(x̄) : graph of the bump

Mapped system
The Laplacian and Gradient operator are transformed:

~¯ = ∇
∇ ~η , ∆
~η +G ¯ = ∆η + Lη ,
1−γ ∂
„ «
~η ∂ ∂
∇ = , ,
∂x L ∂y ∂z
„ «
~η ∂
G = δ(1 − y ) , 0, 0
∂y
∂2 1 − γ 2 ∂2 ∂2
„ «
∆η = 2
+ +
∂x L ∂y 2 ∂z 2
∂2 2
„ «
∂δ 2 ∂ 2 ∂
Lη = −δ (1 − y ) + 2δ (1 − y ) + (δ (1 − y )) ,
∂x ∂y ∂x∂y ∂y 2
and
∂η 1
δ=
∂x η − L
Mapped system
Navier-Stokes system in the computational coordinates
∂~u ~ η )~u
~ η )~u + (~u .G ~ η p + 1 ∆η~u + 1 Lη~u
~ ηp − G
+ (~u .∇ = −∇
∂t Re Re
~ η .~u
∇ ~
= −Gη .~u
∆η p = −Lη p + J( u, v , w )
1 − γ ∂v ∂u 1 − γ ∂u ∂v
» „ « „ «
∂u ∂v
J(u, v , w ) = 2 + δ(1 − y ) − + δ(1 − y )
L ∂y ∂x ∂y L ∂y ∂x ∂y
„ « „ «
∂u ∂w ∂w ∂w ∂u ∂u
− + δ(1 − y ) + + δ(1 − y )
∂z ∂x ∂y ∂z ∂x ∂y
1 − γ ∂w ∂v 1 − γ ∂v ∂w
–
− + .
L ∂y ∂z L ∂y ∂z
Direct resolution.
Smooth geometries (derivatives of η(x))
Time discretization
Velocity :
~¯ n+1 + f n,n−1 with τ = Re

(∆η − 3τ )~u n+1 = Re ∇p
2∆t
~ η )~u ]n,n−1 .
~ η )~u + (~u .G
f n,n−1 = −4τ~u n + τ~u n−1 − [Lη ~u ]n,n−1 + Re[(~u .∇
Pressure :
∆η p n+1 = [ −Lη p + J(u, v , w )]n,n−1
n+1
~ ∗ 3~u − 4~u n + ~u n−1 h ~ ~ η )~u
∇η p .~n = − − (~u .∇η )~u + (~u .G
2∆t
in,n−1
~ η p + 1 (∇
h
+G ~¯ × (∇~¯ × ~u ) )]n,n−1 .(~n + ~n ) − ∇ ~ ηp .~nη
η
Re
Fourier space → parallelization
Velocity :
~¯ n+1 + fˆn,n−1
((∆η )k − 3τ )~ûkn+1 = Re ∇p̂k k
~
fˆkn,n−1 = −4τ ~ûkn + τ ~ûkn−1 − [Lη ~ûk ]n,n−1 + Re [Nˆk ]n,n−1
~ = (~u .∇
with N ~ η )~u
~ η )~u + (~u .G
Pressure :
(∆η )k p̂kn+1 = [ −Lη p̂k + Jˆk (u, v , w )]n,n−1
Nz two dimensional (x, y ) independent subsystems

Parallelization → Nz /Np modes per processor
~
Nonlinear terms Nˆk and Jˆk → MPI communications
NEC-SX8 : code implementation
Previous work :
2D computations on NEC-SX5
Parallel 3D computations on NEC-SX5 (2-8 procs, 1 node)
Parallel 3D computations on IBM-SP (16-256 procs)
Porting to NEC-SX8 :
Vectorization :
Resolution of the Helmholtz and Poisson equations
Derivatives in the normal y -direction : Dy
Fast Fourier Transforms (FFT) in the spanwise directions
Performance of MPI Communications on the NEC-SX8 ?
FFT and Dy
FFT :
Sine and cosine transform : 576 modes
VFFTPACK : multiple 1D transform
Performance : 7-8 Gflops
Normal derivatives Dy : Matrix multiply or FFT ?

Matrix multiply (matmul) : 14 Gflops
FFT + recursive computation : 7 Gflops
do j=ny-3,0,-1 do i=0,nx-1 dya(i,j)=dya(i,j+2)+2.d0*(real(j)+1.d0)*a(i,j+
enddo enddo
Matrix diagonalization
Problems to solve :
∂2u ∂2u T
2
+ 2
− σu = f → Dy2 U + U Dx2 − σ U = F
∂x ∂y
Uji = [u(yj , xi )] , Fji = [f (yj , xi )] , 1 ≤ j ≤ Ny , 1 ≤ i ≤ Nx

Dy2 : discretized collocation-Chebyshev operator
Dx2 : discretized finite-differences operator
Eigenvalues and eigenvectors decomposition : Dy2 = PΛP −1
T
ΛÛ + ÛDx2 − σ Û = F̂
Û = P −1 U , F̂ = P −1 F
Matrix diagonalization
Problems to solve :
∂2u ∂2u T
2
+ 2
− σu = f → Dy2 U + U Dx2 − σ U = F
∂x ∂y
Uji = [u(yj , xi )] , Fji = [f (yj , xi )] , 1 ≤ j ≤ Ny , 1 ≤ i ≤ Nx

Dy2 : discretized collocation-Chebyshev operator
Dx2 : discretized finite-differences operator
Eigenvalues and eigenvectors decomposition : Dy2 = PΛP −1
T
ΛÛ + ÛDx2 − σ Û = F̂
Û = P −1 U , F̂ = P −1 F
MPI communications
Nonlinear terms : go to physical space by FFT to avoid convolution

sums in Fourier space

FFT
û1,k −−−→ u1 (zj ) FFT

u1 (zj )u2 (zj ) −−−→ u\
1,k u2,k
FFT
û −−−→ u (z )
2,k 2 j

y y
z z
x x
Spectral space Physical space

Nx × Ny × NNprocs
z Nx
Nprocs × Ny × Nz
Gather and Scatter
P0 A0 P0
P1 A1 GATHER P1 A0 A1 A2 A3
−−−−−−→
P2 A2 SCATTER P2
←−−−−−−−
P3 A3 P3
IBM-SP :
Non-blocking synchronous send and receive :
MPI_ISSEND and MPI_IRECV
NEC-SX8 :
blocked and buffered send and receive
No overlapping of communications and computations !
New Gather routine
do i=1,npe
call MPI_GATHER(u((i-1)*nx*ny/npe+1,1),long1, &
type_bloc1,ut(1,1),long2, &
type_bloc2,i-1,MPI_COMM_WORLD,code)
enddo
do i=1,npe
if (rang==(i-1)) then
do j=1,npe
call MPI_IRECV(ut(1,(j-1)*nz+1),long2,type_bloc2, &
j-1,tag,MPI_COMM_WORLD,ireq(j,2),ierror)
enddo
call MPI_ISSEND(u((i-1)*nx*ny/npe+1,1),long1,type_bloc1, &
i-1,tag,MPI_COMM_WORLD,ireq(i,1),ierror)
else
call MPI_ISSEND(u((i-1)*nx*ny/npe+1,1),long1,type_bloc1, &
i-1,tag,MPI_COMM_WORLD,ireq(i,1),ierror)
endif
enddo
MPI performance on NEC SX-8
IXS Memory Crossbar: maximum speed of communication

32Gb/s for internal 8 CPU nodes communications
8Gb/s for cross node communications
Several stages of data copying
from the send array to a send-side buffer,
from that buffer to a receive buffer at the other node,
from the receive side buffer to the receive array
Some of these steps can be avoided by using
“Global Memory”(higher Latency)
MPI performance
Performance with packet size of 1Mb (results in Gb/s)

Gather and Scatter 1 : original MPI routines
Gather and Scatter 2 : rewrite with synchronous send and receive
Proc. 8 16 32 64
Gather 1 8.0 4.2 1.6 1.4
Scatter 1 8.1 1.0 0.1 0.05
Gather 2 11.2 4.6 1.4 1.2
Scatter 2 14.9 4.7 1.8 1.3
Performance of original MPI scatter?

MPI performance (2 processors NEC SX-8 IDRIS)






Performance of the code for DEISA Project (NEC

SX-8)
Production jobs features :

Processors : 64
Time per job : 22 hours
Memory : 390 Gb (6 Gb per processor)
Production performance :
Total : 640 Gflops
10 Gflops per process
Time ratios :
Helmholtz and Poisson equations : 55%
FFT : 20%
MPI communications : 13%
Performance of the code

Processor Proc. Grid resolution Mem./proc. CPU /iter time/point/iter Perf.
PERFORMANCES ON SCALAR PROCESSORS (without mapping)
Same physical problem
IBM P4 P655 64 1024 × 193 × 512 1.2 Go 35.8 s 22.59 µs
IBM P4 P655 128 1024 × 193 × 512 0.6 Go 19.0 s 15.87 µs
IBM P4 P655 256 1024 × 193 × 512 0.3 Go 7.2 s 18.17 µs
IBM P6 P575 128 2048 × 193 × 1024 2.4 Go 33.9 s 10.72 µs
IBM P6 P575 256 2048 × 193 × 1024 1.2 Go 16.5 s 10.43 µs
IBM P6 P575 512 2048 × 193 × 1024 0.6 Go 11.5 s 14.59 µs
Same CPU cost per processor
IBM P4 P655 64 1024 × 193 × 512 1.2 Go 35.8 s 22.59 µs 1.2 GFlops
IBM P6 P575 128 2048 × 193 × 1024 2.4 Go 33.9 s 10.72 µs
IBM P6 P575 256 2048 × 193 × 2048 2.4 Go 32.4 s 10.25 µs
IBM P6 P575 512 2048 × 193 × 4096 2.4 Go 50.4 s 15.93 µs
IBM BlueGene P 256 4096 × 33 × 1024 0.4 Go 12.2 s 22.51 µs
Increase of memory and transfer
IBM P4 P655 256 1024 × 193 × 512 0.3 Go 7.2 s 18.17 µs
IBM P4 P655 256 1024 × 193 × 1024 0.6 Go 14.0 s 17.66 µs
IBM P4 P655 256 1024 × 193 × 2048 1.2 Go 34.5 s 21.76 µs
Same problem on different processors
IBM P4 P655 256 1024 × 193 × 512 0.3 Go 7.2 s 18.17 µs
PERFORMANCES ON VECTORIAL PROCESSORS (with mapping)
NEC SX8 64 2304 × 385 × 576 6 Go 18.19 s 2.35 µs 10 GFlops
Peak Perf. (GFlops): BlueGene P: (850Mhz) 3.4 , Power4+ (1.7GHz): 6.8 , Power6 (4.7GHz): 18.8 , NEC-SX8 (2Ghz): 16
Performance of the code: conclusions
Good vectorial performance (up to 65% of peak on NEC SX-8)

Correct scalar performance (up to 20% of peak on IBM SP4)
Good parallelization for low performance processors
MPI communications not fully hide by computation with high
performance processors
Minimization of the number and volume of communications
Work on the algorithm to hide more communications
Future improvements
Finite differences discretization (Compact finite differences)

New time integration scheme (3rd order ?)
Multi-domain in the 3 directions
2
1
y
0
0 1 2 3 4 5 6 7 8 9 10 11 12
x
Parallel in the three direction with less communications :
→ Massively parallel supercomputer (> 10000 CPUs, Blue-Gene)
→ Mixed parallelization MPI/OpenMP ?
More complex geometries (hide blocks)
Future improvements

2
1
y
0
0 1 2 3 4 5 6 7 8 9 10 11 12
x
Parallel in the three coordinates with less communications :
Future improvements

2
1
y
0
0 1 2 3 4 5 6 7 8 9 10 11 12
x
Future improvements

2
1
y
0
0 1 2 3 4 5 6 7 8 9 10 11 12
x
OpenMP: drawback
More complex for long codes

number of loops and/or routines
recursive algorithms, ...
May use additional memory (copy of arrays)
Problem limited by shared memory (usually < 256 Gb)
OpenMP: routine parallelization
!-----------------------------------
! Multiple FFT procedure !
!-----------------------------------
real : a(0:l1+1,0:l2,0:l3) ! array to be transformed

real : wk(4*(l1+2)**2,0:1 ! working array
complex : ex(0:l1) ! initialized vector
integer : ifax(0:l1) ! initialized vector
integer : inc ! distance between two elements to be transformed
integer : jump ! distance between two vector elements to be transformed
integer : l1 ! size of the FFT
integer : nft ! number of simultaneous FFT
integer : isign ! sign of FFT (+1 or -1)
!$OMP PARALLEL PRIVATE(wk)

!$OMP DO SCHEDULE(RUNTIME)
do j=j1,j2
call rfftmlt(a(0,j,0),wk,ex,ifax,inc,jump,l1,nft,isign)
enddo
!$OMP END DO
!$OMP END PARALLEL
OpenMP: loop parallelization
!-----------------------------------
! Laplacian procedure !
!-----------------------------------
!$OMP PARALLEL DO SCHEDULE(RUNTIME)
do iz=0,n3-1
do iy=0,n2-1
do ix=0,n1/2-1
xk2 = k2x(ix)+k2y(iy)+k2z(iz)
vx(ix,iy,iz) = -xk2*vx(ix,iy,iz)
vy(ix,iy,iz) = -xk2*vy(ix,iy,iz)
vz(ix,iy,iz) = -xk2*vz(ix,iy,iz)
enddo
enddo
enddo
!$OMP END PARALLEL DO

OpenMP: conclusions
Good solution for existing code

Useful for OpenMP/MPI parallelization
Optimization of performances (fast processors)
No transfer within a node (6= MPI)
Easy to adapt to the architectures
Not very useful on actual BlueGene
4 cores per nodes
2 Gb of shared memory only
Important to choose the right data format

Huge amount of data ( up to 100Tb by simulations)
Data shared among several teams (through databases)
Data will be post-processed many times (reading speed)
Data maybe used for more than 20 years (DNS of Spalart 1988)
Data Format: NetCDF
Documentation on NetCDF:
http://www.unidata.ucar.edu/content/software/netcdf
free (developed by UCAR)
binary format (compact)
self descriptive (header, attributes)
parallel version available
free tools to manipulate files (simple operations, concatenation)
libraries included in most of Post-processing software
(IDL, Matlab, OpenDX, Paraview, Vapor, ... but not Tecplot ?)
used by many organizations in several research domains (Atmospheric
Science, PIV)
Data Format: NetCDF
ncdump -h NetCDF file.nc

netcdf NetCDF_file.nc {
dimensions:
resolution_x = 2304 ;
resolution_y = 385 ;
resolution_z = UNLIMITED ; // (576 currently)
variables:
float grid_x(resolution_x) ;
float grid_yx(resolution_y, resolution_x) ;
float grid_z(resolution_z) ;
float velocity_x(resolution_z, resolution_y, resolution_x) ;
float velocity_y(resolution_z, resolution_y, resolution_x) ;
float velocity_z(resolution_z, resolution_y, resolution_x) ;
float pressure(resolution_z, resolution_y, resolution_x) ;
// global attributes:
:title = "3D Simulation of Navier Stokes in channel" ;
:X_lenght = 12.56f ;
:Y_lenght = 2.f ;
:Z_lenght = 3.141f ;
:Input_Reynolds = 12600.f ;
}
Post-Processing
Detection of turbulent structures

3D vorticity
Q = Ω2 − S 2
λ2
Up to 4th order statistics (flatness)
hui i → 3 variables
hui uj i → 6 variables
hui uj uk i → 10 variables
Problem of memory (number of variables)

Numerous possible errors (number of variables)
Problem of optimization (ex: vectorization)
⇒ Usually not possible in the course of the simulation
Parallelization of Post-Processing
Why specific (parallel) code for post-processing ?

Problem of CPU (can be larger than the simulation)
Problem of memory (distributed memory)
Adapt to the local resources (long lifetime)
Not dependent of present CPU resources
LML Post-Processing Codes
LML solution:
single processor on a limited streamwise domain
concatenation of all sub-domains at the end
Post-processing computations with 16 processors and 64 Gb RAM

Post-Processing: Reynolds stress budget
∂hui0 uj0 i ∂hui0 uj0 i

+ huk i = Pij + Tij + Dij + Dρ,ij + Φij − ij
∂t ∂xk
0 0 ∂hui i 0 0 ∂huj i
Production: Pij = −huj uk i − hui uk i ,
∂xk ∂xk
∂hui0 uj0 uk0 i
Turbulent Transport: Tij = − ,
∂xk
∂ 2 hui0 uj0 i
Viscous Diffusion: Dij = ν ,
∂xk ∂xk
∂huj0 p 0 i
!
1 ∂hui0 p 0 i
Pressure Diffusion: Dρ,ij = − + ,
ρ ∂xi ∂xj
0
* !+
p0 ∂ui0 ∂uj
Pressure Strain: Φij = + ,
ρ ∂xj ∂xi
0
* +
∂ui0 ∂uj
Dissipation: ij = 2ν .
∂xk ∂xk
Post-Processing: frame of reference
 
+cos(α) +sin(α) 0
A =  −sin(α) +cos(α) 0 
0 0 1
∂ui` ∂uj` ` ` ` X ∂upg ∂uqg g g g

(x , x , x ) = alk ark api aqj (x , x , x )
∂xk` ∂xk` 1 2 3 ∂xlg ∂xrg 1 2 3
l,r ,p,q
| {z }
34 summations
`: local frame, g : global frame

Parameters of DNS
Computation at HLRS (Germany): 64 Processors NEC-SX8
(Distributed European Infrastructure for Supercomputing Applications)
• Reynolds: Reτ = 617 at the inlet (from DNS)

• Domain: 4π × 2 × π
• Resolution: 2304 × 384 × 576 (510 Millions grid points)
• Integration : 450 000 time steps (160000 CPU hours)
• Memory: 400 Gb
• Storage : 932 fields 3D (u,v,w,p) > 7.0 Tb
60 × 48 × 31 time evolutions > 800 Gb
(every other 16 meshes in each direction)

Spatial resolution
(∆x + )max = 10.7, (∆y + )max = 7.9, (∆z + )max = 7.4

(∆x + )inlet = 5.1, [(∆y + )inlet ]max = 5.1, (∆z + )inlet = 3.4
Spatial resolution
«1/4
ν3
„
η= : Kolmogorov scale

Maximum mesh size: ∼ 4η in the region of maximum of k

∼ 6η at the wall
Pressure coefficients
Skin-friction coefficients
Vortices
Iso-value of Q = Ω2 − S 2 .
Vortices in detachment region
Iso-value of Q = Ω2 − S 2 in the separation region.

Streaks
3D Streaks modeled by tubes
3D modeling of streaks using image processing algorithms

(opening-closing, thinning, pruning, ...)
Reynolds Stresses: curved wall (Reτ = 617)








Balance of turbulent kinetic energy (Reτ = 617)








Filtering definition
We will focus on filters for homogeneous isotropic turbulence.

Small scales and large scales are separated using a low pass filter:
In physical space, convolution filter can be defined:
Z +∞
Φ(x) = Φ(x, t)G (x − ξ)d 3 ξ (26)
−∞
In spectral space, the same filter reduce to a single product
Φ̂(k) = Ĝ (k)Φ̂(k) (27)

Filtering
Filtering properties
In order to deal with filtered Navier stokes equations, the filter must satisfy
the following properties: 1. Preserving constants:
Z +∞
a = a ⇐⇒ G (x − ξ)d 3 ξ = 1 (28)
−∞
2. Linearity
φ+ψ =φ+ψ (29)
3. Commutativity with spacial and temporal derivatives
∂φ ∂φ
= , s = x, t (30)
∂s ∂s
Top Hat Filter

1/∆ si |x| ≤ ∆/2
G (x) = (31)
0 sinon
sin(k∆/2)
Ĝ (k) = (32)
k∆/2
Gaussian Filter
1/2
−γ|x|2

γ
G (x) = 2 exp 2 (33)
π∆ ∆
2
!
−∆ k 2
Ĝ (k) = exp (34)
4γ
Cutoff Filter
sin(kc x)
G (x) = , kc = π/∆ (35)
kc x

1 si |k| ≤ kc = π/∆
Ĝ (k) = (36)
0 sinon
Filtering results
The “cut-off ” filter is the ideal one to separate the small scales from
the large scales
The “Gaussien ” filter support if infinite both in physical space and in
spectral space, but it represents a good compromise.
Several other families of filters are possible (implicit filters) and may be
more relevant for LES
LES equations
In order to obtain LES equations, a filter is applied to the Navier Stokes

equations. When the filter has the good properties, we obtain:

∂u i ∂ ∂p ∂ ∂u i ∂u j
+ (ui uj ) = − +ν + , (37)
∂t ∂xj ∂xi ∂xj ∂xj ∂xi
∂u i
= 0 (38)
∂xi
defining ui0 = ui − u i the non-linear term can be decomposed into:

ui uj = (u i + ui0 ) u j + uj0 (39)
= u i u j + u i uj0 + ui0 u j + ui0 uj0 (40)
| {z } |{z}
Cij Rij
Rij : Reynolds Stress Tensor

Cij : Cross Stress Tensor
LES equations
Introducing the non-linear term u i u j computed with filtered quantities, one

can reshape the LES equations:
∂τij?

∂u i ∂ ∂p ∂ ∂u i ∂u j
+ (u i u j ) = − +ν + − , (41)
∂t ∂xj ∂xi ∂xj ∂xj ∂xi ∂xj
τij? = ui uj − u i u j = Cij + Rij (42)
where the subgrid tensor τij? is the only term to model as it is composed
with the unfiltered field.
LES equations
The following decomposition can also be used:
ui uj = (u i u j − u i u j ) +u i u j (43)
| {z }
Lij
Lij is called the Leonard term [8] and it represents the interactions between
large scales. The momentum equation becomes:

∂u i ∂ ∂p ∂ ∂u i ∂u j ∂τij
+ (u i u j ) = − +ν + − (44)
∂t ∂xj ∂xi ∂xj ∂xj ∂xi ∂xj
τij = Lij + Cij + Rij (45)
j u i u j is a quadratic term which contains scales smaller than the

filtered scales u i (and maybe smaller than the mesh size).
LES equations in spectral space

In spectral space the LES equations becomes:
∂ û i (k)
= Tir (k) + Tisgs (k) + νk 2 û i (k) (46)
∂t
where
ZZ
Tir (k) = Mijm û j (p)û m (q) δ(k − p − q) d 3 p d 3 q
ZZ n
Tisgs (k) = Mijm û j (p)uˆ0 m (q) + uˆ0 j (p)û m (q)
o
+uˆ0 j (p)uˆ0 m (q) δ(k − p − q) d 3 p d 3 q
Tir (k) is the resolved part (r) of the nonlinear term (involving only
resolved quantities).
Tisgs (k) is the subgrid scale (SGS) part of the nonlinear terms (i.e. to
be modeled).
Energy transfer (physical space)
Energy equation for the resolved scales qr2 = u i u i /2:

∂qr2 ∂u i ∂u i ∂u i ∂u i ∂
= ui uj + τij −ν − (u i p)
∂t ∂xj ∂xj ∂xj ∂xj ∂xi
| {z } | {z } | {z } | {z }
I II III IV
∂q 2
„ «
∂ ∂ ∂
+ ν r − (u i u i u j ) − (u i τij ) (47)
∂xi ∂xi ∂xj ∂xj
| {z } | {z } | {z }
V VI VII
I production
II subgrid scale turbulent dissipation (sgs )
III dissipation by molecular viscosity
IV diffusion by the pressure
V diffusion by viscous effect
VI diffusion by interactions between resolved scales
VII diffusion by interactions with subgrid scales
e
II + VII subgrid scale energy transfer (Tsgs )
The equation for Fi,ir

(k) = u i (k)u ∗i (k) is obtained by multiplying the LES
∗
equation by u i (k):
r
∂Fi,i (k)
= Γri,i (k) + Γsgs 2 r
i,i (k) − 2νk Fi,i (k) (48)
∂t
By integrating over a sphere of radius k = |k|
x ∂F r (k) x x
i,i
dS(k) = Γri,i (k) + Γsgs
i,i (k) dS(k) − 2νk 2 Fi,i (k)dS(k)
∂t
we obtain the equation for the 3-dimensional resolved spectral function
Er (k):
∂Er (k)
= Tr (k) + Tsgs (k) − 2νk 2 E(k) (49)
∂t
A-priori and a-posteriori test of SGS models
A-priori test : direct comparison of the real subgrid scale quantities (τij , sgs ,
...) with their modeled formulation
Need for DNS database or well resolved Experiment databases
Statical validation only
Usefull to understand how a model works
A-posteriori test : Comparison of the LES results (using a chosen SGS

model) with a reference (DNS, Experiments).
Need a clean reference (DNS or Experiment)
Dynamical validation (but link with numerics)
Usefull as validation in addition to a-priori tests
A-priori tests
Filter the original DNS (or experiment data) with a suitable filter for
LES to separate between the resolved scales u and the subfilter scales
u0 = u − u
Compute the exact values of the quantities to test

e ∂
τij , sgs = τij S ij , Tsgs = ∂xj [u i τij ] − sgs
Compare the real values with the modeled values

Comparison of plots (visualisation)
Computation of
Pcorrelation coefficients (statistical comparison)
ab
C (a, b) = pP P
a2 b 2
A priori estimation of subgrid terms

A-priori estimation of nonlinear subgrid terms for isotropic turbulence
(Rλ ' 200) [7]
(u, u0 )kc spectrum of the nonlinear term u · ∇u0 using a cutoff filter at kc
Subgrid scale models
1. “Turbulent viscosity model”: In this type of model, only the dissipative

e
part (sgs ) of the subgrid scale energy transfer Tsgs is modeled.
2. “scale similarity models”: Based on this hypothesis that the interaction
between the smallest resolved scales are of the same type than the
interaction between the smallest resolved scales (known) and the largest
subfilter scales (unknown)
3. “Mixtes models”: Combine a turbulent viscosity model and a scale
similarity model.
4. Models based on the (partial) deconvolution of filtered quantities. This
leads to reconstruct the subfilter scales which are not explicitly resolved.
Z There are many other SGS models which do not fall into the above
categories.
Spectral turbulent viscosity
The SGS energy transfer term Tesgs (k) in eq. (49) can be put into a
dissipative form
Tesgs (k) = −2 νe (k|kc ) k 2 Er (k) (50)
where νe (k|kc ) is a function of k and of the characteristic wavenumber of
the filter kc . It can have negative values.
This term can also be decomposed into a positive part and a negative part
Tesgs (k) = Te+ e−

sgs (k) + Tsgs (k)
= −2k 2 E(k) νe+ (k|kc ) + νe− (k|kc )

(51)
Te+
sgs (k) represents the positive energy transfer to the (resolved) modes
k of modulus k due to subgrid scales interactions
Te+
sgs (k) represents the negative energy transfer to the (resolved) modes
k of modulus k due to subgrid scales interactions
Property of turbulent viscosity

The form of turbulent viscosity depends on the parameters of the simulation
and more specifically of the type of filter. However, for an infinite Reynolds
number spectrum Er (k) ∝ k −5/3 , general behavior of turbulent viscosity can
be established:
Example of turbulent viscosity νe+ (k|kc )

(dotted line), −νe− (k|kc ) (dash line) and
νe (k|kc ) (solid line)
We can show that [5], in this theo-

retical case, for k < kc /3, νe (k|kc ) ∝
4/3
1/3 kc . Then, the turbulent viscos-
ity grows rapidly near kc .
Spectral turbulent viscosity model
In the spectral turbulent viscosity model, the energy transfer term Γsgs
i,i (k) is
modeled by a dissipative term for each wavenumber k instead of each k of
the 3-dimensional resolved spectral function Er (k)
Tisgs (k) = −2 νt (k|kc ) k 2 ûi (k) (52)
The momentum equation becomes:
∂ûi (k)
= Tir (k) + (ν + νt (k|kc )) k 2 ûi (k) (53)
∂t
Cholet & Lesieur [6, 9] proposed the following model of turbulent viscosity
1/2
E(kc , t)
νt (k|kc , t) = νt∞ (k|kc ) (54)
kc
with νt∞ (k|kc ) = 0.267 + e −3.03(kc /k) (55)
Characteristics of Spectral turbulent viscosity model
j The only justification of this model is based on the fact that the right
behavior of the subgrid scale transfer is recovered when writing the equation
for the variance. However, only the dissipative effect of the subgrid scale are
modeled from the combined effect of energy transfer
The model can be used only in spectral space (periodic isotropic

turbulence)
Restricted to very specific case (research cases)
Turbulent viscosity depends on the filter shape
The turbulent viscosity can be deduced from analytical development in
some configuration and under specific approximations
Accurate and cheap model in the case it can be used
Similar model can be derived in physical space (based on structure
functions)
Structure function model
Transposition by Métais & Lesieur of the constant spectral eddy

viscosity model in physical space
model based on the energy at the cutoff, but expressed in physical space
The energy at cutoff E (kc ) is evaluated by means of second-order
velocity structure function F2
Z
2
F2 (x, r , t) = [u(x, t) − u(x + x0 , t)] d 3 x0 (56)
|x0 |=r
Using a Kolmogorov spectrum, the turbulent viscosity can be expressed as

q
νsgs (x, r , t) = A(r /∆) ∆ F2 (x, r , t) (57)
where A(x) is a known function. So for r = ∆:

q
νsgs (x, ∆, t) = 0.105 ∆ F2 (x, ∆, t) (58)
Smagorinsky model [7]
the subgrid scale nonlinear term is modeled as a diffusive term:

∂ ∂
τij ∝ νsgs S ij (59)
∂xj ∂xj
Using the fact that the trace τkk of τij must be zero
1 mod
τij − τkk = −2 νsgs S ij (60)
3
Problems:
how to define νsgs ?
Is it universal ?
Does it depends of the filter ?
Smagorinsky model: derivation

Local equilibrium hypothesis:
Injected energy (at scale L) = Dissipated energy (at scale η)
If L >> ∆ _ energy totally transferred to subgrid scales ( = sgs ).
Using dimensional analysis:
' U 3 /L, sgs ' u 3 /∆ (61)
where U etu are the characteristic velocity at integral scale (L) and filter
scale (∆).
The energy dissipated by the model (sgs ) can be estimated by:
sgs ' νsgs u 2 /∆2 (62)
So
νsgs ' U∆4/3 L−1/3 (63)
and the velocity U can be estimated by:
1/2
U ' L Sij Sij ' L|S| (64)
Smagorinsky model: derivation

This leads to the following subgrid scale viscosity
νsgs = (Cs? )2 ∆4/3 L2/3 |S| (65)
However, the Integral scale (L) is difficult to compute, but we can assume
L/Delta = cte (not true in reality). So we can recover the more usual form
of the Smagorinsky model (obtained by dimensional argument too):
νsgs = Cs2 ∆2 |S| (66)
where Cs is a model constant of the order of 0.2.
This constant can be estimated in the case of very large Reynolds number
Isotropic turbulence by:
3/4
1 2
Cs = (67)
π 3Ck
where Ck is the Kolmogorov constant (Ck ' 1.6)
One recover Cs ' 0.165 for Ck ' 1.6
Smagorinsky model: constant
j The Smagorinsky constant is not constant !
the “constant” Cs is function of

the flow (isotropic, boundary layer, ...)
the Reynolds number
the filter (L/∆ and η/∆)
the space and time discretization
(→ numerical dissipation)
The “usual” values of the constant are

Cs ' 0.18 − 0.2 for isotropic turbulence
Cs ' 0.065 − 0.1 for turbulent shear flows
Optimal constant parameter
Isotropic turbulence (Rλ = 100), Second order Finite Differences, Runge

Kutta 2nd order:
From Meyers et al. [9]

Model’s constant dependency
From Meneveau et al. [8]
h ´3/2 i
C (∆) = −hτij S ij i/ ∆2 23/2 h S ij S ij
`
i , Pao’s spectrum (68)
Characteristics of Smagorinsky model
The Smagorinsky model is the most popular subgrid scale model BUT
The subgrid scale tensor is far from its ideal value (τijreal 6= τijsmag )
(a-priori validations) [7, 6].
The model is based on the turbulent viscosity hypothesis but Tsgs is not
only dissipative. → No possibility of “backscatter”(energy transfer
from subgrid scale to Resolved scales)
Leads to very poor results in the near wall region (can not be used in is
standard form) [5]
The constant Cs must be adapted for each flow (difficult for complex
flows).
However, several modifications exist to overcome some of these difficulties

(dynamic procedure to compute Cs ) [5].
A-priori validation of Smagorinsky model
Very weak correlation ! /

”Wall Adaptive Large Eddy” model (WALE) [2]

The WALE model is a ”Turbulent viscosity” model based on a new operator
to recover the right near wall scaling (νsgs ∼ y 3 ).
1
τij − τkk = −2νsgs S ij (69)
3
6
with
q
|G a |
νsgs = Cw2 ∆2 q 5 (70)
|S|5 |G a |
and
1
S ij = (∂i u j + ∂j u i )
2
1
G ij = (∂k u i ∂j u k + ∂k u j ∂i u k )
2
a
G ij = G ij − δij G kk
∆ : characteristic size of the filter
Germano’s dynamic procedure
Objectives: let the model’s constant vary (in space and time) and compute
an estimated value.
The Germano’s identity is based on the hypothesis that the Subgrid Tensors
computed on neighboring scales in the inertial range have a similar behavior.
By applying a second imaginary “test” filter ue to the LES equations, we

have
Tij = ugi uj − ui uj (71)
e e
τij is the subgrid scale tensor when using the filter u

Tij is the subgrid scale tensor when using the filter u
e

Translation of the Germano’s identity:
Lij = i uj − ui uj = Tij − τeij

ug (72)
e e
δij mod
Lij − Lkk = C αij − Cg
βij ' C (αij − β
fij ) (73)
3
with (
f2 e e
αij = 2∆ |S|S ij
2
βij = 2∆ |S|S ij
5 equations for 1 unknown (C ) _ multiply by a tensor (ex: S ij )
1 L S
C= ij ij (74)
2 S α −β
ij ij
fij
Dynamic procedure (Lilly) [1]
The use of S ij leads to locally negative and infinite values of C .
Lilly [1] proposed an other choice by trying to minimize the square of the
residues of the tensorial equation:
1 Lij Mij
C= (75)
2 Mij Mij
where
1
Mij = αij − β
fij (76)
2
→ Mij2 = 0 means that the full tensor M is zero (less often)
Dynamic procedure
The dynamic procedure can generate negative values of C

(negative viscosity)
Could model energy transfer from subgrid to resolved scales ,
Exponential growth of local velocities: numerically unstable /
Solutions :
_ suppress negative viscosity (C → max(C , 0))
_ average the function C
- over small domains of space or/and time
- in homogeneous directions (if any)
1 hLij S ij i
C =− (77)
2 hS ij αij i − hS ij β
fij i
Comparison WALE / Dynamic Smagorinsky

Comparison WALE / Dynamic Smagorinsky

Scale Similarity Hypothesis
Z The scale similarity hypothesis is equivalent to says that structure

of the subgrid tensor τij involving the subgrid scales is equivalent to the
structure of the same tensor τij built with the smallest resolved scales
Interpretation:
This hypothesis suppose that the energy transfer nearby the cutoff
wavenumber kc concern mostly the smallest resolved scales and the
largest subgrid scales.
This hypothesis suppose that scales of both sides of the cutoff scale are
strongly correlated.
This is an hypothesis which is in agreement with the Kraichnan’s theory
of turbulent viscosity (quick increase of turbulent viscosity near the
cutoff scale).
Bardina’s model [3]
In the Bardina’s model, the subgrid scale ui0 are replaced by the same filtered
scales u 0i . This leads to:
Rij = (u i − u i )(u j − u j ) (78)

Cij = (u i − u i )u j + (u j − u j )u i (79)
Or
Rij + Cij = u i u j − u i u j (80)
By adding the Leonard’s term Lij which is only based on resolved quantities,
we have:
mod
τij = u i u j − u i u j (81)
Therefore the model leads to approximate the total velocity field ui by the
filtered velocity field u i
Scale similarity models: definition
A more general form of the scale similarity model involve a second filter e
a
and a similarity constant C . The model becomes:
mod
i uj − ui uj )
τij = C (ug (82)
ee
Usually, the second filter (e

a) has a smaller cutoff frequency than the
first filter (a)
The Bardina’s model is a particular case with the same filter and with
C =1
Scale similarity models: characteristics
The model behavior is strongly linked to the filter shape and to the
separation between the two filters
Scale similarity models require no alignment between the subgrid tensor
and the strain tensor Sij . The models are not turbulent viscosity type of
models. ,
The model authorize energy backscatter (from subgrid scales to
resolved scales) ,
The modeled subgrid tensor is well correlated with the real solution
(much better a-priori tests than Smagorinsky model) ,
but the model is generally not dissipative enough to obtain a stable
solution /
Mixed models
Scale similarity models are usually associated with a turbulent viscosity

model to form a ”mixte model” [4, 2, 8]
mod
τij − Lij = τij? = −2νt Sij + C (u i u j − u i u j ) (83)
Keep the same good properties of a-priori tests

Add some dissipation with a turbulent viscosity model
Z Despite their good properties, mixed models are less popular than
classical turbulent viscosity models because they are a little bit more
expensive (second test filter) and not so easy to implement.
Mixte models
From Meneveau et al. [7]

A-priori tests of some models
Ex: Two-dimentional cylinder wake [6] (7 models investigated)

A-priori tests: results
Two-dimentional cylinder wake: sgs

A-priori tests: results
e ∂
A-priori test of Tsgs = ∂xj [u i τij ] − sgs , compact filter, kc = kmax /3
DNS Bardina Similarity Smag. Str. Func. Germano Lilly
DNS +1.000
Bardina -0.058 +1.000
Similarity +0.502 -0.080 +1.000
Smag. +0.219 -0.011 +0.297 +1.000
Str. Func. +0.228 -0.003 +0.306 +0.974 +1.000
Germano +0.005 -0.002 +0.020 +0.002 +0.013 +1.000
Lilly +0.113 -0.038 +0.253 -0.178 -0.136 +0.048 +1.000
Z Results function of the filter (shape & kc )

Model based on deconvolution
The aim of this type of model is to (partially) rebuild the total velocity field
ui as a function of the filtered velocity field u i . The filtering procedure being
usually define as a convolution of the filter with the velocity field, the inverse
procedure is called “deconvolution”
For a filter defined (in Fourier space) by:
Φ̂(k) = Ĝ (k)Φ̂(k) (84)

The inverse procedure exists if
the inverse operator Φ̂(k)/Ĝ (k) exists
if Φ̂(k)/Ĝ (k) is integrable.
The first condition is not verified by the ”top hat” filter as
G (k) = sin(k∆/2)/(k∆/2) is zero for several value of k
Inversion by iterative methods
An inversion method is based on an iterative procedure [1, 1, 2]:

X∞
G −1 = (I − G )p ; I = identité (85)
p=0

Φ = Φ + (Φ − Φ) + (Φ − 2Φ + Φ) + · · · (86)
One can obtain a approximated inverse

XN by truncation of the series at order N:
GN−1 = (I − G )p (87)
p=0
By introducing a Gaussian filter, Germano’s has shown, using this method,

that the dominant term of the subgrid tensor is:
∂ui ∂uj
τij ∼ (88)
∂xk ∂xk
Inversion by Taylor expansion

An other inversion method is based on Taylor expansion series of the filtered
quantities:
Z +∞
Φ(x) = Φ(ξ, t)G (x − ξ)dξ (89)
−∞
The Taylor serie expansion of the Φ(ξ, t) term at (x, t) gives
∂Φ(x, t) 1 ∂ 2 Φ(x, t)
Φ(ξ, t) = Φ(x, t) + (ξ − x) + (ξ − x)2 + ... (90)
∂x 2 ∂x 2
Introducing the expansion in eq. 89:
1 ∂ 2 Φ(x, t) +∞
Z
Φ(x) = Φ(x, t) + G (z)dz + ...
2 ∂x 2 −∞
1 ∂ n Φ(x, t) +∞ n
Z
+ z G (z)dz + ... (91)
n! ∂x n −∞
(92)

∞
X α(p) ∂ p Φ(x, t)
Φ(x) = Φ(x, t) + (93)
p=1
p! ∂x p
where α(l) is the lth-order moment of the convolution kernel:

Z +∞
α(p) = (−1)p z p G (z)dz (94)
−∞
which can be rewritten as follows:

Z (−π−x)/∆
p
α(p) = ∆ ξ p G (ξ)dξ (95)
(π−x)/∆
The filtered variable can be expanded as:

∞
X (−1)p ∂p Φ
Φ(x) = ∆p Mp (x) (x) (96)
p=0
p ∂x p
where Mp (x) is the p th moment of the filter G .

The unfiltered field can be expressed as:
N
!−1
X (−1)p p ∂p
Φ(x) = ∆ Mp (x) p Φ(x) (97)
p=0
p ∂x
Pruett et al.[8] proved that the series quickly converge for “Gaussian”and
“Top Hat”filters.
Eventually for a symmetric filter (G (ξ) = G (−ξ)) which have only even
moments,
∂2

1 2
Φ(x) = I − ∆ M2 (x) 2 + ... Φ(x) (98)
2 ∂x
Tensor diffusivity model [7]
The subgrid tensor can be estimated by

1 2 ∂ 2 Φ(x)
Φ0 (x) = ∆ M2 (x) + O(∆4 ) (99)
2 ∂x 2
1 2 ∂2 “ ”
= ∆ M2 (x) 2 Φ(x) + O(∆2 ) = Φ0 (x) + O(∆2 ) (100)
2 ∂x
Applying these results to the different components of τij , we obtain:
1 2 ∂2
Lij = ui uj − ui uj = ∆ M2 2 (u i u j ) + O(∆4 ) (101)
2 ∂x
2
∂ 2 uj ∂ 2 ui
„ «
1 2 ∂
Cij 0 0
= u i uj + ui u j = ∆ M2 2 u i + uj + O(∆4 ) (102)
2 ∂x ∂x 2 ∂x 2
1 “ 2 ”2 ∂ 2 u i ∂ 2 u j
Rij = ui0 uj0 = ∆ M2 + O(∆6 ) (103)
4 ∂x 2 ∂x 2
So ∂u i ∂u j
τij = Lij + Cij + Rij = ∆2 M2 + O(∆4 ) (104)
∂x ∂x
Limit of deconvolution methods

Limit of deconvolution methods
In practice, a cutoff filter is applied on top of the Gaussian filter (due to

the grid spacing)
On a discretized velocity field, the deconvolution is not able to rebuild
scales smaller than the mesh size
In the case of “optimal deconvolution” to the original grid
(k < km = 1/∆) the unfiltered velocity field corresponds to the total
field filtered with the cutoff filter at km . If no other model are
introduced, the LES equations reduce to truncated Navier Stokes
equations (unresolved DNS) /
In practical use, the deconvolution procedure is only partial
(Approximate Deconvolution Model). Therefore, the approximation is
used as a model.
An additional dissipative model can be associated to the ADM model
Model with estimation of subgrid scales
Model based on a fractal interpolation procedure of subgrid scales. This

model is only based on geometrical considerations (see [6]) .
Model based on an approximation of subgrid scales using u0a = θa N 0
where θa is a characteristic time and N 0 is a growing rate of subgrid
scales
∂ uei
Ni0 = (I − G ) ? −(uej − uj )
∂xj
e = Gd−1 u is the unfiltered field on a finer grid (see [2])
u
Multiscale model based on a deconvolution of the Navier Stokes
equations on a network of embedded grid (see [3, 3])
Multi-level Simulations
Relies on the resolution of an equation for u0a on a auxiliary grid.
Let’s consider N filters G1 , ..., GN with associated cutoff lengths

∆1 ≤ ... ≤ ∆N .
un = Gn ? ... ? G1 ? u = G1n ? u (105)

vn = un − un+1 = (G1n − G1n+1 ) ? u = Fn ? u (106)
where
un : resolved field at the nth level of filtering
vn : the nth level “details”
So we have the following decomposition
X
un = un−k + vn−l (107)
l=1,k
The evolution equations are:

∂u
+ N S(un ) = −τ n = [G1n ?, N S](u), n ∈ [1, N] (108)
∂t
where N S is the symbolic Navier Stokes operator and for the “details”:
∂v
+ N S(un ) − N S(un+1 ) = −τ n + τ n+1 , n ∈ [1, N − 1] (109)
∂t
There are several possibilities for reducing the complexity of the simulation
with respect to DNS:
Multi-level Simulations: quasi-static approximation

for high frequencies
Z Cycling strategy between the different grid levels freezing the high
frequency details over some time while integrating the equations for the low
frequency part.
Problem: how to determine the characteristic time over which the high
frequency can be frozen ?
Several models:
- Multi-mesh method of Voke
- Non-Linear Galerkin methods
- Incremental Unknowns technique
- Liu’s multigrid methods
- Multi-level algorithm proposed by Terracol
- ....
Z An other way is to reduce the complexity is to use a simplified equation

(linear) for the “details” (instead of the nonlinear eq. 109).
- Non-linear Galerkin methods
- Early version of Variational Multiscale approach (Hughues et al.)
- Langevin model of Dubrulle et al.
Z An other way is to limit the number of filtering levels. In this case, even
at the finest description level, subgrid-scale exist and have to be
parametrized. However, it is assumed that a simplified subgrid scale model
can be used at the finest filtering levels (closer to isotropy & less energy
than at coarser level)
- Multi-level algorithm proposed by Terracol
- Modified Estimation Procedure of Domaradzki
- Resolvable Subfilter Scale model (RSFS)
Two-level Simulations
Let’s consider the two-level version. The velocity field can be decomposed
into:
u2 : the resolved filtered field at the coarser level:
v1 : the part of unresolved field at the finest level which is resolved at
the finest level (u1 = (u2 − v1 ) being the resolved filtered field at the
finest level)
v0 the unresolved field at the finest level
If we assume that the filtering operator perfectly commute with the

differential operators, the coupling between the three level comes from the
nonlinear convective term.
B(u, u) = B(u2 + v1 + v0 , u2 + v1 + v0 ) (110)
where B is the bi-linear form defined as B(ui , uj ) = ui uj

At the coarser resolution

2 2 2 2 2
B(u, u) = B(u2 , u2 ) + B(u2 , v1 ) + B(v1 , u2 ) + B(v1 , v1 )
| {z } | {z }
I II
2 2
+ B(u2 + v1 , v0 ) + B(v0 , u2 + v1 + v0 ) (111)
| {z }
III
(I) can be computed directly at coarser grid level

(II) direct coupling between the two levels (can be computed exactly)
(III) direct coupling with the true subgrid scales
At the finest resolution

1 1 1 1 1
B(u, u) = B(u2 , u2 ) + B(u2 , v1 ) + B(v1 , u2 ) + B(v1 , v1 )
| {z } | {z } | {z }
IV V VI
1 1 1
2 2
+ B(u + v1 , v0 ) + B(v0 , u ) + B(v0 , v0 ) (112)
| {z }
VII
(IV) coupling of coarser scales at finest level

(V) coupling between the coarsest and the finest levels
(VI) nonlinear self-interaction between detail v1
(VII) interaction with scales v0 (true subgrid scales): must be modeled
Variational Multiscale Methods (VMS) of Hugues

et al.[9]
2 1
In practice, ∆ = 2∆
The coupling term III is neglected
The term VII is parametrized using a Smagorinsky-like method
!
∂vj1 ∂vi1
(VII )ij = −2νsgs +; (113)
∂xi ∂xj
- The small-small model
!
1 2p ∂vj1 ∂v 1
νsgs = (Cs ∆ ) 2|S 1 |, Sij1 = +; i (114)
∂xi ∂xj
- The large-small model
!
∂u 2j ∂u 2
q
1 2 2 2
νsgs = (Cs ∆ ) 2|S |, S ij = +; i (115)
∂xi ∂xj
Dynamic Subfilter scale Model of Dubrulle et al. [4]
Linearized version of the three-levels approach.
The nonlinear term VI is neglected

The subgrid term VII is not taken into account
⇒ Equation similar to Rapid Distortion Theory
Same idea used for the LES-Langevin model of Laval et al. [6].
Equation for v1 replaced by a Langevin equation for a linearized
equation for the term V
The equivalent of term VI is neglected
The subgrid term VII is modeled by a spectral eddy viscosity model
Tested in homogeneous isotropic turbulence
Generic nonlinear model of Lund & Novikov [3]
Assume that the deviator of the subgrid-scale tensor can be expressed as a

function of the resolved velocity field gradients (Sij , Ωij ), the unit tensor (Id ),
and the square of the cutoff scale (∆2 ):
1
τij − τkk δij ≡ τijd = F (Sij , Ωij , δij , ∆2 ) (116)
3
The most general form is a polynomial of infinite degree of tensor whose

a1 a2 a3 a4
terms are of the form S Ω S S where the ai are positive integers
This series can be reduced to a finite number of linearly independent
terms by the Cayley-Hamilton theorem.
As τ d is symmetrical, only symmetrical terms are retained
⇒ leads to the definition of eleven tensors m1 , m2 , ..., m11
Generic nonlinear model of Lund & Novikov [3]
The eleven linearly independent tensors are:

2
m1 = S m2 = S
2
m3 = Ω m4 = SΩ − ΩS
2 2
m5 = S Ω−ΩS m6 = Id
2 2 2 2 (117)
m7 = S Ω −Ω S m8 = ΩS Ω −Ω S Ω
2 2 2 2 2 2
m9 = S ΩS −S ΩS m10 = S Ω −Ω S
2 2 2 2
m11 = ΩS Ω −Ω S Ω
Under some assumptions on the flow (not axi-symmetrical shear & single
rotation axis aligned with one eigenvectors of S), 5 terms are enough to
represent the deviator part of the tensor:
2 2
τd = C1 ∆2 |S|S + C2 ∆2 (S )d + C3 ∆2 (Ω )d (118)
2 2 2 2
+C4 ∆ (S Ω − Ω S) + C5 ∆ (S Ω − S Ω )/|S|
Nonlinear models
First term: subgrid viscosity model for the forward energy transfer
The anisotropy of the normal stresses is including in the following terms
General model but

Very costly (leads to good results on isotropic homogeneous turbulence)
/
Must determine the 5 constants C1 , ..., C5 /
⇒ Horiuti have shown that the term (SΩ − ΩS) is responsible for a
significant improvement of the correlation with the true subgrid tensor. ,
Deardorf’s models [1]
The Deradorf’s model is based on the resolution of an equation for each

component of the subgrid scale tensor:
∂τij ∂ ∂u j ∂u i
= − (u k τij ) − τik − τjk
∂t ∂xk ∂xk ∂xk
∂uj0
0
∂ 0 0 0 0 ∂ui
− u u u +p +
∂xk i j k ∂xj ∂xi
∂ 0 0 ∂ 0 0 ∂u 0 ∂uj0
− ui p − uj p − 2ν i (119)
∂xj ∂xi ∂xk ∂xk
Deardorff’s models
Deardorff suggested the following model:

∂uj0
p
∂ui0 2
„ « „ «
qsgs 2 2 2 2
p0 + = −Cm τij − qsgs δij + qsgs S ij , (120)
∂xj ∂xi ∆ 3 5
∂ui0 ∂uj0 2
(qsgs )3/2
ν = δij Ce , (121)
∂xk ∂xk ∆
„ «
∂τjk ∂τik ∂τij
ui0 uj0 uk0 = −C3m ∆ + + , (122)
∂xi ∂xj ∂xk
p 0 ui0 ' 0. (123)
For isotropic turbulence, the following parameters can be used:
Cm = 4.13, Ce = 0.70, C3m = 0.2

Some solutions for anisotropic flows
Filter choice:
1/3
∆(x) =
∆1 (x)∆2 (x)∆3 (x) (124)
r
2 2 2
∆(x) = ∆1 (x) + ∆2 (x) + ∆3 (x) /3 (125)

∆(x) = max ∆1 (x), ∆2 (x), ∆3 (x) (126)
∆(x) = ...
Modification of the model
Proposition of Schumann for wall bounded flows:

splitting of τijd between the isotropic and the non-isotropic part
modeling of each part with a different turbulent viscosity
Anisotropic flows
τijd = −2νsgs Sij − hSij i − 2νsgs

a

hSij i (127)
with
2 q
νsgs = C1 ∆ 2 S ij − hS ij i S ij − hS ij i (128)
a
2 q
νsgs = C2 ∆z 2hS ij ihS ij i (129)
and
1/3
∆(z) = ∆1 ∆2 ∆3 (1 − exp (zuτ /25ν)) (130)
∆z (z) = ∆3 (1 − exp (zuτ /25ν)) (131)
C1 = 0.1, C2 = 0.254 for Horiuti [7]

C1 = C2 = 0.254 for Moin & Kim. [1]
Symmetry preserving of subgrid scale models

Navier Stokes equations preserves some symmetries (symmetry group):
the time translation
(t, x, u, p) → (t + a, x, u, p)
the pressure translation
(t, x, u, p) → (t, x, u, p + ξ(t))
the rotation
(t, x, u, p) → (t, Rx, Ru, p)
the generalized Galilean transformation
¨
(t, x, u, p) → (t, x + α(t), u + α̇(t), p − ρα̈(t) − 1/2ρα(t)α(t))
the the first scaling transformation
(t, x, u, p, ν) → (e 2a t, e a x, e −a u, e −2a p)
the the second scaling transformation
(t, x, u, p, ν) → (t, e a x, e a u, e 2a p, e 2a ν)
the reflexion
(t, x, u, p) → (t, Γx, Γu, p)
the material indifference in the limit of a 2D flow
Symmetry preserving of subgrid scale models

The filtered equations must also be preserved by all of these transformations,
since the solution (u, p) of the filtered equations is expected to be a good
approximation of (u, p). Using the hypothesis that the test filters do not
destroy the symmetry properties, we have:
From Razafindralandy et al.[4]

LES applications: wall bounded flows

Required resolution for inner layer (Reδ1.8 ) and outer layer (Reδ0.4 ):
LES capability comes from the inner layer only !

LES applications: wall bounded flows [3]
Problematic of LES of wall bounded flows:

Wall Resolved LES are unaffordable for large Reynolds number
Classical Wall model (based on log law) are ussually not applicable
(APG, separation)
Need to introduce a more elaborated wall treatment
Two-Layer model (TMLES): a fine 1D grid embbeded between the wall
and the first grid point
Wall model LES (WMLES): RANS equations are used for the inner layer
But this is a source of errors:

Inacuracy of the wall layer modelling assumptions (non equilibrium)
Inapplicability of the SGS models in the near wall regions
Increase of numerical error with sharp increase of resolution near the
wall
LES applications: hybrid RANS/LES
Two types of RANS/LES coupling:
Implicit coupling: Dettached Eddy Simulation (DES)

Same model formulation for RANS and LES (Spalart Almaras) [9]
Switch from RANS to LES automatically (function of mesh size /)
Explicit coupling: Two domains, two models

Need to define LES to RANS coupling and RANS to LES coupling
How to deal with fluctuations ? /
LES applications: Combustion Chamber of

Helicopter Engine
Benefit of LES in Industrial Context:
M. Boileau and G. Staffelbach (CERFACS and Turbomeca)
Z Nearly impossible without LES

N. A. Adams and S. Stolz.

Deconvolution methods for subgrid-scale approximation in large-eddy simulation.
In B. Geurts, editor, Modern Simulation Strategies for Turbulent Flow, pages 21–41. R. T. Edwards, 2001.
G. Araya, L. Castillo, C. Meneveau, and K. Jansen.

A dynamic multi-scale approach for turbulent inflow boundary conditions in spatially-developing flows.
to appear in J. Fluid Mech., 2010.
J. Bardina, J. H. Ferziger, and W. C. Reynolds.

Improved subgrid models for large eddy simulation.
AIAA paper, 80:1357, 1980.
J. Bardina, J. H. Ferziger, and W. C. Reynolds.

Improved turbulence models based on large eddy simulation of homogeneous, incompressible, turbulent flows.
Technical report, Technical Report TF-19, Thermal sciences div., Dept. of Mech. Engg., Stanford, 1983.
J. Bodart, J.-B. Cazalbou, and L. oly.

Large scale simulation of turbulence using a hybrid spectral/finite difference solver.
In 21st International Conference on Parallel Computational Fluid Dynamics (ParCFD), pages 1–5, Moffet Field, USA, May 18-22 2009.
J.-P. Chollet and M. Lesieur.

Parametrization of small scales of three-dimensional isotropic turbulence utilizing spectral closure.
J. Atmos. Sci., 38:2747–2757, 1981.
R. Clark, J. Ferziger, and W. Reynolds.

Evaluation of subgrid-scale models using an accurately simulated turbulent flow.
J. Fluid Mech., 91:1–16, 1979.
Pruett C. D., Sochacki J. S., and N. A. Adams.

On taylor-series approximations of the residual stress.
Phys. Fluids, 13:2587–2589, 2001.
J. W. Deardorff.
The use of subgrid transport equations in a three dimensional model of atmospheric turbulence.
ASME J. Fluid Eng., 95:429–438, 1973.
J. A. Domaradzki and E. M. Saiki.

A subgrid-scale model based on the estimation of unresolved scales of turbulence.
Phys. Fluids A, 9:2148–2164, 1997.
T. Dubois and F. Jauberteau.

A dynamical multilevel model for the simulation of the small structures in the dimensional homogenous isotropic turbulence.
Journal of Scientific Computing, 13:323–367, 1998.
B. Dubrulle, J.-P. Laval, S. V. Nazarenko, and N. K.-R. Kevlahan.

A dynamic subfilter-scale model for plane parallel flows.
Phys. Fluids, 13:2045–2064, 2001.
M. Germano, U. Piomelli, P. Moin, and W. H. Cabot.

A dynamical subgrid-scale eddy viscosity model.
Phys. Fluids A, 3:1760–1765, 1991.
M. A. Gonze.
Revue, amélioration et validation de modélisations sous-maille.
Rapport de stage post-doctoral, 1994.
K. Horiuti.
Comparison of the conservative and rotational forms in large eddy simulation of turbulent channel flows.
J. Comp. Phys., 71:343–370, 1987.
K. Horiuti.
A proper velocity scale for the modeling subgrid-scale eddy viscosities in large eddy simulation.
Phys Fluids, 5:146, 1993.
T. J. R. Hugues, L. Mazzei, and K. E. Jansen.

Large eddy simulation and the variational multiscale method.
Comp. Visual. Sci., 3:47–59, 2000.
N. Jarrin, S. Benhamadouche, D. Laurence, and R. Prosser.

A synthetic-eddy-method for generating inflow conditions for large-eddy simulations.
International journal of heat and fluid flow, 27(4):585–593, 2006.
J. Jiménez.
Computing high-Reynolds-number turbulence: will simulations ever replace experiments.
Journal of Turbulence, 4(22):1–14, 2003.
J. Jiménez, S. Hoyas, M. P. Simens, and Yoshinori Mizuno.

Comparison of turbulent boundary layers and channels from direct numerical simulation.
In Sixth International Symposium on Turbulence and Shear Flow Phenomena, pages 289–294, Seoul, Korea, 22-24 June, 2009, 2009.
A. Keating, U. Piomelli, E. Balaras, and H.-J. Kaltenbach.

A priori and a posteriori tests of inflow conditions for large-eddy simulation.
Phys. Fluids., 16(12):4696–4712, 2004.
R. H. Kraichnan.
Eddy viscosity in two and three-dimensions.
J. Atmosph. Science., 33:1521, 1976.
J.-P. Laval and B. Dubrulle.

A LES-langevin model for turbulence.
Eur. Phys. J. B., 49:471–481, 2006.
J.-P. Laval, B. Dubrulle, and J. C. McWilliams.

Langevin models of turbulence: Renormalization group, distant interaction algorithms or rapid distortion theory?
Phys. Fluids, 15(5):1327–1339, 2003.
A. Leonard.
Energy cascade in large-eddy simulations of turbulent flows.
Adv. in Geophys. A, 18:237–248, 1980.
M. Lesieur.
Turbulence in Fluids.
2nd edn. Kluwer, 1990.
D. K. Lilly.
A proposed modification of the germano subgrid-scale closure method.
Phys. Fluids A, 4:633–635, 1992.
S. Liu, C. Meneveau, and J. Katz.

On the properties of similarity subgrid-scale models as deduced from measurments in a turbulent jet.
J. Fluid Mech., 275:83–119, 1994.
T. S. Lund and E. A. Novikov.

Parametrization of subgrid-scale stress by the velocity gradient tensor.
Center for Turbulence Research, Annual Research Briefs, pages 23–43, 1992.
T.S. Lund, X. Wu, and K.D. Squires.

Generation of turbulent inflow data for spatially-developping boundary layer simulations.
J. Comp. Phys., 140:233–258, 1998.
P. J. Mason and D. J. Thomson.

Stochastic backscatter in large-eddy simulations of boundary layers.
J. Fluid Mech., 242:51–78, 1992.
C. Meneveau.
Statistics of turbulence subgrid scale stresses: necessary conditions and experimental tests.
Phys. Fluids, 6:815–833, 1994.
C. Meneveau and J. Katz.

Scale-inveriance and turbulence models for large eddy simulation.
Annu. Rev. Fluid Mech., 32:1–32, 2000.
C. Meneveau and T. S. Lund.

The dynamic smagorinsky model and scale-dependent coefficients in the viscous range of turbulence.
Phys. Fluids, 9(12):3932–3934, 1997.
J. Meyers, P. Sagaut, and B. J. Geurts.

Optimal model parameters for multi-obective large-eddy simulations.
Phys. Fluids, 18:095103, 2006.
P. Moin and J. Kim.

Numerical investigation of turbulent channel flow.
J. Fluid. Mech., 118:341–377, 1982.
F. Nicoud and F. Ducros.

Subgrid-scale stress modelling based on the square of the velocity gradient tensor.
Flow, Turbulence and Combustion, 62:183–200, 1999.
U. Piomelli.
Wall-layer models for large-eddy simulations.
Progress in Aerospace Sciences, 44:437–446, 2008.
D. Razafindralandy and A. Hamdouni.

Consequences of symmetries on the analysis and construction of turbulent models.
Symmetry, Integrality and Geometry: Methods and Applications, 2(052):1–20, 2006.
P. Schlatter and R Örlü.

Assesment of direct numerical simulation data of turbulent boundary layers.
Journal of Fluid Mechanics, 659:116–126, 2010.
A Scotti and C. Meneveau.

A fractal model for large eddy simulation of turbulent flow.
Physica D, 127:198–232, 1999.
J. Smagorinsky.
General circulation experiment with the primitive equations, part i: The basic experiment.
Mon. Wea. Rev., 91:99, 1963.
C. E. Smith, N. Beratlis, E. Balaras, K. Squires, and M. Tsunoda.

Numerical investigation of the flow over a golf ball in the subcritical and supercritical regimes.
Int. J. Heat Fluid Flows, 31:262–273, 2010.
P. R. Spalart and S. R. Allmaras.

One equation turbulence model for aerodynamic flows.
AIAA Paper 92-439, 30th Aerospace Sciences Meeting and Exhibit, 1992.
S. Stolz and N. A. Adams.

An approximate deconvolution procedure for the large-eddy simulation.
Phys. Fluids, 11:1699–1701, 1999.
S. Stolz, N. A. Adams, and L. Kleiser.

An approximate deconvolution model for large-eddy simulation with application to incompressible wall bounded flows.
Phys. Fluids, 13(4):997–1015, 2001.
R. Temam.
Multilevel methods for the simulation of turbulence. a simple model.
Journal of computational physics, 127:309–315, 1996.
X. H. Wu and P. Moin.
Direct numerical simulation of turbulence in a nominally zero-pressure-gradient flat-plate boundary layer.
J. Fluid. Mech., 630:5–41, 2009.
J. Sillero, G. Borrell, J. Jimenez, R.D. Moser.

Hybrid Open-MPI Turbulent Boundary Layer Code Over 32k Cores.
Proceeding EuroMPI’11 Proceedings of the 18th European MPI Users’
Group conference on Recent advances in the message passing interface., 218–227, 2011.
Laizet, S. and N. Li.

Incompact3d: A powerful tool to tackle turbulence problems with up to O(105 ) computational cores
Int. J. Numer. Meth. Fluids, 67:1735 1757, 2011.

Trans Les Dns PDF

Uploaded by

Copyright:

Available Formats

Trans Les Dns PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Trans Les Dns PDF

Uploaded by

Copyright:

Available Formats

Intro. CFD DNS Capacity Ex.

LML DNS Managing Databases LES models

International Master on Turbulence

Laboratoire de Mécanique de Lille (LML), CNRS

Objectives of the course

Problematic of numerical simulations of (turbulent) flows

Need to simulate complex flows

⇒ Need fast and robust methodologies.

Restricted to simple (or simplified) geometries

⇒ Numerical simulation is a tool to understand the physics.

Industrial and Research practices

Research in DNS and LES

From J. ISI Web of Science

Top500 List of supercomputers

Top500 List of supercomputers

Massively Parallel Supercomputer

Extreme parallelism and hybrid

Incompressible Navier Stokes equations

where the (strain rate tensor Sij is defined by:

Energy spectrum: definition

For isotropic homogeneous turbulence, one can define the 3D isotropic

dS(k) is the surface element of the sphere

Energy spectrum of turbulent flows

⇒ km is the inverse of the energy injection scale

Why do we need models ?

Direct Numerical Simulation (DNS) truncated simulation

⇒ Simulation of the statistical average

⇒ Unsteady simulation of large scales

⇒ Unsteady simulation of the largest scales

Navier Stokes equation in Fourier Space

is the Fourier transform of the nonlinear term (convolution product) and

Energy equation in spectral space

By integrating over a sphere of radius k = |k|

Energy equation in spectral space

(large scale) + (large scale) ⇒ (large scale)

Characteristic scales (isotropic turbulence)

• Dissipative scale (“Kolmogorov scale”) η is defined by

• Taylor micro-scale λ is defined such that

and, for isotropic turbulence, λ can be estimated by

Reynold based on integral scale (V rms = [ 13 u 2 ]1/2 ):

DNS at Reτ = 150

DNS at Reτ = 150

DNS at Reτ = 1500

DNS at Reτ = 1500

Resolution for a DNS at Reτ = 1500

For a DNS to be correctly resolved, one need to resolved scales

This leads to the following estimation:

This means 1015 mesh points for Reλ = 1500

DNS of Isotropic turbulence: history

From Y. Kaneda, “Computing Turbulence in the 40963 Range”

Largest DNS of Isotropic turbulence

Simulation of Forced Isotropic Turbulence performed on “Earth

N Reλ kmax ν hi L λ η

Largest DNS of Isotropic turbulence

DNS 40963 gris points Kaneda et al.[?]

DNS of Isotropic turbulence

Memory requirement (time integration, Poisson equation):

M ' 40N 3 (bytes) (19)

Q ' 10N 3 log2 N (20)

N Reλ kmax ν hi L λ η