Performance Analysis at Full Power: Julien Rouhaud

Performance analysis at full power
Julien Rouhaud
pgconf.eu 2019
Oct. 16th 2019
1/40
Julien Rouhaud Performance analysis at full power
Who am I
Julien Rouhaud, from France

Working with PostgreSQL since 2008
DBA, consulting, developer
Author of HypoPG and other tools
Some contributions to PostgreSQL
2/40
Why this talk
My own experience
Based on my experience as database administrator

(subset of) Existing (or new) facilities I find most
useful
Open source
For performance analysis !
There are many other facilities availables and other
approaches
Sometime complementary (some info are only
available in the logs, pgBadger is so useful)
3/40
Why this talk
PostgreSQL’s moving fast
PostgreSQL changes
New features for better performance
New bottlenecks
New performance counters
Lot of metrics available on the OS side
top, perf, iostat...
PostgreSQL’s core statistics
some metrics available
Cumulated statistics
No underlying system metrics
but extensible, there are tools to help !
4/40
PostgreSQL statistics
How it works
Some in core, some in contrib, some in external

extensions
Almost all of them are cumulated counters over
time
Usually store information in shared memory
Accessible with views or Set Returning Functions
5/40
List of in-core views
6/40
The limits
No historisation done by PostgreSQL

You know the cumulated counters since the last
reset
Are those counters always increasing the same
way ?
What happened yesterday between 9AM and 2PM ?
7/40
Raw data
8/40
The solution
Get all metrics every few minutes, and store it

somewhere
You can do that manually with cron or custom script
Or use PoWA
Extensible infrastructure to historize multiple data
sources
optional background worker for a self contained
solution
optional daemon for more complex setup
Custom UI to vizualize and analyze metrics
9/40
Time visualisation
10/40
pg_stat_statements
Must have extensions
Official contrib
Global view of what’s happening on your server
Query normalization, based on object identifiers
Cumulate many statistics per queryid, userid,
dbid
cumulated runtime and number of execution
min, max, mean time
shared/local buffers access (hit, read, dirtied,
written)
temps files
IO timing (depending on track_io_timing)
11/40
pg_stat_statements
What can we learn ?
Most frequent queries

Slowest queries
Queries generating most amount of temporary files
Per-query hit-ratio
Queries requiring more work_mem
...
12/40
pg_stat_statements
Query example
SELECT round(total_time::numeric/calls, 2) AS avg_time, mean_time,

rows/(calls) AS avg_rows,
shared_blks_hit * 100 / (shared_blks_hit+shared_blks_read) AS hit_ratio,
query
FROM pg_stat_statements s JOIN pg_database d ON d.oid = s.dbid
WHERE datname = 'bench' AND (shared_blks_hit+shared_blks_read) > 0
ORDER BY total_time / calls DESC;
13/40
pg_stat_statements
Over time
Query runtime per second (kind of SQL load)
global, per-database or per-query

14/40
pg_stat_statements
On time interval
And general consumption over a specific interval
Drill-down approach to investigate performance

issues
15/40
pg_stat_statements
Identify slow queries
16/40
pg_stat_kcache
Kernel metrics
github.com/powa-team/pg_stat_kcache
Wrapper around get_rusage(2)
Gives access to kernel metrics, aggregated per
(queryid, dbid, userid) :
Physical disk reads and writes
User and system CPU
Context switches, page faults
17/40
pg_stat_kcache
What can we learn ?
"Real" hit-ratio : shared_buffers vs OS cache vs Disk

access
CPU intensive queries
Too high number of active queries
18/40
pg_stat_kcache
Examples - per database
19/40
pg_stat_kcache
Examples - per query
20/40
pg_wait_sampling
Wait events monitoring
github.com/postgrespro/pg_wait_sampling/
Developed by Postgres Professional
Efficient high frequency sampling of wait events
Default period is 10ms, customisable
Aggregated per queryid, dbid
For 9.6+ only, when Wait Events were introduced
21/40
pg_wait_sampling
What can we learn ?
Low level bottlenecks that can’t be seen at SQL

level
Costly parts of a query execution
Lightweight locks contention (Buffer mapping, WAL
write lock. . .)
IPC, IO and other events
22/40
pg_wait_sampling
Examples
Per database :
23/40
pg_wait_sampling
Examples
Per query :
24/40
pg_qualstats
Statistics on predicates
github.com/powa-team/pg_qualstats
Gather statistics on predicates (WHERE / JOIN
clauses)
Number of underlying query executions
Number of predicate’s operator execution
Selectivity
Sequential scan or index scan
Per queryid, userid, dbid
Sampled to avoid overhead (default is 1 /
max_connections)
25/40
pg_qualstats
What can we learn ?
Detect missing indexes

Differentiate most executed, most/least filtering,
most frequent constants
Detect possible partial indexes
If sampled over time, avoid suggesting indexes for
night batches
26/40
pg_qualstats
Constant distribution
27/40
pg_qualstats
Index suggestion
28/40
HypoPG
Hypothetical indexes
github.com/HypoPG/hypopg
Hypothetical indexes, aka. "What if this index
existed ?"
Create "fake" indexes instantly, without any
resource consumption
EXPLAIN can use such index
29/40
pg_qualstats + HypoPG
Index validation
30/40
Global index suggestion
Get all executed queries on the given time interval

Get all interesting predicates (seq scan, filtering at
least 30%. . .
Get information about indexing capabilities
(operators, datatype, opclass. . .)
Analyze and suggest indexes to optimize all queries
with the least amount of indexes
Check with HypoPG that indexes would be used
31/40
Global index suggestion
32/40
pg_track_settings
History of configuration changes
github.com/rjuju/pg_track_settings/
SQL only extension
detect and store the settings changed since last call
both global and object specific (eg. ALTER
DATABASE SET)
and also postgres restart
33/40
pg_track_settings
Example
What changed since yesterday ?
# SELECT * FROM pg_track_settings_diff(now() - interval '1 day', now());

name | from_setting | from_exists | to_setting | to_exists
---------------------+--------------|-------------|------------|----------
checkpoint_segments | 30 | t | 35 | t
(1 row)
34/40
pg_track_settings
Example
What’s the full history for a specific setting ?
# SELECT * FROM pg_track_settings_log('checkpoint_segments');

ts | name | setting_exists | setting
---------------------------+---------------------+----------------+---------
2015-01-25 01:01:42.58+01 | checkpoint_segments | t | 35
2015-01-25 01:00:37.44+01 | checkpoint_segments | t | 30
(2 rows)
35/40
pg_track_settings
Example
What was the configuration like at a specific

timestamp ?
# SELECT * FROM pg_track_settings('2015-01-25 01:01:00');

name | setting
------------------------------+---------
[...]
checkpoint_completion_target | 0.9
checkpoint_segments | 30
checkpoint_timeout | 300
[...]
36/40
pg_track_settings
Graph annotation
Available in PoWA, filtered by database if applicable
37/40
Demo
Demo
dev-powa.anayrat.info
(not credential required, just click connect)
38/40
Conclusion
A lot of tool are there to help

Can be used alone or together
Or even integrated in your own solution
39/40
Questions ?
rjuju.github.io
@rjuju123
Ïpowateam (pg12 compatible)
40/40

Performance Analysis at Full Power: Julien Rouhaud

Uploaded by

Copyright:

Available Formats

Performance Analysis at Full Power: Julien Rouhaud

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Performance Analysis at Full Power: Julien Rouhaud

Uploaded by

Copyright:

Available Formats

Performance analysis at full power

Oct. 16th 2019

Julien Rouhaud, from France

Based on my experience as database administrator

Some in core, some in contrib, some in external

No historisation done by PostgreSQL

Get all metrics every few minutes, and store it

Most frequent queries

SELECT round(total_time::numeric/calls, 2) AS avg_time, mean_time,

Query runtime per second (kind of SQL load)

global, per-database or per-query

And general consumption over a specific interval

Drill-down approach to investigate performance

"Real" hit-ratio : shared_buﬀers vs OS cache vs Disk

Low level bottlenecks that can’t be seen at SQL

Detect missing indexes

Get all executed queries on the given time interval

What changed since yesterday ?

# SELECT * FROM pg_track_settings_diff(now() - interval '1 day', now());

What’s the full history for a specific setting ?

# SELECT * FROM pg_track_settings_log('checkpoint_segments');

What was the configuration like at a specific

# SELECT * FROM pg_track_settings('2015-01-25 01:01:00');

Available in PoWA, filtered by database if applicable

A lot of tool are there to help

You might also like