Modern Python Cookbook: 130+ updated recipes for modern Python 3.12 with new techniques and tools

Modern Python Cookbook

Third Edition

130+ updated recipes for modern Python 3.12 with new techniques and tools

Steven F. Lott

Packt and this book are not officially connected with Python. This book is an effort from the Python community of experts to help more developers.

Modern Python Cookbook

Third Edition

Copyright © 2024 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Denim Pinto Swaroop Singh Parvathy Nair Elliot Dallow Safis Editing Karan Sonawane Safis Editing Subalakshmi Govindhan Ganesh Bhadwalkar Vignesh Raju

First published: November 2016 Second edition: July 2020 Third edition: July 2024 Production reference: 1 260724

Published by Packt Publishing Ltd. Grosvenor House 11 St Paul’s Square Birmingham B3 1RB, UK.

ISBN 978-1-83546-638-4

www.packt.com

Modern Python Cookbook

Modern Python Cookbook

Preface

What you need for this book

Who this book is for

What this book covers

To get the most out of this book

Chapter 1 Numbers, Strings, and Tuples

1.1 Choosing between float, decimal, and fraction

1.2 Choosing between true division and floor division

1.3 String parsing with regular expressions

1.4 Building complicated strings with f-strings

1.5 Building complicated strings from lists of strings

1.6 Using the Unicode characters that aren’t on our keyboards

1.7 Encoding strings – creating ASCII and UTF-8 bytes

1.8 Decoding bytes – how to get proper characters from some bytes

1.9 Using tuples of items

1.10 Using NamedTuples to simplify item access in tuples

Chapter 2 Statements and Syntax

2.1 Writing Python script and module files – syntax basics

2.2 Writing long lines of code

2.3 Including descriptions and documentation

2.4 Writing better docstrings with RST markup

2.5 Designing complex if...elif chains

2.6 Saving intermediate results with the := walrus operator

2.7 Avoiding a potential problem with break statements

2.8 Leveraging exception matching rules

2.9 Avoiding a potential problem with an except: clause

2.10 Concealing an exception root cause

2.11 Managing a context using the with statement

Chapter 3 Function Definitions

3.1 Function parameters and type hints

3.2 Designing functions with optional parameters

3.3 Using super flexible keyword parameters

3.4 Forcing keyword-only arguments with the * separator

3.5 Defining position-only parameters with the / separator

3.6 Picking an order for parameters based on partial functions

3.7 Writing clear documentation strings with RST markup

3.8 Designing recursive functions around Python’s stack limits

3.9 Writing testable scripts with the script-library switch

Chapter 4 Built-In Data Structures Part 1: Lists and Sets

4.1 Choosing a data structure

4.2 Building lists – literals, appending, and comprehensions

4.3 Slicing and dicing a list

4.4 Shrinking lists – deleting, removing, and popping

4.5 Writing list-related type hints

4.6 Reversing a copy of a list

4.7 Building sets – literals, adding, comprehensions, and operators

4.8 Shrinking sets – remove(), pop(), and difference

4.9 Writing set-related type hints

Chapter 5 Built-In Data Structures Part 2: Dictionaries

5.1 Creating dictionaries – inserting and updating

5.2 Shrinking dictionaries – the pop() method and the del statement

5.3 Writing dictionary-related type hints

5.4 Understanding variables, references, and assignment

5.5 Making shallow and deep copies of objects

5.6 Avoiding mutable default values for function parameters

Chapter 6 User Inputs and Outputs

6.1 Using the features of the print() function

6.2 Using input() and getpass() for user input

6.3 Debugging with f{value=} strings

6.4 Using argparse to get command-line input

6.5 Using invoke to get command-line input

6.6 Using cmd to create command-line applications

6.7 Using the OS environment settings

Chapter 7 Basics of Classes and Objects

7.1 Using a class to encapsulate data and processing

7.2 Essential type hints for class definitions

7.3 Designing classes with lots of processing

7.4 Using typing.NamedTuple for immutable objects

7.5 Using dataclasses for mutable objects

7.6 Using frozen dataclasses for immutable objects

7.7 Optimizing small objects with __slots__

7.8 Using more sophisticated collections

7.9 Extending a built-in collection – a list that does statistics

7.10 Using properties for lazy attributes

7.11 Creating contexts and context managers

7.12 Managing multiple contexts with multiple resources

Chapter 8 More Advanced Class Design

8.1 Choosing between inheritance and composition – the is-a question

8.2 Separating concerns via multiple inheritance

8.3 Leveraging Python’s duck typing

8.4 Managing global and singleton objects

8.5 Using more complex structures – maps of lists

8.6 Creating a class that has orderable objects

8.7 Deleting from a list of complicated objects

Chapter 9 Functional Programming Features

9.1 Writing generator functions with the yield statement

9.2 Applying transformations to a collection

9.3 Using stacked generator expressions

9.4 Picking a subset – three ways to filter

9.5 Summarizing a collection – how to reduce

9.6 Combining the map and reduce transformations

9.7 Implementing there exists processing

9.8 Creating a partial function

9.9 Writing recursive generator functions with the yield from statement

Chapter 10 Working with Type Matching and Annotations

10.1 Designing with type hints

10.2 Using the built-in type matching functions

10.3 Using the match statement

10.4 Handling type conversions

10.5 Implementing more strict type checks with Pydantic

10.6 Including run-time valid value checks

Chapter 11 Input/Output, Physical Format, and Logical Layout

11.1 Using pathlib to work with filenames

11.2 Replacing a file while preserving the previous version

11.3 Reading delimited files with the CSV module

11.4 Using dataclasses to simplify working with CSV files

11.5 Reading complex formats using regular expressions

11.6 Reading JSON and YAML documents

11.7 Reading XML documents

11.8 Reading HTML documents

Chapter 12 Graphics and Visualization with Jupyter Lab

12.1 Starting a Notebook and creating cells with Python code

12.2 Ingesting data into a notebook

12.3 Using pyplot to create a scatter plot

12.4 Using axes directly to create a scatter plot

12.5 Adding details to markdown cells

12.6 Including Unit Test Cases in a Notebook

Chapter 13 Application Integration: Configuration

13.1 Finding configuration files

13.2 Using TOML for configuration files

13.3 Using Python for configuration files

13.4 Using a class as a namespace for configuration

13.5 Designing scripts for composition

13.6 Using logging for control and audit output

Chapter 14 Application Integration: Combination

14.1 Combining two applications into one

14.2 Combining many applications using the Command design pattern

14.3 Managing arguments and configuration in composite applications

14.4 Wrapping and combining CLI applications

14.5 Wrapping a program and checking the output

Chapter 15 Testing

15.1 Using docstrings for testing

15.2 Testing functions that raise exceptions

15.3 Handling common doctest issues

15.4 Unit testing with the unittest module

15.5 Combining unittest and doctest tests

15.6 Unit testing with the pytest module

15.7 Combining pytest and doctest tests

15.8 Testing things that involve dates or times

15.9 Testing things that involve randomness

15.10 Mocking external resources

Chapter 16 Dependencies and Virtual Environments

16.1 Creating environments using the built-in venv

16.2 Installing packages with a requirements.txt file

16.3 Creating a pyproject.toml file

16.4 Using pip-tools to manage the requirements.txt file

16.5 Using Anaconda and the conda tool

16.6 Using the poetry tool

16.7 Coping with changes in dependencies

Chapter 17 Documentation and Style

17.1 The bare minimum: a README.rst file

17.2 Installing Sphinx and creating documentation

17.3 Using Sphinx autodoc to create the API reference

17.4 Identifying other CI/CD tools in pyproject.toml

17.5 Using tox to run comprehensive quality checks

Other Books You May Enjoy

Index

Landmarks

Title Page

Cover

Table of Contents

Modern Python Cookbook

Third Edition

130+ updated recipes for modern Python 3.12 with new techniques and tools

Steven F. Lott

Packt and this book are not officially connected with Python. This book is an effort from the Python community of experts to help more developers.

Modern Python Cookbook

Third Edition

Copyright © 2024 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Senior Publishing Product Manager:: Denim Pinto

Acquisition Editor – Peer Reviews:: Swaroop Singh

Project Editor:: Parvathy Nair

Senior Development Editor:: Elliot Dallow

Copy Editor:: Safis Editing

Technical Editor:: Karan Sonawane

Proofreader:: Safis Editing

Indexer:: Subalakshmi Govindhan

Presentation Designer:: Ganesh Bhadwalkar

Developer Relations Marketing Executive:: Vignesh Raju

First published: November 2016

Second edition: July 2020

Third edition: July 2024

Production reference: 1260724

Published by Packt Publishing Ltd.

Grosvenor House

11 St Paul’s Square

Birmingham

B3 1RB, UK.

ISBN 978-1-83546-638-4

www.packt.com

Contributors

About the author

Steven F. Lott has been turning coffee into software since the days when computers were large, expensive, and rare. Working for decades in high tech has given him exposure to a lot of ideas and techniques, some bad, but most are useful and helpful to others.

Steven has been working with Python since the ‘90s, building a variety of tools and applications. He’s written a number of books, and sometimes manages to attend conferences.

Steven is currently a tech nomad who lives in various places on the East Coast of the US. He tries to live by the words, Don’t come home until you have a story. You can find him online at https://fosstodon.org/@slott56.

About the reviewers

Abhijeet Jagutai Dada Mote is a software engineer with expertise in data engineering, data analysis, data science, and automation. He has worked in AdTech, semiconductors, IoT, and media industries and is currently working on a cutting-edge CTV ad platform. He is proficient in Python, development of automation testing frameworks, advanced big data pipelines, and AI-driven anomaly detection systems. He has also contributed to smart city projects, holds various AI related patents, is working on several AI research papers, and delivered a workshop at PyCon Italia. Find out more at: https://scholar.google.com/citations?user=v4tFOAgAAAAJ

I would like to thank my wife, Priyanka, and our one-year-old child, Advaith, for their support and patience. I also thank my Aai-Baba, friends, and colleagues who have helped me become who I am today.

Francisco Benavides has over 20 years of experience in programming, with the last 10 focused on Python development for ETL server applications. He has been part of worldwide teams spanning five continents, has been a team leader and specialist in the telecom sector, a Python developer, and has now been a data engineer in the energy sector for over 5 years, where he has developed a multitude of data ingestion platforms using Linux, Azure, and AWS.

I thank God for the opportunities he has allowed me to take and a special thank you for the blessings I received from my wife and kids, who have endured, at times, long months apart, working and encouraging me to keep going. I also thank the editorial team for trusting me and allowing me to collaborate in the making of this book.

Join our community Discord space

Join our Python Discord workspace to discuss and learn more about the book: https://packt.link/dHrHU

Preface

Python is the preferred language choice of developers, engineers, data scientists, and hobbyists everywhere. It is a great scripting language that can power your applications and provide speed, safety, and scalability. By exposing Python as a series of simple recipes, this book can help you gain insights into specific language features in a concrete context. The idea is to avoid abstract discussions of language features and focus on applying the language to concrete data and processing problems.

What you need for this book

All you need to follow through the examples in this book is a computer running any Python, version 3.12 or newer. Many of the examples can be adapted to work with Python 3 versions prior to 3.12. Material in Chapter 10 describes the

match

statement, introduced with Python 3.10.

We strongly encourage installing a fresh copy of Python, avoiding any pre-installed operating system Python. The language run-time can be downloaded from https://www.python.org/downloads/. An alternative is to start with the Miniconda tool ( https://docs.conda.io/en/latest/miniconda.html) and use conda to create a Python 3.12 (or newer) environment.

Python 2 cannot be used anymore. Since 2020, Python 2 is no longer an alternative.

Who this book is for

The book is for web developers, programmers, enterprise programmers, engineers, and big data scientists. If you are a beginner, this book can get you started. If you are experienced, it will expand your knowledge base. A basic knowledge of programming will help; while some foundational topics are covered, this is not a tutorial on programming or Python.

What this book covers

There are over 130 recipes in this book. We can decompose them into four general areas:

Python Fundamentals

Chapter 1, Numbers, Strings, and Tuples, will look at the different kinds of numbers, how to work with strings, how to use tuples, and how to use the essential built-in types in Python. We will also show ways to exploit the full power of the Unicode character set.

Chapter 2, Statements and Syntax, will cover some basics of creating script files. Then we’ll move on to looking at some of the complex statements, including

if

,

while

,

for

,

break

,

try

,

raise

, and

with

.

Chapter 3, Function Definitions, will look at a number of function definition techniques. We’ll devote several recipes to type hints for a variety of types. We’ll also address an element of designing a testable script by using functions and a main-import-switch.

Chapter 4, Built-In Data Structures Part 1: Lists and Sets, begins an overview of the built-in data structures structures that are available and what problems they solve. This includes a number of recipes showing list and set operations, including list and set comprehensions.

Chapter 5, Built-In Data Structures Part 2: Dictionaries, continues examining the built-in data structures, looking at dictionaries in detail. This chapter will also look at some more advanced topics related to how Python handles references to objects. It also shows how to handle mutable objects as function parameter default values.

Chapter 6, User Inputs and Outputs, explains how to use the different features of the

print()

function. We’ll also look at the different functions used to provide user input. The use of f-strings for debugging and the

argparse

module for command-line input are featured.

Object-Oriented and Functional Design Approaches

Chapter 7, Basics of Classes and Objects, begins the coverage of object-oriented programming. It shows how to create classes and the type hints related to class definitions. This section has been expanded from previous editions to cover dataclasses. It shows how to extend built-in classes, and how to create context managers to manage resources.

Chapter 8, More Advanced Class Design, continues the exploration of object-oriented design and programming. This includes an exploration of the composition vs. inheritance question, and shows how to manage the duck typing principle of Python.

Chapter 9, Functional Programming Features, looks at Python’s functional programming features. This style of programming emphasizes function definitions and stateless, immutable objects. The recipes look at generator expressions, using the

map()

,

filter()

, and

reduce()

functions. We also look at ways to create partial functions and some examples of replacing stateful objects with data structures built from collections of immutable objects.

More Sophisticated Designs

Chapter 10, Working with Type Matching and Annotations, looks more closely at type hints and the

match

statement. This includes using Pydantic to create classes with more strict run-time type-checking. It also looks at introspection of annotated types.

Chapter 11, Input/Output, Physical Format, and Logical Layout, will work with paths and files in general. It will look at reading and writing data in a variety of file formats, including CSV, JSON (and YAML), XML, and HTML. The HTML section will emphasize using BeautifulSoup for extracting data.

Chapter 12, Graphics and Visualization with Jupyter Lab, will use Jupyter Lab to create notebooks that use Python for data analysis and visualization. This will show ways to ingest data into a notebook to create plots, and how to use Markdown to create useful documentation and reports from a notebook.

Chapter 13, Application Integration: Configuration, will start looking at ways that we can design larger applications. The recipes in this chapter address different ways to handle configuration files and how to manage logging.

Chapter 14, Application Integration: Combination, will continue looking at ways to create composite applications from smaller pieces. This will look at object-oriented design patterns and Command-LineInterface (CLI) applications. It will also look at using the

subprocess

module to run existing applications under Python’s control.

Completing a Project: Fit and Finish

Chapter 15, Testing, provides recipes for using the built-in

doctest

and

unittest

testing frameworks used in Python. Additionally, recipes will cover the pytest tool.

Chapter 16, Dependencies and Virtual Environments, covers tools used to manage virtual environments. The built-in

venv

, as well as conda and poetry will be covered. There are a lot of solutions to managing virtual environments, and we can’t cover all of them.

Chapter 17, Documentation and Style, covers additional tools that can help to create high-quality software. This includes a particular focus on sphinx for creating comprehensive, readable documentation. We’ll also look at tox to automate running tests.

To get the most out of this book

To get the most out of this book you can download the example code files and the color images as per the instructions below.

Download the example code files

The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Modern-Python-Cookbook-Third-Edition. This repository is also the best place to start a conversation about specific topics discussed in the book. Feel free to open an issue if you want to engage with the authors or other readers. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://packt.link/gbp/9781835466384.

Conventions used

In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning.

Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: "We can include other contexts through the use of the

include

directive."

A block of code is set as follows:

if distance is None: 

    distance = rate * time 

elif rate is None: 

    rate = distance / time 

elif time is None: 

    time = distance / rate

Any command-line input or output is written as follows:

>>> import math 

>>> math.factorial(52) 

80658175170943878571660636856403766975289505440883277824000000000000

New terms and important words are shown in bold.

Warnings or important notes appear like this.

Tips and tricks appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: Email

[email protected]

, and mention the book’s title in the subject of your message. If you have questions about any aspect of this book, please email us at [email protected].

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book we would be grateful if you would report this to us. Please visit, http://packtpub.com/support/errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.

Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit http://authors.packtpub.com.

Share your thoughts

Once you’ve read Modern Python Cookbook, Third Edition, we’d love to hear your thoughts! Scan the QR code below to go straight to the Amazon review page for this book and share your feedback or leave a review on the site that you purchased it from.

https://packt.link/r/1835466389

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere? Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

Follow these simple steps to get the benefits:

Scan the QR code or visit the link below

https://packt.link/free-ebook/9781835466384

Submit your proof of purchase

That’s it! We’ll send your free PDF and other benefits to your email directly

1

Numbers, Strings, and Tuples

This chapter will look at some of the central types of Python objects. We’ll look at working with different kinds of numbers, working with strings, and using tuples. These are the simplest kinds of data that Python works with. In later chapters, we’ll look at data structures built on these foundations.

While these recipes start with a beginner’s level of understanding of Python 3.12, they also provide some deeper background for those familiar with the language. In particular, we’ll look at some details of how numbers are represented internally, because this can help when confronted with more advanced numerical programming problems. This will help us distinguish the uses cases for the rich variety of numeric types.

We’ll also look at the two different division operators. These have distinct use cases, and we’ll look at one kind of algorithm that demands truncated division.

When working with strings, there are several common operations that are important. We’ll explore some of the differences between bytes—as used by our OS files—and strings used to represent Unicode text. We’ll look at how we can exploit the full power of the Unicode character set.

In this chapter, we’ll show the recipes as if we’re working from the

>>>

prompt in interactive Python. This is the prompt that’s provided when running

python

from the command line or using the Python console in many Integrated Development Environment (IDE) tools. This is sometimes called the read-evaluate-print loop (REPL). In later chapters, we’ll change the style so it looks more like a script file. One goal of this chapter is to encourage interactive exploration because it’s a great way to learn the language.

We’ll cover these recipes to introduce basic Python data types:

Choosing between float, decimal, and fraction

Choosing between true division and floor division

String parsing with regular expressions

Building complicated strings with f-strings

Building complicated strings from lists of strings

Using the Unicode characters that aren’t on our keyboards

Encoding strings – creating ASCII and UTF-8 bytes

Decoding bytes – how to get proper characters from some bytes

Using tuples of items

Using NamedTuples to simplify item access in tuples

We’ll start with numbers, work our way through strings, and end up working with simple combinations of objects in the form of tuples and

NamedTuple

objects.

1.1 Choosing between float, decimal, and fraction

Python offers several ways to work with rational numbers and approximations of irrational numbers. We have three basic choices:

Float

Decimal

Fraction

When we have choices, it helps to have some criteria for making a selection.

1.1.1 Getting ready

There are three general cases for expressions that involve numbers beyond integers, which are:

Currency: Dollars, cents, euros, and so on. Currency generally has a fixed number of decimal places and rounding rules to properly quantize results.

Rational Numbers or Fractions: When we scale a recipe that serves eight, for example, down to five people, we’re doing fractional math using a scaling factor of 5 8 .

Floating Point: This includes all other kinds of calculations. This also includes irrational numbers, like π, root extraction, and logarithms.

When we have one of the first two cases, we should avoid floating-point numbers.

1.1.2 How to do it...

We’ll look at each of the three cases separately.

Doing currency calculations

When working with currency, we should always use the

decimal

module. If we try to use the values of Python’s built-in

float

type, we can run into problems with the rounding and truncation of numbers:

To work with currency, import the

Decimal

class from the

decimal

module:

>>> from decimal import Decimal

We need to create

Decimal

objects from strings or integers. In this case, we want 7.25%, which is 7.25 100 . We can compute the value using

Decimal

objects:

>>> tax_rate = Decimal(’7.25’)/Decimal(100) 

>>> purchase_amount = Decimal(’2.95’) 

>>> tax_rate * purchase_amount 

Decimal(’0.213875’)

We could also use

Decimal(’0.0725’)

instead of doing the division explicitly.

To round to the nearest penny, create a

penny

object:

>>> penny = Decimal(’0.01’)

Quantize the result using the

penny

object:

>>> total_amount = purchase_amount + tax_rate * purchase_amount 

>>> total_amount.quantize(penny) 

Decimal(’3.16’)

This uses the default rounding rule of

ROUND_HALF_EVEN

. The

Decimal

module offers other rounding variations. We might, for example, do something like this:

>>> import decimal 

>>> total_amount.quantize(penny, decimal.ROUND_UP) 

Decimal(’3.17’)

This shows the consequences of using a different rounding rule.

Fraction calculations

When we’re doing calculations that have exact fraction values, we can use the

fractions

module to create rational numbers. In this example, we want to scale a recipe for eight down to five people, using 58 of each ingredient. When the recipe calls for 2 1 2 cups of rice, what does that scale down to?

To work with fractions, we’ll do this:

Import the

Fraction

class from the

fractions

module:

>>> from fractions import Fraction

Create

Fraction

objects from strings, integers, or pairs of integers. We created one fraction from a string, ’

2.5

’. We created the second fraction from a floating-point expression,

5

 /

 8

. This only works when the denominator is a power of 2:

>>> sugar_cups = Fraction(’2.5’) 

>>> scale_factor = Fraction(5/8) 

>>> sugar_cups * scale_factor 

Fraction(25, 16)

We can see that we’ll use almost a cup and a half of rice to scale the recipe for five people instead of eight. While float values will often be useful for rational fractions, they may not be exact unless the denominator is a power of two.

Floating-point approximations

Python’s built-in

float

type can represent a wide variety of values. The trade-off here is that a float value is often an approximation. There may be a small discrepancy that reveals the differences between the implementation of

float

and the mathematical ideal of an irrational number:

To work with

float

, we often need to round values to make them look sensible. It’s important to recognize that all float calculations are an approximation:

>>> (19/155)*(155/19) 

0.9999999999999999

Mathematically, the value should be 1. Because of the approximations used, the computed result isn’t exactly 1. We can use

round(answer,

 3)

to round to three digits, creating a value that’s more useful:

>>> answer = (19/155)*(155/19) 

>>> round(answer, 3) 

1.0

Approximations have a very important consequence.

Don’t compare floating-point values for exact equality.

Code that uses an exact

==

test between floating-point numbers has the potential to cause problems when two approximations differ by a single bit.

The float approximation rules come from the IEEE, and are not a unique feature of Python. Numerous programming languages work with float approximations and have identical behavior.

1.1.3 How it works...

For these numeric types, Python offers a variety of operators:

+

,

-

,

*

,

/

,

//

,

%

, and

**

. These are for addition, subtraction, multiplication, true division, truncated division, modulo, and raising to a power, respectively. We’ll look at the two division operators,

/

and

//

, in the Choosing between true division and floor division recipe.

Python will do some conversions between the various numeric types. We can mix

int

and

float

values; the integers will be promoted to floating-point to provide the most accurate answer possible. Similarly, we can mix

int

and

Fraction

as well as mixing

int

and

Decimal

. Note that we cannot casually mix

Decimal

with float or

Fraction

; an explicit conversion function will be required.

It’s important to note that float values are approximations. The Python syntax allows us to write floating-point values using base 10 digits; however, that’s not how values are represented internally.

We can write the value 8.066 × 10⁶⁷ like this in Python:

>>> 8.066e+67 

8.066e+67

The actual value used internally will involve a binary approximation of the decimal value we wrote. The internal value for this example is this:

>>> (6737037547376141/(2**53))*(2**226) 

8.066e+67

The numerator is a big number,

6737037547376141

. The denominator is always 2⁵³. This is why values can get truncated.

We can use the

math.frexp()

function to see these internal details of a number:

>>> import math 

>>> math.frexp(8.066E+67) 

(0.7479614202861186, 226)

The two parts are called the mantissa (or significand) and the exponent. If we multiply the mantissa by 2⁵³, we always get a whole number, which is the numerator of the binary fraction.

Unlike the built-in float, a

Fraction

is an exact ratio of two integer values. We can create ratios that involve integers with a very large number of digits. We’re not limited by a fixed denominator.

A

Decimal

value, similarly, is based on a very large integer value, as well as a scaling factor to determine where the decimal place goes. These numbers can be huge and won’t suffer from peculiar representation issues.

1.1.4 There’s more...

The Python

math

module contains several specialized functions for working with floating-point values. This module includes common elementary functions such as square root, logarithms, and various trigonometry functions. It also has some other functions such as gamma, factorial, and the Gaussian error function.

The

math

module includes several functions that can help us do more accurate floating-point calculations. For example, the

math.fsum()

function will compute a floating-point sum more carefully than the built-in

sum()

function. It’s less susceptible to approximation issues.

We can also make use of the

math.isclose()

function to compare two floating-point values, an expression, and a literal

1.0

, to see if they’re nearly equal:

>>> (19/155)*(155/19) == 1.0 

False 

>>> math.isclose((19/155)*(155/19), 1.0) 

True

This function provides us with a way to compare two floating-point numbers meaningfully for near-equality.

Python also offers complex numbers. A complex number has a real and an imaginary part. In Python, we write

3.14+2.78j

to represent the complex number 3.14 + 2.78 √ --- − 1 . Python will comfortably convert between float and complex. We have the usual group of operators available for complex numbers.

To support complex numbers, there’s the

cmath

package. The

cmath.sqrt()

function, for example, will return a complex value rather than raise an exception when extracting the square root of a negative number. Here’s an example:

>>> math.sqrt(-2) 

Traceback (most recent call last): 

... 

ValueError: math domain error 

>>> import cmath 

>>> cmath.sqrt(-2) 

1.4142135623730951j

This module is helpful when working with complex numbers.

1.1.5 See also

We’ll talk more about floating-point numbers and fractions in the Choosing between true division and floor division recipe.

Seehttps://en.wikipedia.org/wiki/IEEE_floating_point.

1.2 Choosing between true division and floor division

Python offers us two kinds of division operators. What are they, and how do we know which one to use? We’ll also look at the Python division rules and how they apply to integer values.

1.2.1 Getting ready

There are several general cases for division:

A div-mod pair: We want both parts – the quotient and the remainder. The name refers to the division and modulo operations combined together. We can summarize the quotient and remainder as q,r = (⌊ a b ⌋,a mod b).

We often use this when converting values from one base into another. When we convert seconds into hours, minutes, and seconds, we’ll be doing a div-mod kind of division. We don’t want the exact number of hours; we want a truncated number of hours, and the remainder will be converted into minutes and seconds.

The true value: This is a typical floating-point value; it will be a good approximation to the quotient. For example, if we’re computing an average of several measurements, we usually expect the result to be floating-point, even if the input values are all integers.

A rational fraction value: This is often necessary when working in American units of feet, inches, and cups. For this, we should be using the

Fraction

class. When we divide

Fraction

objects, we always get exact answers.

We need to decide which of these cases apply, so we know which division operator to use.

1.2.2 How to do it...

We’ll look at these three cases separately.

Doing floor division

When we are doing the div-mod kind of calculations, we might use the floor division operator,

//

, and the modulo operator,

%

. The expression

a

 %

 b

gives us the remainder from an integer division of

a

 //

 b

. Or, we might use the

divmod()

built-in function to compute both at once:

We’ll divide the number of seconds by 3,600 to get the value of hours. The modulo, or remainder in division, computed with the

%

operator, can be converted separately into minutes and seconds:

>>> total_seconds = 7385 

>>> hours = total_seconds // 3600 

>>> remaining_seconds = total_seconds % 3600

Next, we’ll divide the number of seconds by 60 to get minutes; the remainder is the number of seconds less than 60:

>>> minutes = remaining_seconds // 60 

>>> seconds = remaining_seconds % 60 

>>> hours, minutes, seconds 

(2, 3, 5)

Here’s the alternative, using the

divmod()

function to compute quotient and modulo together:

Compute quotient and remainder at the same time:

>>> total_seconds = 7385 

>>> hours, remaining_seconds = divmod(total_seconds, 3600)

Compute quotient and remainder again:

>>> minutes, seconds = divmod(remaining_seconds, 60) 

>>> hours, minutes, seconds 

(2, 3, 5)

Doing true division

Performing a true division calculation gives a floating-point approximation as the result. For example, about how many hours is 7,385 seconds? Here’s 736805 using the true division operator:

>>> total_seconds = 7385 

>>> hours = total_seconds / 3600 

>>> round(hours, 4) 

2.0514

We provided two integer values, but got a floating-point exact result. Consistent with our previous recipe, when using floating-point values, we rounded the result to avoid having to look at tiny error digits.

Rational fraction calculations

We can do division using

Fraction

objects and integers. This forces the result to be a mathematically exact rational number:

Create at least one

Fraction

value:

>>> from fractions import Fraction 

>>> total_seconds = Fraction(7385)

Use the

Fraction

value in a calculation. Any integer will be promoted to a

Fraction

:

>>> hours = total_seconds / 3600 

>>> hours 

Fraction(1477, 720)

The denominator of 720 doesn’t seem too meaningful. Working with fractions like this requires a bit of finesse to find useful denominators that makes sense to people. Otherwise, converting to a floating-point value can be useful.

If necessary, convert the exact

Fraction

into a floating-point approximation:

>>> round(float(hours), 4) 

2.0514

First, we created a

Fraction

object for the total number of seconds. When we do arithmetic on fractions, Python will promote any integers to

Fraction

objects; this promotion means that the math is done as precisely as possible.

1.2.3 How it works...

Python has two division operators:

The

/

true division operator produces a true, floating-point result. It does this even when the two operands are integers. This is an unusual operator in this respect. All other operators preserve the type of the data. The true division operation – when applied to integers – produces a float result.

The

//

truncated division operator always produces a truncated result. For two integer operands, this is the truncated quotient. When floating-point operands are used, this is a truncated floating-point result:

>>> 7358.0 // 3600.0 

2.0

1.2.4 See also

For more on the choice between floating-point and fractions, see the Choosing between float, decimal, and fraction recipe.

See PEP-238.

1.3 String parsing with regular expressions

How do we decompose a complex string? What if we have complex, tricky punctuation? Or—worse yet—what if we don’t have punctuation, but have to rely on patterns of digits to locate meaningful information?