Metaprogramming with Python: A programmer's guide to writing reusable code to build smarter applications

BIRMINGHAM—MUMBAI

Metaprogramming with Python

Copyright © 2022 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Associate Group Product Manager: Gebin George

Publishing Product Manager: Shweta Bairoliya

Senior Editor: Nisha Cleetus

Content Development Editor: Yashi Gupta

Technical Editor: Pradeep Sahu

Copy Editor: Safis Editing

Project Coordinator: Deeksha Thakkar

Proofreader: Safis Editing

Indexer: Hemangini Bari

Production Designer: Prashant Ghare

Marketing Coordinator: Sonakshi Bubbar

First published: August 2022

Production reference: 1110822

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-83855-465-1

www.packt.com

To my husband, Dileep V, and to all my family members, for their sacrifices and for exemplifying the power of determination during one of the toughest times of our lives.

– Sulekha AloorRavi

Contributors

About the author

Sulekha AloorRavi is an engineer and data scientist with a wide technical breadth and deep understanding of many technologies and systems. Her background has led her to working on the advanced Python-based application development in the field of artificial intelligence. She enjoys solving real-world business problems with technology and working with data science and business intelligence teams to deliver real value.

She has 15+ years of experience in software engineering and has worked with major IT solution providers and international banks. She graduated with an engineering degree in information technology and later completed a postgraduate program in big data and machine learning. She also enjoys teaching artificial intelligence and machine learning.

I want to thank the people who have been close to me and supported me, especially my husband, Dileep, my nephew, Sathvik, and all my family members.

About the reviewers

Florian Dahlitz has worked in the IT industry together with companies in the insurance, banking, and public industries to realize digitalization and automation as well as AI projects. He received a BSc in applied computer science from the Baden-Württemberg Cooperative State University and will shortly receive his MSc in information systems engineering and management from the Karlsruhe Institute of Technology (KIT). Florian enjoys teaching others programming in Python and helps them raise their Python skills to the next level. He spends his free time in nature and likes to capture landscapes with his camera. 

Sri Manikanta Palakollu is a full-stack web developer with experience in Java, Python, C, C++, databases, AEM, machine learning, and data science. He is a tech reviewer for various tech book publishers. He has published many articles in various fields, such as data science, programming, and cybersecurity, in publications such as HackerNoon, freeCodeCamp, and DDI. He also wrote a book named Practical System Programming with C, Apress Publications.

Sri Manikanta has won a national-level hackathon and regularly contributes to various open source projects. He has mentored more than 5,000 students in many national- and international-level coding hackathons hosted by multiple organizations, colleges, and universities.

Dr. Madhavi Vaidya is an experienced and qualified academician and researcher with a demonstrated history of working in the education management industry, skilled in various programming languages. 

Dr. Madhavi has an understanding and knowledge of various programming and database technologies, data analytics, information retrieval, software engineering, and project management. She is a strong education professional with a Master of Computer Applications and Doctor of Philosophy in the subject of computer science and engineering. One of the key areas of her research is big data analytics using Hadoop MapReduce and various big data technologies. 

Table of Contents

Preface

Part 1: Fundamentals – Introduction to Object-Oriented Python and Metaprogramming

Chapter 1: The Need for and Applications of Metaprogramming

Technical requirements

An overview of metaprogramming

Metaprogramming – a practical introduction

Metadata of the add function

Resolving type errors using metaprogramming

Understanding why we need metaprogramming

Don’t Repeat Yourself

Exploring the applications of metaprogramming

Summary

Chapter 2: Refresher of OOP Concepts in Python

Technical requirements

Introducing our core example

Creating classes

Understanding objects

Applying methods

Implementing inheritance

Extending to multiple inheritance

Understanding polymorphism

Polymorphism within inheritance

Polymorphism in independent classes

Hiding details with abstraction

Protecting information with encapsulation

Private members

Protected members

Summary

Part 2: Deep Dive – Building Blocks of Metaprogramming I

Chapter 3: Understanding Decorators and their Applications

Technical requirements

Looking into simple function decorators

Understanding function decorators with an application

Exchanging decorators from one function to another

Applying multiple decorators to one function

Exploring class decorators

Understanding class decorators with an application

Getting to know built-in decorators

The static method

The class method

Summary

Chapter 4: Working with Metaclasses

Technical requirements

Overview of metaclasses

The structure of a metaclass

Analyzing the arguments

The application of metaclasses

Inheriting the metaclass

Inheriting as a parent and metaclass

Switching metaclasses

Inheritance in metaclasses

Manipulating class variables

Summary

Chapter 5: Understanding Introspection

Technical requirements

Introducing built-in functions

Using the built-in id function

Debugging unintentional assignments using id

Finding out whether an object is callable

Checking whether an object has an attribute

Checking whether an object is an instance

Checking whether an object is a subclass

Understanding the usage of property

Using property as a decorator

Summary

Chapter 6: Implementing Reflection on Python Objects

Technical requirements

Introducing built-in functions used in reflection

Using id to delete duplicates

Using callable to dynamically check and generate methods

Using hasattr to set values

Using isinstance to modify an object

Using issubclass to modify a class

Applying property on a class

Summary

Chapter 7: Understanding Generics and Typing

Technical requirements

What are generics?

How are generics connected to metaprogramming?

How are generics handled in Python?

What happens when data types are specified?

Type hints as annotations

Typing with explicit type checks – approach 1

Creating a class to implement type checking

Creating a class to test type checking

Typing with explicit type checks – approach 2

Creating a class to implement type checking

Creating a class to test type checking

Adding data types with constraints

Creating a simple custom data type

Creating a domain-specific data type

Summary

Chapter 8: Defining Templates for Algorithms

Technical requirements

Explaining a sequence of operations

Back to our core example

The vegetables and dairy counter

Less than 10 items counter

The greater than 10 items counter

Electronics counter

Defining the sequence of methods

The vegetable counter

Less than 10 items counter

Greater than 10 items counter

The electronics counter

Identifying the common functionalities

Designing templates

Summary

Part 3: Deep Dive – Building Blocks of Metaprogramming II

Chapter 9: Understanding Code through Abstract Syntax Tree

Technical requirements

Exploring the ast library

Inspecting Python code with abstract syntax trees

Reviewing simple code using ast

Modifying simple code using ast

Understanding abstract syntax trees with applications

Understanding the ast of a class

Modifying the ast of a code block by parsing

Modifying the ast of a code block by transforming nodes

Summary

Chapter 10: Understanding Method Resolution Order of Inheritance

Technical requirements

Understanding the MRO of a class

Understanding MRO in single inheritance

Understanding MRO in multiple inheritances

Reviewing MRO in multilevel inheritance

Understanding the importance of modifying the order of inheritance

Impact of unintended change of order in inheritance

Summary

Chapter 11: Creating Dynamic Objects

Technical requirements

Exploring type for dynamic objects

Creating multiple instances of a class dynamically

Creating dynamic classes

Creating multiple dynamic classes

Creating dynamic attributes and methods

Defining attributes dynamically

Defining methods dynamically

Summary

Chapter 12: Applying GOF Design Patterns – Part 1

Technical requirements

An overview of design patterns

Exploring behavioral design patterns

Understanding the chain of responsibility

Learning about the command design pattern

The strategy design pattern

Summary

Chapter 13: Applying GOF Design Patterns – Part 2

Technical requirements

Exploring structural design patterns

Understanding the bridge pattern

Understanding the facade pattern

Understanding the proxy pattern

Exploring creational design patterns

Understanding the factory method

Understanding the prototype method

Understanding the singleton pattern

Summary

Chapter 14: Generating Code from AST

Technical requirements

Generating a simple class with a template

Generating multiple classes from a list

Generating a class with attributes

Generating a class with methods

Generating a class with an init method

Generating a class with a user-defined method

Defining a custom class factory

Developing a code generator to generate a simple library

Summary

Chapter 15: Implementing a Case Study

Technical requirements

Explaining the case study

Defining base classes

Developing a code generator library

Generating code

Designing an execution framework

Summary

Chapter 16: Following Best Practices

Technical requirements

Following PEP 8 standards

Indentation

Neat representation

Writing clear comments for debugging and reusability

Adding documentation strings

Documentation string for metaprogramming

Naming conventions

Class names

Variables

Functions and methods

Avoiding the reuse of names

Avoiding metaprogramming where not required

Summary

Other Books You May Enjoy

Preface

Effective and reusable code makes your application development process seamless and easily maintainable. With Python, you have access to advanced metaprogramming features that you can use to build high-performing applications.

This book starts by introducing you to the need for and applications of metaprogramming, before navigating the fundamentals of object-oriented programming. As you progress, you will learn about simple decorators, then work with meta classes, and later focus on introspection and reflection.

You will also delve into generics and typing, before defining templates for algorithms.

After that, you will understand your code using abstract syntax trees and explore method resolution order. This book also shows you how to create your own dynamic objects before structuring the objects through design patterns. Finally, you will learn about simple code-generation techniques along with best practices and eventually build your own applications.

By the end of this learning journey, you will have the skills and confidence you need to design and build reusable high-performing applications that can solve real-world problems.

Who this book is for

If you are an intermediate-level Python programmer looking to enhance your coding skills by developing reusable and advanced frameworks, this book is for you. Basic knowledge of Python programming will help you get the most out of this learning journey.

What this book covers

Chapter 1, The Need for and Applications of Metaprogramming, explains the need for one of the most advanced features in Python and its practical applications. 

Chapter 2, Refresher of OOP Concepts in Python, gives an overview of the existing OOP concepts, such as classes, methods, and objects, along with examples. 

Chapter 3, Understanding Decorators and Their Applications, covers the concept of decorators on functions and classes with the intent to provide you with a detailed overview of decorators, how to code them, and where to use them. This chapter also covers a detailed code walkthrough of the examples. 

Chapter 4, Working with Metaclasses, covers the concept of base classes and metaclasses with the intent to provide you with a detailed overview of metaclasses, how to code them, and where to use them. This chapter also covers a detailed code walkthrough of the examples. 

Chapter 5, Understanding Introspection, covers the concept of introspection in Python with the intent to provide you with a detailed overview of introspection, how to code it, and where to use it. This chapter also covers a detailed code walkthrough of the examples. 

Chapter 6, Implementing Reflection on Python Objects, covers the concept of reflection in Python with the intent to provide you with a detailed overview of reflection, how to code it, and where to use it. This chapter also covers a detailed code walkthrough of the examples. 

Chapter 7, Understanding Generics and Typing, covers the concept of generics in Python with the intent to provide you with a detailed overview of generics, how to code them, and where to use them. This chapter also covers a detailed code walkthrough of the examples.

Chapter 8, Defining Templates for Algorithms, covers the concept of templates in Python with the intent to provide you with a detailed overview of templates, how to code them, and where to use them. This chapter also covers a detailed code walkthrough of the examples. 

Chapter 9, Understanding Code through Abstract Syntax Trees, covers the concept of abstract syntax trees in Python with the intent to provide you with a detailed overview of what abstract syntax trees are, how to code them, and where to use them. This chapter also covers a detailed code walkthrough of the examples. 

Chapter 10, Understanding Method Resolution Order of Inheritance, covers the concept of method resolution order in Python with the intent to provide you with a detailed overview of method resolution order, how to code it, and where to use it. This chapter also covers a detailed code walkthrough of the examples. 

Chapter 11, Creating Dynamic Objects, covers the concept of dynamic objects in Python with the intent to provide you with a detailed overview of dynamic objects, how to code them, and where to use them. This chapter also covers a detailed code walkthrough of the examples. 

Chapter 12, Applying GOF Design Patterns – Part 1, covers the concept of behavioral design patterns in Python with the intent to provide you with a detailed overview of behavioral design patterns and apply them in different applications. This chapter also covers a detailed code walkthrough of the examples. 

Chapter 13, Applying GOF Design Patterns – Part 2, covers the concept of structural and creational design patterns in Python with the intent to provide you with a detailed overview of structural and creational design patterns and apply them in different applications. This chapter also covers a detailed code walkthrough of the examples. 

Chapter 14, Code Generation, covers the concept of code generation in Python with the intent to provide you with a detailed overview of code generation, how to develop a code generator that generates reusable code, and where to use it. This chapter also covers a detailed code walkthrough of the examples. 

Chapter 15, Development of an End-to-End Case Study-Based Application, covers the implementation of all the concepts we have learned so far by developing a case study-based application and a framework to test it. Detailed code with classes and methods along with an explanation of the code is covered in this chapter. Additionally, the steps on how to package and deploy the developed application into a Python library are also covered. 

Chapter 16, Following Best Practices, covers the best practices that can be followed while implementing the concepts of metaprogramming and answers questions such as where to use and where not to use these concepts in your Python application development life cycle. 

To get the most out of this book

Please install the latest version of Python, preferably Python 3.0 or above, and install the latest version of Anaconda from https://www.anaconda.com/products/distribution. Once installed, open Jupyter Notebook to run the examples provided in this book.

If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book’s GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Metaprogramming-with-Python. If there’s an update to the code, it will be updated in the GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots and diagrams used in this book. You can download it here: https://packt.link/LTQbb.

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: To explain this further, let us look at an example where we will generate a class named VegCounter by parsing a series of strings using the ast module.

A block of code is set as follows:

actualclass = compile(class_tree, 'vegctr_tree', 'exec')

actualclass

When we wish to draw your attention to a particular part of a code block or show the output of a code, the relevant lines or items are set in bold:

at 0x0000028AAB0D2A80, file vegctr_tree, line 1>

Tips or Important Notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [email protected] and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you’ve read Metaprogramming with Python, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Part 1: Fundamentals – Introduction to Object-Oriented Python and Metaprogramming

The objective of this section is to give you an overview of the concept of metaprogramming, its usage, and its advantages in building Python-based applications. This section also covers the basics of object-oriented programming in Python, such as the usage of classes, functions, and objects, to help you familiarize yourself with the basic concepts, before deep diving into the complex properties of metaprogramming. 

This part contains the following chapters:

Chapter 1, The Need for and Applications of Metaprogramming

Chapter 2, Refresher of OOP Concepts in Python

Chapter 1: The Need for and Applications of Metaprogramming

Metaprogramming with Python is a practical guide to learning metaprogramming in Python.

In today’s programming world, Python is considered one of the easiest languages to learn and use to develop useful applications. Understanding the programming concepts and applying them is easier in Python compared to any other programming language. A Python program can be written simply by adding existing libraries and making use of their inbuilt methods. At the same time, the language also has many powerful features that can help in developing robust libraries and applications.

This book covers the