Generators in Python

Last Updated : 18 Dec, 2024

Python generator functions are a powerful tool for creating iterators. In this article, we will discuss how the generator function works in Python.

Generator Function in Python

A generator function is a special type of function that returns an iterator object. Instead of using return to send back a single value, generator functions use yield to produce a series of results over time. This allows the function to generate values and pause its execution after each yield, maintaining its state between iterations.

Basic Code Example:

def fun(max):
    cnt = 1
    while cnt <= max:
        yield cnt
        cnt += 1

ctr = fun(5)
for n in ctr:
    print(n)

Output

Explanation: This generator function fun yields numbers from 1 up to a specified max. Each call to next() on the generator object resumes execution right after the yield statement, where it last left off.

Let’s take a deep dive in python generators:

Table of Content

Generator Function in Python
Create a Generator in Python
Yield vs Return
Python Generator Expression

Create a Generator in Python

Creating a generator in Python is as simple as defining a function with at least one yield statement. When called, this function doesn’t return a single value; instead, it returns a generator object that supports the iterator protocol. The generator has the following syntax in Python:

def generator_function_name(parameters):
# Your code here
yield expression
# Additional code can follow

Example:

In this example, we will create a simple generator that will yield three integers. Then we will print these integers by using Python for loop.

# A generator function that yields 1 for first time,
# 2 second time and 3 third time
def fun():
    yield 1            
    yield 2            
    yield 3            
 
# Driver code to check above generator function
for val in fun(): 
    print(val)

Output

1
2
3

Yield vs Return

yield is used in generator functions to provide a sequence of values over time. When yield is executed, it pauses the function, returns the current value and retains the state of the function. This allows the function to continue from the same point when called again, making it ideal for generating large or complex sequences efficiently.

return, on the other hand, is used to exit a function and return a final value. Once return is executed, the function is terminated immediately, and no state is retained. This is suitable for cases where a single result is needed from a function.

Example with return:

def fun():
    return 1 + 2 + 3

res = fun()
print(res)

Output

Python Generator Expression

Generator expressions are a concise way to create generators. They are similar to list comprehensions but use parentheses instead of square brackets and are more memory efficient.

The generator expression in Python has the following Syntax:

(expression for item in iterable)

Example:

In this example, we will create a generator object that will print the multiples of 5 between the range of 0 to 5 which are also divisible by 2.

sq = (x*x for x in range(1, 6))
for i in sq:
    print(i)

Output

Applications of Generators in Python

Suppose we create a stream of Fibonacci numbers, adopting the generator approach makes it trivial; we just have to call next(x) to get the next Fibonacci number without bothering about where or when the stream of numbers ends. A more practical type of stream processing is handling large data files such as log files. Generators provide a space-efficient method for such data processing as only parts of the file are handled at one given point in time. We can also use Iterators for these purposes, but Generator provides a quick way (We don’t need to write __next__ and __iter__ methods here).

Generators in Python – FAQs

How do generators differ from standard functions?

Generators differ from standard functions in that they allow you to iterate through a sequence of values over time, instead of computing and returning a single result.

Standard Functions: Compute a value and return it once. They use the return statement.
def standard_function():
    return [1, 2, 3]
Generators: Use the yield statement to return values one at a time, allowing iteration over a sequence of values without storing the entire sequence in memory.
def generator_function():
    yield 1
    yield 2
    yield 3

How to use the `yield` statement in generators?

The yield statement is used in a generator function to return a value and pause the function’s execution, preserving its state. When next() is called on the generator, execution resumes right after the yield.
def countdown(n):
    while n > 0:
        yield n
        n -= 1

# Usage
gen = countdown(5)
for num in gen:
    print(num)

What are the benefits of using generators in Python?

Memory Efficiency: Generators generate values on the fly, reducing memory usage compared to storing the entire sequence in memory.

Lazy Evaluation: Values are computed only when needed, which can lead to performance improvements, especially with large datasets.

Maintain State: Generators automatically maintain their state between yield statements, making them suitable for iterative processes.

Convenience: Generators simplify code for iterating over large datasets or streams of data without needing to manage state explicitly.

How to handle state and exceptions in Python generators?

State Management: Generators maintain their state between yields automatically. You can also pass data into the generator using the send() method.

def stateful_generator():
    value = yield  # Receive a value from outside
    while True:
        value = yield value * 2

gen = stateful_generator()
next(gen)          # Start the generator
print(gen.send(10))  # Outputs: 20

Exception Handling: You can handle exceptions within a generator using standard try-except blocks.

def error_handling_generator():
    try:
        yield 1
        yield 2
        raise ValueError("An error occurred")
        yield 3
    except ValueError as e:
        print(e)

gen = error_handling_generator()
print(next(gen))
print(next(gen))
print(next(gen))  # This will trigger the exception

To handle exceptions outside the generator, use generator.throw().

def gen_with_error():
    yield 1
    raise ValueError("Error inside generator")

g = gen_with_error()
print(next(g))
try:
    g.throw(ValueError, "Error thrown")
except ValueError as e:
    print(e)