Iterators and Generators - GenAI Engineering

An iterator is an object that returns one value at a time from a collection, while a generator is a special type of iterator created using the yield keyword. They provide a memory-efficient way to process data without loading everything into memory at once.

Learning Objectives

After completing this lesson, you will be able to:

Understand iterables, iterators, and generators.
Create iterators using iter() and next().
Build custom iterators.
Create generators using yield.
Differentiate between yield and return.
Create generator expressions.
Compare iterators and generators.
Identify real-world use cases of generators.

What is an Iterator?

An iterator is an object that returns one element at a time from a collection. It remembers its current position and produces the next value only when requested. Python uses iterators internally whenever you iterate over a collection using a for loop.

Iterator Protocol

An iterator implements the following special methods:

__iter__() – Returns the iterator object.
__next__() – Returns the next element.

When no more elements are available, __next__() raises a StopIteration exception.

Creating an Iterator

Use the iter() function to create an iterator from an iterable.

numbers = [10, 20, 30]

iterator = iter(numbers)

print(iterator)

Output

<list_iterator object at 0x...>

Retrieving Values

Use the next() function to retrieve values from an iterator.

numbers = [10, 20, 30]

iterator = iter(numbers)

print(next(iterator))
print(next(iterator))
print(next(iterator))

Output

10
20
30

StopIteration

Once all elements are consumed, calling next() again raises a StopIteration exception.

numbers = [10]

iterator = iter(numbers)

print(next(iterator))
print(next(iterator))

Practice

Exercise 1

Predict the output.

numbers = [100, 200, 300]

it = iter(numbers)

print(next(it))
print(next(it))

Solution

100
200

The iterator returns one element at a time and remembers its current position.

Exercise 2

Predict the output.

name = "Python"

it = iter(name)

print(next(it))
print(next(it))
print(next(it))

Solution

P
y
t

Strings are iterable objects, so they can be converted into iterators using iter().

Exercise 3

What exception will be raised by the following code?

numbers = [1]

it = iter(numbers)

print(next(it))
print(next(it))

Solution

A StopIteration exception is raised because the iterator has no more elements to return.

Creating a Custom Iterator

You can create your own iterator by implementing the __iter__() and __next__() methods.

__iter__() returns the iterator object.
__next__() returns the next value.
When all values are consumed, __next__() raises a StopIteration exception.

Example

class Counter:

    def __init__(self, limit):
        self.current = 1
        self.limit = limit

    def __iter__(self):
        return self

    def __next__(self):
        if self.current > self.limit:
            raise StopIteration

        value = self.current
        self.current += 1
        return value


counter = Counter(5)

for number in counter:
    print(number)

Output

How It Works

The Counter object is created.
The for loop calls __iter__() to obtain the iterator.
The loop repeatedly calls __next__().
Each call returns the next value.
When the limit is reached, StopIteration is raised, ending the loop.

Practice

Exercise 1

Predict the output.

counter = Counter(3)

for value in counter:
    print(value)

Solution

1
2
3

The iterator returns values from 1 to 3 and then raises StopIteration.

Exercise 2

What happens if raise StopIteration is removed from the __next__() method?

Solution

The iterator will never indicate that it has finished, causing the loop to continue indefinitely or resulting in incorrect behavior.

Exercise 3

Which two special methods must every custom iterator implement?

Solution

Every custom iterator must implement:

__iter__()
__next__()

What is a Generator?

A generator is a special type of iterator created using a function that contains the yield keyword. Unlike a normal function that returns all values at once, a generator produces one value at a time and automatically remembers its execution state. Generators are easier to write than custom iterators because Python automatically implements the iterator protocol for you.

Creating a Generator

A function becomes a generator as soon as it contains a yield statement.

def numbers():
    yield 1
    yield 2
    yield 3


gen = numbers()

print(gen)

Output

<generator object numbers at 0x...>

Notice that calling the function does not execute it immediately. Instead, it returns a generator object.

Using `next()` with a Generator

The next() function starts the generator and retrieves one value at a time.

def numbers():
    print("Starting")

    yield 1
    yield 2
    yield 3

    print("Ending")


gen = numbers()

print(next(gen))
print(next(gen))
print(next(gen))

Output

Starting
1
2
3

The "Ending" message is not printed because the generator pauses after the third yield. It executes the remaining statements only when resumed again.

Using a Generator with a `for` Loop

Generators can be directly used in a for loop.

def numbers():
    yield 1
    yield 2
    yield 3


for number in numbers():
    print(number)

Output

1
2
3

The for loop automatically calls next() until the generator raises StopIteration.

Practice

Exercise 1

Predict the output.

def demo():
    yield 10
    yield 20

g = demo()

print(next(g))
print(next(g))

Solution

10
20

Each call to next() returns the next value produced by the generator.

Exercise 2

Predict the output.

def greet():
    print("Hello")
    yield 1

g = greet()

print("Generator Created")

Solution

Generator Created

Calling a generator function does not execute its body immediately. It simply creates a generator object. The "Hello" message is printed only when the generator starts executing (for example, by calling next(g) or iterating over it).

Exercise 3

What is the output?

def values():
    yield "A"
    yield "B"

for value in values():
    print(value)

Solution

A
B

The for loop automatically retrieves values from the generator until it is exhausted.

Understanding `yield`

The yield keyword is used to produce a value from a generator. Unlike return, which terminates a function, yield pauses the function and preserves its current state. The next time the generator is resumed, execution continues from the statement immediately after the previous yield.

`yield` vs `return`

`return`	`yield`
Terminates the function	Pauses the function
Returns a single value	Produces one value at a time
Function cannot resume	Function resumes from the same point
Used in normal functions	Used in generator functions

Execution Flow

def greet():

    print("Step 1")
    yield "Hello"

    print("Step 2")
    yield "World"

    print("Step 3")


g = greet()

print(next(g))
print(next(g))
next(g)

Output

Step 1
Hello
Step 2
World
Step 3
Traceback (most recent call last):
...
StopIteration

Notice that the function resumes exactly where it paused after each yield.

State Preservation

One of the biggest advantages of generators is that they automatically preserve the values of local variables.

def counter():

    count = 1

    while count <= 3:
        yield count
        count += 1


g = counter()

print(next(g))
print(next(g))
print(next(g))

Output

1
2
3

The variable count is not reinitialized each time. Its value is preserved between successive calls to next().

Multiple `yield` Statements

A generator can contain multiple yield statements.

def colors():

    yield "Red"
    yield "Green"
    yield "Blue"


for color in colors():
    print(color)

Output

Red
Green
Blue

Each yield produces one value before the generator pauses.

Practice

Exercise 1

Predict the output.

def demo():

    print("A")
    yield 10

    print("B")
    yield 20

    print("C")


g = demo()

print(next(g))
print(next(g))

Solution

Output

A
10
B
20

The generator pauses after each yield. Since the generator is not resumed again, "C" is not printed.

Exercise 2

Predict the output.

def test():

    x = 5
    yield x

    x += 5
    yield x


g = test()

print(next(g))
print(next(g))

Solution

Output

5
10

The value of x is preserved between the two yield statements.

Exercise 3

What is the main difference between return and yield?

Solution

return terminates the function and returns a value.
yield pauses the function, returns a value, preserves its state, and resumes execution when requested again.

Generator Expressions

A generator expression provides a concise way to create generators. It is similar to a list comprehension but uses parentheses () instead of square brackets []. Generator expressions generate values only when required, making them memory efficient.

Syntax

(expression for item in iterable)

Example

squares = (x * x for x in range(5))

for square in squares:
    print(square)

Output

Generator Expression vs List Comprehension

# List Comprehension
numbers = [x * x for x in range(5)]

# Generator Expression
numbers = (x * x for x in range(5))

A list comprehension stores all values in memory.
A generator expression generates values one at a time.

Practice

Exercise 1

Predict the output.

gen = (x + 1 for x in range(3))

print(next(gen))
print(next(gen))

Solution

1
2

Each call to next() computes the next value in the sequence.

Exercise 2

Which symbol is used to create a generator expression?

Solution

Generator expressions use parentheses (), whereas list comprehensions use square brackets [].

Infinite Generators

Generators can produce infinite sequences because values are generated only when requested.

Example

def even_numbers():

    number = 0

    while True:
        yield number
        number += 2


gen = even_numbers()

for _ in range(5):
    print(next(gen))

Output

Practice

Exercise 1

Predict the output.

def numbers():

    n = 1

    while True:
        yield n
        n += 1


gen = numbers()

print(next(gen))
print(next(gen))
print(next(gen))

Solution

1
2
3

The generator can continue producing values indefinitely.

Fibonacci Generator

Generators are commonly used to generate mathematical sequences.

Example

def fibonacci(limit):

    a, b = 0, 1

    while a <= limit:
        yield a
        a, b = b, a + b


for number in fibonacci(20):
    print(number)

Output

Practice

Exercise 1

What is the first value produced by the generator?

Solution

The first value is 0, because the generator yields a before updating its value.

Memory Efficiency

One of the biggest advantages of generators is memory efficiency.

List Example

numbers = [x for x in range(1000000)]

The above statement creates one million values in memory.

Generator Example

numbers = (x for x in range(1000000))

The generator creates only one value at a time, significantly reducing memory usage.

When to Use Generators

Use generators when:

Working with large datasets.
Reading large files.
Processing streaming data.
Producing values on demand.
Creating infinite sequences.

Practice

Exercise 1

Which consumes less memory?

[x for x in range(1000000)]

(x for x in range(1000000))

Solution

The generator expression consumes significantly less memory because values are generated only when needed.

Iterator vs Generator

Feature	Iterator	Generator
Created Using	Class	Function
Implements	`__iter__()` and `__next__()`	`yield`
State Management	Manual	Automatic
Code Size	More	Less
Lazy Evaluation	Yes	Yes
Memory Efficient	Yes	Yes
Best Use Case	Custom iteration logic	Sequential data generation

Remember: Every generator is an iterator, but not every iterator is a generator.

Iterable vs Iterator vs Generator

                Iterable
       (list, tuple, set, dict)
                   │
             iter(iterable)
                   │
                   ▼
               Iterator
       (__iter__ + __next__)
                   ▲
                   │
         Generator Object
        (Created using yield)

Iterable → An object that can produce an iterator.
Iterator → Produces one value at a time.
Generator → A special iterator created using the yield keyword.

Real-World Applications

Generators are commonly used for:

Reading large files line by line.
Processing large datasets.
Streaming data from APIs.
Log processing.
Data pipelines.
Machine learning workflows.
Infinite sequences.

Example: Reading a File

def read_file(filename):

    with open(filename, "r") as file:
        for line in file:
            yield line.strip()


for line in read_file("data.txt"):
    print(line)

Instead of loading the entire file into memory, one line is processed at a time.

Key Takeaways

An iterable is an object that can produce an iterator.
An iterator returns one value at a time using next().
A generator is a simpler way to create an iterator using yield.
The yield keyword pauses execution and preserves the function’s state.
Generator expressions provide a concise syntax for creating generators.
Generators are ideal for processing large datasets because they use lazy evaluation.
Every generator is an iterator, but not every iterator is a generator.

Check Your Understanding

Question 1 What is the purpose of the iter() function?

Solution

The iter() function converts an iterable into an iterator.

Question 2 Which special methods make an object an iterator?

Solution

__iter__() and __next__()

Question 3 What is the purpose of the yield keyword?

Solution

The yield keyword pauses a generator, returns a value, preserves its state, and resumes execution from the same point when requested again.

Question 4 What is the difference between yield and return?

Solution

return terminates the function.
yield pauses the function and allows it to continue later.

Question 5 What is a generator expression?

Solution

A generator expression is a concise way to create a generator using parentheses ().

Question 6 Why are generators memory efficient?

Solution

Generators create values only when they are requested instead of storing all values in memory.

Question 7 Can generators be used in a for loop?

Solution

Yes. A generator is an iterator and can be directly used in a for loop.

Question 8 True or False: Every iterator is a generator.

Solution

False. Every generator is an iterator, but not every iterator is a generator.

Question 9 Name two real-world use cases of generators.

Solution

Examples include:

Reading large files
Processing large datasets
Streaming API data
Log processing
Infinite sequences

Practice & Exercises

To reinforce what you’ve learned in this section (Iterators, Custom Iterators, Generators, and Generator Expressions), practice with these interactive notebooks:

Follow-Along Practice

Practice creating iterators, implementing custom iterator classes, writing generators with yield, and creating generator expressions.💻 VS Code | 🚀 Colab | 📥 Download

Practice Exercises

Test your skills with exercises on custom range iterators, cubes generators, odd numbers generator expressions, and infinite powers of three generators.💻 VS Code | 🚀 Colab | 📥 Download

​Learning Objectives

​What is an Iterator?

​Iterator Protocol

​Creating an Iterator

​Retrieving Values

​StopIteration

​Practice

​Exercise 1

​Exercise 2

​Exercise 3

​Creating a Custom Iterator

​Example

​How It Works

​Practice

​Exercise 1

​Exercise 2

​Exercise 3

​What is a Generator?

​Creating a Generator

​Using next() with a Generator

​Using a Generator with a for Loop

​Practice

​Exercise 1

​Exercise 2

​Exercise 3

​Understanding yield

​yield vs return

​Execution Flow

​State Preservation

​Multiple yield Statements

​Practice

​Exercise 1

​Exercise 2

​Exercise 3

​Generator Expressions

​Syntax

​Example

​Generator Expression vs List Comprehension

​Practice

​Exercise 1

​Exercise 2

​Infinite Generators

​Example

​Practice

​Exercise 1

​Fibonacci Generator

​Example

​Practice

​Exercise 1

​Memory Efficiency

​List Example

​Generator Example

​When to Use Generators

​Practice

​Exercise 1

​Iterator vs Generator

​Iterable vs Iterator vs Generator

​Real-World Applications

​Example: Reading a File

​Key Takeaways

​Check Your Understanding

​Practice & Exercises

Follow-Along Practice

Practice Exercises

Learning Objectives

What is an Iterator?

Iterator Protocol

Creating an Iterator

Retrieving Values

StopIteration

Practice

Exercise 1

Exercise 2

Exercise 3

Creating a Custom Iterator

Example

How It Works

Practice

Exercise 1

Exercise 2

Exercise 3

What is a Generator?

Creating a Generator

Using `next()` with a Generator

Using a Generator with a `for` Loop

Practice

Exercise 1

Exercise 2

Exercise 3

Understanding `yield`

`yield` vs `return`

Execution Flow

State Preservation

Multiple `yield` Statements

Practice

Exercise 1

Exercise 2

Exercise 3

Generator Expressions

Syntax

Example

Generator Expression vs List Comprehension

Practice

Exercise 1

Exercise 2

Infinite Generators

Example

Practice

Exercise 1

Fibonacci Generator

Example

Practice

Exercise 1

Memory Efficiency

List Example

Generator Example

When to Use Generators

Practice

Exercise 1

Iterator vs Generator

Iterable vs Iterator vs Generator

Real-World Applications

Example: Reading a File

Key Takeaways

Check Your Understanding

Practice & Exercises