04-Advanced Python Concepts - GenAI Engineering

This chapter introduces advanced Python concepts that make programs more modular, reusable, and memory efficient. These concepts are widely used in modern Python applications, web frameworks, data processing, and AI libraries. Since functions are objects, they can be assigned to variables, passed as arguments, returned from other functions, and stored in collections. These capabilities form the foundation for higher-order functions, decorators, closures, iterators and generators.

Topics Covered

In this module, you’ll learn:

Functions are Objects
Higher-Order Functions
Closures
Decorators
Iterators
Generators
Context Managers

Try Yourself: 💻 VS Code | 🚀 Colab | 📥 Download
Verify Solutions: 💻 VS Code | 🚀 Colab | 📥 Download

Functions are Objects

Like integers, strings, lists, and dictionaries, functions are also objects. Therefore, a function can:

Be assigned to a variable.
Be passed as an argument.
Be returned from another function.
Be stored in a collection.

Languages that support these capabilities are said to support first-class functions.

Assigning a Function

def greet():
    return "Hello"

message = greet

print(message())

Output ?

Show Output

Hello

Notice the difference:

message = greet      # Function object
message = greet()    # Function call

Storing Functions

def add(a, b):
    return a + b

def subtract(a, b):
    return a - b

operations = [add, subtract]

print(operations[0](10, 5))
print(operations[1](10, 5))

Output ?

Show Output

15
5

Exercise 2

What is the difference between the following statements?

f = greet

f = greet()

Solution

f = greet assigns the function object.
f = greet() calls the function and stores its return value.

Higher-Order Functions

A higher-order function is a function that:

Accepts one or more functions as arguments.
Returns a function.

Since functions are objects, they can be passed to and returned from other functions.

Passing Functions as Arguments

def greet():
    print("Hello")

def execute(func):
    func()

execute(greet)

Output ?

Show Output

Hello

Returning Functions

def outer():

    def inner():
        print("Hello")

    return inner

message = outer()

message()

Output ?

Show Output

Hello

Example

def add(a, b):
    return a + b

def multiply(a, b):
    return a * b

def calculate(operation, a, b):
    return operation(a, b)

print(calculate(add, 10, 5))
print(calculate(multiply, 10, 5))

Output ?

Show Output

15
50

Exercise 2

Predict the output.

def outer():

    def inner():
        return "Hello"

    return inner

func = outer()

print(func())

Output ?

Show Output

Hello

Solution

Hello

Exercise 3

When is a function called a higher-order function?

Solution

A function is called a higher-order function if it:

Accepts one or more functions as arguments.
Returns a function.

Higher-order functions form the foundation for closures and decorators.

Closures

Sometimes we want a function to remember information from previous function calls. A normal function cannot do this because its local variables are destroyed when the function finishes executing.

Example: Normal Function

def counter():

    count = 0

    count += 1

    return count

print(counter())
print(counter())
print(counter())

Output ?

Show Output

1
1
1

Each time counter() is called, the local variable count is created again and initialized to 0. Therefore, the function cannot remember its previous state. To preserve the state between function calls, we can use a closure.

What is a Closure?

A closure is an inner function that remembers the variables of its enclosing function even after the enclosing function has finished executing. A closure is created when:

A function is defined inside another function.
The inner function uses variables from the outer function.
The inner function is returned.

Example

def counter():

    count = 0

    def increment():
        nonlocal count

        count += 1
        return count

    return increment


c = counter()

print(c())
print(c())
print(c())

Output ?

Show Output

1
2
3

Here, the variable count is preserved even after counter() has finished executing. Each call to increment() updates the same variable instead of creating a new one. The nonlocal keyword allows the inner function to modify a variable defined in the enclosing function.

Exercise 2

Predict the output.

def counter():

    count = 10

    def increment():
        nonlocal count
        count += 5
        return count

    return increment


c = counter()

print(c())
print(c())

Output ?

Show Output

15
20

Solution

15
20

The variable count is preserved inside the closure and updated on each function call.

Exercise 3

Why do we use closures?

Solution

Closures allow a function to remember and preserve variables from its enclosing function even after the enclosing function has finished executing.

Closures are widely used for state preservation and form the foundation of decorators, where the wrapper function remembers the original function passed to the decorator.

Decorators

A decorator is a function that extends or modifies the behavior of another function without changing its original code. A decorator is a higher-order function because it:

Accepts a function as an argument.
Returns another function.

Creating a Decorator

def logger(func):

    def wrapper():
        print("Before function")

        func()

        print("After function")

    return wrapper

def greet():
    print("Hello")

greet = logger(greet)

greet()

Output ?

Show Output

Before function
Hello
After function

Instead of modifying greet(), the decorator returns a new function with additional behavior.

Using the `@` Syntax

Python provides the @ syntax as a convenient way to apply decorators.

def logger(func):

    def wrapper():
        print("Before function")

        func()

        print("After function")

    return wrapper

@logger
def greet():
    print("Hello")

greet()

Output ?

Show Output

Before function
Hello
After function

The above code is equivalent to:

def greet():
    print("Hello")


greet = logger(greet)

Exercise 2

Which statement is equivalent to the following code?

@logger
def greet():
    print("Hello")

Solution

def greet():
    print("Hello")


greet = logger(greet)

The @ syntax is a shorthand for applying a decorator.

Decorating Functions with Parameters

The previous decorator works only for functions that do not accept any arguments.

@logger
def greet():
    print("Hello")

Suppose we decorate a function that accepts parameters.

@logger
def add(a, b):
    return a + b

Our decorator is:

def logger(func):

    def wrapper():
        print("Function Started")

        result = func()

        print("Function Completed")

        return result

    return wrapper

When we call:

add(10, 20)

Output ?

Show Output

Error executing: logger.<locals>.wrapper() takes 0 positional arguments but 2 were given

Python actually executes:

wrapper(10, 20)

Output ?

Show Output

Error executing: name 'wrapper' is not defined

Since wrapper() does not accept any arguments, Python raises an error.

TypeError: wrapper() takes 0 positional arguments but 2 were given

One solution is to make the wrapper accept the same parameters.

def logger(func):

    def wrapper(a, b):
        print("Function Started")

        result = func(a, b)

        print("Function Completed")

        return result

    return wrapper

This works only for functions having exactly two parameters. To make the decorator work with any function, Python provides argument packing.

def logger(func):

    def wrapper(*args, **kwargs):
        print("Function Started")

        result = func(*args, **kwargs)

        print("Function Completed")

        return result

    return wrapper

Here,

*args collects all positional arguments.
**kwargs collects all keyword arguments.
func(*args, **kwargs) forwards all arguments to the original function.

Now the decorator can be applied to functions with any number of arguments.

@logger
def multiply(a, b):
    return a * b


@logger
def greet(name):
    print(f"Hello {name}")


print(multiply(10, 5))
greet("Alice")

Output ?

Show Output

Function Started
Function Completed
50
Function Started
Hello Alice
Function Completed

The wrapper() function is defined inside another function and remembers the original func even after the outer function has finished executing. This behavior was possible because of closure.

An iterator is an object that returns one value at a time from a collection, while a generator is a special type of iterator created using the yield keyword. They provide a memory-efficient way to process data without loading everything into memory at once.

Learning Objectives

After completing this lesson, you will be able to:

Understand iterables, iterators, and generators.
Create iterators using iter() and next().
Build custom iterators.
Create generators using yield.
Differentiate between yield and return.
Create generator expressions.
Compare iterators and generators.
Identify real-world use cases of generators.

What is an Iterator?

An iterator is an object that returns one element at a time from a collection. It remembers its current position and produces the next value only when requested. Python uses iterators internally whenever you iterate over a collection using a for loop.

Iterator Protocol

An iterator implements the following special methods:

__iter__() – Returns the iterator object.
__next__() – Returns the next element.

When no more elements are available, __next__() raises a StopIteration exception.

Creating an Iterator

Use the iter() function to create an iterator from an iterable.

numbers = [10, 20, 30]

iterator = iter(numbers)

print(iterator)

Output ?

Show Output

<list_iterator object at 0x...>

Retrieving Values

Use the next() function to retrieve values from an iterator.

numbers = [10, 20, 30]

iterator = iter(numbers)

print(next(iterator))
print(next(iterator))
print(next(iterator))

Output ?

Show Output

10
20
30

StopIteration

Once all elements are consumed, calling next() again raises a StopIteration exception.

numbers = [10]

iterator = iter(numbers)

print(next(iterator))
print(next(iterator))

Output ?

Show Output

10
Error executing: 

Exercise 2

Predict the output.

name = "Python"

it = iter(name)

print(next(it))
print(next(it))
print(next(it))

Output ?

Show Output

P
y
t

Solution

P
y
t

Strings are iterable objects, so they can be converted into iterators using iter().

Exercise 3

What exception will be raised by the following code?

numbers = [1]

it = iter(numbers)

print(next(it))
print(next(it))

Output ?

Show Output

1
Error executing: 

Solution

A StopIteration exception is raised because the iterator has no more elements to return.

Creating a Custom Iterator

You can create your own iterator by implementing the __iter__() and __next__() methods.

__iter__() returns the iterator object.
__next__() returns the next value.
When all values are consumed, __next__() raises a StopIteration exception.

Example

class Counter:

    def __init__(self, limit):
        self.current = 1
        self.limit = limit

    def __iter__(self):
        return self

    def __next__(self):
        if self.current > self.limit:
            raise StopIteration

        value = self.current
        self.current += 1
        return value


counter = Counter(5)

for number in counter:
    print(number)

Output ?

Show Output

How It Works

The Counter object is created.
The for loop calls __iter__() to obtain the iterator.
The loop repeatedly calls __next__().
Each call returns the next value.
When the limit is reached, StopIteration is raised, ending the loop.

Exercise 2

What happens if raise StopIteration is removed from the __next__() method?

Solution

The iterator will never indicate that it has finished, causing the loop to continue indefinitely or resulting in incorrect behavior.

Exercise 3

Which two special methods must every custom iterator implement?

Solution

Every custom iterator must implement:

__iter__()
__next__()

What is a Generator?

A generator is a special type of iterator created using a function that contains the yield keyword. Unlike a normal function that returns all values at once, a generator produces one value at a time and automatically remembers its execution state. Generators are easier to write than custom iterators because Python automatically implements the iterator protocol for you.

Creating a Generator

A function becomes a generator as soon as it contains a yield statement.

def numbers():
    yield 1
    yield 2
    yield 3


gen = numbers()

print(gen)

Output ?

Show Output

<generator object numbers at 0x...>

Notice that calling the function does not execute it immediately. Instead, it returns a generator object.

Using `next()` with a Generator

The next() function starts the generator and retrieves one value at a time.

def numbers():
    print("Starting")

    yield 1
    yield 2
    yield 3

    print("Ending")


gen = numbers()

print(next(gen))
print(next(gen))
print(next(gen))

Output ?

Show Output

Starting
1
2
3

The "Ending" message is not printed because the generator pauses after the third yield. It executes the remaining statements only when resumed again.

Using a Generator with a `for` Loop

Generators can be directly used in a for loop.

def numbers():
    yield 1
    yield 2
    yield 3


for number in numbers():
    print(number)

Output ?

Show Output

1
2
3

The for loop automatically calls next() until the generator raises StopIteration.

Exercise 2

Predict the output.

def greet():
    print("Hello")
    yield 1

g = greet()

print("Generator Created")

Output ?

Show Output

Generator Created

Solution

Generator Created

Calling a generator function does not execute its body immediately. It simply creates a generator object. The "Hello" message is printed only when the generator starts executing (for example, by calling next(g) or iterating over it).

Exercise 3

What is the output?

def values():
    yield "A"
    yield "B"

for value in values():
    print(value)

Output ?

Show Output

A
B

Solution

A
B

The for loop automatically retrieves values from the generator until it is exhausted.

Understanding `yield`

The yield keyword is used to produce a value from a generator. Unlike return, which terminates a function, yield pauses the function and preserves its current state. The next time the generator is resumed, execution continues from the statement immediately after the previous yield.

`yield` vs `return`

`return`	`yield`
Terminates the function	Pauses the function
Returns a single value	Produces one value at a time
Function cannot resume	Function resumes from the same point
Used in normal functions	Used in generator functions

Execution Flow

def greet():

    print("Step 1")
    yield "Hello"

    print("Step 2")
    yield "World"

    print("Step 3")


g = greet()

print(next(g))
print(next(g))
next(g)

Output ?

Show Output

Step 1
Hello
Step 2
World
Step 3
Traceback (most recent call last):
...
StopIteration

Notice that the function resumes exactly where it paused after each yield.

State Preservation

One of the biggest advantages of generators is that they automatically preserve the values of local variables.

def counter():

    count = 1

    while count <= 3:
        yield count
        count += 1


g = counter()

print(next(g))
print(next(g))
print(next(g))

Output ?

Show Output

1
2
3

The variable count is not reinitialized each time. Its value is preserved between successive calls to next().

Multiple `yield` Statements

A generator can contain multiple yield statements.

def colors():

    yield "Red"
    yield "Green"
    yield "Blue"


for color in colors():
    print(color)

Output ?

Show Output

Red
Green
Blue

Each yield produces one value before the generator pauses.

Exercise 2

Predict the output.

def test():

    x = 5
    yield x

    x += 5
    yield x


g = test()

print(next(g))
print(next(g))

Solution

Output ?

Show Output

5
10

The value of x is preserved between the two yield statements.

Exercise 3

What is the main difference between return and yield?

Solution

return terminates the function and returns a value.
yield pauses the function, returns a value, preserves its state, and resumes execution when requested again.

Generator Expressions

A generator expression provides a concise way to create generators. It is similar to a list comprehension but uses parentheses () instead of square brackets []. Generator expressions generate values only when required, making them memory efficient.

Syntax

(expression for item in iterable)

Output ?

Show Output

Error executing: name 'iterable' is not defined

Example

squares = (x * x for x in range(5))

for square in squares:
    print(square)

Output ?

Show Output

Generator Expression vs List Comprehension

# List Comprehension
numbers = [x * x for x in range(5)]

# Generator Expression
numbers = (x * x for x in range(5))

A list comprehension stores all values in memory.
A generator expression generates values one at a time.

Exercise 2

Which symbol is used to create a generator expression?

Solution

Generator expressions use parentheses (), whereas list comprehensions use square brackets [].

Infinite Generators

Generators can produce infinite sequences because values are generated only when requested.

Example

def even_numbers():

    number = 0

    while True:
        yield number
        number += 2


gen = even_numbers()

for _ in range(5):
    print(next(gen))

Output ?

Show Output

Fibonacci Generator

Generators are commonly used to generate mathematical sequences.

Example

def fibonacci(limit):

    a, b = 0, 1

    while a <= limit:
        yield a
        a, b = b, a + b


for number in fibonacci(20):
    print(number)

Output ?

Show Output

Memory Efficiency

One of the biggest advantages of generators is memory efficiency.

List Example

numbers = [x for x in range(1000000)]

The above statement creates one million values in memory.

Generator Example

numbers = (x for x in range(1000000))

The generator creates only one value at a time, significantly reducing memory usage.

When to Use Generators

Use generators when:

Working with large datasets.
Reading large files.
Processing streaming data.
Producing values on demand.
Creating infinite sequences.

Iterator vs Generator

Feature	Iterator	Generator
Created Using	Class	Function
Implements	`__iter__()` and `__next__()`	`yield`
State Management	Manual	Automatic
Code Size	More	Less
Lazy Evaluation	Yes	Yes
Memory Efficient	Yes	Yes
Best Use Case	Custom iteration logic	Sequential data generation

Remember: Every generator is an iterator, but not every iterator is a generator.

Iterable vs Iterator vs Generator

                Iterable
       (list, tuple, set, dict)
                   │
             iter(iterable)
                   │
                   ▼
               Iterator
       (__iter__ + __next__)
                   ▲
                   │
         Generator Object
        (Created using yield)

Iterable → An object that can produce an iterator.
Iterator → Produces one value at a time.
Generator → A special iterator created using the yield keyword.

Real-World Applications

Generators are commonly used for:

Reading large files line by line.
Processing large datasets.
Streaming data from APIs.
Log processing.
Data pipelines.
Machine learning workflows.
Infinite sequences.

Example: Reading a File

def read_file(filename):

    with open(filename, "r") as file:
        for line in file:
            yield line.strip()


for line in read_file("data.txt"):
    print(line)

Output ?

Show Output

Error executing: [Errno 2] No such file or directory: 'data.txt'

Instead of loading the entire file into memory, one line is processed at a time.

Key Takeaways

An iterable is an object that can produce an iterator.
An iterator returns one value at a time using next().
A generator is a simpler way to create an iterator using yield.
The yield keyword pauses execution and preserves the function’s state.
Generator expressions provide a concise syntax for creating generators.
Generators are ideal for processing large datasets because they use lazy evaluation.
Every generator is an iterator, but not every iterator is a generator.

Check Your Understanding

Question 1 What is the purpose of the iter() function?

Solution

The iter() function converts an iterable into an iterator.

Question 2 Which special methods make an object an iterator?

Solution

__iter__() and __next__()

Question 3 What is the purpose of the yield keyword?

Solution

The yield keyword pauses a generator, returns a value, preserves its state, and resumes execution from the same point when requested again.

Question 4 What is the difference between yield and return?

Solution

return terminates the function.
yield pauses the function and allows it to continue later.

Question 5 What is a generator expression?

Solution

A generator expression is a concise way to create a generator using parentheses ().

Question 6 Why are generators memory efficient?

Solution

Generators create values only when they are requested instead of storing all values in memory.

Question 7 Can generators be used in a for loop?

Solution

Yes. A generator is an iterator and can be directly used in a for loop.

Question 8 True or False: Every iterator is a generator.

Solution

False. Every generator is an iterator, but not every iterator is a generator.

Question 9 Name two real-world use cases of generators.

Solution

Examples include:

Reading large files
Processing large datasets
Streaming API data
Log processing
Infinite sequences

Context Managers

Whenever we open external resources such as files or database connections, they should be closed properly. Python provides the with statement to handle this automatically.

Without a Context Manager

file = open("sample.txt", "r")
content = file.read()
print(content)
file.close()

Output ?

Show Output

Error executing: [Errno 2] No such file or directory: 'sample.txt'

If an exception occurs before close(), the file may remain open.

Using a Context Manager

with open("sample.txt", "r") as file:
    content = file.read()
    print(content)

Output ?

Show Output

Error executing: [Errno 2] No such file or directory: 'sample.txt'

The file is automatically closed after leaving the with block.

Custom Context Manager

A context manager implements two special methods:

__enter__()
__exit__()

class Database:
    def __enter__(self):
        print("Connection Opened")
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        print("Connection Closed")

with Database():
    print("Executing Queries")

Show Output

Connection Opened
Executing Queries
Connection Closed

Exercise 1

Open a file using the with statement and display its contents. Sample Input

# Assume file sample.txt exists

Expected Output

Displays the file contents.

Solution

with open("sample.txt", "r") as file:
    print(file.read())

Exercise 2

Create a context manager that prints "Start" on entry and "End" on exit. Sample Input

with Demo():
    print("Inside")

Expected Output

Start
Inside
End

Solution

class Demo:
    def __enter__(self):
        print("Start")
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        print("End")

with Demo():
    print("Inside")

​Topics Covered

​Functions are Objects

​Assigning a Function

​Storing Functions

​Exercise 2

​Higher-Order Functions

​Passing Functions as Arguments

​Returning Functions

​Example

​Exercise 2

​Exercise 3

​Closures

​Example: Normal Function

​What is a Closure?

​Example

​Exercise 2

​Exercise 3

​Decorators

​Creating a Decorator

​Using the @ Syntax

​Exercise 2

​Decorating Functions with Parameters

​Learning Objectives

​What is an Iterator?

​Iterator Protocol

​Creating an Iterator

​Retrieving Values

​StopIteration

​Exercise 2

​Exercise 3

​Creating a Custom Iterator

​Example

​How It Works

​Exercise 2

​Exercise 3

​What is a Generator?

​Creating a Generator

​Using next() with a Generator

​Using a Generator with a for Loop

​Exercise 2

​Exercise 3

​Understanding yield

​yield vs return

​Execution Flow

​State Preservation

​Multiple yield Statements

​Exercise 2

​Exercise 3

​Generator Expressions

​Syntax

​Example

​Generator Expression vs List Comprehension

​Exercise 2

​Infinite Generators

​Example

​Fibonacci Generator

​Example

​Memory Efficiency

​List Example

​Generator Example

​When to Use Generators

​Iterator vs Generator

​Iterable vs Iterator vs Generator

​Real-World Applications

​Example: Reading a File

​Key Takeaways

​Check Your Understanding

​Context Managers

​Without a Context Manager

​Using a Context Manager

​Custom Context Manager

​Exercise 1

​Exercise 2

Topics Covered

Functions are Objects

Assigning a Function

Storing Functions

Exercise 2

Higher-Order Functions

Passing Functions as Arguments

Returning Functions

Example

Exercise 2

Exercise 3

Closures

Example: Normal Function

What is a Closure?

Example

Exercise 2

Exercise 3

Decorators

Creating a Decorator

Using the `@` Syntax

Exercise 2

Decorating Functions with Parameters

Learning Objectives

What is an Iterator?

Iterator Protocol

Creating an Iterator

Retrieving Values

StopIteration

Exercise 2

Exercise 3

Creating a Custom Iterator

Example

How It Works

Exercise 2

Exercise 3

What is a Generator?

Creating a Generator

Using `next()` with a Generator

Using a Generator with a `for` Loop

Exercise 2

Exercise 3

Understanding `yield`

`yield` vs `return`

Execution Flow

State Preservation

Multiple `yield` Statements

Exercise 2

Exercise 3

Generator Expressions

Syntax

Example

Generator Expression vs List Comprehension

Exercise 2

Infinite Generators

Example

Fibonacci Generator

Example

Memory Efficiency

List Example

Generator Example

When to Use Generators

Iterator vs Generator

Iterable vs Iterator vs Generator

Real-World Applications

Example: Reading a File

Key Takeaways

Check Your Understanding

Context Managers

Without a Context Manager

Using a Context Manager

Custom Context Manager

Exercise 1

Exercise 2