Modern applications often process millions of records, stream large files, handle API responses, or train Artificial Intelligence models. Loading all data into memory at once can slow down applications and consume excessive system resources.

Python solves this problem with Iterators and Generators. These powerful features allow data to be processed one item at a time instead of loading everything into memory simultaneously.

Iterators provide a standard way to traverse collections, while generators create values on demand using the yield keyword. These concepts are widely used in AI, Machine Learning, Data Analytics, Web Development, Automation, and Backend Engineering because they improve performance and reduce memory usage.

In this lesson, you’ll learn how iterators work, how to create custom iterators, build generators, use generator expressions, and optimize applications using lazy evaluation.

By the end of this lesson, you’ll be able to process large datasets efficiently while writing scalable, memory-optimized Python applications.

Learning Objectives

After completing this lesson, you will be able to:

• Understand iterators and generators.
• Use iter() and next().
• Create custom iterators.
• Build generators using yield.
• Work with generator expressions.
• Understand lazy evaluation.
• Optimize memory usage.
• Apply generators in real-world applications.

Topics Covered

• Introduction to Iterators
• Iterator Protocol
• iter() Function
• next() Function
• Creating Custom Iterators
• Generators
• yield Keyword
• Generator Expressions
• Lazy Evaluation
• Best Practices

Detailed Lesson Content

Introduction to Iterators

Imagine you’re developing an AI-powered Video Streaming Platform.

Your application contains 5 million videos.

Loading all videos into memory simultaneously would consume enormous system resources.

Instead, the platform loads only the required videos as the user scrolls.

This idea is similar to how Iterators work.

An iterator returns one value at a time rather than the complete collection.

What is an Iterator?

An iterator is an object that allows you to traverse a collection one element at a time.

Many Python collections are iterable:

• Lists
• Tuples
• Strings
• Dictionaries
• Sets

Python automatically uses iterators inside for loops.

The iter() Function

The iter() function converts an iterable into an iterator.

Example

numbers = [10,20,30]

iterator = iter(numbers)

Now iterator can produce one value at a time.

The next() Function

The next() function retrieves the next item from an iterator.

Example

numbers = [10,20,30]

iterator = iter(numbers)

print(next(iterator))

print(next(iterator))

print(next(iterator))

Output

10

20

30

If no values remain:

print(next(iterator))

Output

StopIteration

Python raises a StopIteration exception.

Iterator Protocol

Every iterator follows two methods:

• __iter__()
• __next__()

These methods allow Python to retrieve values one by one.

Creating a Custom Iterator

Example

class Numbers:

    def __iter__(self):

        self.number = 1

        return self

    def __next__(self):

        if self.number <= 5:

            value = self.number

            self.number += 1

            return value

        raise StopIteration

numbers = Numbers()

for value in numbers:

    print(value)

Output

1

2

3

4

5

What is a Generator?

A generator is a simpler way to create iterators.

Instead of using __iter__() and __next__(), generators use the yield keyword.

Generators automatically remember where execution stopped.

The yield Keyword

Unlike return, which ends a function, yield pauses execution and remembers its state.

Example

def numbers():

    yield 1

    yield 2

    yield 3

generator = numbers()

print(next(generator))

print(next(generator))

print(next(generator))

Output

1

2

3

yield vs return

Using return

def demo():

    return 10

    return 20

Only the first return executes.

Using yield

def demo():

    yield 10

    yield 20

Both values are produced one after another.

Generator Expressions

Generator expressions provide a shorter syntax.

List

numbers = [i*i for i in range(5)]

Generator

numbers = (i*i for i in range(5))

Generators consume much less memory.

Lazy Evaluation

Generators create values only when needed.

Instead of creating:

1

2

3

4

...

1000000

Python generates each value only when requested.

This technique is called Lazy Evaluation.

It improves:

• Performance
• Memory Usage
• Scalability

Real-World Example 1: Reading Large Files

def read_file(file):

    for line in file:

        yield line

Instead of loading the entire file, one line is processed at a time.

Real-World Example 2: AI Dataset Processing

def dataset():

    for image in range(1000000):

        yield image

Machine Learning models process one image at a time.

Real-World Example 3: OTP Generator

def otp():

    import random

    while True:

        yield random.randint(100000,999999)

generator = otp()

print(next(generator))

Each call generates a new OTP.

Real-World Example 4: Infinite Sequence

def counter():

    number = 1

    while True:

        yield number

        number += 1

generator = counter()

for i in range(5):

    print(next(generator))

Output

1

2

3

4

5

Iterator vs Generator

Common Mistakes

Using return Instead of yield

Incorrect

def numbers():

    return 1

    return 2

Correct

def numbers():

    yield 1

    yield 2

Calling next() Too Many Times

After all values are consumed:

next(generator)

Python raises:

StopIteration

Converting Large Generators into Lists

Avoid

list(generator)

Doing so defeats the memory-saving benefits of generators.