Unleashing the Power of Python Iterator & Generators

• In this blog post, we'll dive into the world of Python generators, exploring what they are, how they work, and practical use cases that demonstrate their utility.
• We will be learning the difference between iteration and generation in Python and how to construct our own Generators with the yield statement. Generators allow us to generate as we go along, instead of holding everything in memory.

What are Python Generators?

• Let's explore a little deeper. We've learned how to create functions with def and the return statement. Generator functions allow us to write a function that can send back a value and then later resume to pick up where it left off. This type of function is a generator.

• in Python, allowing us to generate a sequence of values over time. The main difference in syntax will be the use of a yield statement.

• In most aspects, a generator function will appear very similar to a normal function. The main difference is when a generator function is compiled they become an object that supports an iteration protocol. 
• That means when they are called in your code they don't actually return a value and then exit. Instead, generator functions will automatically suspend and resume their execution and state around the last point of value generation. 
• The main advantage here is that instead of having to compute an entire series of values up front, the generator computes one value and then suspends its activity awaiting the next instruction. This feature is known as state suspension.

To start getting a better understanding of generators, let's go ahead and see how we can create some.

# Generator function for the cube of numbers (power of 3)
def gencubes(n):
    for num in range(n):
        yield num**3

for x in gencubes(10):
    print(x)

Output: 0 1 8 27 64 125 216 343 512 729

• Great! Now since we have a generator function we don't have to keep track of every single cube we created.
• Generators are best for calculating large sets of results (particularly in calculations that involve loops themselves) in cases where we don’t want to allocate the memory for all of the results at the same time.

Let's create another example generator which calculates fibonacci numbers:

def genfibon(n):
    """
    Generate a fibonnaci sequence up to n
    """
    a = 1
    b = 1
    for i in range(n):
        yield a
        a,b = b,a+b

for num in genfibon(10):
    print(num)

Output : 1 1 2 3 5 8 13 21 34 55

• What if this was a normal function, what would it look like?

def fibon(n):
    a = 1
    b = 1
    output = []
    
    for i in range(n):
        output.append(a)
        a,b = b,a+b
        
    return output

fibon(10)

• Notice that if we call some huge value of n (like 100000) the second function will have to keep track of every single result, when in our case we actually only care about the previous result to generate the next one!

Iterating with Generators

• Iterating through a generator is straightforward. You can use a for loop or call the next() function to retrieve the next value.
• The next() function allows us to access the next element in a sequence. Lets check it out:?

def simple_gen():
    for x in range(3):
        yield x
    # Assign simple_gen 

g = simple_gen()
print(next(g))
print(next(g))
print(next(g))

Output : 0 1 2

print(next(g))

Output:
---------------------------------------------------------------------------
StopIteration                             

Traceback (most recent call last)


<ipython-input-12-1dfb29d6357e> in <module>()

----> 1 print(next(g))

StopIteration: 

• After yielding all the values next() caused a StopIteration error. What this error informs us of is that all the values have been yielded.
• You might be wondering that why don’t we get this error while using a for loop? A for loop automatically catches this error and stops calling next().
• Let's go ahead and check out how to use iter(). You remember that strings are iterables:

s = 'hello'
#Iterate over string
for let in s:
    print(let)

Output  
h 
e
l
l
o

• But that doesn't mean the string itself is an iterator! We can check this with the next() function:

next(s)

Output
---------------------------------------------------------------------------

TypeError                                 

Traceback (most recent call last)

<ipython-input-14-61c30b5fe1d5> in <module>()

----> 1 next(s)

TypeError: 'str' object is not an iterator

• Interesting, this means that a string object supports iteration, but we can not directly iterate over it as we could with a generator function. The iter() function allows us to do just that!

s_iter = iter(s)
next(s_iter)

Output:

'h'

next(s_iter)

Output 

'e'

Use Cases of  Generators

• Stream Processing: Generators are perfect for handling large streams of data, processing one piece at a time without loading the entire dataset into memory.
• Infinite Sequences: Generators can represent infinite sequences, like the Fibonacci sequence, without worrying about running out of memory.
• Memory-Efficient Data Processing: When working with large datasets, generators allow you to read data piece by piece, reducing memory consumption.

Advantage of Generators in Python

• Memory Efficiency: Generators produce values one at a time, conserving memory and allowing you to work with large or infinite datasets.
• Lazy Evaluation: Lazy evaluation means that you don't have to compute and store all values in memory at once, making generators ideal for streaming data.
• Readability: Generators can make your code more readable and elegant, especially when dealing with data processing tasks.

More Example of Python generators

Example  1

• Create a generator that generates the squares of numbers up to some number N.

s_iter = iter(s

Example  2

• Create a generator that yields "n" random numbers between a low and high number (that are inputs).
• Note: Use the random library. 
• For example:

import random
random.randint(1,10)

def rand_num(low,high,n):
    for i in range(n):
        yield random.randint(low, high)

for num in rand_num(1,10,12):
    print(num)

Example 3:

• Use the iter() function to convert the string below into an iterator:

s = 'hello'
s = iter(s)
print(next(s))

Example 4:

• Generator function that generates an infinite sequence of even numbers:

# Generator Function
def generate_even_numbers():
    num = 0
    while True:
        yield num
        num += 2

# Create a generator
even_number_generator = generate_even_numbers()

# Generate and print the first 5 even numbers
for _ in range(5):
    print(next(even_number_generator))

Exercise_1 - File Manager  Application

• File manager that allows users to list and filter files in a directory using generators and iterators.

Functional Requirements:

File Listing:

• Users can input a directory path.
• The system will list all files within the specified directory and its subdirectories using an iterator.

Filter Files by Extension:

• Users can choose to filter files based on a specific file extension (e.g., 'txt', 'jpg'). The system will generate a list of files matching the selected extension using a generator.

Filter Files by Size:

• Users can select a minimum file size (in bytes).
• The system will generate a list of files that meet or exceed the specified size using a generator.

Efficient Data Processing:

• Implement iterators and generators to efficiently process and filter files in the specified directory.
• Use generators for filtering to minimize memory consumption.

import os
# Function to generate a list of files in a directory
def list_files(directory):
    for root, _, files in os.walk(directory):
        for file in files:
            yield os.path.join(root, file)

# Function to filter files by extension using a generator
def filter_files_by_extension(directory, extension):
    for file in list_files(directory):
        if file.endswith(f".{extension}"):
            yield file

# Function to filter files by size using a generator
def filter_files_by_size(directory, size):
    for file in list_files(directory):
        if os.path.getsize(file) >= size:
            yield file

# Main program loop
def main():
    directory = input("Enter the directory path: ")

    print("1. List all files")
    print("2. List files by extension")
    print("3. List files by size")
    choice = input("Enter your choice: ")

    if choice == "1":
        for file in list_files(directory):
            print(file)
    elif choice == "2":
        extension = input("Enter the file extension (e.g., 'txt'): ")
        for file in filter_files_by_extension(directory, extension):
            print(file)
    elif choice == "3":
        size = int(input("Enter the minimum file size (in bytes): "))
        for file in filter_files_by_size(directory, size):
            print(file)
    else:
        print("Invalid choice. Please try again.")

if __name__ == "__main__":
    main()

• Check below link for complete code 
• File_manager_Application

Exercise_2 Log Analyzer

• Check below link for complete code 
• Log Analyzer Application

Conclusion

• Generators in Python are a powerful tool for working with data efficiently and elegantly. 
• Whether you need to process large datasets, create infinite sequences, or simply improve the memory efficiency of your code, understanding and using generators can be a game-changer in your Python programming journey.

More Refrence Link for Python Concepts 

• Python All Concepts

• Python Basic Concepts

• Python Intermediate Concepts

• 100-days-of-Code-Python