Advanced Functions

By the Syncfusion DevTutorials team · Last updated

Lambdas, closures and nonlocal, iterators and StopIteration, generators and yield, and decorators with functools.wraps.

A function you have written so far has a name, a body, and a def above it.

But a function in Python is also a value. You can put one in a variable, pass it to another function, build one inside another one, and return it. Once you see that, five things that look like separate topics turn out to be one idea wearing different hats.

Here is that idea in action — a function stored in a variable and handed to another function:

A function is a value like any other

def shout(text):
    return text.upper()

loud = shout
print(loud("asha"))

def apply_twice(func, text):
    return func(func(text))

print(apply_twice(shout, "asha"))
print(type(shout))

Functions are objects. They can be stored, passed, built at run time and returned — and that one fact is what lambdas, closures, generators and decorators are all built on.

What You Will Learn

In this lesson, you will learn:

  • How to write a function with no name, and when that is honestly better
  • What a closure is, and why a function can remember a value after the function that made it has finished
  • The difference between something you can loop over and the thing that does the looping
  • How a generator produces values one at a time instead of building a list
  • How a decorator wraps a function without changing a line inside it

By the end of this lesson, you will be able to write lambdas where they are the right tool, build functions that remember values through closures, explain iterables against iterators, write generators that produce values one at a time, and wrap functions with decorators.

Anonymous Functions

What Is a Lambda Function?

A lambda is a function written as a single expression, with no name and no def. It exists for the cases where naming a function would be more ceremony than the function is worth — a one-line rule handed to sorted() and never used again.

The same function, written twice

def square(n):
    return n * n

square_lambda = lambda n: n * n

print(square(9))
print(square_lambda(9))
print(type(square_lambda))

Both are functions. The only difference is how they were written.

Lambda Syntax

lambda parameters: a single expression

Three rules, and they are the whole language of lambdas:

  1. No def and no name.
  2. The body is one expression. No if statement, no loop, no return.
  3. The value of that expression is returned automatically.

Parameters work the same way

total = lambda a, b: a + b
greet = lambda name, greeting="Hello": f"{greeting}, {name}"
pass_fail = lambda mark: "pass" if mark >= 35 else "fail"

print(total(78, 85))
print(greet("Asha"))
print(greet("Ravi", "Welcome"))
print(pass_fail(78), pass_fail(20))

That last one uses a conditional expression, which is allowed, because it produces a value. An if statement is not.

Lambda with map()

Lambdas become most useful when they are passed directly into another function. map() is one of the common places where that happens.

map() applies a function to every item and hands back the results one at a time.

map applies a rule to every mark

marks = [78, 85, 60]

moderated = map(lambda mark: mark + 5, marks)
print(moderated)
print(list(moderated))

print(list(map(str, marks)))
print([mark + 5 for mark in marks])

map() gives back a lazy object, not a list — printing it shows that plainly. Wrap it in list() to see the values. The last line is the comprehension that does the same job, and for most readers it is easier to follow.

Lambda with filter()

Where map() changes items, filter() decides which items stay and which are removed.

filter keeps what passes

marks = [78, 20, 85, 30, 60]

print(list(filter(lambda mark: mark >= 35, marks)))
print(list(filter(lambda mark: mark < 35, marks)))
print([mark for mark in marks if mark >= 35])

Lambda with sorted()

This is where a lambda genuinely earns its place. sorted() takes a key — a function that says what to sort by.

Sorting records by a part of them

rows = [("Asha", 78), ("Ravi", 85), ("Meera", 60)]

print(sorted(rows))
print(sorted(rows, key=lambda row: row[1]))
print(sorted(rows, key=lambda row: row[1], reverse=True))

marks = {"Asha": 78, "Ravi": 85, "Meera": 60}
print(sorted(marks, key=marks.get, reverse=True))

Without key, sorting a list of tuples goes by the first item — here, the name. With key=lambda row: row[1] it goes by the mark. Note the last line: marks.get is already a function, so no lambda is needed at all.

Lambda against Normal Functions

Side by side, the differences are few:

lambdadef
Has a nameNo — __name__ is <lambda>Yes
BodyOne expressionAny number of statements
DocstringCannot have oneCan, and should
Shows in a traceback as<lambda>Its own name
Best forA rule passed straight to sorted, map or filterAnything you will call by name, test, or read twice

What a lambda is called

square = lambda n: n * n

def cube(n):
    return n ** 3

print(square.__name__)
print(cube.__name__)

Assigning a lambda to a name, as square = lambda n: n * n, gets you the worst of both: a function that needs a name anyway, but shows as <lambda> in every error message. If it deserves a name, give it a def.

Closures

The lambda captured nothing — it only used its own parameters. A function defined inside another one can also use the outer one's variables, and that is the next idea.

What Is a Closure?

A closure is a function that remembers a value from the function that made it, even after that outer function has finished.

A function that remembers

def make_adder(bonus):
    def add(mark):
        return mark + bonus
    return add

add_five = make_adder(5)
add_ten = make_adder(10)

print(add_five(78))
print(add_ten(78))

make_adder finished running before add_five was ever called. The 5 should be gone — and it is not. add kept it.

Nested Functions

A def inside a def is an ordinary function that happens to be built when the outer one runs. It is a new object every time.

A new function each time

def make_adder(bonus):
    def add(mark):
        return mark + bonus
    return add

a = make_adder(5)
b = make_adder(5)

print(a(78), b(78))
print(a is b)

Same behaviour, two different objects — because the inner def ran twice.

Capturing Variables from the Outer Scope

The captured values are not hidden. Python stores them on the function itself.

Where the remembered value lives

def make_adder(bonus):
    def add(mark):
        return mark + bonus
    return add

add_five = make_adder(5)

print(add_five.__code__.co_freevars)
print(add_five.__closure__[0].cell_contents)

co_freevars names what was captured; the closure cell holds the value.

The nonlocal Keyword

Reading an outer variable works by itself. Changing one does not — Python assumes any name you assign to is local, and then complains that you read it before it had a value.

The mistake, and what Python says

def counter():
    count = 0
    def tick():
        count += 1
        return count
    return tick

tick = counter()

try:
    print(tick())
except UnboundLocalError as error:
    print("UnboundLocalError:", error)

nonlocal is the fix. It says: this name belongs to the enclosing function, not to me.

nonlocal makes the change stick

def counter():
    count = 0
    def tick():
        nonlocal count
        count += 1
        return count
    return tick

attendance = counter()
print(attendance(), attendance(), attendance())

other = counter()
print(other())

Note the last two lines: a second counter starts again at 1, because it has its own count.

A Closure Example

Here is the pattern in use — one rule, two settings:

A grader built once, used many times

def grader(pass_mark):
    def grade(mark):
        return "pass" if mark >= pass_mark else "fail"
    return grade

school = grader(35)
scholarship = grader(80)

for mark in [78, 30, 85]:
    print(mark, school(mark), scholarship(mark))

One rule, two settings, no if repeated anywhere.

Closures against Global Variables

A global variable would also survive. The difference is who can reach it.

ClosureGlobal
Who can change itOnly the inner functionAny line in the file
How many copiesOne per call of the outer functionExactly one, ever
Changed withnonlocalglobal
Found by readingThe enclosing functionThe whole file

When two parts of a program each need their own running total, a closure gives each one its own. A global gives them one shared total and a bug that appears only when both are used.

Iterators

What Is an Iterator?

There are two different things here, and keeping them apart makes the rest easy.

  • An iterable is something you can loop over: a list, a string, a dictionary.
  • An iterator is the thing that does the walking. It remembers where it got to, and it gives one item at a time.

A for loop quietly makes an iterator from your iterable and calls it until it runs out.

iter()

Most of the time Python creates iterators for you, but iter() lets you create one yourself and see how it works.

Getting the walker

marks = [78, 85, 60]

walker = iter(marks)
print(type(marks))
print(type(walker))

next()

Once you have an iterator, next() moves it forward one step at a time. When there are none left it raises StopIteration — which is exactly how a for loop knows to stop.

Walking by hand

marks = [78, 85]
walker = iter(marks)

print(next(walker))
print(next(walker))

try:
    next(walker)
except StopIteration:
    print("StopIteration — nothing left")

print(next(iter([]), "nothing there"))

The second argument to next() is a default to return instead of raising.

Writing Your Own Iterator

An object is an iterator if it has __iter__ (which returns itself) and __next__ (which returns the next item or raises StopIteration).

A countdown you can loop over

class Countdown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        self.current -= 1
        return self.current + 1

print(list(Countdown(3)))

for number in Countdown(3):
    print(number)

An iterator is used up. Once it has raised StopIteration, looping over it again gives nothing at all — it does not start over. A list can be looped over again and again; the iterator made from it cannot.

Used up, not reset

marks = [78, 85]
walker = iter(marks)

print(list(walker))
print(list(walker))
print(list(marks))

Generators

What Is a Generator?

A generator is an iterator written as a function. You do not write __iter__ or __next__, and you do not raise StopIteration — Python does all of it for you. All you do is use yield instead of return.

yield

yield hands back one value and pauses the function, keeping every local variable exactly where it was. The next request carries on from that line.

Watch where it pauses

def names_of(students):
    for student in students:
        print("  (working on", student + ")")
        yield student.upper()

walker = names_of(["asha", "ravi"])
print("nothing has run yet")

print(next(walker))
print(next(walker))

try:
    next(walker)
except StopIteration:
    print("StopIteration")

Read the output in order. Calling names_of ran nothing. The work for each name happened only when it was asked for.

Generator Functions

Any function with a yield in it is a generator function. Calling it gives back a generator; it does not run the body.

A generator function

def countdown(start):
    while start > 0:
        yield start
        start -= 1

print(type(countdown(3)))
print(list(countdown(3)))

for number in countdown(3):
    print(number)

Generator Expressions

A generator expression looks like a list comprehension with round brackets instead of square ones. It produces values one at a time instead of building the whole list.

Brackets change everything

squares_list = [n * n for n in range(5)]
squares_gen = (n * n for n in range(5))

print(squares_list)
print(squares_gen)
print(list(squares_gen))

print(sum(n * n for n in range(5)))

When a generator expression is the only argument, the brackets can be dropped — that is the last line.

Why It Matters

Here is the memory difference between the two, side by side:

A list builds everything; a generator does not

import sys

as_list = [n * n for n in range(100000)]
as_generator = (n * n for n in range(100000))

print("list:", sys.getsizeof(as_list), "bytes")
print("generator:", sys.getsizeof(as_generator), "bytes")
print(sum(as_generator))

The list holds a hundred thousand numbers at once. The generator holds its place in the count and nothing else.

Use a list when you need the values more than once, or need their length, or need to reach item 500 directly. Use a generator when you walk through once — reading a large file line by line is the everyday case.

Decorators

What Are Decorators?

A decorator is a function that takes a function and gives back a new one that does something extra. It is how you add timing, logging or a permission check to a function without opening it.

The @ line is shorthand for one assignment, and seeing that written out is the fastest way to understand the whole topic.

The @ line, written out longhand

def announce(func):
    def wrapper(marks):
        print("  starting")
        result = func(marks)
        print("  finished")
        return result
    return wrapper

def average(marks):
    return sum(marks) / len(marks)

average = announce(average)

print(average([78, 85, 60]))

Creating Decorators

The @ does exactly what that assignment did.

The same thing with @

def announce(func):
    def wrapper(marks):
        print("  starting")
        result = func(marks)
        print("  finished")
        return result
    return wrapper

@announce
def average(marks):
    return sum(marks) / len(marks)

print(average([78, 85, 60]))

Wrapping Functions

A wrapper that only accepts one argument can only decorate functions that take one argument. *args and **kwargs make it work for any function at all.

A wrapper that fits any function

def announce(func):
    def wrapper(*args, **kwargs):
        print("  calling", func.__name__)
        return func(*args, **kwargs)
    return wrapper

@announce
def average(marks):
    return sum(marks) / len(marks)

@announce
def report(name, mark, subject="Maths"):
    return f"{name} scored {mark} in {subject}"

print(average([78, 85]))
print(report("Asha", 78))
print(report("Ravi", 85, subject="Science"))

There is one casualty, and it catches everybody.

The decorated function has lost its name

def announce(func):
    def wrapper(*args, **kwargs):
        return func(*args, **kwargs)
    return wrapper

@announce
def average(marks):
    """Average of the marks."""
    return sum(marks) / len(marks)

print(average.__name__)
print(average.__doc__)

The name is wrapper and the docstring is gone, because average now is the wrapper. functools.wraps copies them across.

functools.wraps keeps the identity

import functools

def announce(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        return func(*args, **kwargs)
    return wrapper

@announce
def average(marks):
    """Average of the marks."""
    return sum(marks) / len(marks)

print(average.__name__)
print(average.__doc__)
print(average([78, 85, 60]))

Always put @functools.wraps(func) on your wrapper. Without it, every decorated function in your program reports itself as wrapper in tracebacks, in help() and in logs — and finding the real one is then your problem.

Passing Arguments to a Decorator

A decorator that takes its own settings needs one more layer: a function that returns the decorator.

A decorator with a setting

import functools

def repeat(times):
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            for _ in range(times):
                func(*args, **kwargs)
        return wrapper
    return decorator

@repeat(3)
def roll_call(name):
    print("Present:", name)

roll_call("Asha")

Read @repeat(3) as: call repeat(3), which gives back a decorator, and apply that.

Multiple Decorators

Several @ lines stack. The one closest to the def is applied first, and the outermost one runs its own code first.

Order matters

def shout(func):
    def wrapper(*args):
        return func(*args).upper()
    return wrapper

def exclaim(func):
    def wrapper(*args):
        return func(*args) + "!"
    return wrapper

@shout
@exclaim
def label(name):
    return name

@exclaim
@shout
def other(name):
    return name

print(label("asha"))
print(other("asha"))

Both printed ASHA! here, because upper-casing and adding ! happen not to clash. With a decorator that trims text, the order would change the answer — so read the stack from the bottom up.

Common Uses

Here is the shape in real use — timing a function without touching a line inside it:

Timing, with nothing changed inside the function

import functools
import time

def timed(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        started = time.perf_counter()
        result = func(*args, **kwargs)
        print(f"  {func.__name__} took {(time.perf_counter() - started) * 1000:.2f} ms")
        return result
    return wrapper

@timed
def total_marks(count):
    return sum(range(count))

print(total_marks(50000))

The everyday jobs for a decorator are all of this shape: timing, logging a call, caching a result, checking a permission before the function runs, and retrying after a failure. In every case the function itself stays unchanged and unaware.

Common mistakes

Before moving on, here are some of the mistakes beginners make with these topics.

Each row is one mistake. The middle column is what you will actually see on screen.

MistakeWhat you seeWhy it happensFix
lambda m: return m * 2SyntaxErrorA lambda body is an expression, not a statementDrop return: lambda m: m * 2
Printing map(...)<map object at 0x...>map and filter are lazyWrap in list()
Reusing a map or filter result twiceThe second use is emptyIt is an iterator, and it is used upBuild a list once
Changing an outer variable without nonlocalUnboundLocalErrorAssigning makes the name localAdd nonlocal count
nonlocal used at the top levelSyntaxError: no binding for nonlocalThere is no enclosing functionUse global, or restructure
lambda: i inside a loopEvery one gives the last iThe lambda looks i up when calledlambda i=i: i
Looping over an iterator twiceNothing the second timeAn iterator does not resetLoop over the list, not the iterator
return where yield was meantA value, not a generatorOne yield is what makes it a generatorUse yield
A wrapper without *argsTypeError: wrapper() takes 1 positional argumentThe wrapper's shape must fit any functiondef wrapper(*args, **kwargs)
A decorator without functools.wraps__name__ is wrapperThe original's identity was not copied@functools.wraps(func)
Forgetting return wrapperThe decorated function becomes NoneThe decorator returned nothingReturn the wrapper

What to Remember

  • A function is a value: it can be stored, passed, built and returned.
  • A lambda is a one-expression function with no name. Use it where it is read, not as a named definition.
  • map() and filter() are lazy and used up once. A comprehension is usually clearer.
  • sorted(..., key=...) is where a lambda earns its place.
  • A closure is an inner function that remembers a value from the function that made it. Each call of the outer function makes a new one.
  • Reading an outer variable is free; changing one needs nonlocal.
  • An iterable can be looped over; an iterator does the walking and is used up once.
  • iter() makes one, next() advances it, StopIteration ends it.
  • A generator is an iterator written with yield. Calling the function runs nothing.
  • A generator expression is a comprehension in round brackets — one value at a time, not a whole list.
  • A decorator returns a replacement function. @thing is just name = thing(name).
  • Give every wrapper *args, **kwargs and @functools.wraps(func).

Next: Class Relationships