Advanced Functions
By the Syncfusion DevTutorials team · Last updated
Lambdas, closures and nonlocal, iterators and StopIteration, generators and yield, and decorators with functools.wraps.
A function you have written so far has a name, a body, and a def above it.
But a function in Python is also a value. You can put one in a variable, pass it to another function, build one inside another one, and return it. Once you see that, five things that look like separate topics turn out to be one idea wearing different hats.
Here is that idea in action — a function stored in a variable and handed to another function:
A function is a value like any other
def shout(text):
return text.upper()
loud = shout
print(loud("asha"))
def apply_twice(func, text):
return func(func(text))
print(apply_twice(shout, "asha"))
print(type(shout))
Functions are objects. They can be stored, passed, built at run time and returned — and that one fact is what lambdas, closures, generators and decorators are all built on.
What You Will Learn
In this lesson, you will learn:
- How to write a function with no name, and when that is honestly better
- What a closure is, and why a function can remember a value after the function that made it has finished
- The difference between something you can loop over and the thing that does the looping
- How a generator produces values one at a time instead of building a list
- How a decorator wraps a function without changing a line inside it
By the end of this lesson, you will be able to write lambdas where they are the right tool, build functions that remember values through closures, explain iterables against iterators, write generators that produce values one at a time, and wrap functions with decorators.
Anonymous Functions
What Is a Lambda Function?
A lambda is a function written as a single expression, with no name and no def. It exists for the cases where naming a function would be more ceremony than the function is worth — a one-line rule handed to sorted() and never used again.
The same function, written twice
def square(n):
return n * n
square_lambda = lambda n: n * n
print(square(9))
print(square_lambda(9))
print(type(square_lambda))
Both are functions. The only difference is how they were written.
Lambda Syntax
lambda parameters: a single expression
Three rules, and they are the whole language of lambdas:
- No
defand no name. - The body is one expression. No
ifstatement, no loop, noreturn. - The value of that expression is returned automatically.
Parameters work the same way
total = lambda a, b: a + b
greet = lambda name, greeting="Hello": f"{greeting}, {name}"
pass_fail = lambda mark: "pass" if mark >= 35 else "fail"
print(total(78, 85))
print(greet("Asha"))
print(greet("Ravi", "Welcome"))
print(pass_fail(78), pass_fail(20))
That last one uses a conditional expression, which is allowed, because it produces a value. An if statement is not.
Lambda with map()
Lambdas become most useful when they are passed directly into another function. map() is one of the common places where that happens.
map() applies a function to every item and hands back the results one at a time.
map applies a rule to every mark
marks = [78, 85, 60]
moderated = map(lambda mark: mark + 5, marks)
print(moderated)
print(list(moderated))
print(list(map(str, marks)))
print([mark + 5 for mark in marks])
map() gives back a lazy object, not a list — printing it shows that plainly. Wrap it in list() to see the values. The last line is the comprehension that does the same job, and for most readers it is easier to follow.
Lambda with filter()
Where map() changes items, filter() decides which items stay and which are removed.
filter keeps what passes
marks = [78, 20, 85, 30, 60]
print(list(filter(lambda mark: mark >= 35, marks)))
print(list(filter(lambda mark: mark < 35, marks)))
print([mark for mark in marks if mark >= 35])
Lambda with sorted()
This is where a lambda genuinely earns its place. sorted() takes a key — a function that says what to sort by.
Sorting records by a part of them
rows = [("Asha", 78), ("Ravi", 85), ("Meera", 60)]
print(sorted(rows))
print(sorted(rows, key=lambda row: row[1]))
print(sorted(rows, key=lambda row: row[1], reverse=True))
marks = {"Asha": 78, "Ravi": 85, "Meera": 60}
print(sorted(marks, key=marks.get, reverse=True))
Without key, sorting a list of tuples goes by the first item — here, the name. With key=lambda row: row[1] it goes by the mark. Note the last line: marks.get is already a function, so no lambda is needed at all.
Lambda against Normal Functions
Side by side, the differences are few:
lambda | def | |
|---|---|---|
| Has a name | No — __name__ is <lambda> | Yes |
| Body | One expression | Any number of statements |
| Docstring | Cannot have one | Can, and should |
| Shows in a traceback as | <lambda> | Its own name |
| Best for | A rule passed straight to sorted, map or filter | Anything you will call by name, test, or read twice |
What a lambda is called
square = lambda n: n * n
def cube(n):
return n ** 3
print(square.__name__)
print(cube.__name__)
Assigning a lambda to a name, as square = lambda n: n * n, gets you the worst of both: a function that needs a name anyway, but shows as <lambda> in every error message. If it deserves a name, give it a def.
Closures
The lambda captured nothing — it only used its own parameters. A function defined inside another one can also use the outer one's variables, and that is the next idea.
What Is a Closure?
A closure is a function that remembers a value from the function that made it, even after that outer function has finished.
A function that remembers
def make_adder(bonus):
def add(mark):
return mark + bonus
return add
add_five = make_adder(5)
add_ten = make_adder(10)
print(add_five(78))
print(add_ten(78))
make_adder finished running before add_five was ever called. The 5 should be gone — and it is not. add kept it.
Nested Functions
A def inside a def is an ordinary function that happens to be built when the outer one runs. It is a new object every time.
A new function each time
def make_adder(bonus):
def add(mark):
return mark + bonus
return add
a = make_adder(5)
b = make_adder(5)
print(a(78), b(78))
print(a is b)
Same behaviour, two different objects — because the inner def ran twice.
Capturing Variables from the Outer Scope
The captured values are not hidden. Python stores them on the function itself.
Where the remembered value lives
def make_adder(bonus):
def add(mark):
return mark + bonus
return add
add_five = make_adder(5)
print(add_five.__code__.co_freevars)
print(add_five.__closure__[0].cell_contents)
co_freevars names what was captured; the closure cell holds the value.
The nonlocal Keyword
Reading an outer variable works by itself. Changing one does not — Python assumes any name you assign to is local, and then complains that you read it before it had a value.
The mistake, and what Python says
def counter():
count = 0
def tick():
count += 1
return count
return tick
tick = counter()
try:
print(tick())
except UnboundLocalError as error:
print("UnboundLocalError:", error)
nonlocal is the fix. It says: this name belongs to the enclosing function, not to me.
nonlocal makes the change stick
def counter():
count = 0
def tick():
nonlocal count
count += 1
return count
return tick
attendance = counter()
print(attendance(), attendance(), attendance())
other = counter()
print(other())
Note the last two lines: a second counter starts again at 1, because it has its own count.
A Closure Example
Here is the pattern in use — one rule, two settings:
A grader built once, used many times
def grader(pass_mark):
def grade(mark):
return "pass" if mark >= pass_mark else "fail"
return grade
school = grader(35)
scholarship = grader(80)
for mark in [78, 30, 85]:
print(mark, school(mark), scholarship(mark))
One rule, two settings, no if repeated anywhere.
Closures against Global Variables
A global variable would also survive. The difference is who can reach it.
| Closure | Global | |
|---|---|---|
| Who can change it | Only the inner function | Any line in the file |
| How many copies | One per call of the outer function | Exactly one, ever |
| Changed with | nonlocal | global |
| Found by reading | The enclosing function | The whole file |
When two parts of a program each need their own running total, a closure gives each one its own. A global gives them one shared total and a bug that appears only when both are used.
Iterators
What Is an Iterator?
There are two different things here, and keeping them apart makes the rest easy.
- An iterable is something you can loop over: a list, a string, a dictionary.
- An iterator is the thing that does the walking. It remembers where it got to, and it gives one item at a time.
A for loop quietly makes an iterator from your iterable and calls it until it runs out.
iter()
Most of the time Python creates iterators for you, but iter() lets you create one yourself and see how it works.
Getting the walker
marks = [78, 85, 60]
walker = iter(marks)
print(type(marks))
print(type(walker))
next()
Once you have an iterator, next() moves it forward one step at a time. When there are none left it raises StopIteration — which is exactly how a for loop knows to stop.
Walking by hand
marks = [78, 85]
walker = iter(marks)
print(next(walker))
print(next(walker))
try:
next(walker)
except StopIteration:
print("StopIteration — nothing left")
print(next(iter([]), "nothing there"))
The second argument to next() is a default to return instead of raising.
Writing Your Own Iterator
An object is an iterator if it has __iter__ (which returns itself) and __next__ (which returns the next item or raises StopIteration).
A countdown you can loop over
class Countdown:
def __init__(self, start):
self.current = start
def __iter__(self):
return self
def __next__(self):
if self.current <= 0:
raise StopIteration
self.current -= 1
return self.current + 1
print(list(Countdown(3)))
for number in Countdown(3):
print(number)
An iterator is used up. Once it has raised StopIteration, looping over it again gives nothing at all — it does not start over. A list can be looped over again and again; the iterator made from it cannot.
Used up, not reset
marks = [78, 85]
walker = iter(marks)
print(list(walker))
print(list(walker))
print(list(marks))
Generators
What Is a Generator?
A generator is an iterator written as a function. You do not write __iter__ or __next__, and you do not raise StopIteration — Python does all of it for you. All you do is use yield instead of return.
yield
yield hands back one value and pauses the function, keeping every local variable exactly where it was. The next request carries on from that line.
Watch where it pauses
def names_of(students):
for student in students:
print(" (working on", student + ")")
yield student.upper()
walker = names_of(["asha", "ravi"])
print("nothing has run yet")
print(next(walker))
print(next(walker))
try:
next(walker)
except StopIteration:
print("StopIteration")
Read the output in order. Calling names_of ran nothing. The work for each name happened only when it was asked for.
Generator Functions
Any function with a yield in it is a generator function. Calling it gives back a generator; it does not run the body.
A generator function
def countdown(start):
while start > 0:
yield start
start -= 1
print(type(countdown(3)))
print(list(countdown(3)))
for number in countdown(3):
print(number)
Generator Expressions
A generator expression looks like a list comprehension with round brackets instead of square ones. It produces values one at a time instead of building the whole list.
Brackets change everything
squares_list = [n * n for n in range(5)]
squares_gen = (n * n for n in range(5))
print(squares_list)
print(squares_gen)
print(list(squares_gen))
print(sum(n * n for n in range(5)))
When a generator expression is the only argument, the brackets can be dropped — that is the last line.
Why It Matters
Here is the memory difference between the two, side by side:
A list builds everything; a generator does not
import sys
as_list = [n * n for n in range(100000)]
as_generator = (n * n for n in range(100000))
print("list:", sys.getsizeof(as_list), "bytes")
print("generator:", sys.getsizeof(as_generator), "bytes")
print(sum(as_generator))
The list holds a hundred thousand numbers at once. The generator holds its place in the count and nothing else.
Use a list when you need the values more than once, or need their length, or need to reach item 500 directly. Use a generator when you walk through once — reading a large file line by line is the everyday case.
Decorators
What Are Decorators?
A decorator is a function that takes a function and gives back a new one that does something extra. It is how you add timing, logging or a permission check to a function without opening it.
The @ line is shorthand for one assignment, and seeing that written out is the fastest way to understand the whole topic.
The @ line, written out longhand
def announce(func):
def wrapper(marks):
print(" starting")
result = func(marks)
print(" finished")
return result
return wrapper
def average(marks):
return sum(marks) / len(marks)
average = announce(average)
print(average([78, 85, 60]))
Creating Decorators
The @ does exactly what that assignment did.
The same thing with @
def announce(func):
def wrapper(marks):
print(" starting")
result = func(marks)
print(" finished")
return result
return wrapper
@announce
def average(marks):
return sum(marks) / len(marks)
print(average([78, 85, 60]))
Wrapping Functions
A wrapper that only accepts one argument can only decorate functions that take one argument. *args and **kwargs make it work for any function at all.
A wrapper that fits any function
def announce(func):
def wrapper(*args, **kwargs):
print(" calling", func.__name__)
return func(*args, **kwargs)
return wrapper
@announce
def average(marks):
return sum(marks) / len(marks)
@announce
def report(name, mark, subject="Maths"):
return f"{name} scored {mark} in {subject}"
print(average([78, 85]))
print(report("Asha", 78))
print(report("Ravi", 85, subject="Science"))
There is one casualty, and it catches everybody.
The decorated function has lost its name
def announce(func):
def wrapper(*args, **kwargs):
return func(*args, **kwargs)
return wrapper
@announce
def average(marks):
"""Average of the marks."""
return sum(marks) / len(marks)
print(average.__name__)
print(average.__doc__)
The name is wrapper and the docstring is gone, because average now is the wrapper. functools.wraps copies them across.
functools.wraps keeps the identity
import functools
def announce(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
return func(*args, **kwargs)
return wrapper
@announce
def average(marks):
"""Average of the marks."""
return sum(marks) / len(marks)
print(average.__name__)
print(average.__doc__)
print(average([78, 85, 60]))
Always put @functools.wraps(func) on your wrapper. Without it, every decorated function in your program reports itself as wrapper in tracebacks, in help() and in logs — and finding the real one is then your problem.
Passing Arguments to a Decorator
A decorator that takes its own settings needs one more layer: a function that returns the decorator.
A decorator with a setting
import functools
def repeat(times):
def decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
for _ in range(times):
func(*args, **kwargs)
return wrapper
return decorator
@repeat(3)
def roll_call(name):
print("Present:", name)
roll_call("Asha")
Read @repeat(3) as: call repeat(3), which gives back a decorator, and apply that.
Multiple Decorators
Several @ lines stack. The one closest to the def is applied first, and the outermost one runs its own code first.
Order matters
def shout(func):
def wrapper(*args):
return func(*args).upper()
return wrapper
def exclaim(func):
def wrapper(*args):
return func(*args) + "!"
return wrapper
@shout
@exclaim
def label(name):
return name
@exclaim
@shout
def other(name):
return name
print(label("asha"))
print(other("asha"))
Both printed ASHA! here, because upper-casing and adding ! happen not to clash. With a decorator that trims text, the order would change the answer — so read the stack from the bottom up.
Common Uses
Here is the shape in real use — timing a function without touching a line inside it:
Timing, with nothing changed inside the function
import functools
import time
def timed(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
started = time.perf_counter()
result = func(*args, **kwargs)
print(f" {func.__name__} took {(time.perf_counter() - started) * 1000:.2f} ms")
return result
return wrapper
@timed
def total_marks(count):
return sum(range(count))
print(total_marks(50000))
The everyday jobs for a decorator are all of this shape: timing, logging a call, caching a result, checking a permission before the function runs, and retrying after a failure. In every case the function itself stays unchanged and unaware.
Common mistakes
Before moving on, here are some of the mistakes beginners make with these topics.
Each row is one mistake. The middle column is what you will actually see on screen.
| Mistake | What you see | Why it happens | Fix |
|---|---|---|---|
lambda m: return m * 2 | SyntaxError | A lambda body is an expression, not a statement | Drop return: lambda m: m * 2 |
Printing map(...) | <map object at 0x...> | map and filter are lazy | Wrap in list() |
Reusing a map or filter result twice | The second use is empty | It is an iterator, and it is used up | Build a list once |
Changing an outer variable without nonlocal | UnboundLocalError | Assigning makes the name local | Add nonlocal count |
nonlocal used at the top level | SyntaxError: no binding for nonlocal | There is no enclosing function | Use global, or restructure |
lambda: i inside a loop | Every one gives the last i | The lambda looks i up when called | lambda i=i: i |
| Looping over an iterator twice | Nothing the second time | An iterator does not reset | Loop over the list, not the iterator |
return where yield was meant | A value, not a generator | One yield is what makes it a generator | Use yield |
A wrapper without *args | TypeError: wrapper() takes 1 positional argument | The wrapper's shape must fit any function | def wrapper(*args, **kwargs) |
A decorator without functools.wraps | __name__ is wrapper | The original's identity was not copied | @functools.wraps(func) |
Forgetting return wrapper | The decorated function becomes None | The decorator returned nothing | Return the wrapper |
What to Remember
- A function is a value: it can be stored, passed, built and returned.
- A lambda is a one-expression function with no name. Use it where it is read, not as a named definition.
map()andfilter()are lazy and used up once. A comprehension is usually clearer.sorted(..., key=...)is where a lambda earns its place.- A closure is an inner function that remembers a value from the function that made it. Each call of the outer function makes a new one.
- Reading an outer variable is free; changing one needs
nonlocal. - An iterable can be looped over; an iterator does the walking and is used up once.
iter()makes one,next()advances it,StopIterationends it.- A generator is an iterator written with
yield. Calling the function runs nothing. - A generator expression is a comprehension in round brackets — one value at a time, not a whole list.
- A decorator returns a replacement function.
@thingis justname = thing(name). - Give every wrapper
*args, **kwargsand@functools.wraps(func).
Next: Class Relationships