class: center, middle, huge #Finding closure with closures ## Thomas Ballinger ## @ballingt --- ??? * A friend of mine was asked what are closures was at a programming interview a few years ago. Who here would have something to say in response to that question? My friend was pretty competent with Python and JavaScript and took advantage of closures in his code all the time. I hope today we leave here with something to say on that topic if we're asked. * I worked for several years at a programming retreat where programmers more familiar with other languages would sometimes ask me "does Python even have closures?" I hope we leave here with something to say to that as well. --- class: huge center #spoiler: yes --- class: listitemsfillwidth * how free variables get resolved * how Python determines variable scope * Python 1.0 functions: not quite closures * Python 2.2 function: they're closures now * Python 3.0 function: now with more closure * how we use closures ??? To find that closure, we'll look at... --- ??? I have two functions for formatting strings. One is for html, the other is for outputting text to the terminal. (you know, the way text in the terminal is sometimes different colors) I'm going to import these two functions, then look at their source code with the inspect module, a Python standard library module. --- class: widecode ~~~ >>> ~~~ ??? I'm going to open up the interactive Python interpreter --- class: widecode ~~~ >>> from htmlformat import bold as htmlbold >>> from terminalformat import bold as termbold >>> ~~~ ??? I have to use the "from, import, as" syntax because these functions have the same name --- class: widecode ~~~ >>> from htmlformat import bold as htmlbold >>> from terminalformat import bold as termbold >>> import inspect >>> ~~~ ??? and now using the inspect module, --- class: widecode ~~~ >>> from htmlformat import bold as htmlbold >>> from terminalformat import bold as termbold >>> import inspect *>>> print(inspect.getsource(htmlbold)) *def bold(text): * return '{}{}{}'.format(BOLDBEFORE, text, BOLDAFTER) >>> ~~~ ??? we'll look at the code for the first function, which looks like a simple one-liner, --- class: widecode ~~~ >>> from htmlformat import bold as htmlbold >>> from terminalformat import bold as termbold >>> import inspect >>> print(inspect.getsource(htmlbold)) def bold(text): return '{}{}{}'.format(BOLDBEFORE, text, BOLDAFTER) *>>> print(inspect.getsource(termbold)) *def bold(text): * return '{}{}{}'.format(BOLDBEFORE, text, BOLDAFTER) >>> ~~~ ??? and the second. They're remarkably similar! --- class: widecode ~~~ def bold(text): return '{}{}{}'.format(BOLDBEFORE, text, BOLDAFTER) def bold(text): return '{}{}{}'.format(BOLDBEFORE, text, BOLDAFTER) ~~~ ??? But when I use them, they're going to have different behavior. --- ~~~ *>>> htmlbold('eggplant') *'
eggplant
' >>> ~~~ --- ~~~ >>> htmlbold('eggplant') '
eggplant
' *>>> termbold('eggplant') *'\x1b[1meggplant\x1b[0m' >>> ~~~ --- ??? How is that possible? How are the two functions different? Here's a similar question: the html bold function uses the variable `BOLDBEFORE`. --- class: widecode ~~~ >>> print(inspect.getsource(htmlbold)) def bold(text): * return '{}{}{}'.format(BOLDBEFORE, text, BOLDAFTER) ~~~ ??? Because it is neither a parameter to the function nor a local variable, we call it a "free variable." --- ~~~ >>> from htmlformat import bold as htmlbold >>> def signbold(phrase): ... BEFOREBOLD = '(in Sharpie) ' ... return htmlbold(phrase) ... >>> signbold('eggplant') ~~~ ??? If we call that function after setting a local variable with that name, will that change its behavior? will it output `'
eggplant
'` or `'(in Sharpie) eggplant'`? The question amounts to whether the Python language uses "open free variables," whose values are determined by looking up the call stack, or with closed free variables that use the value in the environment in which the function was defined. --- background-image: url(./theFunctionOfFunction.png) ??? In a 1970 paper describing implementing these two approaches, Joel Moses points out that although it might be easier to implement a language with the first behavior, programmers are usually interested in the second; they want their functions to use the variables they created for use with that function, not new variables at their functions' call sites. --- ~~~ >>> from htmlformat import bold as htmlbold >>> def signbold(phrase): ... BEFOREBOLD = '(in Sharpie) ' ... return htmlbold(phrase) ... *>>> signbold('eggplant') ~~~ ??? With that in mind, will "in Sharpie" print here, or will we get the old behavior of the bold tags? --- ~~~ >>> from htmlformat import bold as htmlbold >>> def signbold(phrase): ... BEFOREBOLD = '(in Sharpie) ' ... return htmlbold(phrase) ... >>> signbold('eggplant') *'
eggplant
' ~~~ ??? And indeed, Python ignores this new variable and bold tags again sandwich the word eggplant. --- ~~~ >>> from htmlformat import bold as htmlbold *>>> BEFOREBOLD = '(in Sharpie) ' >>> ~~~ ??? What about changing the global variable instead? --- ~~~ >>> from htmlformat import bold as htmlbold >>> BEFOREBOLD = '(in Sharpie) ' >>> htmlbold('eggplant') ~~~ ??? What will this print? It's either "in Sharpie" or "b tag". --- ~~~ >>> from htmlformat import bold as htmlbold >>> BEFOREBOLD = '(in Sharpie) ' >>> htmlbold('eggplant') '
eggplant
' ~~~ ??? That's right, these are different global variables! And now we have a hint of how those two bold functions might differ, let's finally take a look. --- class: twofilescode ~~~ #htmlformat.py #terminalformat.py *BEFOREBOLD = '
' BEFOREBOLD = '\x1b[1m' *AFTERBOLD = '
' AFTERBOLD = '\x1b[0m' def bold(text): def bold(text): return '{}{}{}'.format( return '{}{}{}'.format( BOLDBEFORE, BOLDBEFORE, text, text, BOLDAFTER) BOLDAFTER) ~~~ ??? Different global variables, and it seems like they each use their own global variables! "global" is not a great name. --- class: huge center #global variables aren't ??? They aren't universal across your whole program, they're really module level variables. --- ??? Since we're stuck with the word "global," we should think of each module as being a separate globe... --- class: twofilescode twoplanets ![](./Venus_Earth_Comparison.png) ~~~ #htmlformat.py #terminalformat.py *BEFOREBOLD = '
' BEFOREBOLD = '\x1b[1m' *AFTERBOLD = '
' AFTERBOLD = '\x1b[0m' def bold(text): def bold(text): return '{}{}{}'.format( return '{}{}{}'.format( BOLDBEFORE, BOLDBEFORE, text, text, BOLDAFTER) BOLDAFTER) ~~~ ??? Modules as being separate globes --- ![](./peridotplanet.png) ??? And function objects imported from modules are emissaries from their home modules, always in communication with their homeworld. --- ```python >>> from htmlformat import bold *>>> bold.__module__ 'htmlformat' ``` ??? On the function object we find references both to the name of the module where it was defined, and more importantly to the dictionary of the global variables in the module. --- ```python >>> from htmlformat import bold >>> bold.__module__ 'htmlformat' *>>> bold.__globals__.keys() dict_keys(['BEFOREBOLD', 'AFTERBOLD', 'bold', '__name__', '__builtins__', '__cached__', '__spec__', '__package__', '__doc__', '__loader__', '__file__']) ``` ??? On the function object we find a reference to a dictionary of the global variables --- ```python >>> from htmlformat import bold >>> bold.__module__ 'htmlformat' >>> bold.__globals__.keys() dict_keys(['BEFOREBOLD', 'AFTERBOLD', 'bold', '__name__', '__builtins__', '__cached__', '__spec__', '__package__', '__doc__', '__loader__', '__file__']) >>> import htmlformat *>>> vars(htmlformat) is bold.__globals__ True ``` ??? The other important thing about this is that this is a live link back to that home planet! It's the very same object as the namespace of the module! --- background-image: url(./theFunctionOfFunction.png) ??? In that same paper, Joel Moses described an implementation of this type of behavior. For a function to behave this way, it needs both the code it will execute and the environment which closes the free variables used by that function. He called this... --- class: huge center #Closure: #code + environment ??? A closure! A closure is the combination of code and the data structure storing the environment in which to look up the free variables in that code. So Python functions are looking an awful lot like closures! --- ??? Because this is a live link to that module, what if we were to change a binding in that namespace? --- ~~~ BEFOREBOLD = '
' AFTERBOLD = '
' ~~~ ??? What happens when I add this code to the bottom on htmlformat? --- class: twofilescode twoplanets ![](./Venus_Earth_Comparison.png) ~~~ #htmlformat.py BEFOREBOLD = '
' AFTERBOLD = '
' def bold(text): return '{}{}{}'.format( BOLDBEFORE, text, BOLDAFTER) *BEFOREBOLD = '
' *AFTERBOLD = '
' ~~~ --- ~~~ >>> from htmlformat import bold >>> bold('eggplant') ~~~ ??? Will this print with bold or the new span tags surrounding it? --- ~~~ >>> from htmlformat import bold >>> bold('eggplant') '
eggplant
>>> ~~~ ??? The function is going to use the new values because it uses the value of the variables when we call the function, not when we define the function. So if we step through, --- ~~~ *>>> from htmlformat import bold >>> bold('eggplant') '
eggplant
>>> ~~~ ??? we import the module --- class: twofilescode twoplanets ![](./Venus_Earth_Comparison.png) ~~~ #htmlformat.py *BEFOREBOLD = '
' *AFTERBOLD = '
' def bold(text): return '{}{}{}'.format( BOLDBEFORE, text, BOLDAFTER) BEFOREBOLD = '
' AFTERBOLD = '
' ~~~ --- class: twofilescode twoplanets ![](./Venus_Earth_Comparison.png) ~~~ #htmlformat.py BEFOREBOLD = '
' AFTERBOLD = '
' *def bold(text): * return '{}{}{}'.format( * BOLDBEFORE, * text, * BOLDAFTER) BEFOREBOLD = '
' AFTERBOLD = '
' ~~~ --- class: twofilescode twoplanets ![](./Venus_Earth_Comparison.png) ~~~ #htmlformat.py BEFOREBOLD = '
' AFTERBOLD = '
' def bold(text): return '{}{}{}'.format( BOLDBEFORE, text, BOLDAFTER) *BEFOREBOLD = '
' *AFTERBOLD = '
' ~~~ --- ~~~ >>> from htmlformat import bold *>>> bold('eggplant') '
eggplant
>>> ~~~ --- ??? We can even modify these variables from outside the module! --- ~~~ >>> from htmlformat import bold >>> bold('eggplant') '
eggplant
' *>>> import htmlformat >>> ~~~ --- ~~~ >>> from htmlformat import bold >>> bold('eggplant') '
eggplant
' >>> import htmlformat *>>> htmlformat.BEFOREBOLD = 'READ THIS--> ' >>> ~~~ --- ~~~ >>> from htmlformat import bold >>> bold('eggplant') '
eggplant
' >>> import htmlformat >>> htmlformat.BEFOREBOLD = 'READ THIS--> ' *>>> bold('eggplant') ~~~ ??? Is this going to print a span tag, or "READ THIS-->"? --- ~~~ >>> from htmlformat import bold >>> bold('eggplant') '
eggplant
' >>> import htmlformat >>> htmlformat.BEFOREBOLD = 'READ THIS--> ' >>> bold('eggplant') 'READ THIS--> eggplant' >>> ~~~ ??? The distinction between function definition time and function execution time because important with this live link behavior: as you can see, the value it determined at execution time, but Python analyzes code when the function is defined and determines the scope of each variable. --- class: huge center # Identifying the scope of a variable ??? A Python function object is the result of this process, with each of its attribute storing a different piece of information about the function. -- #`bold.__code__` ??? Most of the most interesting stuff is on the `__code__` object --- ??? For as long as Python has been available on the internet, there have been at least two kinds of variables: --- class: center # local variables: `func.__code__.co_varnames` # "phone home" global variables: `func.__code__.co_names` ??? Local variables (including function parameters) appear in `.__code__.co_varnames` and global variables and a few other things make up `.__code__.co_names`. --- ??? You may already have some good intuition for this, so let's be the interpreter and look for patterns in our decisions about which is which. --- class: guess #Be the interpreter ~~~python >>> def movie_titleize(phrase): ... capitalized = phrase.title() ... return capitalized + ": The Untold Story" ~~~ --- class: guess #Be the interpreter ~~~python >>> def movie_titleize(phrase): ... capitalized = phrase.title() ... return capitalized + ": The Untold Story" ~~~ #phrase ??? Is phrase a local variable, or "global" variable? --- class: guess #Be the interpreter ~~~python >>> def movie_titleize(phrase): ... capitalized = phrase.title() ... return capitalized + ": The Untold Story" ~~~ #capitalized ??? Is capitalized a local variable or global variable? --- class: guess ~~~python >>> def movie_titleize(phrase): ... capitalized = phrase.title() ... return capitalized + ": The Untold Story" ~~~ ??? They are both local. One is a parameter to the function, the other is assigned to on the first line. This type of function, sometimes called a "pure" function, doesn't need its link to its home module for looking up variables. Without this associated environment, the function would not be a closure, and here we find out first fork in the definition of the word. Is a function a closure if it has this link to its defining environment but that environment is never used? So some would say functions require free variables to be closures, others that the combination of code and environment is enough, so long as this link to home module is there. --- class: guess ~~~python >>> def movie_titleize(phrase): ... capitalized = phrase.title() ... return capitalized + ": The Untold Story" ~~~ ??? For an altogether different reason, most would say that none of the functions we have seen so far are closures. So don't touch that dial! --- class: guess #Be the interpreter ~~~python >>> def movie_titleize(phrase): ... capitalized = phrase.title() ... return capitalized + ": The Untold Story" ... >>> movie_titleize.__code__.co_varnames ('phrase', 'capitalized') >>> movie_titleize.__code__.co_names ('title') ~~~ ??? We can check our answer by looking at the attribute of the `__code__` object. Attributes also go in co_names, it's a bit of a catchall --- class: guess #Be the interpreter ~~~python >>> def catchy(phrase): ... options = [phrase.title(), DEFAULT_TITLE] ... options.sort(key=catchiness) ... return options[1] ~~~ ??? How about in this function for finding catchy phrases --- class: guess #Be the interpreter ~~~python >>> def catchy(phrase): ... options = [phrase.title(), DEFAULT_TITLE] ... options.sort(key=catchiness) ... return options[1] ~~~ #phrase ??? Is "phrase" a local variable or a global variable? --- class: guess #Be the interpreter ~~~python >>> def catchy(phrase): ... options = [phrase.title(), DEFAULT_TITLE] ... options.sort(key=catchiness) ... return options[1] ~~~ #options --- class: guess #Be the interpreter ~~~python >>> def catchy(phrase): ... options = [phrase.title(), DEFAULT_TITLE] ... options.sort(key=catchiness) ... return options[1] ~~~ #DEFAULT_TITLE --- class: guess #Be the interpreter ~~~python >>> def catchy(phrase): ... options = [phrase.title(), DEFAULT_TITLE] ... options.sort(key=catchiness) ... return options[1] ~~~ #catchiness ??? Now that we've done some guessing, here's the rule: if it's a parameter or is ever assigned to in the function, it's a local variable. Otherwise it's global. --- class: guess #Be the interpreter ~~~python >>> def catchy(phrase): ... options = [phrase.title(), DEFAULT_TITLE] ... options.sort(key=catchiness) ... return options[1] ... >>> catchy.__code__.co_varnames ('phrase', 'options') >>> catchy.__code__.co_names ('catchiness', 'DEFAULT_TITLE', 'sort', 'catchiness') ~~~ ??? Let's check our answers, yep. This is not a pure function, it has free variables. If you found this function in some code and were to copy and paste it into your file, it wouldn't work right? It wouldn't have the right environment. --- ??? Your programmer intuition might bring you to disagree with Python's categorization in this next example. --- ~~~ >>> HIGH_SCORE = 1000 >>> def new_high_score(score): ... print('congrats on the high score!') ... print('old high score:', HIGH_SCORE) ... HIGH_SCORE = score ... >>> new_high_score(1042) ~~~ ??? It certainly looks like the author wanted `HIGH_SCORE` to be a global variable, but what is it? --- ~~~ >>> HIGH_SCORE = 1000 >>> def new_high_score(score): ... print('congrats on the high score!') ... print('old high score:', HIGH_SCORE) *... HIGH_SCORE = score ... >>> new_high_score(1042) ~~~ ??? Right, it gets assigned to right here, making it local for the entire function! What error will we get here? --- ~~~ >>> HIGH_SCORE = 1000 >>> def new_high_score(score): ... print('congrats on the high score!') ... print('old high score:', HIGH_SCORE) ... HIGH_SCORE = score ... >>> new_high_score(1042) Traceback (most recent call last): File "
", line 1, in
File "
", line 2, in new_high_score UnboundLocalError: local variable 'HIGH_SCORE' referenced before assignment ~~~ ??? let's check our classification of HIGH_SCORE... --- ~~~ >>> HIGH_SCORE = 1000 >>> def new_high_score(score): ... print('congrats on the high score!') ... print('old high score:', HIGH_SCORE) ... HIGH_SCORE = score ... >>> new_high_score.__code__.co_varnames ('score', 'HIGH_SCORE') >>> new_high_score.__code__.co_names ('print',) >>> ~~~ ??? Yep, it's local. How could we somehow tell Python to use make it a global variable instead? --- ~~~ >>> HIGH_SCORE = 1000 >>> def new_high_score(score): ... global HIGH_SCORE ... print('congrats on the high score!') ... print('old high score:', HIGH_SCORE) ... HIGH_SCORE = score ... >>> new_high_score(1042) congrats on the high score! old high score: 1000 ~~~ ??? We can use the global keyword lets us bypass Python's heuristic and manually specify the scope of a this variable. --- ??? How are we doing so far? --- class: center hugelessspace ## In review, the # Scope | Value ## of a variable is determined at # Definition | Execution ## time --- class: certificate background-image: url(./certificate.jpg) #Python 2.0 Scope Expert #MM (the year 2000) ??? Well then congratulations, that's all there is to scope all the way up to Python 2.0. Python functions have always closed over their their module-level environments, and there were no big changes in how scope worked up to Python 2.0. But in Python 2.0 an important method of closing free variables wasn't available to us, one required by most people to classify a function as a closure: closing over outer scopes that are not the global scope. --- ~~~python def tallest_building(): buildings = {'Burj Khalifa': 828, 'Shanghai Tower': 632, 'Abraj Al-Bait': 601} def height(name): return buildings[name] return max(buildings.keys(), key=height) ~~~ ??? Here's a function that may look familiar if you've seen from my favorite of Ned Batchelder's talks, "Loop like a Native." It finds the tallest of several buildings with the `max` function, sorting by the local function height. Let's be the interpreter here: in the inner height function, --- class: guess ~~~python def tallest_building(): buildings = {'Burj Khalifa': 828, 'Shanghai Tower': 632, 'Abraj Al-Bait': 601} * def height(name): return buildings[name] return max(buildings.keys(), key=height) ~~~ #name ??? is name a local or a global variable? --- class: guess ~~~python def tallest_building(): buildings = {'Burj Khalifa': 828, 'Shanghai Tower': 632, 'Abraj Al-Bait': 601} def height(name): * return buildings[name] return max(buildings.keys(), key=height) ~~~ #buildings ??? is buildings a local or a global variable? well that's the trick question, right? We'd like it to be neither, we'd like it to be the buildings variable from surrounding scope. What do you think happens when we run this? --- class: huge widecode ~~~ ... File "tmp3.py", line 7, in height return buildings[name] NameError: global name 'buildings' is not defined ~~~ ??? Since `buildings` is not a local variable it is assumed to be global in Python 2.0 and calling it produces this error. --- # local variables: `func.__code__.co_varnames` # "phone home" global variables: `func.__code__.co_names` # "phone home" outer non-global scopes: `func.__code__.co_freevars` ??? Optionally in Python 2.1 as a `__future__` import, then by default in Python 2.2, variables from outer non-global scopes were added and are found at `.__code__.co_freevars`: --- ~~~python >>> height.__code__.co_varnames ('name',) >>> height.__code__.co_names () >>> height.__code__.co_freevars ('buildings',) ~~~ ??? Once that function finally compiles it looks like this --- ??? Now that we're closing over these scopes, we've got all the outer scopes covered: we're definitely talking about closures now. Usually when people talk about closures, they aren't thinking about module-level scope, or rather they might say that module-level "global" variables are a special case. It's probably fair to treat global variables as a special case because in Python their implementation is a special case; storing the global environment is much easier. --- ??? You may already be familiar with module objects in Python: generally they're singletons, so a given module has only one mapping of variables to values. But a function can be run many times, producing many different mappings of variables to values. A different environment must be tracked for each invocation of the function that produces function objects. --- ~~~python formatters = {} colors = ['red', 'green', 'blue'] for color in colors: def in_color(s): return ('
' + s + '
') formatters[color] = in_color *formatters['green']('hello') ~~~ ??? The code creates several functions for displaying text in color in html. What color will this green formatting function be? --- ~~~python formatters = {} colors = ['red', 'green', 'blue'] for color in colors: def in_color(s): return ('
' + s + '
') formatters[color] = in_color *print(color) formatters['green']('hello') ~~~ ??? It helps to consider the value of the global color variable when the for loop has finished. What's the value of color here? So no matter which formatting function we call, the text is going to be in blue. If we want each function to have its own value associated with the color variable, then we need separate scopes for each function definition: --- ~~~ formatters = {} colors = ['red', 'green', 'blue'] *def make_color_func(color): def in_color(s): return ('
' + s + '
') return in_color for color in colors: formatters[color] = make_color_func(color) formatters['green']('hello') ~~~ ??? Each time the `make_color_func` function is called, a new local mapping is created binding color to one of red, green or blue; a function called `in_color` is defined which references the color variable in this outer scope; and the `in_color` function is returned and stuck in a dictionary. --- class: threemoons ![](./threemoons.jpg) ~~~ formatters = {} colors = ['red', 'green', 'blue'] def make_color_func(color): def in_color(s): return ('
' + s + '
') return in_color for color in colors: formatters[color] = make_color_func(color) formatters['green']('hello') ~~~ ??? So there are three new environments, shown here as three moons of the global scope planet. And unlike with modules, there isn't a corresponding namespace object that just has the bindings. --- class: widecode ~~~ >>> height.__closure__ (
,) ~~~ ??? Precisely how these are maintained by Python is out of scope for this talk, but the `.__closure__` attribute on the three produced functions provides some hint. If a function has this attribute set to something other than None then it has free variables which refer to bindings in outer, non-global scopes. --- ??? We've reached the most common definition of a closure: a function with free variables closed by an outer, non-global scope. However another fork in definitions occurs here: some would call our three color functions closures and but not the earlier height function because it was used in the same scope it was defined. Although CPython doesn't implement the two any differently, you could imagine that it becomes more difficult to maintain the environment a function needs to evaluate its variables once the bindings it needs goes out of scope. The distinction here is that looking up the stack instead of the "closure" solution of code + environment would result in the same behavior in the first case, making whether a function was a closure or not only distinguishable in the second case. --- class: center huge #Python 2.2 functions: #definitely closures now! ??? Since 2001, Python has had closures. --- class: huge center #... ??? Well, yeah, but there is one more thing. --- class: certificate background-image: url(./certificate.jpg) #Python 2.7 Scope Expert #MMVIII (the year 2008) ??? You're a Python 2 scope expert now! But... --- ??? But if rumblings of the insufficiency of Python 2's closures have ever reached your ears, you may not have found your closure yet. --- class: huge center ##Read-only closures ##"weak support" for closures ??? You might have heard that Python has "weak support" for closures, or Python has "read-only" closures, not "full" closures. This comes from an asymmetry between global variables and outer non-global variables, which I will hereto refer to as "nonlocal" variables. There's still one more thing we can't do, a way that our Python closures are more limited than those of some other languages. --- ??? Remember the problem with global variables, where if we want to assign to a global variable in a function we need to say "global"? If we didn't have that global keyword our global variables were "read-only?" Not being able to assign to a nonlocal variables because that would make it a local variables leads to the phrase "read-only" closures. We can access those variables, and if they're mutable objects we can send them messages, we can mutate them - but we can't rebind them. --- class: guess ~~~python >>> def get_number_guesser(answer): ... last_guess = None ... def guess(n): ... if n == last_guess: ... print('already guessed that!') ... last_guess = n ... return n == answer ... >>> guess = get_number_guesser(12) >>> guess(9) ~~~ ??? here's an example of when we might want to, but this isn't going to work. Let's be the interpreter again to see why. --- class: guess ~~~python >>> def get_number_guesser(answer): ... last_guess = None ... def guess(n): ... if n == last_guess: ... print('already guessed that!') ... last_guess = n ... return n == answer ~~~ #n ??? If we're being the Python interpreter here, are these local variables, global variables, or nonlocal variables? --- class: guess ~~~python >>> def get_number_guesser(answer): ... last_guess = None ... def guess(n): ... if n == last_guess: ... print('already guessed that!') ... last_guess = n ... return n == answer ~~~ #answer --- class: guess ~~~python >>> def get_number_guesser(answer): ... last_guess = None ... def guess(n): ... if n == last_guess: ... print('already guessed that!') ... last_guess = n ... return n == answer ~~~ #last_guess ??? It's going to think it's a local variable because we use assignment here. So what error will we get when we run that inner function? --- ~~~python >>> def get_number_guesser(answer): ... last_guess = None ... def guess(n): ... if n == last_guess: ... print('already guessed that!') ... last_guess = n ... return n == answer ... >>> guess = get_number_guesser(12) >>> guess(9) Traceback (most recent call last): File "
", line 1, in
File "
", line 4, in guess UnboundLocalError: local variable 'last_guess' referenced before assignment ~~~ ??? One last puzzle piece: nonlocal! --- ~~~python >>> def get_number_guesser(answer): ... last_guess = None ... def guess(n): ... if n == last_guess: ... print('already guessed that!') ... last_guess = n ... return n == answer ... ~~~ ??? Add a hint to the Python interpreter when we define the function: we mean the outer variable, not a new one. Without `nonlocal`, nonlocal variables cannot be rebound to new values. --- ~~~python >>> def get_number_guesser(answer): ... last_guess = None ... def guess(n): ... nonlocal last_guess ... if n == last_guess: ... print('already guessed that!') ... last_guess = n # modifies variable in outer scope ... return n == answer ... >>> guess = get_number_guesser(12) >>> guess(9) False >>> guess(9) already guessed that! False ~~~ --- class: center, huge #Python functions are definitely closures now! ??? "rebinding closures" As with the global keyword the change in semantics may seem small, but its lack is met with incredulity in Python 2 by those familiar with closures in other languages. As we find our closure with what closures are and whether they exist in Python, a new question arises: how did we get on without them for so long? --- class: certificate background-image: url(./certificate.jpg) #Python 3.6 Scope Expert #MMVIII (the year 2016) ??? Now you're totally up to date with scope in Python. ??? Now we know what know what closures are and that Python has them, what's left is how closures are used in Python. --- class: center #How closures are used in Python ??? Closures are used all over the place. We can inspect a function for its `.__closure__` attribute to see if it contains free variables that are closed by outer, nonlocal scopes. --- class: largelist * inner functions * lambdas for key, cmp functions * inner functions used as callbacks * threading.Thread(target=) * shutil.rmtree(..., onerror=) * signal.signal(..., handler) * atexit * ... * decorators ??? Sometimes these are "pure" functions, but often they use state from arround Whenever you write a function that does something interesting. --- class: huge ~~~ @retry def get_current_time(): return requests.get('http://time.is') def retry(func): def new_func(): try: return func() except: return func() return new_func ~~~ ??? Decorators always take a function as an argument and often define a new function to replace it, which itself typically holds a reference to the old function through a free variable from the outer function scope of the decorator. There is one more aspect I'm staring to want closure on: how did we get along without `nonlocal` for so long? Many people still use Python 2 code, how do they get along without it? And in Python 3 code, --- class: huge center #Why isn't nonlocal used much? ??? Adding the nonlocal keyword took nine years, from Python 2.2 in 2001 to Python 3 in 2008. If it's so important a change, why don't we see a ton of code using it now? Even now that the nonlocal is here, the need for compatibility with Python 2 code that many library authors have prevents some uses. Consider this abridged excerpt from Django: --- class: widecode ~~~python def decorating_function(user_function): ... nonlocal_root = [root] # make updateable non-locally def wrapper(): nonlocal_root[0] = oldroot[NEXT] ... ~~~ heavily abridged [Django code](https://github.com/django/django/blob/master/django/utils/lru_cache.py#L80) from utils/lru_cache.py ??? Real-world examples often use the *pattern* of nonlocal, but do it with a workaround: stick the variable in a mutable object! --- ??? We already have modules and the global keyword, a bigger one is that we have a nice object system (method binding)... If you need a callback that changes some state, instead of using nonlocal we'll often take that state and stick it in an object, and use methods to telegraph that we're modifying state. --- class: widecode ~~~python def tallest_building(): buildings = {'Burj Khalifa': 828, 'Shanghai Tower': 632, 'Abraj Al-Bait': 601} return max(buildings.keys(), key=buildings.get) ~~~ ??? (explain this) And finally I posit a cultural reason: Python programmers tend to be comfortable with private data being externally accessible. Python and JavaScript are relatively similar languages, and both lack (or in certain versions have lacked) private object data which can be accessed by methods of the object by not by outside code, instead using conventions like a single underscore to inform users that such a variable is not part of the public interface with that object. There's a pattern that uses nonlocal that in both languages is possible, pattern is possible, but in JavaScript it is commonplace while in Python it is unheard of. --- ??? The next code sample is the most complicated one we've seen yet. --- class: widecode ~~~python >>> class Person(object): pass >>> def create_person(name): ... age = 10 ... p = Person() ... def birthday(): ... nonlocal age ... age = age + 1 ... p.birthday = birthday ... p.greet = lambda: print("Hi, I'm", age) ... return p ... >>> me = create_person('Tom') >>> me.birthday() >>> me.age Traceback (most recent call last): File "
", line 1, in
AttributeError: 'Problem' object has no attribute 'age' ~~~ ??? Now this is totally possible in JavaScript and Python. But culturally we don't do it in Python. --- class: center #Getting our closure ??? And I think it's fine that we don't use rebinding closures all that much. --- class: center #Getting our closure ##use `nonlocal` when appropriate ??? So why use nonlocal? Because it more precisely expresses what we mean! No need to go looking for uses of it, but they do pop up! --- class: center #Getting our closure ##What are closures and does Python have them? ??? Closures are the combination of code and environment - all Python functions! --- class: center listitemsfillwidth ##Closures only count if... * they refer to variables from outer scope * they refer to variables from outer, non-global scopes * they preserve mappings that have gone out of scope * they rebind outer scope, non-global variables ??? The original definition was just carrying the environment with the function, but new definitions have arisen. Personally, I like "functions which close over nonlocal variables" It's fine if thosed closed over variables don't actually go out of scope, the fact that they could is enough. --- class: center #they refer to variables from outer, non-global scopes ??? those variables don't have to be out of scope yet And I think it's worth keeping module-level scope in mind, it's not the whole environment but it's a piece. --- class: center #Getting our closure ##Does Python even have closures? ??? Python has all of these! -- ##"Yes, Python functions can refer to variables in outer scopes even after those variables have gone out of scope." -- #(since Python 2.2 in 2001) ??? I like the "all Python 2.2 and greater functions are closures" answer, but have found closure in knowing the discussion to have if I were asked. --- class: reflist #Want to find out more? * builtins: last resort of failed global variable lookups * `__closure__` and `__code__.cell_vars`: how closures are implemented * bytecode: what does "compiling" a function really mean? * descriptors and method binding: the dark secret that turns functions into methods * scopes of various comprehensions and generator expressions * "The Function of FUNCTION in LISP..." Joel Moses, 1970 * PEP 227 Statically Nested Scopes: includes notes on closure implementation * Ned Batchelder's names talk ??? Unfortunately we don't have time for questions, but I'd love to take them outside. These are some of the topics I'd love to talk about or recommend for further reading. I hope I've helped you all find closure with closures. Thank so much for coming! --- class: huge center thankyou #Thank you ##Thomas Ballinger ##@ballingt ##ballingt.com thanks to Ned Batchelder for some examples from "Loop like a Native" talk at PyCon NA 2013 Image credits: `http://uncovermichigan.com/content/22900-three-moons-cast-shadow-jupiter-s-surface-simultaneously` `https://en.wikipedia.org/wiki/Colonization_of_Venus#/media/File:Venus_Earth_Comparison.png`