What is "with" in Python for?

Question

I’m doing a study on Python and I came across this guy with.

I know him from Javascript. However, I don’t know how he works in Python.

Most of the examples I’ve seen show it being used in reading files.

Example:

with open("my_file.txt") as file:
    data = file.read()
    # faça algo com "data"

Answer 1

It is used to ensure completion of acquired resources.

A file, for example, is opened. Who guarantees that it will be closed? Even if you explicitly put in the code that it should be closed, if an exception occurs, the code goes out of scope without executing the rest of the code that is in scope, it skips the lock.

To avoid this we use a try finally. The finally ensures completion. As the code gets a little long and this case is quite frequent the language provided a simplified form with the with.

He can manipulate objects that contain the methods __enter__() and __exit__(). They are called internally at the very beginning of the execution of the created block and within the finally internal created in the block.

In the example quoted there must be something like this internally:

try:
    __enter__()
    open("my_file.txt") as file:
        data = file.read()
        #faça algo com "data"
finally:
    __exit__()

I put in the Github for future reference.

Certainly within the __exit__() has a close() to close the file. About the finally you already know how it works according to that question done previously. The operation in PHP and Python as in almost any language that has this feature is virtually identical.

To help what was said in the comments, it is analogous but not identical to the command using C# and VB.NET (do not confuse with the directive) or try () with Resources of Java (not to be confused with try normal).

Documentation.

Answer 2

The command with serves to facilitate the writing of any code block that involves resources that need to be "finalized" (i.e., restored, released, closed, etc... ) after the block is terminated - and it allows this to be done automatically, with the logic of completion within the object used.

So first thing: the completion code of with always runs - no matter if an error occurred inside the with or not.

But this could already be done with the block finally and a clause try...finally - no matter the outcome of the block within the try, the finally always runs.

The great differential of with is that the programmer who is using the objects that need completion don’t have to worry about calling these methods explicitly.

In the case of a file, in general people know that it has to be called the close - but the most common, as is already used the close is that no one ends up programming an excerpt that handles file with call to close, without using the with remembering to put this inside a finally. That is, the with face encourages the code to be written in a way that behaves in the correct way:

wrong way:

f = open("data.txt", "w")
# codigo usando o arquivo f
f.close()

right way, without the with:

try:
   f = open("data.txt", "w")
   # codigo usando o arquivo f
finally:
   f.close()

right way with the with (recommended):

with open("data.txt", "w") as f:
   # codigo usando o arquivo f

(the file is automatically closed at the end of the block).

Then note that the code "right" still gets a shorter line than the "wrong" way. But - as I mentioned above, the close files is not the coolest thing of with - because everyone knows and remembers the close (hopefully) - but is that for any object that needs a shutdown - threads Locks, database transactions, change terminal status to read keys without needing the <enter>, Monkey patch functions and methods in testing, the with performs the completion transparently - without the user needing to know if they need to call the method close, release, undo of each object (in fact, it forces programmers of the object classes that work with to implement a common interface, with the methods __enter__ and __exit__.

What’s more, when you’re programming your own classes that you can use with with can handle within your function code __exit__ the most common, or expected errors that could happen in the block: the method __exit__ receives information about the exception that occurred within the block, and the __exit__ can choose to treat the error and delete the exception, or simply release the resource and pass the error forward: ie a lot of code that would always be the same as someone would have to write within the clause except all once you were using your object can stay inside the __exit__ and people use their object without worrying about the code that would go inside the except.

Here is a simple example of a "bank account" that makes temporary transactions with the balance, but only allows transactions to take place at the end of the with if there is a positive balance in all accounts:

class ContaBancariaTransacional:
    def __init__(self, titular, saldo_inicial=0):
        self.titular = titular
        self.saldo = saldo_inicial

    def deposito(self, valor):
        self.temp += valor

    def retirada(self, valor):
        if self.saldo - self.temp - valor < 0:
            raise ValueError("Não há saldo suficiente")
        self.temp -= valor
        return valor

    def __enter__(self):
        self.temp = 0

    def __exit__(self, tipo_excecao, valor_excecao, traceback):
        if tipo_excecao is None:
            # Não ocorreu nenhum erro no bloco, e podemos
            # atualizar o saldo definitivo
            self.saldo += self.temp
        del self.temp

    def __repr__(self):
        return f"conta de {self.titular}. Saldo: {self.saldo:.02f}"

def transferir(conta1, conta2, valor):
    with conta1, conta2:

        conta2.deposito(conta1.retirada(valor))

And in an interactive session we can see:

In [96]: conta1 = ContaBancariaTransacional("João", 100)

In [97]: conta2 = ContaBancariaTransacional("José", 0)

In [98]: conta1, conta2
Out[98]: (conta de João. Saldo: 100.00, conta de José. Saldo: 0.00)

In [99]: transferir(conta1, conta2, 100)

In [100]: conta1, conta2
Out[100]: (conta de João. Saldo: 0.00, conta de José. Saldo: 100.00)

In [101]: transferir(conta1, conta2, 50)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
(...)

ValueError: Não há saldo suficiente

In [102]: conta1, conta2
Out[102]: (conta de João. Saldo: 0.00, conta de José. Saldo: 100.00)

Disclaimer: that code is not complete to be a "two Phase transaction" - and in fact, it only works because the account the balance comes from is before in charge with - when the __exit__ of the first account is called, it raises the exception, which then prevents the balance increase in the count2. In addition, to put into production it would be nice to have some Ocks per thread in this Account class as well, no matter how toy it is. But I think you can see how with can work with classes defined by the programmer himself.

Now, other than for transfer- imagine that this "account" object is used by a system that is inside a withdrawal machine (in case the access to the account object would be remote, with some methods remote calling protocol, of course) - but assuming a Pythonica library to operate the machine hardware - the code for booty could simply be:

def saque(atm, conta, valor):
    with conta:
        valor = conta.retirada(valor)
        atm.disponibilizar_notas(valor)
        atm.abrir_gaveta()
        atm.esperar(10)
        atm.fechar_gaveta()
        if not atm.gaveta_vazia():
            raise RuntimeError("Usuario nao retirou o dinheiro")

Ready - if any of the methods involving even mechanical parts of the machine give an error, the actual balance of the account is not debited. (Note that if there is no desired value in the account, the withdrawal method itself already aborts the transaction)

Answer 3

Syntax

The syntax of the declaration with is:

with EXPR as VAR:
    BLOCK

with and as are reserved words of language. EXPR is an arbitrary expression (but not a list of expressions), and VAR is a single target for assignment (an object, in a simplified form). VAR cannot be a comma-separated list of variables, but can be a comma-separated list of variables (will be explained later). The part as VAR is optional, so the syntax below remains valid:

with EXPR:
    BLOCK

From version 2.7 of the language it is possible to use multiple contexts managers:

with EXPR1 as VAR1, EXPR2 as VAR2:
    BLOCK

Which is analogous to creating statements with nested:

with EXPR1 as VAR1:
    with EXPR2 as VAR2:
        BLOCK

Functioning

The order of execution is:

1. The expression given by EXPR is evaluated in order to obtain the context manager;
2. The method __exit__ context manager is loaded for future use;
3. The method __enter__ context manager is called;
4. If VAR is defined, is assigned to the VAR the return of the method __enter__;
5. The code block defined by BLOCK is executed;
6. The method __exit__ is called. If any exception was fired by BLOCK, the guy (type), value (value) and tracking information (traceback) of the exception are passed as parameter. Otherwise, if BLOCK is executed without generating errors, is finished with the instructions break, continue or return, the method __exit__ will be executed with the three parameters equal to None.

The closest translation of the first statement about its operation is:

# 1. A expressão é avaliada para obter o gerenciador de contexto:
mgr = (EXPR)

# 2. O método __exit__ é carregado
exit = type(mgr).__exit__

# 3. O método __enter__ é executado:
value = type(mgr).__enter__(mgr)

exc = True
try:
    try:
        # 4. O retorno de __enter__ é atribuído a VAR:
        VAR = value

        # 5. O bloco de código é executado:
        BLOCK
    except:
        # Captura os erros ocorridos durante a execução de BLOCK:
        exc = False

        # 6. Executa o método __exit__ passando as informações de erro:
        if not exit(mgr, *sys.exc_info()):
            raise
finally:
    # 6. Executa o método __exit__ passando None para os parâmetros:
    if exc:
        exit(mgr, None, None, None)

The details of the execution may vary according to the characteristics of the objects involved.

If the expression given in EXPR does not have at least one of the methods __enter__ and __exit__, an exception of the type AttributeError will be fired, first to __exit__, second to __enter__.

Example

Consider the example:

class Foo (object):

    def __enter__ (self):
        print("Enter")
        return 1

    def __exit__ (self, exit, value, exc):
        print("Exit")

with Foo() as foo:
    print(foo)

The exit code will be:

Enter
1
Exit

Code working on Ideone.

The execution is the same as:

import sys

mgr = Foo()
exit = Foo.__exit__
value = Foo.__enter__(mgr)
exc = True
try:
    try:
        foo = value
        print(foo)
    except:
        exc = False
        if not exit(mgr, *sys.exc_info()):
            raise
finally:
    if exc:
        exit(mgr, None, None, None)

Code working on Ideone.

If the method __enter__ return a list of values:

class Foo (object):

    def __enter__ (self):
        print("Enter")
        return 1, 2, 3

    def __exit__ (self, exit, value, exc):
        print("Exit")

It is possible to assign a comma-separated list of variables within the with:

with Foo() as (a, b, c):
    print(a, b, c)

The exit would be:

Enter
1 2 3
Exit

Code working on Ideone.

Compatibility

The statement of with added in version 2.5 of the language, however, it is necessary to enable it through the code:

from __future__ import with_statement

As of version 2.6, the declaration is already enabled by default. Versions prior to 2.5 do not support this statement.

Context Managers

It is possible, in a practical way, to define new contexts managers from the library contextlib, using the decorator contextmanager.

from contextlib import contextmanager

@contextmanager
def tag(name):
    print("<%s>" % name)
    yield
    print("</%s>" % name)

with tag("h1"):
    print("Stack Overflow")

The output generated is:

<h1>
Stack Overflow
</h1>

The functioning is analogous considering what is before yield as the method __enter__ and what is after the method __exit__ (only for merit of result, but are NOT analogous).

Examples

The Abaixos examples were taken from PEP 343 itself, are not entirely functional, but express very well the use, although not limited to, of the statement in question.

1. When the resource used needs to be blocked at the beginning of the execution and released at the end of the execution:

   @contextmanager
   def locked(lock):
       lock.acquire()
       try:
           yield
       finally:
           lock.release()
   
   with locked(myLock):
       # Código executado com myLock bloqueado. O recurso será
       # liberado quando o bloco de código ser executado.
   

2. When a file opened by the program is required to be closed at the end of the program:

   @contextmanager
   def opened(filename, mode="r"):
       f = open(filename, mode)
       try:
           yield f
       finally:
           f.close()
   
   with opened("/etc/passwd") as f:
       for line in f:
           print line.rstrip()
   

This behavior is already native to the function open python.

3. When working with transactions in the database:

   @contextmanager
   def transaction(db):
       db.begin()
       try:
           yield None
       except:
           db.rollback()
           raise
       else:
           db.commit()
   

4. When you wish to temporarily redirect the outgoing target:

   @contextmanager
   def stdout_redirected(new_stdout):
       save_stdout = sys.stdout
       sys.stdout = new_stdout
       try:
           yield None
       finally:
           sys.stdout = save_stdout
   
   with opened(filename, "w") as f:
       with stdout_redirected(f):
           print("Hello world") # Será escrito no arquivo filename
   

Other examples can be found in the other answers to this question, PEP 343 or other references.

References:

PEP 343 -- The "with" Statement

Compound statements - The with statement