Python Attributes and Methods

What is an attribute? Quite simply, an attribute is a way to get from one object to another. Apply the power of the almighty dot - objectname.attributename - and voila! you now have the handle to another object. You also have the power to create attributes, by assignment: objectname.attributename = notherobject.

Which object does an attribute access return, though? And where does the object set as an attribute end up? These questions are answered in this chapter.

>>> class C(object):
...     classattr = "attr on class"  
...
>>> cobj = C()
>>> cobj.instattr = "attr on instance"  
>>>
>>> cobj.instattr  
'attr on instance'
>>> cobj.classattr 
'attr on class'
>>> C.__dict__['classattr'] 
'attr on class'
>>> cobj.__dict__['instattr'] 
'attr on instance'
>>>
>>> cobj.__dict__ 
{'instattr': 'attr on instance'}
>>> C.__dict__ 
{'classattr': 'attr on class', '__module__': '__main__', '__doc__': None}

Ok, I admit 'user-provided attribute' is a term I made up, but I think it is useful to understand what is going on. Note that __dict__ is itself an attribute. We didn't set this attribute ourselves, but Python provides it. Our old friends __class__ and __bases__ (none which appear to be in __dict__ either) also seem to be similar. Let's call them Python-provided attributes. Whether an attribute is Python-provided or not depends on the object in question (__bases__, for example, is Python-provided only for classes).

We, however, are more interested in user-defined attributes. These are attributes provided by the user, and they usually (but not always) end up in the __dict__ of the object on which they're set.

When accessed (for e.g. print objectname.attributename), the following objects are searched in sequence for the attribute:

If all this hunting around fails to find a suitably named attribute, Python raises an AttributeError. The type of the type (objectname.__class__.__class__) is never searched for attribute access on an object (objectname in the example).

The built-in dir() function returns a list of all attributes of an object. Also look at the inspect module in the standard library for more functions to inspect objects.

The above section explains the general mechanism for all objects. Even for classes (for example accessing classname.attrname), with a slight modification: the bases of the class are searched before the class of the class (which is classname.__class__ and for most types, by the way, is <type 'type'>).

Some objects, such as built-in types and their instances (lists, tuples, etc.) do not have a __dict__. Consequently user-defined attributes cannot be set on them.

We're not done yet! This was the short version of the story. There is more to what can happen when setting and getting attributes. This is explored in the following sections.

Continuing our Python experiments:

>>> class C(object):
...     classattr = "attr on class"
...     def f(self):
...             return "function f"
...
>>> C.__dict__ 
{'classattr': 'attr on class', '__module__': '__main__',
 '__doc__': None, 'f': <function f at 0x008F6B70>}
>>> cobj = C()
>>> cobj.classattr is C.__dict__['classattr'] 
True
>>> cobj.f is C.__dict__['f'] 
False
>>> cobj.f 
<bound method C.f of <__main__.C instance at 0x008F9850>>
>>> C.__dict__['f'].__get__(cobj, C) 
<bound method C.f of <__main__.C instance at 0x008F9850>>

It is only the presence of the __get__() method that transforms an ordinary function into a bound method. There is nothing really special about a function object. Anyone can put objects with a __get__() method inside the class __dict__ and get away with it. Such objects are called descriptors and have many uses.

Any object with a __get__() method, and optionally __set__() and __delete__() methods, accepting specific parameters is said to follow the descriptor protocol. Such an object qualifies as a descriptor and can be placed inside a class's __dict__ to do something special when an attribute is retrieved, set or deleted. An empty descriptor is shown below.

class Desc(object):
    "A descriptor example that just demonstrates the protocol"
    
    def __get__(self, obj, cls=None): 
        pass

    def __set__(self, obj, val): 
        pass

    def __delete__(self, obj): 
        pass

What we defined above is a class that can be instantiated to create a descriptor. Let's see how we can create a descriptor, attach it to a class and put it to work.

class C(object):
    "A class with a single descriptor"
    d = Desc() 
    
cobj = C()

x = cobj.d 
cobj.d = "setting a value"  
cobj.__dict__['d'] = "try to force a value" 
x = cobj.d 
del cobj.d 

x = C.d 
C.d = "setting a value on class" 

Note that when accessed from the class itself, only the __get__() method comes in the picture, setting or deleting the attribute will actually replace or remove the descriptor.

Descriptors work only when attached to classes. Sticking a descriptor in an object that is not a class gives us nothing.

In the previous section we used a descriptor with both __get__() and __set__() methods. Such descriptors, by the way, are called data descriptors. Descriptors with only the __get__() method are somewhat weaker than their cousins, and called non-data descriptors.

Repeating our experiment, but this time with non-data descriptors, we get:

class GetonlyDesc(object):
    "Another useless descriptor"
    
    def __get__(self, obj, typ=None):
        pass

class C(object):
    "A class with a single descriptor"
    d = GetonlyDesc()
    
cobj = C()

x = cobj.d 
cobj.d = "setting a value" 
x = cobj.d 
del cobj.d 

x = C.d 
C.d = "setting a value on class" 

Interestingly, not having a __set__() affects not just attribute setting, but also retrieval. What is Python thinking? If on setting, the descriptor gets fired and puts the data somewhere, then it follows that the descriptor only knows how to get it back. Why even bother with the instance's __dict__?

Data descriptors are useful for providing full control over an attribute. This is what one usually wants for attributes used to store some piece of data. For example an attribute that gets transformed and saved somewhere on setting, would usually be reverse-transformed and returned when read. When you have a data descriptor, it controls all access (both read and write) to the attribute on an instance. Of course, you could still directly go to the class and replace the descriptor, but you can't do that from an instance of the class.

Non-data descriptors, in contrast, only provide a value when an instance itself does not have a value. So setting the attribute on an instance hides the descriptor. This is particularly useful in the case of functions (which are non-data descriptors) as it allows one to hide a function defined in the class by attaching one to an instance.

class C(object):
    def f(self):
        return "f defined in class"

cobj = C()

cobj.f() 

def another_f():
    return "another f"

cobj.f = another_f

cobj.f() 

This is the long version of the attribute access story, included just for the sake of completeness.

When retrieving an attribute from an object (print objectname.attrname) Python follows these steps:

Note that Python first checks for a data descriptor in the class (and its bases), then for the attribute in the object __dict__, and then for a non-data descriptor in the class (and its bases). These are points 2, 3 and 4 above.

The descriptor result above implies the result of calling the __get__() method of the descriptor with appropriate arguments. Also, checking a __dict__ for attrname means checking if __dict__["attrname"] exists.

Now, the steps Python follows when setting a user-defined attribute (objectname.attrname = something):

What happens when setting a Python-provided attribute depends on the attribute. Python may not even allow some attributes to be set. Deletion of attributes is very similar to setting as above.

Before you rush to the mall and get yourself some expensive descriptors, note that Python ships with some very useful ones that can be found by simply looking in the box.

class HidesA(object):

    def get_a(self):
        return self.b - 1

    def set_a(self, val):
        self.b = val + 1

    def del_a(self):
        del self.b

    a = property(get_a, set_a, del_a, "docstring") 


    def cls_method(cls):
        return "You called class %s" % cls

    clsMethod = classmethod(cls_method) 


    def stc_method():
        return "Unbindable!"

    stcMethod = staticmethod(stc_method) 

As we saw earlier, Python functions are descriptors too. They weren't descriptors in earlier versions of Python (as there were no descriptors at all), but now they fit nicely into a more generic mechanism.

A property is always a data-descriptor, but not all arguments are required when defining it.

class VariousProperties(object):

    def get_p(self):
        pass

    def set_p(self, val):
        pass

    def del_p(self):
        pass

    allOk = property(get_p, set_p, del_p)  

    unDeletable = property(get_p, set_p) 

    readOnly = property(get_p) 

The getter and setter functions need not be defined in the class itself, any function can be used. In any case, the functions will be called with the instance as the first argument. Note that where the functions are passed to the property constructor above, they are not bound functions anyway.

Another useful observation would be to note that subclassing the class and redefining the getter (or setter) functions is not going to change the property. The property object is holding on to the actual functions provided. When kicked, it is going to say "Hey, I'm holding this function I was given, I'll just call this and return the result.", and not "Hmm, let me look up the current class for a method called 'get_a' and then use that". If that is what one wants, then defining a new descriptor would be useful. How would it work? Let's say it is initialized with a string (i.e. the name of the method to call). On activation, it does a getattr() for the method name on the class, and use the method found. Simple!

Classmethods and staticmethods are non-data descriptors, and so can be hidden if an attribute with the same name is set directly on the instance. If you are rolling your own descriptor (and not using properties), it can be made read-only by giving it a __set__() method but raising AttributeError in the method. This is how a property behaves when it does not have a setter function.

Why do we need Method Resolution Order? Let's say:

There are messy solutions to this problem, and clean ones. Python, obviously, implements a clean one which is explained in the next section.

Let's say:

Using this technique, each method is called only once. It appears clean, but seems to require too much work. Fortunately for us, we neither have to implement find_out_whos_next() for each class, nor set the next_class_list, as Python does both of these things.

Python provides a class attribute __mro__ for each class, and a type called super. The __mro__ attribute is a tuple containing the class itself and all of its superclasses without duplicates in a predictable order. A super object is used in place of the find_out_whos_next() method.

class B(A): 
    def do_your_stuff(self):
        super(B, self).do_your_stuff() 
        # do stuff with self for B

If we're using a class method, we don't have an instance self to pass into the super call. Fortunately for us, super works even with a class as the second argument. Observe that above, super uses self only to get at self.__class__.__mro__. The class can be passed directly to super as shown below.

class A(object):
    @classmethod 
    def say_hello(cls):
        print 'A says hello'

class B(A):
    @classmethod
    def say_hello(cls):
        super(B, cls).say_hello() 
        print 'B says hello'

class C(A):
    @classmethod
    def say_hello(cls):
        super(C, cls).say_hello()
        print 'C says hello'

class D(B, C):
    @classmethod
    def say_hello(cls):
        super(D, cls).say_hello()
        print 'D says hello'

B.say_hello() 
D.say_hello() 

There is yet another way to use super:

class B(A):

    def do_your_stuff(self):
        self.__super.do_your_stuff()
        # do stuff with self for B

B._B__super = super(B) 

When created with only a type, the super instance behaves like a descriptor. This means (if d is an instance of D) that super(B).__get__(d) returns the same thing as super(B,d). In above, we munge an attribute name, similar to what Python does for names starting with double underscore inside the class. So this is accessible as self.__super within the body of the class. If we didn't use a class specific attribute name, accessing the attribute through the instance self might return an object defined in a subclass.

While using super we typically use only one super call in one method even if the class has multiple bases. Also, it is a good programming practice to use super instead of calling methods directly on a base class.

A possible pitfall appears if do_your_stuff() accepts different arguments for C and A. This is because, if we use super in B to call do_your_stuff() on the next class, we don't know if it is going to be called on A or C. If this scenario is unavoidable, a case specific solution might be required.

One question as of yet unanswered is how does Python determine the __mro__ for a type? A basic idea behind the algorithm is provided in this section. This is not essential for just using super, or reading following sections, so you can jump to the next section if you want.

Python determines the precedence of types (or the order in which they should be placed in any __mro__) from two kinds of constraints specified by the user:

In addition, to avoid being ambiguous, Python adheres to the following principle:

We can satisfy the constraints if we build the __mro__ for each new class C we introduce, such that:

Here the same problem is translated into a game. Consider a class hierarchy as follows:

Figure 2.2. A Simple Hierarchy

Since only single inheritance is in play, it is easy to find the __mro__ of these classes. Let's say we define a new class as class N(A,B,C). To compute the __mro__, consider a game using abacus style beads over a series of strings.

Figure 2.3. Beads on Strings - Unsolvable

Beads can move freely over the strings, but the strings cannot be cut or twisted. The strings from left to right contain beads in the order of __mro__ of each of the bases. The rightmost string contains one bead for each base, in the order the bases are specified in the class statement.

The objective is to line up beads in rows, so that each row contains beads with only one label (as done with the O bead in the diagram). Each string represents an ordering constraint, and if we can reach the goal, we would have an order that satisfies all constraints. We could then just read the labels off rows from the bottom up to get the __mro__ for N.

Unfortunately, we cannot solve this problem. The last two strings have C and B in different orders. However, if we change our class definition to class N(A,C,B), then we have some hope.

Figure 2.4. Beads on Strings - Solved

We just found out that N.__mro__ is (N,A,C,B,object) (note we inserted N at the head). The reader can try out this experiment in real Python (for the unsolvable case above, Python raises an exception). Observe that we even swapped the position of two strings, keeping the strings in the same order as the bases are specified in the class statement. The usefulness of this is seen later.

Sometimes, there might be more than one solution, as shown in the following figure.

Figure 2.5. Multiple Solutions

An alternative position for A and C is shown in grey. The order can be kept unambiguous (more correctly, monotonic) if the following policies are followed:

This, essentially, is the idea behind the algorithm used by Python to generate the __mro__ for any new type. The formal algorithm is formally explained elsewhere [mro-algorithm].

In Python, we can use methods with special name like __len__(), __str__() and __add__() to make objects convenient to use (for example, with the built-in functions len(), str() or with the '+' operator, etc.)

class C(object):
    def __len__(self):  
        return 0

cobj = C()

def mylen():
    return 1

cobj.__len__ = mylen 

print len(cobj) 

The same is true for all such methods, putting them on the instance we want to use them with does not work. If it did go to the instance then even something like str(C) (str of the class C) would go to C.__str__(), which is a method defined for an instance of C, and not C itself.

A simple technique to allow defining such methods for each instance separately is shown below.

class C(object):
    def __len__(self):
        return self._mylen() 

    def _mylen(self): 
        return 0

cobj = C()

def mylen():
    return 1

cobj._mylen = mylen 

print len(cobj) 

Subclassing built-in types is straightforward. Actually we have been doing it all along (whenever we subclass <type 'object'>). Some built-in types (types.FunctionType, for example) are not subclassable (not yet, at least). However, here we talk about subclassing <type 'list'>, <type 'tuple'> and other basic data types.

>>> class MyList(list): 
...     "A list that converts appended items to ints"
...     def append(self, item): 
...         list.append(self, int(item)) 
...
>>>
>>> l = MyList() 
>>> l.append(1.3) 
>>> l.append(444) 
>>> l
[1, 444] 
>>> len(l) 
2
>>> l[1] = 1.2 
>>> l
[1, 1.2]
>>> l.color = 'red' 

Basic lists do not have __dict__ (and so no user-defined attributes), but ours does. This is usually not a problem and may even be what we want. If we use a very large number of MyLists, however, we could optimize our program by telling Python not to create the __dict__ for instances of MyList.

class MyList(list):
    "A list subclass disallowing any user-defined attributes"
    
    __slots__ = [] 

ml = MyList()
ml.color = 'red' # raises exception! 

class MyListWithFewAttrs(list):
    "A list subclass allowing specific user-defined attributes"

    __slots__ = ['color'] 

mla = MyListWithFewAttrs()
mla.color = 'red' 
mla.weight = 50 # raises exception! 

The purpose and recommended use of __slots__ is for optimization. After a type is defined, its slots cannot be changed. Also, every subclass must define __slots__, otherwise its instances will end up having __dict__.

We can create a list even by instantiating it like any other type: list([1,2,3]). This means list.__init__() accepts the same argument (i.e. any iterable) and initializes a list. We can customize initialization in a subclass by redefining __init__() and upcalling __init__() on the base.

Tuples are immutable and different from lists. Once an instance is created, it cannot be changed. Note that the instance of a type already exists when __init__() is called (in fact the instance is passed as the first argument). The __new__() static method of a type is called to create an instance of the type. It is passed the type itself as the first argument, and passed through other initial arguments (similar to __init__()). We use this to customize immutable types like a tuple.

class MyList(list):

    def __init__(self, itr): 
        list.__init__(self, [int(x) for x in itr])

class MyTuple(tuple):
    
    def __new__(typ, itr): 
        seq = [int(x) for x in itr]
        return tuple.__new__(typ, seq) 

The __new__() method is not special to immutable types, it is used for all types. It is also converted to a static method automatically by Python (by virtue of its name).