Python Language Features and Tricks
sahandsaba.comWhy are there so many negative comments? Maybe those posters are vastly underestimating how many people that just start out read HN. I think it's a pretty good post to read after something like "X in Y minutes - Python" to get a very quick grasp of what the language is like.
I'm also not ashamed to say that despite having written quite a few LOC of Python I wasn't aware of named slices for some reason and I think they can clear up some chunks of code I have produced (make it more readable)
I've been programming in Python for about 3 years and learned a couple of tricks from this post, which was why I submitted it.
There's always tricks to learn, and it's hard to remember everything when you're first starting out with a new language.
Extended unpacking was a new one on me. Kind of like a simple version of pattern matching. I think this one will prove useful!
(I've been using python for many years, but not full time. It's the language I use for one-off scripts, small tools that make use of its handy standard library, or prototypes for things I'm going to write in some other language. That's my excuse for the gaps in my knowledge.)
I've been using python since the 90s, and I learned a few things from this post.
Right after this string of replies, I was expecting Guido van Rossum would be next, saying he created Python and even he learnt something new...
> ... I wasn't aware of named slices for some reason ....
I wasn't either. But I wouldn't call them a hidden feature of the language. A slice object is an object. So you can make it the value of a named variable, just like any other object. Somehow that wasn't obvious, though.
Now I'm wondering how else I might be able to improve my code by giving names to non-obvious things.
Same here, learned something new today.
A common theme running through named tuples, itertools and collections is Raymond Hettinger. He has also put together lots of tutorials and talks.
Great suggestion. Just searched for him on youtube and a couple of excellent looking talks popped up. Halfway through and this one seems great so far: https://www.youtube.com/watch?v=OSGv2VnC0go
Really enjoy his style as well :)
Thank you sir, something else to go through on a quiet Sunday :).
Python's unpacking is a poor man's pattern matching. I'd really love to see them extend it to support user-defined patterns like Scala's extractors or F#'s active patterns.
I think unpacking is just unpacking, languages that have pattern matching allow you to do something identical:
# let i = (1,2,3) ;;
val i : int * int * int = (1, 2, 3)
# let a,b,c = i;;
val a : int = 1
val b : int = 2
val c : int = 3
Unpacking pretty much does one thing, and it makes that one thing easier and a lot more readable. Combine that with list comprehension (from what I understand C#'s LINQ is similar) and you end up with code that's highly maintainable/readable (as long as you name your variables appropriately, of course).
I believe Python supports pattern matching other than Regex as well.
Python list comprehensions are not lazy, though. If you generate [for x in xrange(1, 10000)] you'll get 10000 elements in your list. My understanding is that LINQ list comprehensions are that.
> I believe Python supports pattern matching other than Regex as well.
That's pattern matching on strings. The kind of pattern matching being discussed here is pattern matching on data types (see [1]).
Python does have generator expressions, for instance what you wrote above is written (lazily) as:(x for x in xrange(1, 10000))
It doesnt have functional style pattern matching though.
'Lazy lists' are called generators in Python. Just use (...) Instead of [...] in your example.
I'm aware of generator expressions, but the fact that list comprehensions are not lazy can trip people if they are used to other languages with this feature, since they are usually lazy.
Reading through http://en.wikipedia.org/wiki/List_comprehension, it seems that there are roughly equal numbers of languages where "list comprehension" produces strict list as languages where it produces lazy lists.
I think it's a lack of naming convention - the word "list" on its own means "strict list" in some languages and "lazy list" in others.
Fair enough, I retract my statement :) Maybe people are not surprised after all.
I don't agree, people are more likely to come from python that to python from a language that has lazy list comprehensions. But regardless, using a language requires that you learn it and I doubt many people think list comp is lazy. Pythonic programming obeys explicit over implicit, which which if you want a lazy version you explicitly use a lazy version.
> Pythonic programming obeys explicit over implicit, which which if you want a lazy version you explicitly use a lazy version.
What does this have to do with anything? If the construct was defined as lazy, then it would be just as explicit. There is nothing about generator expressions that says "here, here, this is lazy".
I've been using Python and Ruby on and off for a couple years (largely because I haven't found the need to use it seriously day job or side projects).
One thing that strikes odd for me is how people describe Python/Ruby are way more readable than Java.
I felt that Python, while more readable than Ruby (because Python uses less symbols), still contain more nifty tricks compare to Java.
It's true that the resulting code is less code but behind that less line of code bugs might linger around because there might be plenty "intents" being hidden deep in the implementation of Python.
The Python way that is touted many times is "explicit is better than implicit" seems to correlate better with the much maligned "Java is too verbose".
Anyhow, the other day I was refreshing my Python skill and learned the default implicit methods that I can override ( those eq, gte, gt, lte, lt) and I wonder how overriding those resulted in less lines of code compare to Java overriding equals, hashCode, and implementing one Comparator method than can return -1, 0, 1 to cover the whole spectrum of gte, gt, lte, (and even equality, given the context).
I suppose everything is relative...
In isolated offline code, I think Ruby approaches the unreadable - but only because DSLs are fashionable, and you have to understand the library involved to understand the magic that's going on behind the scenes. With sufficient online searching, you can understand things at a surface level, and with enough time with a debugger you can appreciate all the mechanics that are happening, but IMHO it still leaves you with an unpleasant taste in your mouth at the sheer quantity of magic.
I do prefer Ruby to Python though. I much prefer monadic enumerators combined with lambdas to list comprehensions, even though these tend to be eager in Ruby but have the option of being lazy in Python. Itertools helps, but is deeply hampered by Python's overly verbose lambda syntax.
Been thinking about Python vs Ruby lately. I dread reading other people's Ruby code (eg. if there is a problem in a library I'm using) but I don't have that same dread reading Python code. Now if Python had blocks and RubyGems then I would never give Ruby a second look.
What does RubyGems provide that isn't covered by pypy/eggs? I thought ruby and python packaging was approximated feature parity here (which is to say, both leave something to be desired, but work quite well).
But that's the thing y'know... It's not just about DSL but it's also Ruby's syntax that include lots of symbols (of course DSL made the learning curve increase as well).
By symbol I mean the use of tilda, pound sign, ampersand colon.
Seems to me that it's more of personal preference than anything else and this is why for me personally, Ruby is pretty close to Perl.
Somehow I prefer verbose than too terse. I prefer to read the code in words than in mix of words and symbols.
The comparison functions are there to address the comparison for equality of things that don't have an order. So in some sense, they are more explicit (a minute with the docs explains that implementations only need to directly define 2 of them). -1,0,1 style comparisons still exist in Python 2.7, they were dropped in 3.0. Probably one of those things that could have gone either way.
I think comparisons about readability aren't very useful (they are subjective, 'enough' counts), but I sure like the relative terseness of Python (to the point that I lament people who come over and write Java in Python (I don't mean to direct that at you, it's just a thing that happens, where people don't take advantage of things that are very idiomatic and thus clear even when terse)).
Fair point and I learned early on during my college years not to import habit from one language to the other one. Mainly because "idiomatic" tend to depend on the internal implementation of the language.
The language has expanded a lot recently (2.x) series. It is no longer the language it once was. It is more advanced and not as svelte.
There's similar thing re eq/gte/etc for Python. Particulars escape me at the moment.
It's important to remember that OrderedDict keeps insertion order, it isn't an implementation of a sorted dictionary.
zip to unzip a dict is a very slow approach to do it
Instead of
mi = dict(zip(m.values(), m.keys()))
mi = {v: k for (k, v) in m.iteritems()}When did support for dictionary comprehensions make it into 2.x? I could've sworn it didn't used to work, but I just tried it in the shell and sure enough, it does in 2.7.3.
Wow I didn't know that! I was always doing something like this
mi = dict((v,k) for k,v in m.iteritems())
more importantly it's wrong - the iteration order of .values() and .keys() is not guaranteed to be consistent
The order of items in the dictionary isn't stable, but the order of .keys() is guaranteed to be the same as the order of .values() (as long as you don't modify the dict in between calling one then the other).
I have two questions.
1. I'm unfamiliar with the term 'unpacking'. Is it any different from pattern matching in, say, Haskell (but perhaps not as feature-rich)?
2. Aren't slices pretty much a staple in Python? I didn't think using them was considered a 'trick'.
Unpacking is a limited form of what is called destructuring in other languages like Clojure. I would say that, in terms of feature-richness: unpacking < destructuring < pattern matching.
I'd extend that one step further
where first-class patterns are increasingly becoming available in some languages which offer pattern matching (in particular, Haskell will have them soon).unpacking < destructuring < pattern matching < first-class patternsI did a little googling, but am finding it difficult to find good clear information - do you have any articles where I can read about first-class patterns?
https://ghc.haskell.org/trac/ghc/wiki/ViewPatterns
That's the haskell extension.
To see a very nice use of them, check out this paper (pdf) http://strictlypositive.org/CJ.pdf
How do view patterns make patterns "first-class"? To me that means able to manipulate them as values, I don't see how view patterns allow that, they're just syntactic sugar for case expressions.
http://www.reddit.com/r/haskell/comments/1vpaey/pattern_syno...
I spoke too eagerly—the new feature is just named and namespaced patterns. It's a bit of a bump in power, but it's not fully general yet.
For true(-ish) first-class patterns take a look at Prisms in the lens package or some of the other first-class pattern libraries.
I don't know Clojure, but I do know Python, and I'd like to say that unpacking is more flexible than you may realize. For example, you can do this:
a,(b,c),d = [1,[2,3],4]I don't mean to belittle Python here, I think its a great language and I find its unpacking useful. I'm only trying to demonstrate that the concept can be (and is in some other languages, like Clojure) taken further to make it more useful still.
I think that Python's unpacking allows most, if not all, of Clojures sequence destructuring for tuples and lists. Clojure takes it a bit further, however, by applying it to all sequences. For example, you could do this:
because strings are also sequences. You can also do something like this (excuse the awkward syntax as I try to express it in pseudo-Python):a,b,c = "XYZ"
This might not be so useful in python, since the list already is d and c is simply slicing the end from the list, but Clojure allows you to destructure function arguments: (defn foo [[a, b & c :as d]] ...) when passed the above list (foo [1 2 3 4 5]) would bind the variables as shown in the above comments.a,b : c as d = [1,2,3,4,5] # a = 1 # b = 2 # c = [3, 4, 5] # d = [1, 2, 3, 4, 5]Where destructuring really shines, though, is that you can destructure maps (dictionaries in Python) and vectors can also be treated as maps (their keys are the indices), so you can do stuff like this:
http://clojure.org/special_forms#binding-formsa, {[b {:keys [c, d]} :foo}, e = [1, {foo: [2, {c: 3, d: 4}, bar: 9}, 5] # a = 1 # b = 2 # c = 3 # d = 4 # e = 5neat!
Tuple unpacking:
Using slices is normal with listst = (1, 2) a, b = t # a == 1 and b == 2Cool like this,hope implement in php. No need to explode
PHP has the list construct which is a little uglier, and has a couple of odd behaviors
php > $t = array(1, 2); php > list($a, $b) = $t; php > echo "$a, $b"; 1, 2
Coming from a long history of languages like BASIC and Pascal, I will bookmark this tutorial. It seems to open up a lot of interesting Python features that were, quite frankly, not always easy to understand when described in plain text, but now seem pretty simple when presented as examples.
I'll also think about the "collection of simple examples" next time I want to document something.
I used to bookmark tutorials like this, but now ...
I just bookmark the HN discussion. Not only does that retain a link to the tutorial, but it also retains a link to a discussion that usually adds value to the tutorial.
I'm assuming that HN discussions remain available for many years. Anyone know for sure if that is/isn't true?
Instapaper has become my bookmarking tool. I don't even really sort it out anymore. I know it should be there when I need it and mostly the actual article is saved.
I hope so.
Nice reference.
1.29 happened to be exactly what I was looking for:
for subset in itertools.chain(*(itertools.combinations(a, n) for n in range(len(a) + 1)))
You can also do
itertools.chain.from_iterable(itertools.combinations(a, n) for n in range(len(a) + 1))
Slice has always been a painful adventure for me. I always forget that [1:3] is not all inclusive. It's actually just range from 1 to 2.
I believe in 2.7 zip is still returning a list rather than an iterator (izip in Python 2, zip in Python 3+).
Another unappreciated stdlib is definitely functools. mock is also another awesome stdlib.
functools, collections and itertools are definitely useful to make things faster. Also check out the list of stdlib. http://docs.python.org/2/library/
> I always forget that [1:3] is not all inclusive. It's actually just range from 1 to 2.
Just remember that for positive indices, len(slice(n,m)) == m - n
Slices make sense when you realize that [n:n] is always empty, and [:n] + [n:] is always the whole list. From these two identities, all else follows.
Great list, I do several mini-tutorials of python at http://runnable.com/u/mamcx. I try to pick several tricks for each theme
I think this is great, I've been doing Python for a while and I knew many of the features but I also learned a few new ones.
I don't understand how this one to flatten lists works:
a = [[1, 2], [3, 4], [5, 6]]
[x for l in a for x in l]
So that flattening can also be written as:
More perversely, we can also flatten with my invention:x = [[1,2], [3,4], [5,6]] [x for x in x for x in x]
More reasonably, of course,list(None for x in x if (yield from x) and False) # Python >=3.3list(itertools.chain.from_iterable(x))The expression is a list comprehension with 2 nested for statements. It is similar to this, which names its result:
result = [] for l in a: for x in l: result.append(x)Yes. And the order of the two for's in the list comprehension was deliberately kept the same as the order of the two for loops in your explicit code, on purpose, for ease of remembering how the former (i.e. list comp) works.
That order is the part that confused me. I expected to read it right-to-left, instead it's left to right for the for statements, then the expression on the left at the end.
I see now that the Python docs explain this very clearly...
Yes. Meant to say that but forgot - that the Python docs explain it. That's where I read it myself :)
A good reference, to be sure, but man, do I resent the term “trick” in programming. It implies a deception, or something clever that you wouldn’t think to look for, like opening a wine bottle with a shoe. These aren’t tricks, they’re (largely) standard library features that you would simply expect to exist. But maybe I’m underestimating the NIH effect.
it's amazing how much work and effort almost any of this examples would take to implement in C
The fair comparison is probably to some data structure (list/array/whatever makes sense) and its rich companion library.
This is awesome, I've been programming python for about 8 years now and a lot of these still surprised me.
Same as php.keep upgrading.
I know bugger all Python, but I know negative indexing.
Good for you. Did you know everything else?
> Good for you. Did you know everything else?
Obviously not, after all I did say "I know bugger all Python".
I did know about slices.
patterns / tricks = language deficiencies
Wake me up when Python will support tail call elimination and will get rid of GIL. For now this language is no better than PHP.
For TRE: http://neopythonic.blogspot.com/2009/04/tail-recursion-elimi...
For the GIL, there are alternative implementations of Python (Jython, IronPython, etc.)
Hm... Python only supports 1000 recursions? That seems unsafe. It seems like anyone who writes functional-style code will run the risk of a stack overflow.
Believe it or not, it's actually deliberate. And not that uncommon in other languages too. The only way you can realistially have 'unlimited' recursion is to use tail call optimisation, which is deliberaterly not implemented in python:
http://neopythonic.blogspot.co.uk/2009/04/final-words-on-tai...
Is Guido's take on it.
Essentially, as I understand it, his take is that massive recursion is confusing, and 'unpythonic'. And with the whole 'explicit is better than implicit' thing, I guess there is a point.
Some times recursion is the best/most obvious solution, and then it is a bit annoying to not have TCO, but you can usually work around it.
You can increase the recursion limit like this:
import sys
sys.setrecursionlimit(x)
Kind of a "hackish" way to do things but you can do it if you need to.
^Although I think the idea is that you shouldn't need to. Any recursive implementation can either be easily translated to work iteratively, or can be implemented in such a way that 1000 levels of recursion should be more than enough.
For example, 1000 levels of recursion is more than enough to count all the nodes in a binary tree unless the tree is extremely poorly balanced or inordinately large.
No better than PHP? That's more than a little ridiculous. Every language has shortcomings, and there's no silver bullet/perfect multitool that can or should be used for everything.
It's too bad it's apparently not good for... whatever it is you do. It is pretty great for some things, and having spent more than a few years with both Python and PHP, I can say without reservation that there is absolutely no comparison in terms of language quality.
I suggest you are suffering from the Blub Paradox[0][1][2].
[0] http://paulgraham.com/avg.html
[1] http://c2.com/cgi/wiki?BlubParadox
[2] http://weblog.raganwald.com/2006/10/are-we-blub-programmers....
> I suggest you are suffering from the Blub Paradox[0][1][2].
Writing some C will cure that disease, but the treatment has possible side effects, like carpal tunnel syndrome.
P.G. is right about PHP. This language is better than Python in various ways. For ex. it does not scare people away if you want to sell your project, because everyone and their grandma knows PHP. Most successful web pages (eg. FB) started out with PHP, not Lisp, C++, Python, Java, but PHP.
Every language has patterns or tricks, and every language has deficiencies, but these aren't the same thing at all.
It seems you have read some generic criticism of Python to copy from, and don't have much first-hand experience
There are only handful of language runtimes that got rid of GIL entirely. Namely JVM, .NET, and Rubinius. Other languages either use non-native threads or doesn't support threading at all. (E.g. Nodejs has no threading.)