PyPy, Jython and CPython 3 on dotCloud
blog.dotcloud.comWhat is the typical use case for Jython? Is it so that people can run Python at companies that only run Java? In that use case, I'm assuming people are creating a python app, and then compiling a War, and then deploying on Tomcat or Jetty?
I can think about the power of Python scripting + the rich world of Java libraries/frameworks.
My main use is for web scraping + javascript support, since I love Python and I like the HTMLUnit headless browser written in Java. I've written two articles using it:
i) http://blog.databigbang.com/web-scraping-ajax-and-javascript...
ii) http://blog.databigbang.com/automated-browserless-oauth-auth...
Yup, banks don't like Python, but Java is OK. So you write your Java in Jython, package it up & deploy.
Here's another use case. At Repustate, we have an API to create PowerPoint presentations via an API call. We use Apache POI (Java) to create & manage PowerPoint slides, but our code base is all Python. So we have a Jython process running on Jetty, but instrument all of the Java bits with Python.
It's pretty amazing to have all of the power of the JVM at your disposal but with the grace of Python.
It's been a while, but I recall a Java shop where the (complicated) build process was written in Jython.
yeah that is pretty cool.
This and if you have a large (sometimes legacy) JVM-based (i.e. not just Java, but Scala or Clojure as well) code base that you want to integrate with Python: Hadoop and Weka come to mind.
I see. Can you access/call the java/scala/closure classes from inside of your python code? If so, that is pretty cool.
I know Quora hired Alex Gaynor to move them to PyPy last summer. Is anyone else actually using non-CPython 2.x in the real world?
I've run one-off internal tasks with it, but for production, our CPU load on CPython is still low enough I can chant "it ain't broke" and do other things.
I have tested our codebase on it to make sure we can deploy if we need to, though.
Did you have any issues running it on PyPy? I know when I tried to run it, I had an issue with one of my requirements not being supported under PyPy.
We have an indirect, semi-optional, largely accidental dependency on eventlet/gevent which won't work in PyPy, but it's easy to just rip out if we go to production with PyPy. We should rip it out anyway, but there's no reason to spend the two hours on it right now.
What would you deploy with? I have gotten it to work with Gunicorn, but if you can't use eventlet/gevent, that doesn't look like a great option. sync mode works aright, but not as good as async mode.
Now that I know about Gunicorn (thanks!) I might deploy with that. I was just going to use multiprocessing + wsgiref.
We don't get significant benefits from async with our current codebase and load characteristics -- like I said, the dependency was accidental.
I don't know about eventlet, but gevent appears to be working in pypy (for some trivial example code from the gevent page).
Have you tried your application since greenlet got built-in to pypy stable?
I hadn't heard of dotCloud before. From their web page, it looks like they might really simplify AWS deployment. Can any current users comment on how it compares to going through Amazon?
The pricing page doesn't indicate how much storage or processing power you get for each price plan. I'm also wondering if they allocate enough resources for a site that does scientific computing on the back end.
I've had a bunch of weirdness deploying a Flask app to the service.
Defining requirements in a yml is nice for auto-install, but the scripts tend to break for me worse than "hand installing" via pip.
The reason I'm moving the app to a micro EC2 instance is that dotCloud actively masks the debug interface, so I can't exactly tell why my app is breaking on their servers, but not my local machine. Plus the fact that dotCloud/Heroku is largely overkill for small, team-focused apps. I can't really justify dropping 99/mo for something only 50 people will be using.
Otherwise, it was an incredibly smooth experience.
EDIT: requirements.txt, not YML
Sorry to hear things that you're having trouble with dotCloud. A couple quick questions:
First, which requirements are you putting in your dotcloud.yml? In general, pip-installable requirements should just go in a standard requirements.txt file, and dotcloud.yml should be used to describe the higher-level structure of your application (ie which service types you need).
Also, I'm not sure what you mean about "masking the debug interface", but all your Flask/WSGI and nginx logs are available with the "dotcloud logs" command, and you can also log into your service instance directly to debug using "dotcloud ssh" (in which case logs can be found in the /var/log directory). You can also certainly use Flask in debug mode, by setting "application.debug = True" in your wsgi.py
It is true that you may have trouble using the interactive, browser-based debugger with the default setup on dotCloud, then but this is a Flask limitation, not a dotCloud one. From http://flask.pocoo.org/docs/quickstart/#debug-mode: "the interactive debugger does not work in forking environments (which makes it nearly impossible to use on production servers)"
Hope that helps, and if you have any other questions, drop us a line: support@dotcloud.com
Thanks for the response! I had to facepalm at the WSGI config, I had been setting it within the app itself: it returned a 500 otherwise, which makes sense now.
Consider my one issue moot! Next time I'll read the docs better before complaining...
Awesome!
Any docs changes we could make to avoid others hitting the same problem?
Nope. The relevant info is literally the first warning callout on the Python guide: http://docs.dotcloud.com/services/python/
Thanks again for taking the time to set me straight on this.
Thanks for the feedback! A few improvements that might make your life easier since you last tried: dotCloud fully supports plain old requirements.txt (consumed by plain old pip). You can access all application logs (the less masking the better as far as we're considered).
Pricing is still the same and admittedly not perfect for small-ish projects on a budget. Our delibarate strategy is to charge businesses for a high-value service (and in our experience they gladly pay for it!), then find ways to subsidize hackers, students and bootstrapped startups with something free or cheap that doesn't cause us to bleed money. That is, in my view, the only way to truly help hackers in the long run - what good is our free service if it gets snatched up and digested by another unrelated business within the year?
dotCloud definitely simplifies deployment over raw AWS, but does cost a bit more on a per-resource-unit basis (understandable, given the platform's value is primarily in the time-savings). dotCloud's focus so far has been on network-based apps (which, in practice means mostly web apps), but there have been people running some big-data/scientific workloads.
As to whether it makes sense for scientific computing, that depends a lot on the type of computation. On the upside, the vast majority of current users are memory-bound, so there is a fair bit of idle CPU sitting around, but even with bursting into other users' unused CPU allocations, if you really need the absolute-highest CPU throughput, you may have to run your own machines.
The resources are sold in "scaling-unit" bundles, which essentially comprise 300MB (burstable) RAM and 10GB (soft limit) disk, with a "reasonable" amount of CPU and bandwidth usage ("reasonable" essentially means unmetered, but monitored for abuse).
That being said, if you have really unusual requirements, drop an email to support@dotcloud.com, and I'm sure we can work something out.
full disclosure: I work at dotCloud
In my experience they have shitty customer service. I wrote them an email a couple of times and never got a reply back.
I'm really surprised and sorry to hear that. We usually have pretty prompt support replies, and generally give really thorough answers, since support is mostly handled by core engineers (we have an engineering support rotation)
I'll admit that the one downside of having an engineering support rotation is that the quality of the written English does vary a bit from day to day because some of our engineers aren't native English speakers. But I'd much rather have a thorough and technically accurate (if somewhat poorly written) support response than a well-written but technically inept one.