Nbterm: Jupyter Notebooks in the Terminal
blog.jupyter.orgVery cool to see this – but will only be successful with great terminal plotting tools. The ones the author mentions like the matplotlib interface clearly won't do[0] due to lack of resolution
A perfect use case for unicode plotting [1] (shameless plug)
I spent a hot second last year thinking about sparklines in a terminal, and it seemed to me that unicode could use some more glyphs to improve this situation of treating a single glyph as a modest array of pixels. I don't know what I googled for, but for some reason I found only about 1/3rd of the characters listed in the final two lines of
http://tamivox.org/dave/boxchar/index.html
Unless I'm wrong about your demo, it seems like you are using some of these? There's a 2x2 pixel grid, and some characters that would be good for bar charts, both horizontal and vertical, and also with negative numbers. That's better than I thought we had, but not a big palette to work with.
Are you aware of any proposals to expand the pixel art capabilities of Unicode? Grids have the problem of requiring 2^(X x Y) characters to represent all states, but diacritical marks don't have that scaling problem. For example it would take 64 characters to represent a 2x3 grid of all on/off states, whereas a 6x8 grid of rectangular diacritical marks would only take 48 glyphs.
That's right, I'm using the 2x2 characters and some off-center horizontal line characters.
Note aware of Unicode expansions, though more and more fonts seem to support the above.
Sparklines in the terminal sounds very cool, did you end up coding something?
Note that one other popular alternative (see for example the Unicode plotting lib for Julia) is using Braille characters, which probably have an even better support across fonts
> Sparklines in the terminal sounds very cool, did you end up coding something?
It seemed to me that the existing tools were about as good as I was going to achieve on my own. I got distracted from my use case and will have to circle back, hopefully soon.
I am a big fan of https://github.com/Evizero/UnicodePlots.jl . It is unexpected how useful these tools can be to track the evolution of numerical experiments/developments right on the terminal.
The shameless plug is well justified. uniplot is seriously impressive!
Thanks! Do let me know if you are still looking for any particular feature
Do you know if Uniplot is able to handle pediatric growth charts ? (https://www.cdc.gov/growthcharts/index.htm) or generally, being able to plot on a background that is always used, instead of the (default) of a blank page ? such as : https://www.cdc.gov/growthcharts/data/who/GrChrt_Boys_24HdCi...
Thanks !
PS: Is the general case of a background image handled by Uniplot, and does Uniplot allow for the idea of taking a PDF, putting the plot data into it, and then creating a new PDF file ?
Thanks for the question! So the labels only depend on the limits of the plotting window, which you can supply as a keyword option. In that sense, and together with using the same gridlines options, you could have the same background in some sense.
What uniplot cannot do today is to have multiple vertical axis (like weight and head circumference in your example).
Hope this helps!
P.S.: Regarding your PDF question, for that in fact I think that some plotting library to graphics (rather than the terminal) might be better
So, the question isn't really about the axes. The question is about the curved lines that are part of the background. These lines allow a medical person to know what the "normal" curves are for a child, and then the plotted points are compared to how well they match the curves (by observation) This tells the medical person if the child is growing normally, or if they are malnourished or have some other problem that should be watched to explain the unusual lack or presence of growth.
Since the data points may be gathered at any time, the possible creation of a combined PDF will allow the medical record to include which chart was used and when, so that folks in the future have a better handle on the history of that child.
Gnuplotlib can do it just fine. https://github.com/dkogan/gnuplotlib/
Kind of agree, but I often use Jupyter just to capture the command history and its (textual) output. For someone like me this is still useful even without the graphical part of Jupyter.
Slightly tangent, but has anyone figured out a good solution for version controlling jupyter notebooks?
The closest thing that we've found has been to use the notebook percent format in a simple .py file [0][1]. It plays with git much nicer than an .ipynb and it is still interactive enough for rapid prototyping. However, it would be nice to have some first-class support from Jupyter on this.
[0] https://jupytext.readthedocs.io/en/latest/formats.html?highl...
[1] https://code.visualstudio.com/docs/python/jupyter-support-py
Until recently, I also thought this was a major problem. My old solution was to always pair .ipynb files with proper .py modules of the same name, so the .ipynb always starts with `%run foo.py` and just calls functions.
However, I recently started using VSCode Insiders the preview release of VSCode, which has amazing support for Jupyter notebooks in the editor. You can use your normally configured linters, auto-formatters, vi-mode keyboard shortcuts (major selling point for me). You even get legible, cell-aware diffs when comparing in git. Now, the edit/git workflow for .ipynb files is so close to parity I've stopped caring whether code is in a proper Python module or not, and I almost never run Jupyter Notebook or Jupyter Lab in the browser.
In some ways this is a loss, because it's nice to have project-wide linting and other tooling that only work with .py files, but using `%run` was always an imperfect abstraction. By default, `%run` executes modules in a new module namespace which is then copied over, so it's not exactly "paste into Jupyter" unless you do `%run -i`. Even then, it's limited by running all at once. Every cell in a typical Notebook is effectively a button that runs an `exec()` statement, and you can't achieve those semantics by calling Python functions.
I've been using the VSCode Insider's release as well and have been loving it and the new native notebook features for all the reasons you've listed already.
That's an interesting solution. I believe this is similar to what Joel Grus does [0], except %s/jupyter/ipython.
Forgive the snark because my suggestion is obviously not an improvement or even a match for target audience, but org-mode files with inline org-babel code-blocks is what I consider to be perfect version controlled notebook.
Pity Emacs is not the best on-boarding experience.
Worth noting that org files can also embed images inline. Unfortunately, I don't think it's enough to attract new users: images are non-interactive, you can't embed any other media, you have to pop a separate window to edit code, you can't embed rendered markdown (ie. headers and paragraphs have mostly the same font style/size), and so on. Sure org, babel, and calc give you a lot of other things to like, but that's if you're already an Emacs user.
Here[0] is a guide that explains syncing ipynb <> py files with Jupytext. I also add ipynb to `.gitignore`. It works well, although the file browser in Jupyter becomes cluttered with every notebook file being doubled. It'd be great to hide the underlying py files.
[0] https://github.com/mwouts/jupytext/blob/master/docs/paired-n...
So far, we try to just use Jupyter notebooks for experimentation, scratch, and poking around.
Real, working code gets checked in to a repository. The only reason to go back to an old notebook after that point is maybe to see how you may have experimented with or poked at some data.
Shameless plug, but with Nextjournal you can use git directly: https://github.nextjournal.com/
And apart from that, we have a normal github component to load your code from a github repository: https://nextjournal.com/help/github
Could you share some of your experience with nbdev? I'm a huge fan of what the fastai team has been doing and I've tried nbdev, but I haven't been convinced yet. Particularly with the pull request experience, it's not very easy to do code reviews.
FWIW my team uses bitbucket and the PR experience is significantly worse than github/gitlab unfortunately.
I just keep them as .ipynb files and then use git's filter and smudge features together with nbconvert's "clear notebook output" preprocessing, ensuring only clean notebooks get added/diff'd/committed.
I think jupytext is already as close to "first-class" support as you're going to get. Personally, I'd be happy to see the project included with Jupyter, but you'll have to pester the Jupyter devs for that ;-)
I would love to see that as well, I'm wondering what has stopped them from integrating it already... Maybe there's room for some contributions from the community here :)
You could try a preprocessing step of converting them to markdown with Pandoc or nbconvert. In Git, you can configure custom diff tools for certain file formats.
You can configure so they don't save their output, then the ipynb diffs will be readable.
Makes me wonder if you could split an ipynb into an in and out file, then add /*.out to .gitignore.
I believe I achieved the no-saving-output affect by adding a python snippet/plugin to Jupyter Lab. So you could program it to do whatever you want. That's what I love about Jupyter Lab, you can turn it into whatever kind of environment you want.
I like this idea, but seems a little backwards. Normally you commit the _source_ and omit the _artifacts_ haha.
That's what they are saying? Doesn't seem backwards to me.
Oh, you may be right. I interpreted it as having a separate build step to generate the *.out files.
There is also Emacs Jupyter if you use Emacs, and all sorts of `org-babel` integrations if you use org-mode.
If you use vim, you can also try https://github.com/hanschen/vim-ipython-cell . It works quite well
If you use emacs, ein is also a good choice: https://github.com/millejoh/emacs-ipython-notebook
I write my code in Python or Hy files and then call functions from the notebook. Makes the editing feedback loop much easier.
Ayyy, first person I see using Hy in the wild. Have been interested for long time to use it as a stepping stone into lisp. But now that I know about Julia, that is a much more likely candidate (or a lispy dialect thereof). The semantics and metaprogramming abilities mirror lisp much more closely.
How have Hy been treating you?
You might also be interested in Nextjournal, it's in clojure (whose notation you should be familiar). (check also https://github.nextjournal.com for a quick test from github code)
Ok, I failed in basic usage: Entering nbterm, I assumed some standard key bindings -- None seemed to work. Neither Ctrl-C, Ctrl-D, Ctrl-x ctrl-c, escape q, escape :q, ... seems to work. So off to google: Escape Ctrl-q should work. But does not.
Ok I'm stuck, let's at least get 1+1 to work. So entering 1+1 and .. hitting Enter? Shift Enter? No. Ok, command mode, Ctrl E.. aaaaand it's stuck.
Looks nice but I just don't seem to get it to work using anaconda on macos..
The project's README might help.
Yes, it would help if those key combinations would work (even though they are still at least unusual). The quit combination does not work and neither does evaluating a cell. Does anybody have luck actually getting it to work?
Edit: The issue is using a new environment without ipykernel installed (not a dependency of the conda package). With ipykernel, I can exit nbterm and even evaluate cells
To be fair the poster you’re replying to seems to have run the correct commands indicated by the README, unless I’m missing the same thing they are.
Thanks, just what I was looking for.
I used to use a matplotlib backend that worked with ITerm2 for coding in a remote server (standard technique, that I also cover in my Hy language book). Adding Nbterm would also be useful. Off topic, but it is sometimes just (much) better to do development on a remote server with much more compute and faster Internet connections.
I'm curious, what version of Hy are you targeting in your book?
The latest public version (pip install Hy).
I don’t generally install from the GitHub repo. I know that Hy is no big deal, just a Lisp skin on top of Python, but I really enjoy using it. Hy is an example of “done” software: it was written years ago, and just does what it is meant to. I have noticed comments on Reddit about Hy not being actively maintained, but as I said, it is “done.”
In my field, I hear the word "legacy" used for applications that just work, and don't require a lot of tweaks because they handle the use cases that come up commonly.
The web interface was the main reason I never really used the tool for more than a few test runs. Writing code in the browser combines bad usability and performance with a lack of useful dev tools.
Which brings me to the question whether I can edit the code in a text editor as well or I have to write it as shown in the animation. It would be really helpful if there was a shortcut to open the current (code) section in $EDITOR, kind of like Git and other tools do it.
Vscode and intelij both have had good support for notebook for a long time. For the web interface there maybe even extension to add intellisense to it or smt
For editing notebooks in vim, I've created https://github.com/goerz/jupytext.vim. Note that this does not allow to run any cells, it just edits the inputs.
The motivation behind this was to have some basic interaction with existing ipynb files on a remote server without having to run the jupyter server (and set up port forwarding etc.) It's worth noting that the `jupytext.vim plugin is most useful if you're actually not running `jupytext` within jupyter; If you are, you could just directly open the .py or .md files linked to any .ipynb in your editor.
I've used `jupytext.vim` to edit existing notebooks and then run them through `jupyter nbconvert --to notebook --execute`. It's also great for refactoring: moving code from a notebook files into a module, between notebooks, or to create a new notebook as a variation of an existing one.
One way I've been exploring, is to use nbconvert [1] to convert notebooks to python scripts and back.
Have you found any Jupyter notebooks that don't convert easily ?
Use the qt-console, best of both worlds.
qt-console is great, but it's not a notebook, ie. you don't have cells you can go back to and edit. It's more of a classic console, with great readline emulation and ability to display images easily.
I used nteract which is like a bit like a notepad for ipnyb files. It is not great software, but it works.
There is the %edit magic
I found that this is almost completely non-working. Ctrl+q doesn't quit, cells don't run. Does it work for others?
It also unfortunately uses shortcuts that are native on Mac (ctrl+up and down do "mission control" things).
Why would one need this when you already have ipython?
IPython doesn’t help you view, edit, or execute .ipynb files.
Afaik IPython is Jupyter's predecessor and superseded by Jupyter (which provides the notebook abstraction for different languages).
IPython was in the beginning Python specific, though its heritage lives on in the new name (the 'py').
Alright. Now get rid of the stupid ipynb format and store the notebooks as God intended (runnable Python files with comments).
You may already be aware of it, but jupytext solves most of the problems ipynb files cause.
Indeed, at which stage one might just use vim or emacs and skip this notebook thing.
But presentations would be less glossy and look like less work ...
We've come full circle
What I thought when I saw the headline; ipynb has since matured so much beyond ipython into its own workflow and ecosystem it makes sense to port it back to the cli.
Would be interesting to spin this into a normal bash terminal where you can edit text in the “cell” like in GUI editor and then execute the cell at the end
VS Code has been making progress to support Jupyter notebooks (https://code.visualstudio.com/docs/python/jupyter-support), but it's not quite there yet. Code completion works, but other features such as user snippets are missing, and it can be quite laggy. Apparently that's because the VS Code team has to reimplement a lot of stuff for Jupyter specifically (compared to adding support for Rust or Go with a language server).
VSCode Insiders has huge improvements in this area. I switched to Insiders specifically for this and it’s great. You get linting, auto-formatting, vim-mode keyboard input, and even git diff for .ipynb. I don’t run Jupyter’s browser UI at all now.
For those that don't know, VSCode Insiders seems to be some beta/alpha channel of releases with features that are under-development. So I guess that means it will end up in VSCode stable in a little bit.
My team is doing a lot of cli-based hacking on notebooks. They’re an interesting developing format.
I would love early HN feedback on Notebook Ninja, which is (pre-)alpha here: https://notebook.ninja
What features does Noteback Ninja have that aren't present in the nbterm system ?
Great question. Right now Ninja is web-only for the public. Internally it's a CLI (releasing soon).
nbterm is "notebooks in the terminal", Ninja is "materialized notebook executions". So, for example, whereas nbterm will display cell outputs, ninja will materialize them to disk and make them available via (eg) HTTP.
Whereas nbterm is a CLI-based notebook development environment, Ninja is a notebook production execution environment.
Ninja continues the thought from "how do I build notebooks better" to "how do I get into production".
So Notebook Ninja has a flag that means write the notebook and its calculated cells to an html page or some other form that would capture a snapshot at a particular step in the process, and thus be suitable to be served by a web server without being "re-executed" ?
Why doesn't nbterm allow for production execution of code ? I thought that was the point of its existence, just that they are done in an terminal window rather than a web window.
In essence, yes. Ninja is based around the idea of reproducable science: a given execution of a notebook is immutable and accessible consistently. It writes out cell outputs depending on their mimetype (this is included in Jupyter Notebook metadata).
It writes out all declared variables, per-cell, and makes them accessible in several formats (for pre-alpha, as JSON objects).
It then exposes all of these via our web servers, so it's really easy to connect them to other services.
Or, for example, generate a permanent link to the execution of a particular cell.
I'm glad to see that you can link to the execution of a particular cell. Just to clarify, if the cell had the execution for the Age of a particular person, I don't know if the link to the cell is of the age of that person as of the particular day and time when it was executed, or if is stored in an unevaluated form, the execution would yield an age dependent upon the calculation and using "Now" as the value that is compared to the Date of Birth to find out what the age may be. (IE: is the "NOW" evaluated lazily to the date+time of execution, and thus the age will change as time passes) OR (is the "NOW" evaluated eagerly to the date+time of when the execution first occurs, and the value of the age of the person as of that date+time is stored unchanging as the value of the AGE cell ?
It's evaluated eagerly. The notebook is executed and materialized at upload.
We're enabling re-triggering notebooks soon, which will allow things like scheduled and parameterized executions, and those will be eagerly evaluated as well.