Clarifying exceptions and visualizing tensor operations in deep learning code
explained.aiBe sure to read the last section - really deep work involved, including writing a Python parser! Here's a snippet:
"I used the built-in Python tokenizer, but built my own parser for the subset of statements (assignments, return) and operations supported by TensorSensor. There is a built-in Python parser, but it generates a different AST than I wanted; plus, filtering the built-in tree structure for just the parts I care about would be about the same amount of work as writing the parser. I built the parser using simple recursive descent, rather than adding another dependency (my friend antlr) to TensorSensor.
Next, we need to augment the AST with the values of all subexpressions so it looks like the AST from the previous section. This is a straightforward bottom-up walk of the AST calling eval() on each subexpression, saving the result in the associated node. In order to execute the various bits of the expression in the proper context after an exception, though, TensorSensor has to walk the call stack back down to the specific context that directly invoked the tensor library. The trick is not chasing the call stack too far, down into the tensor library.
Once the AST is augmented with partial values, we need to find the smallest subexpressions that evaluate to tensors for visualization purposes. That corresponds to the deepest subtrees that evaluate to tensors, which is the role of tsensor.analysis.smallest_matrix_subexpr()."
(Note: the author is also the creator of ANTLR.)
Thanks, Jeremy. :) I didn't go into super huge detail in the article on the implementation part as most readers won't have interest in language nerd details like you and I do.
This kind of code transformation would be an everyday task in a Lisp. Would it make sense to use a Lisp FFI to interface one of tensor libraries and write the transformations there?
One of the biggest challenges when writing code to implement deep learning networks is getting all of the tensor (matrix and vector) dimensions to line up properly, even when using predefined network layers. This article describes a new library called TensorSensor that clarifies exceptions by augmenting messages and visualizing Python code to indicate the shape of tensor variables. It works with Tensorflow, PyTorch, and Numpy, as well as higher-level libraries like Keras and fastai.
This looks like a neat tool, especially for students taking courses like CS 231N! I imagine this already works in Jupyter Notebook? Perhaps this could be integrated into an extension for the VSCode, maybe in the debugger.
I wondered about that. It might work just calling my internal pyviz("some python code in string) function from the debugger. It'll execute in context (maybe?)
This looks really neat, but I've only felt the need for such a tool with tensorflow. With pytorch and numpy you can just use ipdb or any other debugger to be parachuted right at the exception and inspect shape and value of all the variables you need.
True, but you have to do it one at a time to look at the variables. Also, the exception message doesn't tell you which operator among several is the issue. This just makes it easier. :)
Of course, but debugging is a general skill that will work in every case.
Not trying to detract anything from this tool, I'll definitely give it a try it next time I have something that doesn't work :)
BTW, here's a link to the twitter thread.
https://twitter.com/the_antlr_guy/status/1313569854918590465
I believe problems solved by this tool generally disappear when using named tensors
Are people using named tensors a lot these days? I haven't seen a lot of code using them but I also haven't been looking at a lot of tensor code lately.