Creating a Python Code Completer & More Abstract Syntax Tree Projects

How does a code completion tool work? What is an Abstract Syntax Tree, and how is it created in Python? How does an AST help you write programs and projects that inspect and modify your Python code? This week on the show, Meredydd Luff, co-founder of Anvil, shares his PyCon talk, “Building a Python Code Completer.”

Meredydd talks about his experience building a code completion engine for the Anvil platform. The conversation leads us to discuss how Python parses the code that you type. We examine tokenization, abstract syntax trees, and how parsing has changed in Python.

We cover additional projects that you can explore once you have a tool that inspects the Python code you’re writing. Join us as we dive into multiple rabbit holes of research and exploration.

Topics:

00:00:00 – Introduction 00:02:11 – Building a Python Code Completer - Talk 00:08:48 – How is recursion being used? 00:10:28 – Abstract Syntax Trees, Parsers, and Tokens 00:14:58 – Sponsor: CData Software 00:15:39 – Is parsing into tokens a separate step? 00:21:45 – What makes the new PEG parser unique? 00:25:32 – Using the AST doesn’t have to be scary. 00:27:33 – More on the PEG parser 00:29:54 – Keeping scope and holding completions 00:39:43 – Video Course Spotlight 00:41:15 – Was this your first expirement in creating a code completer? 00:46:46 – How can adding type hints assist a completer? 00:50:53 – Projects you can build using AST 00:53:02 – Linter project 00:54:15 – Automatic code formatter 00:54:59 – Code testing and the pytest project 00:56:22 – Security tools 00:57:11 – Friendlier tracebacks 00:59:57 – Code completion vs GitHub Copilot 01:04:14 – Code completion in a text editor vs IDE 01:08:53 – What are you excited about in the world of Python? 01:09:54 – What do you want to learn next? 01:11:20 – How can people follow your work? 01:12:31 – Thanks and goodbye

Show Links:

Talk - Meredydd Luff: Building a Python Code Completer - YouTube Anvil | Build Web Apps with Nothing but Python Episode #63: Create Web Applications Using Only Python With Anvil – The Real Python Podcast “Writing a PEG parser for fun and profit” - Guido van Rossum (North Bay Python 2019) - YouTube PEG Parsing Series Overview. My series of blog posts about PEG | by Guido van Rossum Abstract syntax tree - Wikipedia Parse tree - Wikipedia ast — Abstract Syntax Trees — Python 3.10.6 documentation Full Grammar specification — Python 3.10.6 documentation Python AST Explorer Examples of working with ASTs — Green Tree Snakes 1.0 documentation bellybutton: Custom Python linting through AST expressions black · PyPI pytest: helps you write better programs — pytest documentation friendly-traceback: Friendlier Python tracebacks. Raspberry Pi Pico – Raspberry Pi Raspberry Pi Pico W : ID 5526 : $6.00 : Adafruit Industries, Unique & fun DIY electronics and kits State management - Wikipedia Recoil Meredydd Luff (@meredydd) / Twitter

Level up your Python skills with our expert-led courses:

Python Assignment Expressions and Using the Walrus Operator Testing Your Code With pytest Writing Beautiful Pythonic Code With PEP 8

Support the podcast & join our community of Pythonistas