Bowler builds on top of the Concrete Syntax Tree (CST) provided by lib2to3 in the standard library. This enables direct modifications to the tree, without losing formatting, comments, or whitespace, and without needing to directly support every new version of Python as they get released.
Fixers are a core concept of refactoring in lib2to3, and they contain both
a grammar-based search "pattern" for finding nodes in the CST as well as methods for
transforming nodes that match the pattern. They are relatively self-contained, and
form the core component of the
2to3 tool used for migrating older Python structures
to modern Python 3 analogues.
Bowler wraps much of its framework around generating a set of fixers, and then using a modified version of the lib2to3 refactoring framework to apply these fixers to the target Python source files. This allows Bowler to reuse and benefit from most of the features in lib2to3 while still allowing it to provide custom behaviors and functionality not found in the standard library.
Bowler operates by building "queries" against the CST. Queries consist of one or more "transforms", which specify what elements of the syntax tree to modify, as well as how to modify those elements. Each transform represents a search against the syntax tree, along with a programmatic modification to any matched tree elements. These transforms generate a fixer for lib2to3, and Bowler will execute multiple transforms simultaneously on each file processed. By combining multiple transforms together, Bowler enables a wide range of refactoring, from simple renames to complicated upgrades from old APIs to their modern replacements.
The primary mechanism for building queries and transforms in Bowler is by using a "fluent" API:
A fluent interface is a method for designing object oriented APIs based extensively on method chaining with the goal of making the readability of the source code close to that of ordinary written prose, essentially creating a domain-specific language within the interface. -- Fluent Interface
This allows you to chain method calls to the
Query class one after the other, without
needing to assign a query object to a variable or reference that variable for every
method call. In practice, that looks something like this:
( Query() .select(...) .modify(...) .execute() )
Transforms are implicitly created by the
Query API while defining the desired query.
Each transform begins with a "selector" (the syntax tree pattern to search for), or an
existing fixer class that already contains a selector pattern. Then, any number of
"filters" are specified, followed by one or more "modifiers". Each time a selector or
fixer is added to the query, a new transform is generated, and subsequent filters or
modifiers are added to the transform.
Repeating this pattern of selector, filters, and modifiers will generate further transforms, each of which will be executed as elements in the syntax tree match the selector patterns for a given transform.
These represents lib2to3 search patterns to select syntax tree nodes matching the given pattern, while capturing relevant child nodes or leaves when available. Bowler includes a set of common selectors that attempt to find all definitions or references of a specific type, and uses a consistent naming scheme for all captured elements.
An example selector to find all uses of the
pattern = """ power< "print" trailer< "(" print_args=any* ")" > """ ( Query() .select(pattern) ... )
This example finds all function calls named
Once elements are found matching the active selector, filters can further limit which
elements are passed to the modifiers by inspecting the original matches and returning
True if it still matches, or
False if the element should be excluded. Elements
will only be considered as matches if they successfully pass all filter functions.
If an element fails one filter function, it will not be passed or considered for any
other filter functions.
An example filter, to pair with the example selector above, that only matches calls to the print function with string literal arguments:
def print_string(node: LN, capture: Capture, filename: Filename) -> bool: args = capture.get("print_args") # args will be a list because we captured `any*` if args: return len(args) == 1 and args.type == Token.STRING: return False ( Query() ... .filter(print_string) ... )
This filter function will only return
True if the matched print call contained
exactly one argument, and that argument's type was a string literal.
After matched elements have been filtered, all remaining matches will be passed to all modifier functions in sequence. The modifier function may then transform the syntax tree at any point from the root to the leaves, without being restricted to the branch corresponding to the matched element. Transformations can include modifying existing nodes or leaves, removing or replacing elements, or inserting elements into the tree.
An example modifier, which builds the above selector and filter examples, transforms
the string literal arguments of the print function to wrap them in a hypothetical
translation function (
def translate_string(node: LN, capture: Capture, filename: Filename) -> None: args = capture.get("print_args") # make sure the arguments haven't already been modified if args and args.type == Token.STRING: args.replace( Call( Name("tr"), args=[args.clone()], ) ) ( Query() ... .modify(translate_string) ... )
The modifier function replaces the existing string literal element with the nested
nodes to represent a call to the hypothetical translation function (
tr), with a clone
of the existing string literal as the only argument.
Note: take care to ensure that matched and captured elements still meet the expectations of your modifier function. They may have already been transformed (or entirely removed or replaced) by previous modifiers.o
Bowler recognizes that there are cases where it's useful to post-process final changes to the codebase, and provides a mechanism for attaching "processors" to the query. These are functions that will be executed for each individual "hunk", or change, to a file – the same "hunks" that you would see in a unified diff format – and optionally decide whether that hunk should be applied or skipped. This allows for behavior such as extra logging, selective application of hunks based on conditions, record keeping for future queries, and much more.
An example processor that keeps track of every file touched by the query:
MODIFIED: Set[Filename] = set() def modified_files(filename: Filename, hunk: Hunk) -> bool: MODIFIED.add(filename) return True ( Query() ... .process(modified_files) ... )
After building a query with some combination of selectors, filters, modifiers, and/or
processors, there are a few different execution commands that Bowler takes to determine
how it will apply the query to the codebase. The default behavior when calling
.execute() on the query is to generate an interactive diff – similar to that provided
git add -p – and show each hunk in series, asking the user to apply or skip that
hunk. This is the "safest" option, as the user can verify that each change is intended
and meets the expectations of the refactoring. There are also options to just generate
a diff without applying hunks, or to apply all hunks without asking, and there are
shortcut methods for each of these options as well.
( Query() ... .execute( interactive = True, # ask about each hunk write = False, # automatically apply each hunk silent = False, # don't ask or print hunks at all ) )