import database
import pytest
@pytest.fixture()
def test_a(db):
return db
def test_b(db):
return db
database = "database"
class database:
pass
which seems roughly like what the author is after. Mentions of "database" outside function definitions are not modified. That sort of logic I always found hard to replicate in basic GNU-like tools. If run without stdin, the above command runs recursively, in-place (careful with that one!).
Note: I just wrote this, and version 0.13.2 is required for the above to work.
Interesting use of treesitter. But I’m a little surprised that treesitters built in query language wasn’t used.
There’s no need to manually iterate through the tree, and use if statements to select nodes. Instead you can just write a couple of simple queries (and even use treesitters web UI to test the queries), and have the treesitter just provide all the nodes for you.
Having no experience with treesitter I find the query language rather hard to parse. From a practical point of view and experimenting with the library I’m not surprised to go with this nested For loop approach.
The query language is definitely underdocumented. In case it helps you, what helped me was realizing it’s basically a funky pattern language, à la the match pattern sublanguages in OCaml/Haskell/Rust.
But the syntax for variable binding is idiosyncratic and the opposite of normal pattern languages. Writing “x” doesn’t bind the thing at the position to the variable x; instead, you have to write e.g. foo @x to bind x to the child of type foo. Insanely, some Scheme dialects use @ with the exact opposite semantics!! There’s also a bizarre # syntax for conditionals and statements.
Honestly there isn’t really an excuse for how weird they made the pattern syntax given that people have spent decades working on pattern matching for everything from XML to objects (even respecting abstraction!). I’ve slowly been souring on treesitter in general, but paraphrasing Stroustrup: there are things people complain about, and then there are things nobody uses.
Not really. It uses S-expressions but Scheme pattern matching is totally different. The most common Scheme pattern matching syntax is basically the same as pattern matching in any other language: x means “bind the value at this position to x”, not “the child node of type”. See: https://www.gnu.org/software/guile/manual/html_node/Pattern-... or syntax-rules.
It’s as much a Scheme dialect as WASM’s S-expression form is a Scheme dialect.
Treesitter’s query syntax is slightly understandable in the sense that having x match a node among siblings of type x works well for extracting values out of sibling lists. Most conventional pattern syntaxes struggle with this, e.g. how do you match the string “foo” inside of a list of strings in OCaml or Rust without leaving the match expression and resorting to a loop?
But you could imagine a syntax-rules like use of ellipses …. There’s also a more powerful pattern syntax someone worked on for implementing Scheme-like macros in non-S-expression based languages whose name escapes me right now.
I’ve been looking at codemod tools recently, just as a way to extend my editing toolbox. I came across https://ast-grep.github.io/, which looks like it might address part of this problem. My initial test case was to locate all calls to a method where a specific argument was ‘true’, and it handled that well - that’s the kind of thing an IDE seems to struggle with. I’m not yet sure whether it could handle renaming a variable though.
I guess what I’m looking for is something that
* can carry out the kind of refactorings usually reserved for an IDE
* has a concise language for representing the refactorings so new ones can be built quite easily
* can apply the same refactoring in multiple places, with some kind of search language (addressing the task of renaming a test parameter in multiple methods)
* ideally does this across multiple programming languages
# Iterate through all files in the codebase
for file in codebase.files:
# Check for functions with the pytest.fixture decorator
for function in file.functions:
if any(d.name == "fixture" for d in function.decorators):
# Rename the 'db' parameter to 'database'
db_param = function.get_parameter("db")
if db_param:
db_param.set_name("database")
# Log the modification
print(f"Modified {function.name}")
Tree-sitter is really powerful, but it's worth people learning a few methods they prefer to use because there are going to be situations where one method works better than another. Things I have found useful in the past include
- perl -pi -e 's/foo/bar/g' files
"-pi" means "in place edit" so it will change the files in place. If you have a purely mechanical change like he's doing here it's a very reasonable choice. If you're not as much of a cowboy as I am, you can specify a suffix and it will back the files up, so something like
perl -p -i.bak -e 's/db/database/g' py
For example then all your original '.py' files will be copied to '.py.bak' and the new renamed versions will be '.py'
For vim users (I know emacs has the same thing but I don't remember the exact invocation because it has been >20years since I used emacs as my main editor) it's worth knowing the "global" command. So you can execute a particular command only on lines that match some regex. So say you want to delete all the lines which mention cheese
:%g/cheese/d
Say you want to replace "db" with "database" but only on lines which start with "def"
:%g/^def/s/db/database/
OK cool. Now if you go 'vim *py' you can do ":argdo g/^def/s/db/database/ | update" and it will perform that global command across all the files in the arg list and save the ones which have changed.
I realise that and like the article. I was trying to convey in my response that devs should have these things in their toolkit not that you "did the wrong thing"[1] somehow by using treesitter for this.
I was thinking too as I read that AST could be swapped in for tree sitter and I think it'd work more or less the same (not sure it'd have an advantage though, unless you preferred using standard library tools where possible)
> I do wish tree-sitter had a mechanism to directly manipulate the AST. I was unable to simply rename/delete nodes and then write the AST back to disk. Instead I had to use Jedi or manually edit the source (and then deal with nasty off-set re-parsing logic).
> LibCST parses Python 3.0 -> 3.12 source code as a CST tree that keeps all formatting details (comments, whitespaces, parentheses, etc). It’s useful for building automated refactoring (codemod) applications and linters.
> Finding Occurrences: The find occurrences command (C-c f by default) can be used to find the occurrences of a python name. If unsure option is yes, it will also show unsure occurrences; unsure occurrences are indicated with a ? mark in the end. Note that ropevim uses the quickfix feature of vim for marking occurrence locations. [...]
> Rename: When Rename is triggered, rename the symbol under the cursor. If the symbol under the cursor points to a module/package, it will move that module/package files
> Rename symbol: Renaming is a common operation related to refactoring source code, and VS Code has a separate Rename Symbol command (F2). Some languages support renaming a symbol across files. Press F2, type the new desired name, and press Enter. All instances of the symbol across all files will be renamed
Yeah, there `sed` and `git diff` with one or more filenames in a variable might do.
Because pytest requires a preprocessing step, renaming fixtures is tough, and also for jupyter notebooks %%ipytest is necessary to call functions that start with test_ and upgrade assert keywords to expressions; e.g
`assert a == b, error_expr` is preprocessed into `assertEqual(a,b, error_expr)` with an AssertionError message even for comparisons of large lists and strings.
There is a Python library/tool called Bowler (https://pybowler.io/docs/basics-intro) that allows selecting and transforming elements on a concrete syntax tree. From my limited experience with it, I guess it would have been a nice fit for this refactoring.
I've always wanted to do mechanical refactors and recently ran into the problem the author ran into where tree-sitter can't write back the AST as source. Is there an alternative that is able to do this for most programming languages?
In the JavaScript world, jscodeshift and its upstream tool recast are frequently used. I believe you could do the same thing with esbuild and some Rust based tools, but these two are probably the most popular.
I think this particular case would be difficult to refactor even in an IDE like PyCharm, which afaik is the best at refactoring Python (might be outdated).
PyCharm understands pytest fixtures and if this is really just about a single fixture called "database", it takes 3 seconds to do this refactoring by just renaming it.
I believe the idea is that those identifiers are semantically related: that fixture decorator inspects the formal parameter names so that it can pass the appropriate arguments to each test when the tests are run. A sufficiently smart IDE and/or language server would thus know that these identifiers are related, and performing a rename on one instance would thus rename all of the others.
And maybe you were being facetious, but an IDE is an “Integrated Development Environment”.
In PyCharm: Move cursor on any occurence or definition of "database" fixture, press the "Rename" hotkey (Shift+F6), delete old name and type new name, press Enter key to confirm.
A single fixture, yes. If there are many fixtures of the same name in different test modules, it wouldn't work, but that's not how I understood the problem in the blog post, which says
>rename every instance of a pytest fixture from database -> db
Every instance of a fixture, not every instance of all fixtures of the same name.
I don’t think that this will handle what the author wants, though. If each function takes db as an argument and uses it internally, those don’t count as references that you can change globally, right?
An input like
is turned into which seems roughly like what the author is after. Mentions of "database" outside function definitions are not modified. That sort of logic I always found hard to replicate in basic GNU-like tools. If run without stdin, the above command runs recursively, in-place (careful with that one!).Note: I just wrote this, and version 0.13.2 is required for the above to work.
[0]: https://github.com/alexpovel/srgn