Antibody — sqlparse

Of the bugs this project already fixed, how many could come back without a test noticing?

Before: 13 of 20 caught. After: 18 of 20 caught.

Y = 20: the number of past bugs where the patch applied, tests ran, and a check was possible (CAUGHT + NOW CAUGHT + STILL EXPOSED + NO CHANGE FOUND). Excluded, NO DATA, and NOT A BUG rows are not counted.

Status Description Fix Issue
NOW CAUGHTReindenting large tuple lists caused quadratic CPU consumption (GHSA-cfqr-cjx5-5jcm); fixed by measuring offsets backwards; now protected by test_a51df6d9e2.py.
Evidence

Catching tests:

tests/antibody/test_a51df6d9e2.py

New test:

import timeit
import sqlparse


def test_reindent_tuple_list_not_quadratic():
    def make_sql(n):
        values = ", ".join(f"({i}, {i+1})" for i in range(n))
        return f"SELECT a, b FROM t WHERE (a, b) IN ({values})"

    def time_op(n):
        sql = make_sql(n)
        return min(timeit.repeat(lambda: sqlparse.format(sql, reindent=True), number=1, repeat=3))

    n = 100
    t_n = time_op(n)
    t_4n = time_op(n * 4)
    ratio = t_4n / t_n
    assert ratio < 9, f"growth ratio {ratio:.1f} suggests quadratic behaviour"

a51df6d9
NO DATADollar-quoted literals and multiline comments triggered O(n²) re-scanning on adversarial input (GHSA-prg7-hcfm-mfcr); fixed by a pre-processing pass; code has since moved so the fix cannot be probed against its original location.
code-moved: git apply -R failed
d1d80602
NOW CAUGHTA comment-only statement caused quadratic DoS in group_comments (GHSA-f2ff-p2ww-7p4p); fixed by processing tokens in one forward pass; now protected by test_ef2012a5ee.py.
Evidence

Catching tests:

tests/antibody/test_ef2012a5ee.py

New test:

import timeit
import sqlparse

def test_strip_comments_not_quadratic():
    def make_sql(n):
        return "\n".join(f"-- comment {i}" for i in range(n))

    def time_op(n):
        sql = make_sql(n)
        return min(timeit.repeat(lambda: sqlparse.format(sql, strip_comments=True), number=1, repeat=3))

    n = 500
    t_n = time_op(n)
    t_4n = time_op(n * 4)
    ratio = t_4n / t_n
    assert ratio < 9, f"growth ratio {ratio:.1f} suggests quadratic behaviour"

ef2012a5
CAUGHTA BETWEEN bound written as a leading-dot float (e.g. .03) demoted the preceding keyword from Keyword to Name; fixed by adding the pattern to the keyword regex; caught by an existing test.
Evidence

Catching tests:

tests/test_regressions.py::test_between_leading_dot_float_issue601[a BETWEEN .03 AND .06]
tests/test_regressions.py::test_between_leading_dot_float_issue601[a between .03 and .06]

a194d318
CAUGHTget_real_name() returned the wrong component for identifiers with more than two dotted parts (e.g. db.schema.tbl.col returned 'schema' instead of 'col'); fixed by iterating to the last real name; caught by an existing test.
Evidence

Catching tests:

tests/test_parse.py::test_get_real_name_multi_part_dotted

f66d12c2
CAUGHTMATERIALIZED was tokenised as Name instead of Keyword, leaving it unaffected by keyword_case formatting; fixed by adding it to KEYWORDS; caught by an existing test.
Evidence

Catching tests:

tests/test_regressions.py::test_materialized_view_issue752

ac3b9e0b
CAUGHTLowercase 'as' in CREATE TABLE AS SELECT left has_as False, causing the function-grouping guard to fire and skip all function grouping; fixed by case-folding the check; caught by an existing test.
Evidence

Catching tests:

tests/test_grouping.py::test_grouping_alias_ctas_lowercase_as

111b35cf
CAUGHTROW_FORMAT was tokenised as Name, causing the grouper to treat it as a table alias; fixed by adding it to KEYWORDS; caught by an existing test.
Evidence

Catching tests:

tests/test_regressions.py::test_alter_table_row_format_issue773

26d7d652
NO DATAAnonymous BEGIN...END blocks were incorrectly split into multiple statements; fixed by a stack-based splitter rewrite; code has since moved so the fix cannot be probed against its original location.
code-moved: git apply -R failed
8f978caa
NO DATAWhen the grouping-limit was exceeded, functions silently returned None instead of raising SQLParseError; fixed by adding an explicit error raise; code has since moved.
code-moved: git apply -R failed
da67ac16
NO DATABEGIN TRANSACTION...END TRANSACTION blocks were not split correctly, a regression introduced in v0.5.4; fixed by handling the TRANSACTION keyword; code has since moved.
code-moved: git apply -R failed
5ca50a2e
NO DATASemicolons inside BEGIN...END blocks were treated as statement terminators, splitting a single statement into multiple fragments; fixed by tracking block depth; code has since moved.
code-moved: git apply -R failed
1a3bfbd5
NO DATAIF EXISTS inside a BEGIN...END block caused incorrect statement merging; fixed by adding IF to the keyword table for the splitter; code has since moved.
code-moved: git apply -R failed
e92a032c
CAUGHTstrip_comments=True left a stray '--B' token when the statement contained only comments; fixed by a one-line filter correction; caught by an existing test.
Evidence

Catching tests:

tests/test_format.py::TestFormat::test_strip_comments_single

73c8ba3a
NOT A BUGAdd pre-commit hook support (fixes #537)31903e09537
EXCLUDEDAdd type annotations to public API functions
non-behavioural: Source changes are identical after stripping non-behavioural elements
f1d2d688
NOW CAUGHTATTACH and DETACH were not in KEYWORDS_PLPGSQL, so PostgreSQL ALTER TABLE DETACH PARTITION statements parsed them as Name tokens; fixed by adding the two keywords; now protected by test_694747fdbe.py.
Evidence

Catching tests:

tests/antibody/test_694747fdbe.py

New test:

import sqlparse
from sqlparse import tokens as T


def test_attach_detach_are_keywords():
    # ATTACH and DETACH should be recognized as Keyword tokens in PostgreSQL dialect
    for word, sql in [
        ("DETACH", "ALTER TABLE atable DETACH PARTITION atable_p00"),
        ("ATTACH", "ALTER INDEX atable_acolumn_idx ATTACH PARTITION atable_p01_acolumn_idx"),
    ]:
        parsed = sqlparse.parse(sql)[0]
        found_type = None
        for token in parsed.flatten():
            if token.normalized.upper() == word:
                found_type = token.ttype
                break
        assert found_type == T.Keyword, (
            f"Expected {word} to have token type {T.Keyword}, got {found_type}"
        )

694747fd
EXCLUDEDFixed 'trailing' typo in split method documentation
non-behavioural: Source changes are identical after stripping non-behavioural elements
27431095
EXCLUDED[fix] issue262 adding comment and tests
non-behavioural: Source changes are identical after stripping non-behavioural elements
8eb778ca
NO DATASQL hints (/*+ ... */) were deleted by strip_comments, causing loss of query optimiser hints; fixed by preserving hint-style block comments; code has since moved.
code-moved: git apply -R failed
af6df6c6
NOW CAUGHTEXTENSION was tokenised as Identifier rather than Keyword, producing wrong token types in DROP/CREATE EXTENSION statements; fixed by adding it to KEYWORDS; now protected by test_e48000a5d7.py.
Evidence

Catching tests:

tests/antibody/test_e48000a5d7.py

New test:

import sqlparse
from sqlparse import tokens as T


def test_extension_tokenized_as_keyword():
    """EXTENSION should be tokenized as a Keyword, not part of an Identifier."""
    stmt = "DROP EXTENSION pgcrypto"
    parsed = sqlparse.parse(stmt)[0]

    extension_token = None
    for token in parsed.flatten():
        if token.normalized.upper() == 'EXTENSION':
            extension_token = token
            break

    assert extension_token is not None, "EXTENSION token not found in parsed statement"
    assert extension_token.ttype is T.Keyword, (
        f"Expected EXTENSION to have ttype {T.Keyword}, got {extension_token.ttype}"
    )

e48000a5
NO DATAASC/DESC NULLS FIRST/LAST were tokenised inconsistently, breaking ORDER BY formatting; fixed by adding NULLS FIRST/LAST as compound keywords; code has since moved.
code-moved: git apply -R failed
b126ba5b
CAUGHTstrip_whitespace=True failed to descend into sub-groups (e.g. CTE parentheses), leaving trailing spaces inside nested tokens; fixed by recursing into sub-groups; caught by an existing test.
Evidence

Catching tests:

tests/test_format.py::test_strip_ws_removes_trailing_ws_in_groups

0c4902f3
NO DATAMultiple CASE clauses inside a BEGIN block caused a single statement to be split into several; fixed by tracking CASE depth in the splitter; code has since moved.
code-moved: git apply -R failed
791e25de
NO DATANo compact formatting option existed, leaving users with no way to suppress excessive newlines; fixed by adding reindent_aligned/compact; code has since moved.
code-moved: git apply -R failed
bf74d8bc
NO DATAStripping an inline comment directly adjacent to a value fused it with the next keyword (e.g. '1AS foo'); fixed by preserving grouping boundaries in Comment handling; code has since moved.
code-moved: git apply -R failed
8b034273
CAUGHTFunction.get_parameters() raised an error when the function had an OVER clause because it assumed the last token was always a Parenthesis; fixed by stopping at the first Parenthesis; caught by an existing test.
Evidence

Catching tests:

tests/test_grouping.py::test_grouping_function

e03b74e6
NO DATAWindow functions were not grouped with their OVER clause into a single Identifier; fixed by adding OVER-clause grouping; code has since moved.
code-moved: git apply -R failed
617b8f6c
CAUGHTPRIMARY KEY was not recognised as a compound keyword, producing incorrect token structure in CREATE TABLE statements; fixed by adding a regex pattern; caught by an existing test.
Evidence

Catching tests:

tests/test_regressions.py::test_primary_key_issue740

46971e5a
NOW CAUGHTTRUNCATE was classified as DML instead of DDL, and GRANT/REVOKE as DML instead of DCL, causing ttype checks to return wrong results; fixed by correcting the keyword classifications; now protected by test_d76e8a4425.py.
Evidence

Catching tests:

tests/antibody/test_d76e8a4425.py

New test:

import sqlparse
from sqlparse import tokens as T


def test_truncate_ddl_grant_revoke_dcl():
    """TRUNCATE should be Keyword.DDL; GRANT and REVOKE should be Keyword.DCL."""
    def first_ttype(sql):
        return sqlparse.parse(sql)[0].tokens[0].ttype

    truncate_ttype = first_ttype("TRUNCATE")
    grant_ttype = first_ttype("GRANT")
    revoke_ttype = first_ttype("REVOKE")

    assert truncate_ttype == T.Keyword.DDL, (
        f"TRUNCATE ttype is {truncate_ttype}, expected {T.Keyword.DDL}"
    )
    assert grant_ttype == T.Keyword.DCL, (
        f"GRANT ttype is {grant_ttype}, expected {T.Keyword.DCL}"
    )
    assert revoke_ttype == T.Keyword.DCL, (
        f"REVOKE ttype is {revoke_ttype}, expected {T.Keyword.DCL}"
    )

d76e8a44
NO CHANGE FOUNDJSON operators (-> and ->>) in SELECT caused incorrect reindent alignment; fixed by grouping JSON-operator expressions before indenting; the check found no observable difference in current code.
check: .antibody/sqlparse/checks/8c24779e02.py
Evidence

Probe: .antibody/sqlparse/checks/8c24779e02.py
With fix:

DIFFERENCE FOUND: indents=[0, 7] vs all-equal
Without fix:
DIFFERENCE FOUND: indents=[0, 7] vs all-equal

8c24779e
CAUGHTThe -> operator was not recognised as a single token, so use_space_around_operators=True split it and produced invalid SQL; fixed by adding JSON-operator patterns; caught by an existing test.
Evidence

Catching tests:

tests/test_format.py::test_format_json_ops
tests/test_parse.py::test_json_operators[->]
tests/test_parse.py::test_json_operators[->>]
tests/test_parse.py::test_json_operators[#>]
tests/test_parse.py::test_json_operators[#>>]
tests/test_parse.py::test_json_operators[@>]
tests/test_parse.py::test_json_operators[<@]

6b05583f
NO DATAThe Lexer singleton was stored before keywords were fully loaded, causing intermittent failures in multi-threaded use; fixed by acquiring a lock during initialisation; code has since moved.
code-moved: git apply -R failed
5bb129d3
NO DATAStatements following a GO keyword in Transact-SQL were typed as UNKNOWN instead of their true DML type; fixed by treating GO as a statement splitter; code has since moved.
code-moved: git apply -R failed
7334ac99
CAUGHTgroup_order() was not recursive, so ORDER BY modifiers (DESC) inside sub-queries were not grouped; fixed by adding the @recurse decorator; caught by an existing test.
Evidence

Catching tests:

tests/test_grouping.py::test_grouping_nested_identifier_with_order

39b5a025
CAUGHTcopy.deepcopy() on a parsed statement raised TypeError because _TokenType.__deepcopy__ was missing and the class matched as callable; fixed by ignoring dunder attributes in __getattr__; caught by an existing test.
Evidence

Catching tests:

tests/test_regressions.py::test_copy_issue672

fac38cd0
NOT A BUGCleanup regex for detecting keywords (fixes #709).fc76056f709
CAUGHTget_type() returned UNKNOWN when a comment appeared between the WITH keyword and the CTE identifier; fixed by skipping comment tokens in the type-detection walk; caught by an existing test.
Evidence

Catching tests:

tests/test_regressions.py::test_comment_between_cte_clauses_issue632

dd9d5b91
NOT A BUGSwitch to pyproject.toml (fixes #685).8b789f28685
NO DATAThe IN keyword was not uppercased by keyword_case='upper' after a regression introduced in 0.4.0; fixed by reverting the erroneous Comparison classification; code has since moved.
code-moved: git apply -R failed
e9241945
NO DATAUnicode characters outside the ASCII range (e.g. Chinese ideographs) were not recognised as valid identifier characters, breaking reindentation; fixed by using \w in the regex; code has since moved.
code-moved: git apply -R failed
b72a8ff4
NO DATALowercase 'create table' statements were not parsed correctly because keyword comparisons lacked .upper() calls; fixed by normalising case; code has since moved.
code-moved: git apply -R failed
403de6fc
NO DATADISTINCTROW (an MS Access keyword) was not in the keyword table, so it was tokenised as a Name; fixed by adding it to KEYWORDS; code has since moved.
code-moved: git apply -R failed
0bbfd5f8
STILL EXPOSEDThe INDICATOR keyword was misspelled as 'INDITCATOR' in the main keywords dict; fixed by correcting the typo; INDICATOR is also listed in a second keyword table, so it is a keyword with or without the fix, and taking the fix out changes nothing a test could catch.
Evidence

Note: INCONCLUSIVE (data-only change)

e58781dd
NO DATAThe scientific-notation regex used \d* (zero-or-more), causing aliases like 'e2' to be misclassified as float literals; fixed by changing to \d+; code has since moved.
code-moved: git apply -R failed
e6604678

Run it on your own project

python3 .bob/skills/recurrence-audit/scripts/antibody.py setup <name> <git-url> [--rev SHA] [--deps PKG ...]
python3 .bob/skills/recurrence-audit/scripts/antibody.py candidates <name>
python3 .bob/skills/recurrence-audit/scripts/antibody.py run <name>
python3 .bob/skills/recurrence-audit/scripts/antibody.py ledger <name>

Needs Python 3.12 or newer. Run from the root of the repository that holds .bob/.

Method

For each past bug fix: apply the reverse patch, run the test suite, restore the code. A CAUGHT row means at least one test failed when the bug was put back. A NOW CAUGHT row means a new test was proved and just re-passed that check. A STILL EXPOSED row means the patch applied, all tests stayed green, and a probe confirmed the behaviour changed (or no check could tell either way). The row says which. NO CHANGE FOUND: a probe ran but both outputs matched (listed under “Excluded after a check”, with the check and both outputs). NO DATA: the patch could not be applied or no tests ran (with its reason).

What this can’t tell you

Rows with no data are unknown, never safe. “Caught” means a test noticed the regression — not that the test is perfect or complete. A green suite on a reverted patch is a lower bound, not a guarantee.