Python tools

urdunlp

Urdu and Roman Urdu text processing · Python 3.10+ · no dependencies

Normalises Urdu text (including text copied out of PDFs), transliterates Roman Urdu to Urdu script and back, tells Urdu from the ten other languages written in the same script, finds English inside Roman Urdu, reads numbers written as words, and tokenises, stems and removes stopwords.

Input → output

>>> transliterate_to_urdu("mujhe kal lahore jana hai")
'مجھے کل لاہور جانا ہے'
>>> transliterate_to_roman("آج موسم بہت اچھا ہے")
'aaj musam bohat acha hai'
>>> normalize("كتاب") == normalize("کتاب")   # Arabic kaf/yeh unified
True

Held-out evaluation on Google's Dakshina hand-romanised sentences (52,087 words). The weak numbers are published too: stemming adds only +0.008 recall@10, and a script check calls 99.9% of Persian "Urdu".

blast-radius

What a dependency upgrade really changes · installs both versions side by side

Reads the public API of two versions of a package and sorts every change by how likely it is to reach production unnoticed: a removed name (an ImportError tells you), a reshaped signature (a type checker tells you), a widened one (harmless), and the same signature with a different answer, which nothing tells you. For those it runs both versions on the same inputs.

27 real upgrades: how often a release broke exported API

One of them: urllib3 2.2.1 → 2.2.2, a patch release

2.2.1  (*, headers=None, status, version,                 reason, decode_content, ...)
2.2.2  (*, headers=None, status, version, version_string, reason, decode_content, ...)

A new required keyword-only parameter: every subclass calling super().__init__(...) now raises TypeError, and positional callers have reason land silently in the new slot. No changelog entry says so.

flake-detective

Names the cause of a flaky pytest test · Python 3.11+ · no dependencies

Runs a suite many times, changing one thing per batch of runs (test order, PYTHONHASHSEED, the frozen wall-clock date) and reports which change makes each flaky test flip, with the failure rate under every condition beside the verdict. Instead of "test_x is flaky" you get "test_x fails after test_y".

4 flaky tests:
    1  order
    1  hash-seed
    1  clock
    1  nondeterminism
--------------------------------------------------------------------------
test                                          baseline    order hashseed    clock
test_order_dependent.py::test_aaa_first_one_wins   0.0      0.4      0.0      0.0
    ORDER: failed 0 of 7 baseline runs, 3 of 7 shuffled: its result
    depends on which tests run before it.

suite-auditor

Which tests check nothing · mutation testing that reports only what it can prove

Mutates each function a suite covers. A surviving mutant counts as a gap only once there is a concrete call on which the mutant and the original return different things; survivors it cannot separate stay "unproven" and never reach the headline.

Two real audits

SUITE AUDIT - pricing
  mutants scored     : 17
  killed by the suite: 13  (76.5%)
  PROVEN GAPS        : 2  (2 with an unarguable witness)
  unproven survivors : 2  (possibly equivalent; not counted)
  uncovered functions: 1  (no test reaches them at all)