fix(python): a script's import of a module beside it in a package folder links again

CHANGELOG.md

@@ -420,7 +420,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 - A call on a variable or attribute named after a class, like `client` or `nodelist`, now prefers the class it spells whole, `Client` or `NodeList`, over another that only shares a word with it, like `RedisCacheClient`.
 - A call through an attribute of an imported value, like flask's `app.jinja_env.filters.keys()`, no longer links to the value's own `jinja_env`: the method called is `keys`. Re-index to update an existing project.
 - A call through a class name, like `Logger.log()`, or through a variable named after one, like `logger.log()`, now links to that class's own method or the one it inherits, such as Django's `form.is_valid()` reaching `BaseForm`, instead of the same-named method of a class whose name merely contains it, such as `FileLogger`. These calls also no longer link to a class a test declares inside one of its functions, or, from your own code, to a class in a test suite. Re-index to update an existing project. Thanks @inth3shadows.
-- Python imports now resolve the way Python finds modules: from your project's package roots — the repository root, `src/` layouts, each service with its own `pyproject.toml`, `setup.cfg` or `setup.py`, and the package directories those declare — so `from shop.cart import total` reaches `src/shop/cart.py`, `from pkg import name` reaches what the package's `__init__.py` defines or re-exports (respecting `__all__`), and `import json` or `import requests` no longer links to a same-named file nested in a test fixture, an example, a vendored copy or another package. A script can still import a module from a folder above it, like `from test import TestCase` in a `tests/` folder with the runner `test.py` one level up, when that module defines every name the script takes from it. After a module, package or `__init__.py` is added or removed, a package's `__init__.py` is edited, or a build config is edited, the next sync brings Python imports back in line with a fresh index; the file watcher doesn't react to a build-config edit by itself, so run `codegraph sync` after one. Re-index Python projects after upgrading. (#1704)
+- Python imports now resolve the way Python finds modules: from your project's package roots — the repository root, `src/` layouts, each service with its own `pyproject.toml`, `setup.cfg` or `setup.py`, and the package directories those declare — so `from shop.cart import total` reaches `src/shop/cart.py`, `from pkg import name` reaches what the package's `__init__.py` defines or re-exports (respecting `__all__`), and `import json` or `import requests` no longer links to a same-named file nested in a test fixture, an example, a vendored copy or another package. A script can still import a module from a folder above it, like `from test import TestCase` in a `tests/` folder with the runner `test.py` one level up, or from beside it in a folder with an `__init__.py`, like `from pysim import run` next to `pysim.py`, when that module defines every name the script takes from it and isn't named after a standard library module or a package it wraps. After a module, package or `__init__.py` is added or removed, a package's `__init__.py` is edited, or a build config is edited, the next sync brings Python imports back in line with a fresh index; the file watcher doesn't react to a build-config edit by itself, so run `codegraph sync` after one. Re-index Python projects after upgrading. (#1704)
 - An Express route keeps its named handler when a middleware before it contains an arrow function. `rateLimit({ keyGenerator: (req) => req.ip })` or `(req, res, next) => next()` no longer hides the handler at the end of the route (re-index to update an existing project).
 - A TypeScript type re-exported through a barrel (`export type { Foo } from`, `export { type Foo } from`, `export type * from`) now links to its real declaration instead of a same-named type elsewhere in the project, and import lists with comments, string names, or a JSDoc `@import` are read correctly. Re-index to update an existing project.
 - In TypeScript, a file that re-exports a whole module's types, like `export type * from './types'` or `export type * as Types from './types'`, now parses cleanly. Before, the parser couldn't read that line, so a barrel made of re-exports was reported as producing no symbols and `codegraph status` listed it among the files it could not parse. Re-index a project that uses it after upgrading. Thanks @gavinwilliams for the report and @danusha2345 for the pull request. (#2474)

__tests__/python-module-resolution.test.ts

@@ -900,6 +900,193 @@ def go():
     });
   });
 
+  // A script's folder is first on `sys.path` whether or not it holds an
+  // `__init__.py`: rocksdb's tools/block_cache_analyzer/block_cache_pysim_test.py
+  // and the block_cache_pysim.py beside it. A file of a real package reaches
+  // a bare name through the roots instead, so the names it takes must prove it.
+  describe('a module beside a script in a package folder', () => {
+    const pysim = `class HashTable:
+    def size(self):
+        return 0
+
+
+def run(n):
+    return n
+
+
+kLimit = 3
+`;
+
+    it('binds the names a script imports from the module beside it', async () => {
+      write({
+        'tools/an/__init__.py': '',
+        'tools/an/pysim.py': pysim,
+        'tools/an/pysim_test.py': `from pysim import HashTable, kLimit, run
+
+
+def test_all():
+    table = HashTable()
+    return run(kLimit)
+`,
+      });
+      cg = await CodeGraph.init(tempDir, { index: true });
+
+      expect(targets('test_all', 'tools/an/pysim_test.py', 'calls')).toEqual(['run@tools/an/pysim.py']);
+      expect(targets('pysim_test.py', 'tools/an/pysim_test.py', 'imports').sort()).toEqual([
+        'HashTable@tools/an/pysim.py', 'kLimit@tools/an/pysim.py', 'pysim.py@tools/an/pysim.py', 'run@tools/an/pysim.py',
+      ]);
+    });
+
+    it('leaves a bare name to the top-level module a root provides', async () => {
+      write({
+        'utils.py': 'def load(n):\n    return "top"\n',
+        'pkg/__init__.py': '',
+        'pkg/utils.py': 'def load(n):\n    return "beside"\n',
+        'pkg/mod.py': `from utils import load
+
+
+def go():
+    return load(1)
+`,
+      });
+      cg = await CodeGraph.init(tempDir, { index: true });
+
+      expect(targets('go', 'pkg/mod.py', 'calls')).toEqual(['load@utils.py']);
+    });
+
+    it('binds nothing when the module beside it lacks a name the file imports', async () => {
+      write({
+        'pkg/__init__.py': '',
+        'pkg/yaml.py': 'def dump(x):\n    return x\n',
+        'pkg/mod.py': `from yaml import dump, safe_load
+
+
+def go():
+    return safe_load(dump(1))
+`,
+      });
+      cg = await CodeGraph.init(tempDir, { index: true });
+
+      expect(targets('go', 'pkg/mod.py', 'calls')).toEqual([]);
+      expect(targets('mod.py', 'pkg/mod.py', 'imports')).toEqual([]);
+    });
+
+    it('never takes a file for the package it is named after', async () => {
+      write({
+        'pkg/__init__.py': '',
+        // A wrapper of the same name: its import is the real package's.
+        'pkg/client.py': `from client import Client
+
+
+class Client(Client):
+    pass
+
+
+def make():
+    return Client()
+`,
+      });
+      cg = await CodeGraph.init(tempDir, { index: true });
+
+      expect(targets('client.py', 'pkg/client.py', 'imports')).toEqual([]);
+    });
+
+    it('leaves a name to the package a wrapper beside the file wraps', async () => {
+      write({
+        'pkg/__init__.py': '',
+        'pkg/yaml.py': `import yaml
+
+
+def load(text):
+    return yaml.safe_load(text)
+`,
+        'pkg/mod.py': `from yaml import load
+
+
+def go():
+    return load("a: 1")
+`,
+      });
+      cg = await CodeGraph.init(tempDir, { index: true });
+
+      expect(targets('go', 'pkg/mod.py', 'calls')).toEqual([]);
+      expect(targets('mod.py', 'pkg/mod.py', 'imports')).toEqual([]);
+    });
+
+    // As `from html import escape` reads in django/utils/text.py, beside
+    // django/utils/html.py and its own `escape`.
+    it.each([
+      ['beside the file', 'pkg/__init__.py', 'pkg/html.py', 'pkg/text.py'],
+      ['in a folder above the file', 'tools/other.py', 'tools/html.py', 'tools/sub/text.py'],
+    ])('leaves a stdlib name to the stdlib, with a module of that name %s', async (_label, other, module, importer) => {
+      write({
+        [other]: '',
+        [module]: 'def escape(text):\n    return text\n',
+        [importer]: `from html import escape
+
+
+def go():
+    return escape("a")
+`,
+      });
+      cg = await CodeGraph.init(tempDir, { index: true });
+
+      expect(targets('go', importer, 'calls')).toEqual([]);
+      expect(targets(importer.slice(importer.lastIndexOf('/') + 1), importer, 'imports')).toEqual([]);
+    });
+
+    it('takes a dotted name below its own folder, the module proved the same way', async () => {
+      write({
+        'pkg/__init__.py': '',
+        'pkg/sub/__init__.py': '',
+        'pkg/sub/helper.py': 'def assist(n):\n    return n\n',
+        'pkg/run.py': `from sub.helper import assist
+
+
+def go():
+    return assist(1)
+`,
+      });
+      cg = await CodeGraph.init(tempDir, { index: true });
+
+      expect(targets('go', 'pkg/run.py', 'calls')).toEqual(['assist@pkg/sub/helper.py']);
+    });
+
+    it('binds nothing for a plain import, which takes no names to prove the module', async () => {
+      write({
+        'pkg/__init__.py': '',
+        'pkg/helper.py': 'def assist(n):\n    return n\n',
+        'pkg/mod.py': `import helper
+
+
+def go():
+    return helper.assist(1)
+`,
+      });
+      cg = await CodeGraph.init(tempDir, { index: true });
+
+      expect(targets('go', 'pkg/mod.py', 'calls')).toEqual([]);
+      expect(targets('mod.py', 'pkg/mod.py', 'imports')).toEqual([]);
+    });
+
+    it('looks in no package above the file, only in its own folder', async () => {
+      write({
+        'pkg/__init__.py': '',
+        'pkg/helper.py': 'def assist(n):\n    return n\n',
+        'pkg/sub/__init__.py': '',
+        'pkg/sub/mod.py': `from helper import assist
+
+
+def go():
+    return assist(1)
+`,
+      });
+      cg = await CodeGraph.init(tempDir, { index: true });
+
+      expect(targets('go', 'pkg/sub/mod.py', 'calls')).toEqual([]);
+    });
+  });
+
   describe('sync', () => {
     /** Every Python calls/imports edge, as a sorted list of readable triples. */
     function pythonEdges(graph: CodeGraph): string[] {
@@ -1157,6 +1344,174 @@ def go():
         expect(fresh.join('\n')).not.toContain('assist');
         expect(pythonEdges(cg)).toEqual(fresh);
       });
+
+      it.each([
+        ['a module in a folder below it', 'from lib.helper import assist', 'tools/lib/helper.py', 'tools/lib/other.py'],
+        // A package directly below `tools` would make it a root.
+        ['a package below it', 'from x.lib import assist', 'tools/x/lib/__init__.py', 'tools/x/lib/other.py'],
+      ])('follows an edit of %s', async (_label, line, moduleFile, otherFile) => {
+        write({
+          'tools/sub/run.py': `${line}\n\n\ndef go():\n    return assist(1)\n`,
+          [otherFile]: '',
+          [moduleFile]: 'def other():\n    return 0\n',
+        });
+        cg = await CodeGraph.init(tempDir, { index: true });
+        expect(pythonEdges(cg).join('\n')).not.toContain('assist');
+
+        write({ [moduleFile]: 'def assist(n):\n    return n\n' });
+        await cg.sync();
+        let fresh = await freshEdges();
+        expect(fresh).toContain(`calls go@tools/sub/run.py -> function:assist@${moduleFile}`);
+        expect(pythonEdges(cg)).toEqual(fresh);
+
+        write({ [moduleFile]: 'def other():\n    return 0\n' });
+        await cg.sync();
+        fresh = await freshEdges();
+        expect(fresh.join('\n')).not.toContain('assist');
+        expect(pythonEdges(cg)).toEqual(fresh);
+      });
+
+      it('re-opens nothing when an edited module has no importer below it that spells it', async () => {
+        write({
+          'tools/helper.py': 'def assist(n):\n    return n\n',
+          'tools/sub/run.py': 'def go():\n    return 1\n',
+          'other/run.py': 'from helper import assist\n\n\ndef go():\n    return assist(1)\n',
+        });
+        cg = await CodeGraph.init(tempDir, { index: true });
+        const spy = spyReopen(cg);
+
+        write({ 'tools/helper.py': 'def assist(n):\n    return n + 1\n' });
+        await cg.sync();
+
+        expect(spy.narrow).not.toHaveBeenCalled();
+      });
+
+      it('re-opens nothing when a root answers the import, not a script\'s folder', async () => {
+        write({
+          'svc/pyproject.toml': '[project]\nname = "svc"\n',
+          'svc/helper.py': 'def assist(n):\n    return n\n',
+          'svc/sub/run.py': 'from helper import assist\n\n\ndef go():\n    return assist(1)\n',
+        });
+        cg = await CodeGraph.init(tempDir, { index: true });
+        expect(pythonEdges(cg)).toContain('calls go@svc/sub/run.py -> function:assist@svc/helper.py');
+        const spy = spyReopen(cg);
+
+        write({ 'svc/helper.py': 'def assist(n):\n    return n + 1\n' });
+        await cg.sync();
+
+        expect(spy.narrow).not.toHaveBeenCalled();
+      });
+    });
+
+    describe('a module beside a script in a package folder', () => {
+      const importer = `from helper import assist
+
+
+def go():
+    return assist(1)
+`;
+
+      it('follows an edit that gives the module the name, or takes it away', async () => {
+        write({
+          'pkg/__init__.py': '',
+          'pkg/run.py': importer,
+          'pkg/helper.py': 'def other():\n    return 0\n',
+        });
+        cg = await CodeGraph.init(tempDir, { index: true });
+        expect(pythonEdges(cg).join('\n')).not.toContain('assist');
+
+        write({ 'pkg/helper.py': 'def assist(n):\n    return n\n' });
+        await cg.sync();
+        let fresh = await freshEdges();
+        expect(fresh).toContain('calls go@pkg/run.py -> function:assist@pkg/helper.py');
+        expect(pythonEdges(cg)).toEqual(fresh);
+
+        write({ 'pkg/helper.py': 'def other():\n    return 0\n' });
+        await cg.sync();
+        fresh = await freshEdges();
+        expect(fresh.join('\n')).not.toContain('assist');
+        expect(pythonEdges(cg)).toEqual(fresh);
+      });
+
+      it('links it when the module is added, and unlinks it when removed', async () => {
+        write({ 'pkg/__init__.py': '', 'pkg/run.py': importer });
+        cg = await CodeGraph.init(tempDir, { index: true });
+        expect(pythonEdges(cg).join('\n')).not.toContain('assist');
+
+        write({ 'pkg/helper.py': 'def assist(n):\n    return n\n' });
+        await cg.sync();
+        let fresh = await freshEdges();
+        expect(fresh).toContain('calls go@pkg/run.py -> function:assist@pkg/helper.py');
+        expect(pythonEdges(cg)).toEqual(fresh);
+
+        fs.rmSync(path.join(tempDir, 'pkg/helper.py'));
+        await cg.sync();
+        fresh = await freshEdges();
+        expect(fresh.join('\n')).not.toContain('assist');
+        expect(pythonEdges(cg)).toEqual(fresh);
+      });
+
+      // The name below the folder, and what the folder's own `__init__.py` holds.
+      it.each([
+        ['a module in a package below it', 'from sub.helper import assist', 'pkg/sub/helper.py', 'pkg/sub/__init__.py'],
+        ['a package below it', 'from sub import assist', 'pkg/sub/__init__.py', 'pkg/sub/mod.py'],
+      ])('follows an edit of %s', async (_label, line, moduleFile, otherFile) => {
+        write({
+          'pkg/__init__.py': '',
+          [otherFile]: '',
+          'pkg/run.py': `${line}\n\n\ndef go():\n    return assist(1)\n`,
+          [moduleFile]: 'def other():\n    return 0\n',
+        });
+        cg = await CodeGraph.init(tempDir, { index: true });
+        expect(pythonEdges(cg).join('\n')).not.toContain('assist');
+
+        write({ [moduleFile]: 'def assist(n):\n    return n\n' });
+        await cg.sync();
+        let fresh = await freshEdges();
+        expect(fresh).toContain(`calls go@pkg/run.py -> function:assist@${moduleFile}`);
+        expect(pythonEdges(cg)).toEqual(fresh);
+
+        write({ [moduleFile]: 'def other():\n    return 0\n' });
+        await cg.sync();
+        fresh = await freshEdges();
+        expect(fresh.join('\n')).not.toContain('assist');
+        expect(pythonEdges(cg)).toEqual(fresh);
+      });
+
+      it('re-opens nothing when an edited module has no such importer', async () => {
+        write({
+          'pkg/__init__.py': '',
+          'pkg/helper.py': 'def assist(n):\n    return n\n',
+          'pkg/run.py': 'from .helper import assist\n\n\ndef go():\n    return assist(1)\n',
+          // Spelled the same, but from a folder that doesn't hold the module.
+          'other/run.py': 'from helper import assist\n\n\ndef go():\n    return assist(1)\n',
+          // Beside it, but neither a Python import nor an import at all.
+          'pkg/widget.ts': "import { assist } from 'helper';\nexport const go = assist;\n",
+          'pkg/named.py': 'def helper():\n    return 0\n',
+        });
+        cg = await CodeGraph.init(tempDir, { index: true });
+        const spy = spyReopen(cg);
+
+        write({ 'pkg/helper.py': 'def assist(n):\n    return n\n\n\ndef more():\n    return 2\n' });
+        await cg.sync();
+
+        expect(spy.narrow).not.toHaveBeenCalled();
+      });
+
+      it('re-opens nothing for a module named after the stdlib module its neighbours import', async () => {
+        write({
+          'pkg/__init__.py': '',
+          'pkg/types.py': 'class Thing:\n    pass\n',
+          'pkg/run.py': 'from types import ModuleType\nfrom .types import Thing\n\n\ndef go():\n    return Thing()\n',
+        });
+        cg = await CodeGraph.init(tempDir, { index: true });
+        const spy = spyReopen(cg);
+
+        write({ 'pkg/types.py': 'class Thing:\n    pass\n\n\nclass More:\n    pass\n' });
+        await cg.sync();
+
+        expect(spy.narrow).not.toHaveBeenCalled();
+      });
     });
 
     it('re-opens nothing on a sync that changes nothing', async () => {

src/index.ts

@@ -1680,8 +1680,8 @@ export class CodeGraph {
               this.resolver.clearCaches();
               for (const f of this.orchestrator.resurrectPythonPackageImporters(this.resolver.getPythonPackageImporters(editedInits))) reopened.add(f);
             }
-            // An edit can give a module above a script the names its importers
-            // take from it, or take them away.
+            // An edit can give a module that only a script's folder provides the
+            // names its importers take from it, or take them away.
             const scriptModules = this.resolver.getPythonScriptModuleScope(
               (result.changedFilePaths ?? []).filter((f) => pythonFilesBefore.has(f) && after.has(f))
             );