|
| 1 | +# The grammars of the PHPDoc language |
| 2 | + |
| 3 | +The files in this directory describe the language `phpstan/phpdoc-parser` reads, |
| 4 | +in the [PP3 format](https://phplrt.org/docs/basics/grammar) the |
| 5 | +[phplrt](https://phplrt.org) compiler reads. They are the specification of the |
| 6 | +language, and they are what `tests/PHPStan/Parser/FuzzyTest.php` writes its |
| 7 | +corpus from. |
| 8 | + |
| 9 | +| File | What it holds | |
| 10 | +|---------------------|-----------------------------------------------------------------| |
| 11 | +| `lexemes.pp3` | Every token the language is read into | |
| 12 | +| `common.pp3` | What the grammars share: names, brackets, line breaks | |
| 13 | +| `types.pp3` | The type language of `TypeParser` | |
| 14 | +| `const-expr.pp3` | The constant expressions of `ConstExprParser` | |
| 15 | +| `phpdoc-block.pp3` | The PHPDoc itself: its tags, its text, its Doctrine annotations | |
| 16 | +| `type.pp3` | The entry point starting at `Type` | |
| 17 | +| `constant-expr.pp3` | The entry point starting at `ConstantExpr` | |
| 18 | +| `phpdoc.pp3` | The entry point starting at `PhpDoc` | |
| 19 | + |
| 20 | +## What they are for |
| 21 | + |
| 22 | +A grammar says what a PHPDoc may be written as, so it can be walked the other |
| 23 | +way round and asked for PHPDocs instead of being asked about one: |
| 24 | + |
| 25 | + make grammars-install # the toolchain, which asks for PHP 8.4 |
| 26 | + make fuzz # write a corpus into temp/fuzzy |
| 27 | + php vendor/bin/phpunit --filter FuzzyTest |
| 28 | + |
| 29 | +`tools/phplrt/fuzz.php` compiles a grammar, walks its rules at random and writes |
| 30 | +down what comes out. `FuzzyTest` then asks the parser to read every one of them |
| 31 | +in full, and to read a type back as the very same type once it has been printed. |
| 32 | +That is a great deal more of the language than a hand-written corpus covers, and |
| 33 | +it is what replaced the `abnfgen`-driven fuzzer this project used before. |
| 34 | + |
| 35 | +Nothing in `src/` reads these files, and the library needs neither the toolchain |
| 36 | +nor PHP 8.4: where either is missing, `FuzzyTest` skips itself. Only the |
| 37 | +`Grammars` job of `.github/workflows/build.yml` runs it for real. |
| 38 | + |
| 39 | +## How they are written |
| 40 | + |
| 41 | +The rules are named after the methods of `PhpDocParser`, `TypeParser` and |
| 42 | +`ConstExprParser` they stand for and are written in the order those methods try |
| 43 | +things in, so that a grammar and the parser it describes can be read side by |
| 44 | +side. |
| 45 | + |
| 46 | +### What the grammars describe |
| 47 | + |
| 48 | +**A PHPDoc that is written correctly.** The parser reads a broken one as well, |
| 49 | +by turning whatever it cannot read into an `InvalidTagValueNode` carrying the |
| 50 | +very error it has raised, and a grammar has no way of writing that error down. |
| 51 | +So what a broken PHPDoc means is left to the parser, and everything the grammars |
| 52 | +describe is something the parser has to read in full. |
| 53 | + |
| 54 | +Two things follow from wanting that to hold for **every** input rather than for |
| 55 | +most of them: |
| 56 | + |
| 57 | +- **A place the parser raises an error at is written as something the grammar |
| 58 | + cannot recognize.** Most of them are written as a `!` predicate forbidding |
| 59 | + whatever the error would have been raised on. For instance a name followed by |
| 60 | + a `<` has to go on into a generic type or into a callable, because `Foo<` is |
| 61 | + an error rather than the type `Foo` followed by something else: |
| 62 | + |
| 63 | + ``` |
| 64 | + IdentifierAtomic |
| 65 | + : ... |
| 66 | + | !ShapeBrace() Identifier() !<T_DOUBLE_COLON> ( IdentifierSuffix() | !<T_OPEN_ANGLE_BRACKET> ) |
| 67 | + ; |
| 68 | + ``` |
| 69 | + |
| 70 | + The same predicate is what keeps a rule from **giving back** what it has read. |
| 71 | + `@template T of` is an error rather than a template named `T` with the |
| 72 | + description `of`, so the bound is written as "either a bound or no `of` at |
| 73 | + all": |
| 74 | + |
| 75 | + ``` |
| 76 | + TemplateUpperBound |
| 77 | + : <T_KEYWORD_OF> Type() |
| 78 | + | <T_KEYWORD_AS> Type() |
| 79 | + | !<T_KEYWORD_OF> !<T_KEYWORD_AS> |
| 80 | + ; |
| 81 | + ``` |
| 82 | + |
| 83 | +- **A rule reads exactly the tokens its method reads**, down to the line breaks |
| 84 | + around it. |
| 85 | + |
| 86 | +### The tokens are not read by phplrt |
| 87 | + |
| 88 | +A grammar of this directory is not read by the lexer it declares: it is read by |
| 89 | +the very tokens `PHPStan\PhpDocParser\Lexer\Lexer` produces, handed over by |
| 90 | +`tools/phplrt/Fuzzer/TokenStream.php`. |
| 91 | + |
| 92 | +The `%token` declarations therefore name the tokens and document the language |
| 93 | +without being what reads it. Some of them describe something the lexer never |
| 94 | +reads as a token of its own, and `TokenStream` is what tells those apart in the |
| 95 | +stream: |
| 96 | + |
| 97 | +- a word the parser compares by value (`is`, `array`, `covariant`, `static`, …) |
| 98 | + — every one of them is still an ordinary name as well, which is why they are |
| 99 | + all listed among the alternatives of `Identifier`; |
| 100 | +- a tag whose value a rule of its own reads (`@param`, `@return`, …), told apart |
| 101 | + from the tags nothing reads the value of; |
| 102 | +- a bracket or an asterisk whose neighbouring whitespace decides what it means, |
| 103 | + which is what tells `array{a: int}` from the type `array` followed by a brace, |
| 104 | + and `Foo[0]` from `Foo [0]`; |
| 105 | +- a tag a space is written before, which is what tells the `@since` of |
| 106 | + `@author Foo @since 1.0` from the `@baz` of `@author Foo <foo@baz.com>`; |
| 107 | +- a `<` opening what the parser recognizes as an HTML tag, so that |
| 108 | + `@return Foo<br>see below</br>` keeps meaning the type `Foo` followed by a |
| 109 | + description. |
| 110 | + |
| 111 | +Telling them apart there is what lets the grammars be written without semantic |
| 112 | +predicates, which the PP3 format has none of. |
| 113 | + |
| 114 | +### What is left out |
| 115 | + |
| 116 | +Three corners of the language are left out on purpose, because a grammar cannot |
| 117 | +say what the parser does there. Each of them is written up where the rule that |
| 118 | +skirts it is written: |
| 119 | + |
| 120 | +- a description that ends at a tag written in the middle of a line, which the |
| 121 | + parser decides by reading the tag and looking at what it turns out to be; |
| 122 | +- the same, on a line after the first, where the parser reads that line twice: |
| 123 | + once as part of the description and again as whatever comes next; |
| 124 | +- a tag whose value a rule reads, written with a parenthesis after it, where |
| 125 | + whether the description ends there depends on whether that value can be read |
| 126 | + at all. |
0 commit comments