OverviewFilesCommitsIssuesDocsReleasesPeers

codetree commits / 0691230

GDScript expression floor, and the grammar ceiling it runs into
lib/derive-gdscript.js derives the same model as the JS floor — operations, wires, named wires, zones, exits — so the renderer never has to care which language produced a file. test/derive-gdscript.test.js asserts that directly: the SAME program written twice agrees on zone count, zone shapes, exit kinds, carried state and the loop's driving iterable. Three grammar facts established by dumping real parse trees, not by reading the node-name table: - `break` and `continue` have NO node in this grammar. Both arrive as IdentifierExpressionNode > Identifier and are identified by TEXT. Miss it and every loop exit silently becomes a free variable read. - `elif`/`else` are already SIBLING clauses of IfNode, so GDScript models Z1's bands natively. There is no ladder to absorb, unlike JS's nested `else if`. - A declaration and a write are the same node type: `var x = 1` and `x = 1` are both AssignmentExpressionNode, and only the first child differs. ## The ceiling, measured The vendored grammar is @gdquest/lezer-gdscript 0.1.2, and 0.1.2 is the LATEST published version — there is no upgrade to take, so closing this means forking a grammar. Of 33 GDScript 4 constructs, 11 emit error nodes: `:=` inferred typing, `static func`, member/index access on a call result (`get_tree().process_frame`), `enum`, ternary `a if c else b`, `Array[int]`, `^"path"`, `&"name"`, lua-style dicts, inline get/set. `await` itself is fine — the earlier suspicion was actually the member-on-call underneath it. On 300 real .gd files nothing threw, but 261 of 300 contained error nodes. ## E7 — a mis-parse is marked, never hidden lezer RECOVERS instead of failing, so a mis-parsed file yields a model that looks complete while structure has gone missing: a `for` whose Body was lost stops being a zone, and the `continue` inside it then lands nowhere. Any model with grammar error nodes or an exit landing nowhere now carries `partial: true` and a reason, and `codetree derive` prints it at the top. That is not hypothetical. material-maker/parse_args.gd derives with 111 error nodes, several loops reported as spanning a single line, and three `continue`s whose enclosing `for` was lost entirely. It is deliberately NOT a fixture: pinning tests to a repo we do not own is what has parser/traverse/tree red. ## Also fixed A foreach zone reported the wrong driving input. GDScript returned the item binding (`row`) and JS returned nothing at all; both now return the ITERABLE (`rows`), which is what drives the zone, and the cross-language test pins it. deriveFile() now dispatches on extension and names the language in the model; an unsupported extension says so instead of returning an empty model that looks like a file with no code in it. Tests: 14 GDScript tests / 166 asserts. The suite's 15 pre-existing failures (parser 8, traverse 2, tree 5) are unchanged and still the stale ../hyper-orgs pins. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
06912309685319c17ed8f2d438203ad39ca8d654
by plankatron <fleeky@gmail.com> · 2026-09-20T21:26:53.000Z

Changed files (7) +1067 -6

docs/EXPRESSION-FLOOR.md +53 -3
……@@ -262,9 +262,59 @@ four can reach the floor. Verified node names:
262262| `do..while` | `DoWhileStatement` | — | `DoStatement` | `do_statement` |
263263| labelled exit | `LabeledStatement` + labelled jump | unlabelled only | jumps + `goto` | `break`/`continue` |
264264
265JS is nearly free: its parser already walks every node. GDScript and C++ visit
266only top-level declarations today and need a body descent. GDScript has exactly
267two loop forms and is the cheapest to add.
265JS is nearly free: its parser already walks every node. C++ and GLSL visit only
266top-level declarations today and need a body descent.
267
268### GDScript: implemented, with a hard ceiling
269
270`lib/derive-gdscript.js` produces the same model, and `test/derive-gdscript.test.js`
271asserts that the SAME program in both languages agrees on zones, shapes, exits and
272carried state. Three grammar facts that are not guessable and were established by
273dumping real parse trees, not by reading the node-name table:
274
2751. **`break` and `continue` have no node.** Both arrive as
276 `IdentifierExpressionNode > Identifier` and must be identified BY TEXT. Miss it
277 and every loop exit silently becomes a free variable read.
2782. **`elif` / `else` are already sibling clauses** of `IfNode`, so GDScript models
279 bands the way Z1 wants. There is no ladder to absorb, unlike JS.
2803. **A declaration and a write are the same node type.** `var x = 1` and `x = 1`
281 are both `AssignmentExpressionNode`; only the first child differs.
282
283**The ceiling.** The vendored grammar is `@gdquest/lezer-gdscript` 0.1.2, and 0.1.2
284is the LATEST published version — there is no upgrade to take, so closing this
285means forking a grammar. Measured 2026-09-20 against 33 GDScript 4 constructs, 11
286produce error nodes:
287
288| broken | note |
289|---|---|
290| `var x := 0` | inferred typing, including `:=` defaults in params |
291| `static func` | |
292| member/index on a call result | `get_tree().process_frame`, `a().b()`, `a()[0]` — very common |
293| `enum E { A, B }` | |
294| ternary `a if c else b` | despite `TernaryOperatorExpressionNode` existing in the grammar |
295| `var a: Array[int]` | typed collections |
296| `^"path"` / `&"name"` | NodePath and StringName literals |
297| `{ a = 1 }` | lua-style dictionaries |
298| inline `get:` / `set:` | |
299
300Clean: plain/typed `var`, `const`, `func`, `->` return types, `@export`/`@tool`,
301`signal` with args, inner `class`, `await` itself, lambdas, `$Node`, `%Unique`,
302`match`, `for`, `while`, `super`, `assert`, multi-line arrays, `is`/`as`, `%`
303formatting, `breakpoint`.
304
305On 300 real `.gd` files nothing threw, but **261 of 300 contained error nodes**.
306
307### E7. A mis-parse is marked, never hidden
308
309lezer RECOVERS rather than failing, so a mis-parsed file yields a model that looks
310complete while structure has silently gone missing — a `for` whose Body was lost
311stops being a zone, and the `continue` inside it then lands nowhere. Any model with
312grammar error nodes or an exit that lands nowhere carries `partial: true` and a
313`partialReason`. Nothing downstream may treat a partial model as authoritative.
314
315Evidence for why this matters: `material-maker/parse_args.gd` derives with 111
316error nodes, several loops reported as spanning a single line, and three
317`continue`s whose enclosing `for` was lost entirely.
268318
269319---
270320
index.js +8 -1
……@@ -621,7 +621,14 @@ async function cmdDerive () {
621621 const rel = path.split('/').slice(-2).join('/')
622622 console.log(rel + ' — ' + model.nodes.length + ' nodes · ' + model.zones.length +
623623 ' zones · ' + model.exits.length + ' exits · ' + model.names.length +
624 ' names · ' + model.functions.length + ' functions')
624 ' names · ' + model.functions.length + ' functions' +
625 (model.language ? ' [' + model.language + ']' : ''))
626
627 // A partial model looks complete. Say so loudly, at the top (spec E7).
628 if (model.partial) {
629 console.log(' PARTIAL — ' + model.partialReason +
630 '; structure is missing, do not trust this model')
631 }
625632
626633 const zoneById = new Map(model.zones.map(z => [z.id, z]))
627634 const line = (r) => String(r.startLine).padStart(4)
lib/derive-gdscript.js +690 -0
……@@ -0,0 +1,690 @@
1/**
2 * The expression floor for GDScript.
3 *
4 * Same model as lib/derive.js produces for JS — operations, wires, named wires,
5 * zones, exits — because the renderer must not care which language a file is.
6 * Specified in docs/EXPRESSION-FLOOR.md.
7 *
8 * Three things about this grammar (vendor/lezer-gdscript) that are NOT guessable
9 * and were established by dumping a real parse tree:
10 *
11 * 1. `break` and `continue` have NO node in the grammar. Both arrive as
12 * `IdentifierExpressionNode > Identifier` and are identified BY TEXT. Miss
13 * this and every loop exit silently becomes a free variable read.
14 * 2. `elif` / `else` are already SIBLING clauses of `IfNode` (IfClause,
15 * ElifClause, ElseClause), so GDScript models bands the way the spec wants.
16 * There is no ladder to absorb, unlike JS where `else if` nests.
17 * 3. A declaration and a write are the SAME node type. `var x = 1` and `x = 1`
18 * are both `AssignmentExpressionNode`; only the first child differs
19 * (`VariableNode`/`ConstantNode` vs a bare target).
20 *
21 * Also: `var x := 0` predates this grammar, which emits a zero-width error node
22 * inside `TypeCast`. That is counted and reported, never fatal.
23 *
24 * Offsets are source offsets (UTF-16 code units), as they are on the JS side.
25 * The field names say `byte` for model compatibility.
26 */
27
28import { parser } from '../vendor/lezer-gdscript/lezer-gdscript.mjs'
29
30// R1 — native name -> the normalized kind the model speaks.
31const OPS = new Map([
32 ['CallExpressionNode', 'CallExpression'],
33 ['PreloadExpressionNode', 'CallExpression'],
34 ['AssertNode', 'CallExpression'],
35 ['BinaryOperatorExpressionNode', 'BinaryExpression'],
36 ['TypeTestExpressionNode', 'BinaryExpression'],
37 ['UnaryOperatorNode', 'UnaryExpression'],
38 ['TernaryOperatorExpressionNode', 'ConditionalExpression'],
39 ['ArrayExpressionNode', 'ArrayExpression'],
40 ['DictionaryExpressionNode', 'ObjectExpression'],
41 ['ReturnNode', 'ReturnStatement'],
42 ['AssignmentExpressionNode', 'AssignmentExpression']
43])
44
45// R2 — endpoints. A path, a reference, a literal: never a node.
46const WIRES = new Set([
47 'IdentifierExpressionNode', 'VariableName', 'LiteralExpressionNode',
48 'SelfExpressionNode', 'GetNodeExpressionNode', 'SubscriptExpressionNode',
49 'BuiltinConstants', 'String', 'True', 'False', 'Null', 'Identifier'
50])
51
52// R5 — transparent wrappers: unwrap to what they wrap, keep the marker.
53const FOLDED = new Set([
54 'AwaitExpressionNode', 'CastExpressionNode', 'GroupedExpressionNode'
55])
56
57const ZONES = new Set(['IfNode', 'ForNode', 'WhileNode', 'MatchNode'])
58const FUNCS = new Set(['FunctionNode', 'LambdaExpressionNode'])
59const OPERATORS = new Set(['ArithOp', 'BitOp', 'CompareOp', 'AndOp', 'OrOp', 'LogicOp', 'NotOp', 'in', 'is'])
60
61function isExpression (name) {
62 return OPS.has(name) || WIRES.has(name) || FOLDED.has(name) || FUNCS.has(name)
63}
64
65function children (node) {
66 const out = []
67 let c = node.firstChild
68 while (c) { out.push(c); c = c.nextSibling }
69 return out
70}
71
72function childByName (node, name) {
73 for (const c of children(node)) if (c.name === name) return c
74 return null
75}
76
77export function deriveGDScript (source, filePath) {
78 const model = {
79 filePath,
80 language: 'gdscript',
81 functions: [],
82 nodes: [],
83 names: [],
84 zones: [],
85 exits: [],
86 errors: 0,
87 partial: false,
88 partialReason: null
89 }
90
91 let tree
92 try {
93 tree = parser.parse(source)
94 } catch (err) {
95 return { ...model, error: err.message }
96 }
97
98 // --- positions. Lezer hands back offsets; the model wants lines too.
99 const lineStarts = [0]
100 for (let i = 0; i < source.length; i++) {
101 if (source[i] === '\n') lineStarts.push(i + 1)
102 }
103 function posAt (offset) {
104 let lo = 0
105 let hi = lineStarts.length - 1
106 while (lo < hi) {
107 const mid = (lo + hi + 1) >> 1
108 if (lineStarts[mid] <= offset) lo = mid; else hi = mid - 1
109 }
110 return { line: lo + 1, col: offset - lineStarts[lo] }
111 }
112 function rangeOf (node) {
113 const s = posAt(node.from)
114 const e = posAt(node.to)
115 return {
116 startByte: node.from,
117 endByte: node.to,
118 startLine: s.line,
119 startCol: s.col,
120 endLine: e.line,
121 endCol: e.col
122 }
123 }
124 const idOf = (node) => filePath + ':' + node.from + '-' + node.to
125 const textOf = (node) => source.slice(node.from, node.to)
126
127 // --- scopes (E5)
128 const scopes = [new Map()]
129 const declare = (name, kind) => { if (name) scopes[scopes.length - 1].set(name, kind) }
130 function resolve (name) {
131 for (let i = scopes.length - 1; i >= 0; i--) {
132 const kind = scopes[i].get(name)
133 if (kind) return kind
134 }
135 return 'free'
136 }
137
138 // R5 — a property path is one label. `a.b[c]` folds to `a.b[..]`.
139 function pathOf (node) {
140 if (!node) return null
141 const name = node.name
142 if (name === 'Identifier' || name === 'VariableName') return textOf(node).trim()
143 if (name === 'SelfExpressionNode') return 'self'
144 if (name === 'IdentifierExpressionNode') {
145 const id = childByName(node, 'Identifier')
146 return id ? textOf(id) : textOf(node).trim()
147 }
148 if (name === 'GetNodeExpressionNode') return textOf(node).trim()
149 if (name === 'LiteralExpressionNode') return textOf(node).trim()
150 if (name === 'SubscriptExpressionNode') {
151 let out = ''
152 let bracket = false
153 for (const c of children(node)) {
154 if (c.name === '[') { bracket = true; out += '[..]'; continue }
155 if (c.name === ']') { bracket = false; continue }
156 if (bracket) continue
157 if (c.name === '.') { out += '.'; continue }
158 const part = pathOf(c)
159 if (part === null) return null
160 out += part
161 }
162 return out
163 }
164 return null
165 }
166
167 // The identifier a path is rooted in — what gets resolved.
168 function rootOf (node) {
169 let cur = node
170 while (cur) {
171 if (cur.name === 'SubscriptExpressionNode') { cur = cur.firstChild; continue }
172 if (cur.name === 'IdentifierExpressionNode' || cur.name === 'VariableName') {
173 const id = childByName(cur, 'Identifier')
174 return id || cur
175 }
176 return cur
177 }
178 return null
179 }
180
181 function unwrap (node) {
182 const flags = {}
183 let cur = node
184 while (cur && FOLDED.has(cur.name)) {
185 if (cur.name === 'AwaitExpressionNode') flags.awaited = true
186 const next = children(cur).find(c => isExpression(c.name))
187 if (!next) break
188 cur = next
189 }
190 return { node: cur, flags }
191 }
192
193 function operatorLabel (node) {
194 for (const c of children(node)) if (OPERATORS.has(c.name)) return textOf(c)
195 return node.name
196 }
197
198 function labelFor (node, kind) {
199 switch (node.name) {
200 case 'CallExpressionNode': {
201 const callee = children(node).find(c => isExpression(c.name))
202 const p = callee ? pathOf(callee) : null
203 return p === null ? '(expr)' : p
204 }
205 case 'PreloadExpressionNode': return 'preload'
206 case 'AssertNode': return 'assert'
207 case 'BinaryOperatorExpressionNode': return operatorLabel(node)
208 case 'TypeTestExpressionNode': return 'is'
209 case 'UnaryOperatorNode': return operatorLabel(node)
210 case 'TernaryOperatorExpressionNode': return '?:'
211 case 'ArrayExpressionNode': return '[]'
212 case 'DictionaryExpressionNode': return '{}'
213 case 'ReturnNode': return 'return'
214 case 'AssignmentExpressionNode': return '='
215 default: return kind
216 }
217 }
218
219 // `var x = 1` and `x = 1` are the same node type; the first child decides.
220 function declaratorOf (node) {
221 if (node.name !== 'AssignmentExpressionNode') return null
222 const first = node.firstChild
223 if (!first) return null
224 if (first.name === 'VariableNode' || first.name === 'ConstantNode') return first
225 return null
226 }
227
228 function declaredName (declNode) {
229 const id = childByName(declNode, 'Identifier')
230 return id ? textOf(id) : null
231 }
232
233 function inputFor (raw, slot) {
234 const { node, flags } = unwrap(raw)
235 if (!node) return null
236 const base = { slot }
237 if (flags.awaited) base.awaited = true
238
239 if (node.name === 'LiteralExpressionNode' || node.name === 'String' ||
240 node.name === 'BuiltinConstants' || node.name === 'True' ||
241 node.name === 'False' || node.name === 'Null') {
242 return { ...base, name: textOf(node).trim(), literal: true }
243 }
244 if (WIRES.has(node.name)) {
245 const path = pathOf(node)
246 const root = rootOf(node)
247 if (path !== null) {
248 const rootName = root ? textOf(root).trim() : null
249 return {
250 ...base,
251 name: path,
252 source: root && root.name === 'SelfExpressionNode' ? 'this' : resolve(rootName)
253 }
254 }
255 }
256 if (FUNCS.has(node.name)) return { ...base, name: '(fn)', source: 'closure', node: idOf(node) }
257 if (OPS.has(node.name)) {
258 return { ...base, name: labelFor(node, OPS.get(node.name)), source: 'node', node: idOf(node) }
259 }
260 return { ...base, name: node.name, source: 'expr' }
261 }
262
263 function operandsOf (node) {
264 const kids = children(node)
265 switch (node.name) {
266 case 'CallExpressionNode': {
267 const out = []
268 const callee = kids.find(c => isExpression(c.name))
269 if (callee) out.push([callee, 'callee'])
270 const params = childByName(node, 'CallParamsExpressionNode')
271 if (params) {
272 let i = 0
273 for (const c of children(params)) {
274 if (!isExpression(c.name)) continue
275 out.push([c, 'arg' + i])
276 i++
277 }
278 }
279 return out
280 }
281 case 'PreloadExpressionNode':
282 case 'AssertNode': {
283 const out = []
284 let i = 0
285 for (const c of kids) {
286 if (c.name === 'CallParamsExpressionNode') {
287 for (const p of children(c)) {
288 if (!isExpression(p.name)) continue
289 out.push([p, 'arg' + i]); i++
290 }
291 continue
292 }
293 if (!isExpression(c.name)) continue
294 out.push([c, 'arg' + i]); i++
295 }
296 return out
297 }
298 case 'BinaryOperatorExpressionNode': {
299 const ops = kids.filter(c => isExpression(c.name))
300 return ops.length >= 2 ? [[ops[0], 'left'], [ops[1], 'right']] : ops.map(o => [o, 'value'])
301 }
302 case 'TypeTestExpressionNode': {
303 const ops = kids.filter(c => isExpression(c.name))
304 return ops.length > 0 ? [[ops[0], 'value']] : []
305 }
306 case 'UnaryOperatorNode': {
307 const ops = kids.filter(c => isExpression(c.name))
308 return ops.length > 0 ? [[ops[0], 'value']] : []
309 }
310 case 'TernaryOperatorExpressionNode': {
311 const ops = kids.filter(c => isExpression(c.name))
312 const slots = ['then', 'test', 'else']
313 return ops.map((o, i) => [o, slots[i] || ('item' + i)])
314 }
315 case 'ArrayExpressionNode':
316 return kids.filter(c => isExpression(c.name)).map((c, i) => [c, 'item' + i])
317 case 'DictionaryExpressionNode': {
318 const out = []
319 let i = 0
320 for (const entry of kids) {
321 if (entry.name !== 'DictionaryEntry') continue
322 const parts = children(entry).filter(c => isExpression(c.name))
323 const value = parts.length > 1 ? parts[parts.length - 1] : parts[0]
324 if (value) { out.push([value, 'item' + i]); i++ }
325 }
326 return out
327 }
328 case 'ReturnNode': {
329 const ops = kids.filter(c => isExpression(c.name))
330 return ops.length > 0 ? [[ops[0], 'value']] : []
331 }
332 case 'AssignmentExpressionNode': {
333 // Only the VALUE flows in; the target is a write.
334 const ops = kids.filter(c => isExpression(c.name))
335 const value = ops.length > 1 ? ops[ops.length - 1] : (declaratorOf(node) ? ops[0] : null)
336 return value ? [[value, 'value']] : []
337 }
338 default:
339 return []
340 }
341 }
342
343 // Z1 — shape follows control flow. A body holding exactly one jump is a guard.
344 function bodyJumpOnly (clause) {
345 const body = childByName(clause, 'Body')
346 if (!body) return false
347 const jumps = []
348 let count = 0
349 const visit = (n) => {
350 for (const c of children(n)) {
351 if (c.name === 'SimpleStatement' || c.name === 'CompoundStatement') count++
352 if (c.name === 'ReturnNode') jumps.push('return')
353 if (c.name === 'IdentifierExpressionNode') {
354 const text = textOf(c).trim()
355 if (text === 'break' || text === 'continue') jumps.push(text)
356 }
357 if (c.name === 'SuiteNodeBody' || c.name === 'SuiteNode' ||
358 c.name === 'SimpleStatement' || c.name === 'SmallStatement') visit(c)
359 }
360 }
361 visit(body)
362 return count === 1 && jumps.length === 1
363 }
364
365 function shapeOfZone (node) {
366 if (node.name === 'ForNode') return 'foreach'
367 if (node.name === 'WhileNode') return 'repeat'
368 if (node.name === 'MatchNode') return 'switch'
369 if (node.name === 'IfNode') {
370 const kids = children(node)
371 const hasElse = kids.some(c => c.name === 'ElifClause' || c.name === 'ElseClause')
372 if (hasElse) return 'two-exit'
373 const ifClause = childByName(node, 'IfClause')
374 if (ifClause && bodyJumpOnly(ifClause)) return 'guard'
375 return 'single-branch'
376 }
377 return node.name
378 }
379
380 // The zone's driving input. For a branch that is the test; for a foreach it is
381 // the ITERABLE, which means skipping past the item binding — `for row in rows`
382 // is driven by `rows`, not by `row`.
383 function conditionOf (node) {
384 const holder = node.name === 'IfNode' ? childByName(node, 'IfClause') : node
385 if (!holder) return null
386 if (node.name === 'ForNode') {
387 let seenIn = false
388 for (const c of children(holder)) {
389 if (c.name === 'in') { seenIn = true; continue }
390 if (!seenIn) continue
391 if (isExpression(c.name)) return { text: textOf(c).replace(/\s+/g, ' '), range: rangeOf(c) }
392 }
393 return null
394 }
395 for (const c of children(holder)) {
396 if (isExpression(c.name)) return { text: textOf(c).replace(/\s+/g, ' '), range: rangeOf(c) }
397 }
398 return null
399 }
400
401 // ForNode: `for` Identifier `in` <iterable> Body — the item is the Identifier.
402 function itemOf (node) {
403 if (node.name !== 'ForNode') return null
404 const id = childByName(node, 'Identifier')
405 return id ? textOf(id) : null
406 }
407
408 function bandsOf (node) {
409 if (node.name === 'IfNode') {
410 const out = []
411 for (const c of children(node)) {
412 if (c.name === 'IfClause') out.push({ kind: 'then', test: conditionOf(c), range: rangeOf(c) })
413 if (c.name === 'ElifClause') out.push({ kind: 'else-if', test: conditionOf(c), range: rangeOf(c) })
414 if (c.name === 'ElseClause') out.push({ kind: 'else', test: null, range: rangeOf(c) })
415 }
416 return out.length > 1 ? out : null
417 }
418 if (node.name === 'MatchNode') {
419 const out = []
420 for (const c of children(node)) {
421 if (c.name !== 'MatchBranchNode') continue
422 const pattern = childByName(c, 'PatternNode')
423 out.push({
424 kind: 'case',
425 test: pattern ? { text: textOf(pattern).trim(), range: rangeOf(pattern) } : null,
426 range: rangeOf(c)
427 })
428 }
429 return out.length > 0 ? out : null
430 }
431 return null
432 }
433
434 // --- the walk
435 const writes = []
436 const reads = []
437 const declaredAt = new Map()
438 const zoneStack = []
439 const fnStack = []
440
441 function nearestLoop () {
442 for (let i = zoneStack.length - 1; i >= 0; i--) {
443 if (zoneStack[i].kind === 'loop') return zoneStack[i]
444 }
445 return null
446 }
447 function nearestBreakTarget () {
448 for (let i = zoneStack.length - 1; i >= 0; i--) {
449 const z = zoneStack[i]
450 if (z.kind === 'loop' || z.shape === 'switch') return z
451 }
452 return null
453 }
454
455 // Hoist one scope's declarations before walking it, so a method that reads a
456 // class var declared further down still resolves (E5).
457 function hoist (node) {
458 for (const c of children(node)) {
459 if (c.name === 'SimpleStatement' || c.name === 'SmallStatement' ||
460 c.name === 'CompoundStatement' || c.name === 'StatementGroup') {
461 hoist(c)
462 continue
463 }
464 if (c.name === 'AssignmentExpressionNode') {
465 const decl = declaratorOf(c)
466 if (decl) declare(declaredName(decl), 'local')
467 continue
468 }
469 if (c.name === 'VariableNode' || c.name === 'ConstantNode') {
470 declare(declaredName(c), 'local')
471 continue
472 }
473 if (c.name === 'FunctionNode') {
474 const id = childByName(c, 'Identifier')
475 if (id) declare(textOf(id), 'function')
476 continue
477 }
478 if (c.name === 'SignalStatement') {
479 const id = childByName(c, 'Identifier')
480 if (id) declare(textOf(id), 'signal')
481 continue
482 }
483 if (c.name === 'ClassNameStatement' || c.name === 'ExtendsStatement') {
484 const id = childByName(c, 'Identifier')
485 if (id) declare(textOf(id), 'type')
486 }
487 }
488 }
489
490 function walk (node) {
491 if (node.name === '⚠') model.errors++
492
493 let pushedScope = false
494 let pushedZone = null
495 let pushedFn = null
496
497 if (FUNCS.has(node.name)) {
498 const id = childByName(node, 'Identifier')
499 const fn = {
500 id: idOf(node),
501 name: id ? textOf(id) : '(lambda)',
502 kind: node.name,
503 range: rangeOf(node),
504 params: [],
505 async: false,
506 generator: false
507 }
508 scopes.push(new Map())
509 pushedScope = true
510 const params = childByName(node, 'CallParamsParameterNode')
511 if (params) {
512 for (const p of children(params)) {
513 if (p.name !== 'ParameterNode') continue
514 const pid = childByName(p, 'Identifier')
515 const name = pid ? textOf(pid) : '(pattern)'
516 declare(name, 'param')
517 fn.params.push(name)
518 }
519 }
520 const body = childByName(node, 'Body')
521 if (body) hoist(body)
522 model.functions.push(fn)
523 fnStack.push(fn)
524 pushedFn = fn
525 }
526
527 if (ZONES.has(node.name)) {
528 const parent = zoneStack.length > 0 ? zoneStack[zoneStack.length - 1] : null
529 const zone = {
530 id: idOf(node),
531 type: node.name,
532 kind: node.name === 'ForNode' || node.name === 'WhileNode' ? 'loop' : 'branch',
533 shape: shapeOfZone(node),
534 range: rangeOf(node),
535 depth: parent ? parent.depth + 1 : 1,
536 parent: parent ? parent.id : null,
537 condition: conditionOf(node),
538 item: itemOf(node),
539 bands: bandsOf(node),
540 carried: [],
541 exits: []
542 }
543 model.zones.push(zone)
544 zoneStack.push(zone)
545 pushedZone = zone
546 if (zone.item) {
547 scopes.push(new Map())
548 declare(zone.item, 'local')
549 zone.__scoped = true
550 }
551 }
552
553 // A declaration names a wire (R3) and is NOT an operation.
554 const decl = declaratorOf(node)
555 if (decl) {
556 const name = declaredName(decl)
557 const value = children(node).filter(c => isExpression(c.name)).pop()
558 const entry = { name, range: rangeOf(node), from: 'none', node: null }
559 if (value) {
560 const inner = unwrap(value).node
561 if (inner.name === 'LiteralExpressionNode' || inner.name === 'String' ||
562 inner.name === 'BuiltinConstants' || inner.name === 'True' ||
563 inner.name === 'False' || inner.name === 'Null') entry.from = 'literal'
564 else if (OPS.has(inner.name)) { entry.from = 'node'; entry.node = idOf(inner) }
565 else if (FUNCS.has(inner.name)) { entry.from = 'closure'; entry.node = idOf(inner) }
566 else entry.from = 'wire'
567 }
568 if (name) {
569 model.names.push(entry)
570 declaredAt.set(name, node.from)
571 }
572 } else if (node.name === 'VariableNode' || node.name === 'ConstantNode') {
573 // `var x` with no value, standing alone.
574 const parentIsAssign = node.parent && node.parent.name === 'AssignmentExpressionNode'
575 const name = declaredName(node)
576 if (!parentIsAssign && name) {
577 model.names.push({ name, range: rangeOf(node), from: 'none', node: null })
578 declaredAt.set(name, node.from)
579 }
580 } else if (OPS.has(node.name)) {
581 const kind = OPS.get(node.name)
582 const op = {
583 id: idOf(node),
584 kind,
585 label: labelFor(node, kind),
586 range: rangeOf(node),
587 zone: zoneStack.length > 0 ? zoneStack[zoneStack.length - 1].id : null,
588 fn: fnStack.length > 0 ? fnStack[fnStack.length - 1].id : null,
589 inputs: []
590 }
591 for (const [operand, slot] of operandsOf(node)) {
592 const input = inputFor(operand, slot)
593 if (input) op.inputs.push(input)
594 }
595 if (node.name === 'AssignmentExpressionNode') {
596 const target = children(node).find(c => isExpression(c.name))
597 if (target) {
598 const path = pathOf(target)
599 const root = rootOf(target)
600 if (path) op.writes = path
601 if (root) {
602 const rootName = textOf(root).trim()
603 op.writesRoot = rootName
604 writes.push({ name: rootName, at: rangeOf(node) })
605 }
606 }
607 }
608 model.nodes.push(op)
609 }
610
611 // --- exits (Z4). `return` has a node; break/continue do NOT (see header).
612 let exitKind = null
613 if (node.name === 'ReturnNode') exitKind = 'return'
614 else if (node.name === 'IdentifierExpressionNode') {
615 const text = textOf(node).trim()
616 if (text === 'break') exitKind = 'break'
617 else if (text === 'continue') exitKind = 'continue'
618 }
619
620 if (exitKind) {
621 const fn = fnStack[fnStack.length - 1]
622 let lands = { on: 'unresolved', id: null }
623 if (exitKind === 'return') {
624 lands = fn ? { on: 'function-output', id: fn.id } : { on: 'module-output', id: filePath }
625 } else if (exitKind === 'continue') {
626 const loop = nearestLoop()
627 if (loop) lands = { on: 'zone-entry', id: loop.id }
628 } else if (exitKind === 'break') {
629 const target = nearestBreakTarget()
630 if (target) lands = { on: 'zone-exit', id: target.id }
631 }
632 const exitId = idOf(node) + ':exit'
633 model.exits.push({
634 id: exitId,
635 kind: exitKind,
636 node: node.name === 'ReturnNode' ? idOf(node) : null,
637 range: rangeOf(node),
638 zone: zoneStack.length > 0 ? zoneStack[zoneStack.length - 1].id : null,
639 lands
640 })
641 if (zoneStack.length > 0) zoneStack[zoneStack.length - 1].exits.push(exitId)
642 }
643
644 if (node.name === 'Identifier') {
645 const text = textOf(node)
646 if (text !== 'break' && text !== 'continue') reads.push({ name: text, at: node.from })
647 }
648
649 for (const c of children(node)) walk(c)
650
651 if (pushedZone) {
652 if (pushedZone.__scoped) { scopes.pop(); delete pushedZone.__scoped }
653 zoneStack.pop()
654 }
655 if (pushedFn) fnStack.pop()
656 if (pushedScope) scopes.pop()
657 }
658
659 const root = tree.topNode
660 hoist(root)
661 walk(root)
662
663 // A mis-parse is not a parse failure: lezer recovers and hands back a tree with
664 // error nodes in it, so the model looks complete while structure has silently
665 // gone missing — a `for` whose Body was lost stops being a zone, and the
666 // `continue` inside it then lands nowhere. Mark the model PARTIAL so nothing
667 // downstream treats it as authoritative. See the capability table in
668 // docs/EXPRESSION-FLOOR.md: 11 of 33 GDScript 4 constructs trip this grammar.
669 const orphanExits = model.exits.filter(e => e.lands.on === 'unresolved').length
670 if (model.errors > 0 || orphanExits > 0) {
671 model.partial = true
672 model.partialReason = (model.errors > 0 ? model.errors + ' grammar error node(s)' : '') +
673 (model.errors > 0 && orphanExits > 0 ? '; ' : '') +
674 (orphanExits > 0 ? orphanExits + ' exit(s) landing nowhere' : '')
675 }
676
677 // Z3 — a zone's outputs are its phis. Identical rule to the JS side.
678 for (const zone of model.zones) {
679 const carried = new Set()
680 for (const w of writes) {
681 if (w.at.startByte < zone.range.startByte || w.at.endByte > zone.range.endByte) continue
682 const at = declaredAt.get(w.name)
683 if (at !== undefined && at >= zone.range.startByte) continue
684 if (reads.some(r => r.name === w.name && r.at > zone.range.endByte)) carried.add(w.name)
685 }
686 zone.carried = [...carried].sort()
687 }
688
689 return model
690}
lib/derive.js +45 -2
……@@ -22,6 +22,7 @@
2222 */
2323
2424import { parse as acornParse } from '../vendor/acorn/dist/acorn.mjs'
25import { deriveGDScript } from './derive-gdscript.js'
2526
2627const PARSE_OPTS = {
2728 ecmaVersion: 'latest',
……@@ -251,8 +252,14 @@ function shapeOfZone (node) {
251252 return node.type
252253}
253254
255// The zone's driving input: a branch's test, a switch's discriminant, or — for
256// for-of/for-in, which have neither — the ITERABLE being walked. Without that
257// last case a foreach zone reported no driver at all, and the GDScript side
258// reported the item binding instead, so the two languages disagreed on the same
259// program.
254260function conditionOf (node, source) {
255 const test = node.test || node.discriminant
261 const test = node.test || node.discriminant ||
262 ((node.type === 'ForOfStatement' || node.type === 'ForInStatement') ? node.right : null)
256263 if (!test) return null
257264 return { text: source.slice(test.range[0], test.range[1]), range: rangeOf(test) }
258265}
……@@ -269,9 +276,10 @@ function itemOf (node) {
269276 return null
270277}
271278
272export function deriveFile (source, filePath) {
279export function deriveJS (source, filePath) {
273280 const empty = {
274281 filePath,
282 language: 'javascript',
275283 functions: [],
276284 nodes: [],
277285 names: [],
……@@ -609,3 +617,38 @@ export function deriveFile (source, filePath) {
609617
610618 return model
611619}
620
621/**
622 * Pick a derivation by extension.
623 *
624 * The model shape is identical whichever language answers — that is the whole
625 * point, because the renderer positions nodes by source range and must not care
626 * what produced them. A language we cannot derive yet says so in `error` rather
627 * than returning an empty model that looks like a file with no code in it.
628 */
629export function deriveFile (source, filePath) {
630 const path = String(filePath || '')
631 const dot = path.lastIndexOf('.')
632 const ext = dot === -1 ? '' : path.slice(dot).toLowerCase()
633
634 switch (ext) {
635 case '.js':
636 case '.mjs':
637 case '.cjs':
638 return deriveJS(source, filePath)
639 case '.gd':
640 return deriveGDScript(source, filePath)
641 default:
642 return {
643 filePath,
644 language: null,
645 functions: [],
646 nodes: [],
647 names: [],
648 zones: [],
649 exits: [],
650 error: 'no derivation for ' + (ext || 'a file with no extension') +
651 ' — JS and GDScript only (C++ and GLSL keep a full tree and can follow)'
652 }
653 }
654}
test/derive-gdscript.test.js +237 -0
……@@ -0,0 +1,237 @@
1import { test } from 'brittle'
2import { resolve, dirname } from 'path'
3import { fileURLToPath } from 'url'
4import fs from 'fs'
5import { deriveFile } from '../lib/derive.js'
6
7const HERE = dirname(fileURLToPath(import.meta.url))
8const FIXTURE = resolve(HERE, 'fixtures/derive')
9
10// Ground truth for fixtures/derive/sample.gd, counted by hand.
11//
12// tally()
13// rows == null binary op + GUARD zone (return null)
14// return null return op + exit -> function output
15// for row in rows FOREACH zone, item 'row', carried ['total']
16// row.skip a path, folds to a socket label (R5)
17// continue exit -> the foreach zone's entry, in a GUARD zone
18// (parses as IdentifierExpressionNode: this grammar
19// has NO break/continue node at all)
20// total + row.count binary op
21// total = ... assign op (writes the class-level 'total' wire)
22// total > limit binary op + GUARD zone (break)
23// break exit -> the foreach zone's exit
24// total > 0 binary op
25// if/else TWO-EXIT zone, 2 bands; both paths exit
26// return total, return 0 two return ops + two exits
27//
28// label()
29// Time.get_ticks_msec() call op, 'Time' is a FREE wire
30// str(row.name, at) call op, 'str' is FREE
31// return return op + exit
32//
33// Totals: 2 functions, 11 operation nodes, 5 zones, 6 exits, 2 named wires
34// ('total' at class scope, 'at' in label; the loop's 'row' is the
35// zone's item, not a named wire)
36
37function derive (name) {
38 const path = resolve(FIXTURE, name)
39 return deriveFile(fs.readFileSync(path, 'utf8'), path)
40}
41
42test('derive gd: the counts the fixture documents', (t) => {
43 const m = derive('sample.gd')
44
45 t.absent(m.error, 'no error')
46 t.is(m.language, 'gdscript', 'reports its language')
47 t.is(m.functions.length, 2, '2 functions')
48 t.is(m.nodes.length, 11, '11 operation nodes')
49 t.is(m.zones.length, 5, '5 zones')
50 t.is(m.exits.length, 6, '6 exits')
51 t.is(m.names.length, 2, '2 named wires')
52})
53
54test('derive gd: Z1 — three conditional shapes', (t) => {
55 const m = derive('sample.gd')
56 const shapes = {}
57 for (const z of m.zones) shapes[z.shape] = (shapes[z.shape] || 0) + 1
58
59 t.is(shapes.guard, 3, '3 guards')
60 t.is(shapes.foreach, 1, '1 foreach loop')
61 t.is(shapes['two-exit'], 1, '1 two-exit conditional')
62
63 const cond = m.zones.find(z => z.shape === 'two-exit')
64 t.is(cond.bands.length, 2, 'then + else')
65})
66
67test('derive gd: break and continue resolve despite having no grammar node', (t) => {
68 const m = derive('sample.gd')
69 const loop = m.zones.find(z => z.shape === 'foreach')
70
71 const cont = m.exits.find(e => e.kind === 'continue')
72 const brk = m.exits.find(e => e.kind === 'break')
73 t.ok(cont, 'found the continue')
74 t.ok(brk, 'found the break')
75 t.is(cont.lands.on, 'zone-entry', 'continue lands on a zone entry')
76 t.is(cont.lands.id, loop.id, 'the enclosing loop')
77 t.is(brk.lands.on, 'zone-exit', 'break lands on a zone exit')
78 t.is(brk.lands.id, loop.id, 'the enclosing loop')
79
80 // And they are NOT mistaken for operations or wires.
81 const labels = m.nodes.map(n => n.label)
82 t.absent(labels.includes('break'), 'break is not an operation')
83 t.absent(labels.includes('continue'), 'continue is not an operation')
84})
85
86test('derive gd: Z3 — carried state', (t) => {
87 const m = derive('sample.gd')
88 const loop = m.zones.find(z => z.shape === 'foreach')
89 t.alike(loop.carried, ['total'], 'the loop carries total')
90 t.is(loop.item, 'row', 'and binds row as its item')
91
92 const cond = m.zones.find(z => z.shape === 'two-exit')
93 t.alike(cond.carried, [], 'the two-exit zone converges nothing')
94})
95
96test('derive gd: R5 — a property path folds into a socket label', (t) => {
97 const m = derive('sample.gd')
98 const add = m.nodes.find(n => n.label === '+')
99 t.ok(add, 'found the + node')
100 const names = add.inputs.map(i => i.name)
101 t.ok(names.includes('total'), 'left input is the total wire')
102 t.ok(names.includes('row.count'), 'right input is the folded path')
103
104 const kinds = m.nodes.map(n => n.kind)
105 t.absent(kinds.includes('SubscriptExpressionNode'), 'no path node survives')
106})
107
108test('derive gd: R3 — declarations name wires', (t) => {
109 const m = derive('sample.gd')
110 const names = m.names.map(n => n.name).sort()
111 t.alike(names, ['at', 'total'], 'total and at')
112 t.is(m.names.find(n => n.name === 'total').from, 'literal', 'total from a literal')
113 t.is(m.names.find(n => n.name === 'at').from, 'node', 'at from an operation')
114})
115
116test('derive gd: E5 — references resolve or are marked free', (t) => {
117 const m = derive('sample.gd')
118 const call = m.nodes.find(n => n.label === 'Time.get_ticks_msec')
119 t.ok(call, 'found the call')
120 t.is(call.inputs[0].source, 'free', 'Time is free — a GDScript global')
121
122 const add = m.nodes.find(n => n.label === '+')
123 const totalIn = add.inputs.find(i => i.name === 'total')
124 t.is(totalIn.source, 'local', 'the class-level var resolves')
125
126 for (const n of m.nodes) {
127 for (const i of n.inputs) {
128 if (i.literal) continue
129 t.ok(i.source, 'every non-literal input names where it came from')
130 }
131 }
132})
133
134test('derive gd: E2/R7 — positions and derived ids', (t) => {
135 const m = derive('sample.gd')
136 for (const list of [m.nodes, m.zones, m.exits, m.names]) {
137 for (const item of list) {
138 t.ok(Number.isInteger(item.range.startByte), 'has startByte')
139 t.ok(item.range.startLine >= 1, 'has a 1-based line')
140 t.ok(item.range.endLine >= item.range.startLine, 'endLine follows')
141 }
142 }
143 for (const n of m.nodes) {
144 t.is(n.id, m.filePath + ':' + n.range.startByte + '-' + n.range.endByte, 'id is file:start-end')
145 }
146})
147
148test('derive gd: E1 — the derivation is pure', (t) => {
149 const a = derive('sample.gd')
150 const b = derive('sample.gd')
151 t.is(JSON.stringify(a), JSON.stringify(b), 'byte-identical model')
152})
153
154test('derive gd: an error node does not stop the derivation', (t) => {
155 const m = derive('inferred.gd')
156 t.ok(m.errors > 0, 'the := is reported as a grammar error')
157 t.absent(m.error, 'but the file still derives')
158 t.alike(m.names.map(n => n.name), ['count'], 'and the wire is still named')
159 t.ok(m.functions.length === 1, 'and the function is still found')
160})
161
162test('derive gd: a mis-parse marks the model PARTIAL', (t) => {
163 // lezer RECOVERS from a construct it does not know, so the model looks
164 // complete while structure has gone missing. That must be visible.
165 const clean = derive('sample.gd')
166 t.absent(clean.partial, 'the clean fixture is not partial')
167
168 const m = derive('inferred.gd')
169 t.ok(m.partial, ':= makes the model partial')
170 t.ok(m.partialReason.includes('grammar error'), 'and says why: ' + m.partialReason)
171})
172
173test('derive gd: `static func` is mis-parsed, so the model says PARTIAL', (t) => {
174 const src = [
175 'static func f(rows):',
176 '\tfor row in rows:',
177 '\t\tif row.skip:',
178 '\t\t\tcontinue',
179 '\treturn 0',
180 ''
181 ].join('\n')
182 const m = deriveFile(src, resolve(FIXTURE, 'synthetic-static.gd'))
183
184 t.ok(m.errors > 0, 'the grammar reports errors on `static func`')
185 t.ok(m.partial, 'so the model is partial')
186 // Recovery is good enough HERE that the loop survives and the continue still
187 // lands. On real code it is not: material-maker/parse_args.gd has 111 error
188 // nodes, loops that come back spanning a single line, and three continues
189 // whose enclosing `for` was lost entirely. That file is not a fixture on
190 // purpose — pinning a test to a repo we do not own is the mistake that has
191 // parser/traverse/tree red right now.
192 t.is(m.exits.filter(e => e.lands.on === 'unresolved').length, 0,
193 'this small case still resolves, which is why PARTIAL is the signal, not the exits')
194})
195
196test('derive gd: an exit with nowhere to land is reported, never dropped (E3)', (t) => {
197 // A `continue` outside any loop. Invalid GDScript, which the grammar happily
198 // reads as an identifier — so this is also how a LOST loop presents.
199 const m = deriveFile('func f():\n\tcontinue\n', resolve(FIXTURE, 'synthetic-orphan.gd'))
200
201 const orphans = m.exits.filter(e => e.lands.on === 'unresolved')
202 t.is(orphans.length, 1, 'the continue is still reported')
203 t.is(orphans[0].kind, 'continue', 'as a continue')
204 t.is(orphans[0].lands.id, null, 'with no landing id')
205 t.ok(m.partial, 'and the model is partial')
206 t.ok(m.partialReason.includes('landing nowhere'), 'saying so: ' + m.partialReason)
207})
208
209// The point of the model: the SAME program in two languages agrees on control
210// flow. Node counts differ (JS has a template literal), the structure does not.
211test('derive: the JS and GDScript twins agree on structure', (t) => {
212 const js = derive('sample.js')
213 const gd = derive('sample.gd')
214
215 const shapes = (m) => {
216 const out = {}
217 for (const z of m.zones) out[z.shape] = (out[z.shape] || 0) + 1
218 return out
219 }
220 const exits = (m) => {
221 const out = {}
222 for (const e of m.exits) out[e.kind] = (out[e.kind] || 0) + 1
223 return out
224 }
225
226 t.is(gd.zones.length, js.zones.length, 'same zone count')
227 t.alike(shapes(gd), shapes(js), 'same zone shapes')
228 t.alike(exits(gd), exits(js), 'same exit kinds')
229 t.is(gd.functions.length, js.functions.length, 'same function count')
230
231 const gdLoop = gd.zones.find(z => z.shape === 'foreach')
232 const jsLoop = js.zones.find(z => z.shape === 'foreach')
233 t.alike(gdLoop.carried, jsLoop.carried, 'same carried state')
234 t.is(gdLoop.item, jsLoop.item, 'same loop item')
235 t.is(gdLoop.condition.text, 'rows', 'the foreach is driven by the ITERABLE')
236 t.is(jsLoop.condition.text, 'rows', 'and JS agrees')
237})
test/fixtures/derive/inferred.gd +8 -0
……@@ -0,0 +1,8 @@
1# `:=` inferred typing. The vendored lezer-gdscript grammar predates it and
2# emits an error node inside TypeCast — the derivation must survive that and
3# still name the wire.
4var count := 0
5
6func bump():
7 count = count + 1
8 return count
test/fixtures/derive/sample.gd +26 -0
……@@ -0,0 +1,26 @@
1extends Node
2class_name Sample
3
4# The GDScript twin of sample.js. Same program, same control flow, so the two
5# models must agree on ZONES, SHAPES, EXITS and CARRIED STATE. Node counts
6# legitimately differ: JS has a template literal here, GDScript does not.
7
8var total = 0
9
10func tally(rows, limit):
11 if rows == null:
12 return null
13 for row in rows:
14 if row.skip:
15 continue
16 total = total + row.count
17 if total > limit:
18 break
19 if total > 0:
20 return total
21 else:
22 return 0
23
24func label(row):
25 var at = Time.get_ticks_msec()
26 return str(row.name, at)