For the last few weeks I worked on an automated scorer for my model, Transcendental Pluralism (TP), built in turns with Claude and GPT.
The program parses each sentence, cuts it into units, decides which are abstract and assigns a stage (1.4 to 3.4) and a quarter, the kind of link between the units. It is not a trained model: it works by almost 200 explicit rules (+1740 if/enif and 3640 extra conditions), each derived from me and my scoring decisions and each switchable, so every score can be traced to the rule behind it. A language model answers only one interpretive question, whether a word is meant abstractly within a specific context. Its answers are cached, so scores are reproducible.
On 28 texts (791 sentences) it matches my centre of gravity in all 28, my stage in 87% and my stage plus quarter in 78% of sentences. On three children's texts in the range of my 3.1 and 3.2 Stages which I scored blind first it reached 88% stage, 72% stage plus quarter, centre of gravity 3 out of 3.
Pure AI raters, given the manual and my examples, were as good on stage (about 90%) but weaker and unstable on the quarter: one rater agreed with itself on only 58% of quarters. A three-agent debate panel scored worse than a single rater.
My current takeaway: rules for structure, language models for interpretation might allow a stably scoring and human aligned automated scorer outside of the common logic to build hughe dictionaries or use a lot of training data.
Still this is oriented towards the earlier stages in the spectrum of my model and the question remains how well it will translate to increasingly more nuanced forms of syntax.