# Method
How a body of writing becomes fifteen numbers, how two texts are compared, and
why the comparison deserves belief. The terms are defined on the
[Concepts](/concepts) page and the commands are in [Developers](/developers).
## What a parser can measure
Ask a model to write like you and it copies your topics and your favourite
words. Those copy easily, and a reader who knows your writing recognises
something else: how long a sentence runs before its main verb arrives, how deep
the clauses embed, whether ideas get joined or subordinated, how often a claim
becomes a noun. A dependency parser produces each of those for any passage, so
a style can be written down as a specification and the result checked against
it afterwards.
## The fifteen features
Six are measured once per sentence and kept as distributions: sentence length
in tokens, embedding depth, finite clauses, subordinate clauses, coordinated
items, and tokens before the main verb. Each is recorded as a mean, a standard
deviation and the tenth, fiftieth and ninetieth percentiles.
Seven are rates over the whole corpus: fragments per sentence, passives and
agentless passives per finite verb, colon and dash predications per sentence,
pronouns per hundred tokens, and subordinate clauses and coordinated items per
sentence.
Two are categorical distributions: the share of tokens in each part of speech,
and the frequency of the twenty five most common function words.
One sentence's tree describes that sentence, the way one coin flip describes
that flip. A style is a distribution over many sentences, so a profile needs a
corpus. Below roughly a dozen sentences the spread is noise, and the profile
carries a warning that says so.
Every target is a range, because a writer who alternates a long subordinated
sentence with a short flat one has a rhythm, and a generator that hits the mean
every time flattens it. Uniformity is a claim about variance, so the spread is scored
as its own part of each feature, and sentence length variation below a quarter
of the mean is reported as a finding. Where a corpus is itself uniform, the
specification tells the generator to vary anyway.
## Voice match
Each of the six distributions contributes a distance on mean and spread
together, with the spread weighted so that a rewrite cannot score well by
landing on the mean every sentence:
Each rate contributes its absolute difference divided by the target rate, with
a floor of 0.5 on the divisor so that a rare feature cannot dominate. The two
categorical distributions contribute total variation distance. Divergence
between a target profile P and a candidate C is the mean of the fifteen, and
the reported score squashes it onto a readable scale:
D(P, C) = (1/15) · Σi=1..15 di(P, C)
similarity = 100 · e−1.4 D
Identical profiles score 100. The second step is strictly decreasing in D, so
the squashing changes how the number reads and never changes an ordering.
## Content kept
A rewrite may fix structure and may not invent content. The second number
checks the second half of that, because the promise was previously a line in a prompt plus the
model's own account of whether it had obeyed, and a model that fabricates is
not a reliable witness to its own fabrication.
Numbers, dates and named entities in the source are anchors, compared after
normalisation, so *two*, *2* and *$4.2M* against *4,200,000* all match. The
score is anchor recall, less fifteen points per introduced item to a cap of
forty five, less a penalty for novel content words above a ten percent free
allowance, less a penalty for source sentences the output does not cover, less
ten points per unexplained change in negation count.
The pass or fail verdict is stricter than the score and does not average. It
requires no introduced item, no unexplained dropped anchor, a novel word rate
at or below twenty five percent, sentence coverage at or above eighty percent,
and no change in negation count that a negative verb does not account for.
Where the analysis finds missing information, a superlative with no comparison
set, a passive with a suppressed agent, a pronoun with no recoverable referent,
that stays missing and is reported as a gap. Dropping can be legitimate: an
anchor named in a reported gap is excused. Introducing never is, and a self
reported inference does not overturn a measurement.
## Why structure carries a voice
Three small experiments on two public domain authors, Austen and Twain, run
while the measure was being built.
### Sentence length alone
| | n | mean | sd |
|---|---|---|---|
| Austen | 15 | 28.67 | 16.30 |
| Twain | 11 | 27.64 | 10.62 |
Welch's t = 0.19, Cohen's d = 0.07. By this feature the two authors are the
same.
### Same length, different parse
Two sentences carrying the same facts at almost the same length:
“The note went out unchanged, and the number reached the board, and nobody
wrote a correction.” against “The note, which nobody had corrected,
went out unchanged, so that the number reached the board before anyone
noticed.” Sixteen tokens against nineteen. By length the two are within a
fifth of each other; the parses share only the root.
### Pooling weak features
Take Austen's profile and score two held out passages against it, one of
Austen's and one of Twain's. For each feature, the distance to Austen's own
passage should be the smaller one.
| feature | own | other | gap |
|---|---|---|---|
| punctuation predication | 0.134 | 0.800 | +0.666 |
| subordination per sentence | 0.309 | 0.771 | +0.462 |
| depth | 0.052 | 0.447 | +0.395 |
| subordination spread | 0.356 | 0.682 | +0.326 |
| coordination per sentence | 0.420 | 0.712 | +0.292 |
| tokens before the verb | 0.447 | 0.738 | +0.291 |
| passive per finite verb | 0.056 | 0.342 | +0.286 |
| function words | 0.345 | 0.551 | +0.206 |
| coordination spread | 0.372 | 0.536 | +0.164 |
| pronouns per 100 tokens | 0.235 | 0.374 | +0.139 |
| sentence length | 0.176 | 0.306 | +0.130 |
| agentless passive | 0.196 | 0.292 | +0.096 |
| finite clauses | 0.110 | 0.203 | +0.093 |
| part of speech mix | 0.146 | 0.188 | +0.042 |
| fragment rate | 0.000 | 0.000 | 0.000 |
Fourteen of fifteen point the right way and one ties. If structure carried
nothing about authorship, each feature would be a coin flip:
A sign test assumes nothing about how the distances are distributed, and these
are not normally distributed. Sentence length, which separated nothing as a
mean, contributes +0.130 once it is measured as a distribution.
Averaging k roughly independent features keeps the signal and shrinks the noise
by the square root of k:
signal to noise(k) = √k · signal to noise(1)
So the metric averages all fifteen at equal weight, and a text that happens to
be flat on two or three of them still scores on the rest.
### Held out passages against each profile
| held out | vs Austen | vs Twain |
|---|---|---|
| **Austen** | **73.1** | 53.4 |
| **Twain** | 52.3 | **65.7** |
The diagonal wins in both directions. If a writer's unseen prose sat closer to
a stranger's profile than to their own, nothing built on the measure would hold,
so this matrix is computed before anything else.
The diagonal is 73.1 and 65.7. A profile estimated from a few dozen sentences
carries sampling error, and writers vary between passages, so an author's own
unseen prose scores below 100 against their own profile. That figure is the
ceiling a restyle can reach, and the [benchmark](/benchmark) reports its
results against a ceiling computed the same way. The [Concepts](/concepts)
page repeats the matrix for three Howard Marks memos.
## The parser learned on newswire
`en_core_web_sm` was trained on published news, and it misreads note register
English in repeatable ways. Five repairs run before any diagnostic, and each one
that fires appears in the report: ISO dates and hyphenated compounds kept as
single tokens, a head noun misread as a verb, a subject arc landing on a nominal
head, `one` as a pronoun against `one` as a numeral, an accusative pronoun parsed
as a subject.
Each sentence also carries a list of structural reasons to distrust its parse.
The rewrite may dispute a finding on the grounds that the parser, and not the
prose, went wrong. One such dispute turned out to be right and is now a
regression test.
## One measurement, two uses
The machinery that finds defects also describes styles. Five nominalisations is
a bug report when auditing a draft and a fingerprint when imitating an author.
The profile records what is there.
Where a measured feature collides with a rule, the rule wins and the report
prints the override. The profile measures dash predication, because it is a
complete description, and marks it as not a target, because the rules ban it.
## Limits
The features are not independent, so the square root of k is an upper bound on
the gain. Fifteen features and a handful of passages is a small experiment. The
measure says nothing about lexical choice, humour, argument, or the decision
about what to leave out, and a passage can match every feature and read like
nobody, so the interface shows the prose above the score.