Article X-Ray

spaCy · en_core_web_sm · a real run

Who, where, when, how much — in ….

A paragraph of business news goes in. Out come the people, the companies, the countries, the money, the percentages, the dates and the statute — each span found, labelled and located, by a 12 MB statistical model running on one CPU core. Nothing on this page was written by a language model. Everything on it was measured.

…

01 — the centrepiece

The annotated article

This is the model's actual output, rebuilt from the ordered segment list in results.json — the site never re-parses anything. Press the toggle to strip the highlighting: the difference between those two states is the entire product.

    02 — grouped and counted

    Entity summary

    Every span, grouped under a human-readable category and counted. ×N is the number of separate mentions — repetition is how you tell the article's protagonists from its scenery.

    03 — noun chunks

    Key phrases

    Noun chunks taken straight off the dependency parse, lowercased, then filtered to drop pronoun and stopword heads. Sized by mention count. This is what a search index would key on.

    04 — dependency parse

    Sentence skeletons

    Each sentence stripped to subject, root verb lemma and object. It looks trivial written down, and it is exactly the shape a knowledge graph stores. A dash means the parse found no such argument in that sentence.

    SubjectVerb (lemma)Object

    05 — why this matters

    Extraction as infrastructure

    Extraction is the layer nobody demos, and the layer everything sits on. Before a search engine can rank a document it has to know what the document is about; before a compliance pipeline can flag a filing it has to know which parties, amounts and dates the filing names — and that reduction from prose to structure is a job, not a side effect. A generative model can do it, at roughly a thousand times the latency and a per-token bill, with an output you then have to validate; this pipeline does it in milliseconds on a CPU, deterministically, for the cost of the electricity. The trade is honest, and it goes both ways: en_core_web_sm is a small statistical model, it will mislabel an unfamiliar company as a product and a city as a person, and every serious deployment pairs it with rules, gazetteers and a human review queue for the cases that matter. But at a hundred million documents the question stops being which model is cleverest and becomes which model you can afford to run on all of them — and that is why the un-flashy extraction layer is still, quietly, underneath the interesting things.

    06 — the source

    The extraction core

    Verbatim from article_xray.py — the whole pipeline call, the entity grouping, the noun-chunk filter, the skeletons, and the reverse-offset annotation trick. Stdlib plus spaCy, nothing else.

    article_xray.py — excerpt 
    # This single line runs the whole pipeline: tokenizer -> tagger -> parser -> NER.
start = perf_counter()
doc = nlp(text)
pipeline_ms = (perf_counter() - start) * 1000

# --- ENTITIES ---------------------------------------------------------
# Group every entity under its human label, counting repeats so that the
# article's actual protagonists float to the top.
by_label = {}
for ent in doc.ents:
    human = LABEL_HUMAN.get(ent.label_, ent.label_.title())
    by_label.setdefault(human, Counter())[ent.text.strip()] += 1

# --- KEY NOUN PHRASES -------------------------------------------------
# noun_chunks come straight off the dependency parse; filtering out pronoun
# and stopword heads leaves the phrases a reader would actually index on.
phrase_counts = Counter()
for chunk in doc.noun_chunks:
    phrase = chunk.text.lower().strip()
    root = chunk.root
    if len(phrase) <= 3 or root.is_stop or root.pos_ == "PRON":
        continue
    phrase_counts[phrase] += 1

# --- SENTENCE SKELETONS -----------------------------------------------
# Dependency parsing doing information extraction: strip each sentence down
# to who did what to whom, which is the shape a knowledge graph stores.
skeletons = []
for sent in sents[:MAX_SKELETONS]:
    root = sent.root
    subject = next((t.text for t in root.lefts if "subj" in t.dep_), "-")
    obj = next((t.text for t in root.rights
                if t.dep_ in ("dobj", "pobj", "attr")), "-")
    skeletons.append({"subject": subject, "verb": root.lemma_, "object": obj})

# --- ANNOTATED TEXT ---------------------------------------------------
# Inserting markers shifts every character after the insertion point, so we
# walk each sentence's entities in REVERSE: the classic trick that keeps the
# earlier offsets valid while we rewrite from the end backwards.
for sent in sents[:MAX_ANNOTATED_SENTS]:
    marked = sent.text
    for ent in reversed(sent.ents):
        s = ent.start_char - sent.start_char
        e = ent.end_char - sent.start_char
        marked = f"{marked[:s]}[{ent.text}|{ent.label_}]{marked[e:]}"
    print(f"  {' '.join(marked.split())}")

    07 — reproduce it

    Run it yourself

    Four commands. The numbers you get will differ from the ones on this page only in the millisecond count, which depends on your CPU.

    1. Create an environment

      Python 3.9 or newer. A virtualenv keeps the model out of your system site-packages.

      python3 -m venv .venv && source .venv/bin/activate

    2. Install spaCy

      The library only. It ships no models — that is the next step, and the one people forget.

      pip install spacy

    3. Download the model

      en_core_web_sm is about 12 MB. Without it, spacy.load() raises OSError and the script prints this exact command back at you.

      python -m spacy download en_core_web_sm

    4. X-ray something

      No argument runs the built-in sample. Pass a path to read a file, and add --json to write the results.json this page is built from.

      python article_xray.py python article_xray.py my_article.txt --json