refactor(ontology): SchemaOrgCatalog live-fetches from schema.org (drops bundled snapshot)
Replaces the bundled ``schema_org.json`` snapshot + offline generator
script with a live-fetch SchemaOrgCatalog that pulls Schema.org's
published JSON-LD vocabulary on first use and caches the *processed*
result under ``$XDG_CACHE_HOME/graphrag-sdk/schema_org.json``. The
source of truth is now Schema.org itself — there is no in-package
data file to drift from upstream.
What's gone:
- ``graphrag_sdk/src/graphrag_sdk/data/schema_org.json`` — bundled
92 KB snapshot. Deleted entirely. No more data file shipping
inside the package.
- ``scripts/generate_schema_org_data.py`` — the offline generator
whose only purpose was to keep that bundled snapshot in sync.
Deleted; its parsing logic now lives directly in the catalog.
What's new in SchemaOrgCatalog:
- ``url=`` parameter (default
``https://schema.org/version/latest/schemaorg-current-https.jsonld``)
- ``cache_path=`` parameter (default
``$XDG_CACHE_HOME/graphrag-sdk/schema_org.json``)
- ``cache_ttl_days=`` parameter (default 30; ``None`` = never expire;
``0`` = force re-fetch on every constructor)
- ``curated_types=`` parameter to widen/narrow the in-scope subset
(default = the same 10 common types we shipped before)
- ``SchemaOrgFetchError`` raised when the network fails AND the cache
is missing/stale. No offline fallback by design.
JSON-LD processing (walk ``@graph``, build ``parents_by_class`` for
``rdfs:subClassOf``, propagate inherited members into curated
subclasses, classify each property's ranges into primitive→attribute
or curated-entity→relation) all moves into the catalog. The on-disk
cache stores the *processed* output, so subsequent loads skip both the
network round-trip and the parsing pass.
Concurrency: a ``threading.Lock`` guards the first-call fetch so two
concurrent callers don't both download.
Test rewrite: previously the catalog tests called
``SchemaOrgCatalog()`` and read the bundled JSON. With no bundled
file, they now mock ``urllib.request.urlopen`` with a tiny Schema.org
JSON-LD fixture (5 types, 4 properties — enough to exercise every
parsing branch including ``subClassOf`` inheritance and the
attribute/relation split) and redirect ``cache_path`` to ``tmp_path``
for hermetic, network-free runs. New coverage:
- cache hit on second construction (mocked urlopen sentinels would
raise if called again — proves the cache short-circuits)
- network failure + missing cache → ``SchemaOrgFetchError`` with the
expected URL and "no offline fallback" wording
- ``cache_ttl_days=0`` forces re-fetch every time
Also retains (from the prior round of review fixes that survive this
refactor):
- Bridge-relation fix in ``discover_grounded``: relation lookup now
unions detected types with ``existing.entities`` labels so
``Person→Organization`` is surfaced when ``Person`` is in
``existing`` and only ``Organization`` is in the corpus. Test:
``test_bridge_relations_to_existing_labels_are_surfaced``.
- Subclass-inheritance regression test, retargeted at the in-catalog
parser:
``test_subclass_inherits_base_class_attributes``.
- Docs cleanup: the "Catalogs" section now describes the live-fetch
behaviour (URL, cache location, TTL, no-offline-fallback) instead
of the bundled-snapshot story. The misleading
``catalog.as_ontology()`` reference is gone.
Local full suite: 1013 passed (was 1011), 29 skipped, no regressions.
Lint + format both clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> G
Gal Shubeli committed
5f7afec8990730fbaad01b8f91be39d9ecb4a75c
Parent: 4a7aa62