Extracting Renewal Options with spaCy and Regex for ASC 842 / IFRS 16 Lease Term

Problem Statement #Link to this section

A single missed renewal option is the difference between a correct lease term and a restatement. This page answers one precise engineering-plus-compliance question: how do you extract the duration and frequency of a renewal option from raw lease prose — using spaCy dependency parsing to locate the clause and a compiled regex to pull the exact number and unit — so that the extended period lands in the lease term boundary that drives the present-value calculation? Under ASC 842 and IFRS 16 the lease term is the non-cancellable period plus any renewal periods the lessee is reasonably certain to exercise, and that term sets the number of discounting periods, the straight-line expense denominator, and the amortization horizon of the right-of-use asset. Pure regex breaks on syntactic variation (tenant may elect to extend, lessee shall have the option to renew); pure spaCy is unreliable at exact numeric capture. The extraction step here is one stage inside the broader lease document extraction and clause parsing pipelines, and its output is a single decision: how many periods to add to the term.

Standard Anchor #Link to this section

Two provisions govern this narrow extraction directly, and both turn on the same phrase:

• ASC 842-10-30-1 — the lease term is the non-cancellable period together with periods covered by an option to extend if the lessee is reasonably certain to exercise that option (and periods covered by a termination option the lessee is reasonably certain not to exercise). ASC 842-10-55-26 lists the economic factors that inform "reasonably certain."
• IFRS 16.18–19 — an entity determines the lease term as the non-cancellable period plus periods covered by an extension option if the lessee is reasonably certain to exercise it, plus periods covered by a termination option if reasonably certain not to exercise it.

The extraction engine does not decide "reasonably certain" — that is a judgment recorded elsewhere. Its job is to reliably surface every extension option (duration, frequency, and any conditional trigger) so that the judgment is applied to a complete set of candidates rather than to whatever a human reviewer happened to notice.

Formula / Algorithm Specification #Link to this section

Let $T_{base}$ be the non-cancellable term in whole periods and let each captured, reasonably-certain renewal add $T_{ext}$ periods. The lease term used downstream is:

$$N=T_{base}+\sum _{k=1}^{m}1[{\text{certain}}_k]\cdot T_{ext,k}$$

where $m$ is the number of extracted extension options and $1[\cdot ]$ is the reasonably-certain indicator. Because contracts state renewals in mixed units, every captured span is normalized to years before it can enter $N$:

$$T_{ext}^{\text{years}}=\frac{q\cdot u}{12},u=\{\begin{array}{ll}12& \text{unit}=\text{year}\\ 1& \text{unit}=\text{month}\end{array}$$

with $q$ the captured integer quantity. That $N$ then feeds the present value calculation logic for the lease liability:

$$PV=\sum _{t=1}^N\frac{PM{T}_t}{(1+r{)}^t}$$

where $PM{T}_t$ is the period payment and $r$ is the periodic discount rate. Getting $N$ wrong by one renewal period shifts every discounted term, so the extraction algorithm is, in effect, a control over the summation bound.

The extraction itself is a two-stage pipeline:

1. Locate — run the text through a spaCy pipeline and use the dependency parse plus a Matcher to isolate sentences whose head verbs/objects include renewal lemmas (renew, extend, option, term, period).
2. Capture — run a compiled, anchored regex against each matched span to pull $q$ (quantity), the unit, and the frequency, then normalize to years.

Annotated Python Snippet #Link to this section

The example below is single-responsibility: it locates renewal sentences with spaCy, captures duration and unit with an anchored regex, normalizes to years with decimal.Decimal, and asserts the extracted term matches the expected value. It uses en_core_web_sm so it runs without the transformer download; swap in en_core_web_trf for production contextual accuracy.

import re
import spacy
from decimal import Decimal
from spacy.matcher import Matcher

nlp = spacy.load("en_core_web_sm")

# 1. LOCATE: flag sentences whose lemmas signal a renewal/extension option.
matcher = Matcher(nlp.vocab)
matcher.add("RENEWAL", [[{"LEMMA": {"IN": ["renew", "extend", "option"]}}]])

# 2. CAPTURE: anchored pattern pulls quantity + unit, ignoring filler words.
CLAUSE_RE = re.compile(
    r"(?i)(?:renew|extend|option|term)\s+(?:for|of|by)?\s*(?:an?\s+)?"
    r"(\d{1,3})\s*(?:additional|further|extra)?\s*(year|month|period)s?"
)
UNIT_MONTHS = {"year": Decimal(12), "month": Decimal(1), "period": Decimal(12)}

def extract_renewal_years(text: str) -> list[Decimal]:
    """Return each renewal option's duration in years, normalized from the span."""
    doc = nlp(text)
    match_sents = {doc[s:e].sent for _, s, e in matcher(doc)}  # dedupe by sentence
    results: list[Decimal] = []
    for sent in match_sents:
        m = CLAUSE_RE.search(sent.text)      # anchor regex to the spaCy span
        if not m:
            continue                          # UnitMissingError path in production
        qty, unit = Decimal(m.group(1)), m.group(2).lower()
        results.append((qty * UNIT_MONTHS[unit]) / Decimal(12))
    return sorted(results)

clause = ("The Tenant shall have the option to renew for one additional "
          "term of 60 months, and may further extend for 3 years thereafter.")
years = extract_renewal_years(clause)

assert years == [Decimal("3"), Decimal("5")], years   # 60 months -> 5.0 yrs
print(years)  # [Decimal('3'), Decimal('5')]

The assertion is the guardrail: 60 months must normalize to 5.0 years, and 3 years must stay 3.0, so the two options add 8 reasonably-certain candidate years to $T_{base}$ — not 63, which is the classic unit-confusion bug the normalization step exists to prevent.

spaCy-Only vs Regex-Only vs Hybrid #Link to this section

The hybrid column is the only one that satisfies both audiences at once: engineers get deterministic numeric capture, and accountants get a defensible dependency path and source span for every extracted option.

Gotcha: Unit Confusion and Unanchored Regex #Link to this section

The two failure modes that silently corrupt the term are a missing temporal unit and an unanchored regex that reads a number from an adjacent clause. Walk this checklist before trusting an extracted $T_{ext}$:

1. Assert a unit was captured. If the regex matches a bare integer with no year/month/period inside a ~50-character window, raise UnitMissingError and fall back to the base term until a human confirms — never assume years.
2. Anchor the regex to the spaCy span, not the whole document. Searching the full text lets renew ... 30 in one sentence pair with days from the next; scope every CLAUSE_RE.search to sent.text.
3. Normalize before summing. Convert every capture to years with the shared unit map; do not add a months value to a years value.
4. Route conditionals to review. Use spaCy's neg, mark, and advmod dependencies to detect options contingent on landlord approval or market-rate resets, and hold them out of the reasonably-certain set.

Before (unanchored, no unit guard — reads the wrong number and assumes years):

m = re.search(r"(\d{1,3})", full_document_text)   # WRONG: no unit, no span anchor
t_ext_years = int(m.group(1))                      # 60 (months!) treated as 60 years

After (span-anchored, unit-validated, normalized):

m = CLAUSE_RE.search(sent.text)                    # anchored to the spaCy sentence
if not m:
    raise UnitMissingError(sent.text)              # conservative fallback path
t_ext_years = (Decimal(m.group(1)) * UNIT_MONTHS[m.group(2).lower()]) / Decimal(12)

The extracted, normalized options then serialize into the standard clause object (clause_type, duration_years, confidence_score, source_span, dependency_path) that flows into payment schedule data normalization and, at portfolio scale, through async batch processing for lease portfolios.

spaCy locates the renewal sentence and a span-anchored regex captures the number and unit; the unit gate is the control point — a clean capture normalizes months to years and extends the term N that bounds the PV summation, while a missing unit raises UnitMissingError and routes the span to review rather than assuming years.

Frequently Asked Questions #Link to this section

Why combine spaCy and regex instead of using one of them alone?

spaCy's dependency parse reliably locates renewal language even under syntactic variation like "tenant may elect to extend," but it is unreliable at pulling exact integers and units. A compiled regex is precise at numeric capture but greedily crosses clause boundaries and misses non-standard phrasing. The hybrid pipeline lets spaCy isolate the sentence span and then anchors the regex to that span, giving deterministic capture without cross-clause contamination.

Does the extractor decide whether a renewal is "reasonably certain"?

No. ASC 842-10-30-1 and IFRS 16.18–19 require a "reasonably certain" judgment based on economic incentives, and that judgment is recorded separately. The extractor's job is to surface every extension option — its duration, frequency, and any conditional trigger — so the reasonably-certain test is applied to a complete candidate set rather than to whichever options a reviewer happened to spot.

How are renewal durations stated in months normalized for the lease term?

Every captured quantity is converted to years before it enters the term: a 60-month renewal normalizes to 5.0 years using a shared unit map, and a "3 years" capture stays 3.0. Normalizing with decimal arithmetic prevents the unit-confusion bug where 60 months is treated as 60 years, which would inflate the number of discounting periods and misstate the present value.

What happens when the regex captures a number but no time unit?

The parser raises a UnitMissingError and falls back to a conservative base-term calculation until a human verifies the clause, rather than assuming years. Documents with conditional triggers (landlord approval, market-rate resets) or conflicting options are routed to a manual review queue with the highlighted span and dependency path, preserving audit defensibility.

Related #Link to this section

• Sibling: Normalizing irregular payment schedules into JSON — the next stage, where the extracted term aligns with period granularity.
• Parent: NLP Clause Extraction & Tagging — the full clause-classification layer this renewal extractor plugs into.
• Section: Lease Document Extraction & Clause Parsing Pipelines — ingestion through normalization and sync into the measurement chain.