Applied NLP Architecture

From free text to traceable value signals.

Teams describe value in hundreds of ways. This system turns those phrases into a consistent set of business-value tags — while keeping the evidence, the scores, and the decision path visible for review.

↓ See the system ↓ Skip to the decision logic

Ali Reza Rashidi6 min readUpdated 6 August 2026

Not magical precision — a repeatable, reviewable layer between unstructured work descriptions and portfolio-level analysis.

In this piece

The system — five stages, fully inspectable
The principles — design for review, not for trust
Sources — the methods behind the architecture

01—The system

Five stages, fully inspectable#

The mission, the pipeline, the ontology, the decision policy, and the output contract — each designed to be reviewed by a human, not trusted blindly.

Teams describe value in hundreds of different ways: “reduce latency,” “speed up delivery,” “strengthen controls.” The goal is not to make subjective language look magically precise — it is to create a repeatable, reviewable layer between unstructured work descriptions and portfolio-level analysis.

InNatural-language tasksShort descriptions of work, outcomes, improvements, or risks.

⚙Extraction & mappingIdentify the value phrase, score it, compare with ontology concepts.

✓Policy decisionAccept, reject, or route to human review using thresholds.

OutAuditable tagsThe tag, evidence, scores, and review status — as JSON.

Value extraction.

Find the phrase that describes the expected benefit or avoided harm.

Evidence scoring.

Estimate how clearly the phrase is supported by the original text.

Semantic mapping.

Compare the phrase with the definitions and examples in the ontology.[1]

Policy decision.

Accept, reject, or route the candidate to human review using thresholds.

Structured output.

Store the tag, evidence, similarity, confidence, and decision status.

The ontology holds ten recurring value concepts; every accepted result cites one of them, with evidence.[3] Four of the ten:

EfficiencyCost, effort, or resource reduction.

AgilitySpeed, adaptability, and time-to-market.

ReliabilityStability, availability, and defect reduction.

Security & complianceProtection, controls, and obligations.

Chart 01 · the ontology in use

Efficiency · 148Agility · 122Reliability · 96Security & compliance · 74Cost reduction · 58Quality · 52Customer experience · 40Innovation · 30Risk avoidance · 24Growth · 16

How to read this: frequency, not importance — all ten concepts remain valid even when one appears less often in a particular dataset.

Chart 02 · decision policy

How to read this: auto-accept only when both thresholds are met. High similarity with weak evidence — or strong evidence with an ambiguous tag — goes to review.

Thresholds should be selected on labelled validation data — not chosen because they “look right.”[2]Review rate, false acceptance, false rejection, and calibration all need monitoring in production.

Chart 03 · aggregated output

How to read this: in this sample, Efficiency and Agility dominate — a portfolio tilting toward optimization and delivery speed over innovation-oriented work.

The output contract

Field	Meaning
value_phrase	The exact benefit or risk phrase extracted from the source
tag	The selected ontology concept
similarity	Semantic proximity to the tag definition and examples
confidence	Support score for the extraction or classification
decision	Accepted, rejected, or needs review

record.json

{
  "source_text": "Reduced deployment time by automating checks",
  "value_phrase": "reduced deployment time",
  "tag": "Agility",
  "similarity": 0.88,
  "confidence": 0.91,
  "decision": "accepted",
  "evidence": "Reduced deployment time"
}

02—The principles

Design for review, not for trust#

The system’s value is not that it is always right — it is that it is never opaque. Every acceptance carries its evidence; every uncertainty carries a route to a human.

Extract before you classify.

Language understanding and business taxonomy are separate problems — keep them separate.

Threshold on evidence AND similarity.

One strong signal is never enough for an autonomous decision.

Calibrate on labelled data.

A confidence score is a system score until you have proven otherwise.

Keep the receipt.

Tag, phrase, scores, decision — stored together, or it didn’t happen.

The portfolio effect: once every task speaks the same value language, leadership can finally ask — and answer — “what is the organization actually working on?”

03—Grounding

Sources#

The numbers in the charts are illustrative samples. The architecture is not invented — these are the methods it stands on.

Semantic similarity between short texts. Reimers & Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks” (EMNLP-IJCNLP 2019) — the standard recipe for comparing phrases by cosine similarity over sentence embeddings. arxiv.org/abs/1908.10084
Why a confidence score is not a probability. Guo, Pleiss, Sun & Weinberger, “On Calibration of Modern Neural Networks” (ICML 2017) — modern models are systematically over-confident until calibrated on held-out data, which is why thresholds must be selected on labelled validation sets. arxiv.org/abs/1706.04599
What an ontology is. Gruber, “A Translation Approach to Portable Ontology Specifications,” Knowledge Acquisition 5(2):199–220 (1993) — the classic definition: an explicit specification of a shared conceptualization. doi.org/10.1006/knac.1993.1008

Intelligent Value Extraction

LLM Evaluation

LangGraph vs LangChain and more

From free text to traceable value signals.

Five stages, fully inspectable#

Design for review, not for trust#

Sources#

Ali Reza Rashidi

Leave a Reply Cancel reply

Intelligent Value Extraction

LLM Evaluation

LangGraph vs LangChain and more

From free text to traceable value signals.

Five stages, fully inspectable#

Design for review, not for trust#

Sources#

Ali Reza Rashidi

Related posts

LangGraph vs LangChain and more

LLM Evaluation

LLM thinking

Leave a Reply Cancel reply