SENTINEL is a local-first LLMOps platform that continuously evaluates Large Language Models, diagnoses the root causes of failures, and automatically attempts to self-heal them.
The formal academic paper detailing SENTINEL's underlying mathematical models, hybrid dense-sparse vector embedding algorithms, and autonomous prompt self-healing mutation engine is published and available for public review:
- 🌐 ResearchGate Publication: SENTINEL: Autonomous LLMOps with Hybrid Subword-Dense Embedding Cosine Similarity, Real-Time Faithfulness Verification, and Closed-Loop Prompt Self-Healing
- 📖 Full Research Paper (Markdown): RESEARCH_PAPER.md
- 📄 PDF Preprint Manuscript: SENTINEL_Research_Paper.pdf
- 💡 Unique Features & Novel Contributions: Key Differentiators vs Ragas, TruLens, DeepEval & LangSmith
- 📦 PyPI Package:
sentinel-eval-sdkv3.0.0 on PyPI - 🐍 Python SDK Repository:
sentinel-eval-sdkPython Client Directory
Large Language Models can produce incorrect, inconsistent, hallucinated, or poorly grounded responses even when the underlying application is functioning correctly.
Traditional CI/CD pipelines primarily test whether the software itself is working:
Code
↓
Unit Tests
↓
Integration Tests
↓
Build
↓
Deploy
However, an LLM application can pass all traditional software tests while its actual AI quality degrades.
SENTINEL introduces an AI quality layer into the software lifecycle:
Code / Prompt / Model Change
↓
LLM Evaluation
↓
Failure Detection
↓
Diagnosis
↓
Self-Healing Attempt
↓
Verification
↓
Quality Gate
↓
Deploy / Block
SENTINEL is designed as a local-first platform, using Ollama to run local LLMs without requiring paid inference APIs or API keys.
SENTINEL aims to provide four major capabilities:
Continuously evaluate LLM responses using multiple quality metrics.
Determine the likely root cause when an LLM response fails.
Automatically apply an appropriate recovery strategy.
Integrate AI evaluation into CI/CD so degraded prompts or models can be blocked before deployment.
SENTINEL evaluates LLM responses across multiple dimensions:
- Semantic correctness
- Hallucination
- Faithfulness
- Relevance
- Consistency
- Toxicity
- Latency
The evaluation engine combines:
- Golden test suites
- Embedding-based semantic similarity
- Document grounding
- LLM-as-a-Judge
- Custom evaluation metrics
SENTINEL uses multiple layers instead of relying on a single evaluation technique.
Manually defined test cases contain:
Question
Expected Answer
Category
Metadata
The generated answer is compared against the expected answer using semantic similarity.
Example:
Expected:
Paris is the capital of France.
LLM:
The capital city of France is Paris.
Although the wording differs, the semantic meaning is almost identical.
SENTINEL therefore uses embeddings instead of exact string matching.
For applications using RAG, the answer is compared with retrieved document chunks.
Question
↓
Retriever
↓
Documents
↓
LLM
↓
Answer
↓
Compare Answer ↔ Retrieved Context
If the response is not sufficiently supported by the retrieved context, SENTINEL can flag a potential hallucination or grounding failure.
For difficult or ambiguous cases, a second LLM evaluates:
Question
+
Generated Answer
+
Reference / Context
and produces a structured judgment with a reason.
This provides an additional evaluation signal when simple semantic similarity is insufficient.
A major feature of SENTINEL is that it does not stop at:
Evaluation → FAIL
Instead, it attempts to determine:
Why did the model fail?
The initial diagnosis framework contains three major failure categories.
The same question is sent to multiple models:
Llama 3.1
Mistral
Phi-3
Their responses are converted into embeddings.
SENTINEL measures semantic agreement between the responses.
Conceptually:
Question
│
├──→ Llama
├──→ Mistral
└──→ Phi
│
▼
Embeddings
│
▼
Similarity Analysis
│
▼
Model Agreement
Low agreement can be evidence that the prompt is ambiguous or underspecified.
The system should combine this signal with other evidence rather than treating one threshold as absolute proof.
A model can fail because the information required to answer the question does not exist in the knowledge base.
SENTINEL analyzes:
Correct / expected information
↓
Embedding
↓
Compare against retrieved documents
↓
Knowledge coverage
If relevant information cannot be found in the available knowledge base, the failure may be classified as:
KNOWLEDGE_GAP
Some models may consistently perform poorly on particular topics.
For example:
Model: Llama 3.1
Topic: Financial Reasoning
Evaluations: 150
Failures: 118
Failure Rate: 78.6%
SENTINEL uses historical evaluation data to identify persistent model weaknesses.
If a model consistently fails on a topic, SENTINEL can create a model weakness flag and recommend considering another model.
Once a failure is diagnosed, SENTINEL attempts to select an appropriate recovery strategy.
Failure
↓
Diagnosis
│
├── Vague Prompt
│ ↓
│ Prompt Healing
│
├── Knowledge Gap / Retrieval Failure
│ ↓
│ RAG Healing
│
└── Model Weakness
↓
Model Recommendation
For vague or poorly structured prompts, SENTINEL uses a meta-LLM to generate an improved prompt.
Input:
Original Prompt
+
Incorrect Response
+
Expected Response
+
Failure Information
↓
Meta-LLM
↓
Improved Prompt
The new prompt is then evaluated against the original.
Original Prompt
│
├──────────────┐
│ │
▼ ▼
Evaluation Evaluation
│ │
▼ ▼
Original Score Healed Score
│ │
└──────┬───────┘
▼
Comparison
↓
Better Version
When the problem is related to missing or poorly retrieved information, SENTINEL attempts to improve retrieval.
Original Question
↓
Generate Alternative Queries
↓
Multiple Search Queries
↓
ChromaDB Retrieval
↓
Candidate Documents
↓
Embedding Similarity
↓
Re-ranking
↓
Top Relevant Documents
↓
LLM
↓
Re-evaluation
The goal is to improve the quality of context supplied to the LLM.
Model weakness cannot always be automatically fixed.
Instead, SENTINEL:
- Records the failure.
- Calculates historical failure rates.
- Creates a weakness flag.
- Generates an alert.
- Recommends an alternative model.
- Allows the result to be displayed in the dashboard.
SENTINEL evaluates original and healed versions against the same evaluation set.
Evaluation Set
│
┌────────┴────────┐
▼ ▼
Original Prompt Healed Prompt
│ │
▼ ▼
Evaluation Evaluation
│ │
└────────┬────────┘
▼
Comparison
↓
Winner Selected
Metrics can include:
- Correctness
- Hallucination
- Faithfulness
- Consistency
- Toxicity
- Latency
- Overall evaluation score
The goal is to ensure that a healing operation actually improves the system rather than simply changing its behavior.
SENTINEL is designed to run locally.
The basic installation can contain:
SENTINEL Desktop
│
├── React
├── Tauri
├── FastAPI
├── SQLite
├── ChromaDB
├── Sentence Transformer
└── Ollama
├── Llama 3.1
├── Mistral
└── Phi-3
This provides:
- Local inference
- No mandatory paid LLM APIs
- No mandatory API keys
- Privacy-preserving evaluation
- Offline-capable evaluation
- Local experimentation
Actual model performance will depend on the user's available CPU, GPU, RAM, and storage.
SENTINEL
│
┌─────────┴─────────┐
│ │
Desktop UI Backend
│ │
React + Tauri FastAPI
│
▼
Evaluation Service
│
▼
Evaluation Engine
│
┌───────────────────┼──────────────────┐
│ │ │
▼ ▼ ▼
LLM Embeddings RAG
│ │ │
▼ ▼ ▼
Ollama MiniLM-L6-v2 ChromaDB
│
┌─────────┼─────────┐
▼ ▼ ▼
Llama Mistral Phi-3
│
▼
Evaluation
│
▼
Diagnosis
│
┌────────┼────────┐
▼ ▼ ▼
Vague Knowledge Model
Prompt Gap Weakness
│ │ │
▼ ▼ ▼
Prompt RAG Alert /
Healing Healing Recommendation
│ │ │
└────────┼────────┘
▼
Verification
│
▼
A/B Test
│
▼
Persistence
│
▼
SQLite / PostgreSQL
│
▼
Dashboard
The core SENTINEL pipeline is:
INGEST
↓
EVALUATE
↓
CLASSIFY
↓
DIAGNOSE
↓
HEAL
↓
VERIFY
↓
STORE
↓
ALERT
Each stage has a specific responsibility.
- Python
- Ollama
- Llama 3.1
- Mistral
- Phi-3
- DeepEval
- Sentence Transformers
- all-MiniLM-L6-v2
- scikit-learn
- LlamaIndex
- ChromaDB
- Python
- FastAPI
- Pydantic
- SQLAlchemy
- SQLite
- PostgreSQL
- REST APIs
- Background workers
- React
- TypeScript
- Tailwind CSS
- D3.js / Recharts
- Tauri
- MLflow
- DVC
- Git
- GitHub Actions
- Docker
- Docker Compose
- Kubernetes
- Redis
- Prometheus
- Grafana
SENTINEL/
│
├── frontend/ → UI + Tauri
├── backend/ → FastAPI + database + APIs
├── ai/ → entire AI/ML system
├── data/ → datasets + benchmarks
├── prompts/ → prompt assets/versioning
├── research/ → experiments + paper
├── infrastructure/ → Docker + Kubernetes
├── mlops/ → MLflow + DVC + evaluation gates
├── tests/ → all testing
├── scripts/ → architecture + documentation
├── docs/ → setup/utilities
│
├── .github/
├── README.md
├── LICENSE
├── CONTRIBUTING.md
├── pyproject.toml
├── docker-compose.yml
├── dvc.yaml
├── params.yaml
└── .gitignore
| Directory | Responsibility |
|---|---|
frontend/ |
React + Tauri desktop interface |
backend/ |
FastAPI, APIs, database and application services |
ai/ |
LLM evaluation, diagnosis, healing and RAG |
data/ |
Golden datasets, benchmarks and experiment data |
prompts/ |
System, evaluation and healing prompts |
research/ |
Experiments, analysis, results and research paper |
infrastructure/ |
Docker, Kubernetes and monitoring |
mlops/ |
MLflow, DVC and evaluation gates |
tests/ |
Unit, integration and end-to-end tests |
scripts/ |
Setup, model and benchmark utilities |
docs/ |
Architecture and technical documentation |
.github/ |
CI/CD workflows |
SENTINEL is also designed as an experimental research project.
The primary research question is:
Can an autonomous system accurately diagnose the root causes of LLM failures and select an appropriate self-healing strategy without human intervention?
Potential research questions include:
- Can multi-model semantic disagreement identify vague prompts?
- Can retrieval analysis distinguish knowledge gaps from other failure types?
- Can historical failure patterns identify model-specific weaknesses?
- Can automatic prompt healing improve LLM evaluation scores?
- Can RAG healing improve grounding and faithfulness?
- Can the system automatically select an appropriate healing strategy?
- Does self-healing improve quality without unacceptable latency overhead?
SENTINEL will be evaluated against baseline systems.
Potential baselines include:
Baseline 1
LLM without self-healing
Baseline 2
LLM + standard RAG
Baseline 3
LLM + generic prompt rewriting
Baseline 4
LLM + fixed retrieval strategy
SENTINEL
Evaluation
→ Diagnosis
→ Targeted Healing
→ Verification
Evaluation metrics may include:
- Accuracy
- Semantic correctness
- Hallucination rate
- Faithfulness
- Consistency
- Toxicity
- Latency
- Diagnosis precision
- Diagnosis recall
- Diagnosis F1
- Healing success rate
- Regression rate
- Healing latency overhead
- Token overhead
To understand which components contribute to performance, SENTINEL can be evaluated with individual components removed.
Examples:
Full SENTINEL
SENTINEL without multi-model diagnosis
SENTINEL without RAG re-ranking
SENTINEL without historical model analysis
SENTINEL without prompt healing
SENTINEL without verification
The results can be compared to determine the contribution of each component.
Each experiment should record information such as:
Model
Model Version
Prompt Version
Dataset Version
Question
Category
Response
Latency
Evaluation Scores
Diagnosis
Diagnosis Confidence
Healing Strategy
Healed Response
Before Score
After Score
MLflow can be used for experiment tracking, while Git and DVC can be used for code, prompt, and dataset versioning.
The goal is reproducibility.
SENTINEL is designed around local inference.
When running in local mode:
Questions
Documents
Prompts
LLM Responses
Embeddings
Evaluation Results
can remain on the user's machine.
No external LLM API is required for the core local evaluation workflow.
Although SENTINEL is local-first, the architecture can later support centralized deployment.
Load Balancer
│
▼
FastAPI Instances
│
┌────────┴────────┐
▼ ▼
Redis PostgreSQL
│
▼
Evaluation Workers
│
▼
Ollama / vLLM Servers
│
▼
LLMs
This enables:
- Team-wide evaluation
- Centralized experiment tracking
- Multiple workers
- Horizontal scaling
- Shared dashboards
- CI/CD integration
SENTINEL extends traditional CI/CD by evaluating AI quality before deployment.
Developer Commit
↓
Unit Tests
↓
Integration Tests
↓
Build
↓
LLM Evaluation Suite
↓
Compare Against Baseline
↓
Quality Gate
│
┌───┴────┐
▼ ▼
PASS FAIL
│ │
▼ ▼
Deploy Block
Example quality policies:
Correctness below threshold
→ FAIL
Hallucination above threshold
→ FAIL
Faithfulness below threshold
→ FAIL
Significant latency regression
→ FAIL
This allows AI quality regression to become a deployment concern.
- Ollama setup
- Local model integration
- FastAPI
- SQLite
- Basic evaluation engine
- Semantic similarity
- Latency evaluation
- Initial hallucination/grounding evaluation
- Docker development environment
- DeepEval integration
- DVC
- MLflow
- Redis/background workers
- Expanded evaluation metrics
- Evaluation datasets
- Multi-model diagnosis
- Knowledge-gap detection
- Model weakness detection
- Prompt healer
- RAG healer
- Verification
- A/B testing
- GitHub Actions
- Automated evaluation
- Baseline comparison
- Quality gates
- Deployment blocking
- Regression testing
- Desktop application
- React dashboard
- Evaluation visualization
- Failure analysis
- Healing history
- Model comparison
- Monitoring
- Packaging
- Large-scale benchmark
- Model comparison
- Ablation studies
- Error analysis
- Statistical analysis
- Research report
- Paper preparation
- Demo
- Documentation
Responsible for:
- LLM integration
- Evaluation engine
- Embeddings
- Evaluation metrics
- Diagnosis
- RAG
- Prompt healing
- RAG healing
- Model weakness detection
- A/B testing
- AI experiments
- Research methodology
Responsible for:
- FastAPI
- API design
- Database
- SQLAlchemy
- Application services
- Evaluation job management
- Result persistence
- Backend testing
Responsible for:
- Docker
- Docker Compose
- GitHub Actions
- DVC infrastructure
- MLflow infrastructure
- Kubernetes
- CI/CD
- Evaluation gates
- Monitoring
- Deployment
SENTINEL uses a feature-branch workflow.
main
│
├── feature/ai-*
├── feature/backend-*
└── feature/devops-*
Development process:
Create Feature Branch
↓
Develop
↓
Run Tests
↓
Commit
↓
Push
↓
Pull Request
↓
Code Review
↓
CI Checks
↓
Merge into main
The main branch should remain stable.
SENTINEL uses multiple testing levels.
Test individual:
- Metrics
- Embedding functions
- Diagnosis logic
- Healing logic
- API services
Test:
API
↓
Evaluation Service
↓
AI Engine
↓
Database
Test the complete workflow:
Question
↓
LLM
↓
Evaluation
↓
Diagnosis
↓
Healing
↓
Verification
↓
Stored Result
This project is licensed under the MIT License.
See LICENSE for details.
SENTINEL is currently under active development.
The architecture and research methodology may evolve as experiments are conducted and results are analyzed.
The current development priority is:
Core Infrastructure
↓
LLM Evaluation
↓
Failure Diagnosis
↓
Self-Healing
↓
LLMOps / CI-CD
↓
Research Benchmark
SENTINEL aims to move LLM applications from:
"Does the application work?"
to:
"Is the AI system still performing correctly,
why did it fail, can it fix itself,
and should this version be deployed?"
The long-term goal is to provide a local-first, autonomous quality and reliability layer for LLM applications.
SENTINEL — Autonomous LLMOps and Self-Healing Evaluation Platform
Built as a collaborative AI/ML, backend, and DevOps engineering project.
Status: 🚧 Under Development