Creating and Managing Embeddings

Creating and managing embeddings in Aimogen is how you give AI accurate, searchable, domain-specific memory without training models or bloating prompts. Embeddings are treated as a managed knowledge layer: you decide what goes in, when it updates, and how it is used.

This section explains the lifecycle of embeddings from creation to maintenance.

---

What You Are Actually Managing #

When you manage embeddings, you are managing three things:

• the source content
• the embedding index
• the update lifecycle

Embeddings are not “fire and forget”. They require intentional curation.

---

Where Embeddings Are Managed #

Embeddings are managed from the Aimogen admin interface.

Path:
Aimogen → Embeddings

From there you can:

• create new embeddings
• choose content sources
• regenerate embeddings
• delete outdated indexes
• inspect embedding status

Embeddings are global assets, not tied to a single chatbot or assistant.

---

Creating Embeddings: The Core Process #

Creating embeddings follows a deterministic flow.

1. select the content source
2. split content into chunks
3. generate embeddings using an embedding model
4. store vectors in an index

Once stored, embeddings are available immediately to features that use them.

---

Choosing Content to Embed #

You should embed only content that is factual, stable, and useful as reference.

Good candidates:

• documentation
• FAQs
• knowledge base articles
• product descriptions
• policies
• help guides
• internal manuals

Poor candidates:

• blog opinion pieces
• marketing copy
• transient announcements
• low-quality or duplicated text
• user-generated noise

Embeddings amplify content quality, good or bad.

---

Chunking Strategy (Critical) #

Before embedding, content is split into chunks.

Why this matters:

• smaller chunks improve retrieval precision
• large chunks dilute relevance
• overly small chunks lose context

Good chunking:

• keeps one idea per chunk
• avoids breaking sentences mid-thought
• balances context and precision

Chunking strategy matters more than model choice.

---

Selecting an Embedding Model #

Embedding models are chosen independently of text-generation models.

Consider:

• language support
• embedding dimensionality
• cost
• provider compatibility

Once embeddings are generated, changing the model requires regenerating the entire index.

---

Storing and Indexing #

Generated embeddings are stored in an index.

An index:

• holds vectors and metadata
• supports similarity search
• is queried at runtime by chatbots, assistants, or workflows

Indexes are isolated. One index does not affect another.

---

Using Embeddings After Creation #

Once created, embeddings can be used by:

• chatbots for factual answers
• AI Assistants for domain knowledge
• AI Forms for context-aware reasoning
• OmniBlocks for retrieval steps

Nothing automatically “uses” embeddings unless a feature is configured to do so.

---

Updating Embeddings When Content Changes #

Embeddings do not update automatically.

If content changes:

• existing embeddings become stale
• AI may retrieve outdated information

You must:

• regenerate embeddings for updated content
• remove obsolete indexes if needed

This is a deliberate maintenance step.

---

Regenerating vs Appending #

Two common update strategies:

Regenerate

• replaces all vectors
• ensures full consistency
• costs more
• safest option

Append

• adds new content only
• faster
• riskier if old content changed

For production knowledge bases, regeneration is usually preferred.

---

Deleting Embeddings Safely #

Deleting an embedding index:

• removes it immediately
• breaks any feature relying on it
• does not affect original content

Before deletion:

• confirm no chatbot or assistant depends on it
• update references if needed

There is no soft-delete fallback.

---

Embeddings and Assistants #

Embeddings are not automatically attached to assistants.

To use embeddings with an assistant:

• configure retrieval logic
• ensure the assistant is instructed to use retrieved context
• combine with strong system instructions

Embeddings provide data. Assistants decide how to use it.

---

Embeddings and Chatbots #

For chatbots:

• user query is embedded
• similar chunks are retrieved
• retrieved content is injected into AI context
• AI answers grounded in real data

If embeddings are poorly curated, chatbot quality suffers.

---

Performance and Cost Considerations #

Embedding creation:

• costs tokens
• is usually a one-time or occasional cost

Embedding querying:

• is fast
• is cheap
• scales well

Good embeddings reduce:

• prompt size
• hallucinations
• repeated AI calls

---

Logging and Monitoring #

You should monitor:

• which embeddings are queried
• retrieval quality
• missing or irrelevant matches
• outdated content usage

Poor retrieval usually means poor chunking or poor source selection.

---

What Managing Embeddings Does Not Do #

Managing embeddings does not:

• change model behavior
• enforce business rules
• validate correctness
• replace prompts or instructions
• auto-detect content changes
• guarantee perfect answers

Embeddings improve relevance, not judgment.

---

Common Mistakes #

• embedding too much content
• embedding low-quality text
• forgetting to regenerate after updates
• mixing unrelated topics in one index
• assuming embeddings equal memory
• skipping chunking strategy

Embeddings reward discipline.

---

Best Practices #

Embed only authoritative content, chunk carefully, regenerate when content changes, monitor retrieval quality, and combine embeddings with clear system instructions. Treat embeddings like a curated knowledge base, not a dumping ground.

---

Summary #

Creating and managing embeddings in Aimogen means building and maintaining a semantic knowledge layer that AI can query at runtime. By carefully selecting content, chunking it intelligently, generating embeddings, and keeping them up to date, you enable chatbots, assistants, and workflows to answer accurately and consistently without retraining models. Properly managed embeddings turn AI from a guesser into a grounded, reliable system.