Skip to main content
Version: August 2026 - Dify 1.16.1

📄 2: Build Your Knowledge Base

Welcome to the second step of your Testus Patronus journey! In this exercise, you'll learn how to build a Knowledge Base that your AI assistant can search through using retrieval-augmented generation (RAG).

This is where your assistant starts to understand your testing documentation, requirements, and historical issues.

Retrieval

Release baseline: This exercise targets Dify 1.16.1. The Knowledge labels, settings, indexed documents, and retrieval evidence below were verified on the authorized disposable instance.

📌 What You'll Learn​

🧠 Embeddings​

When a document is ingested, it's transformed into a vector representation using an embedding model. These vectors help the AI understand meaning and similarity between pieces of text.

Embeddings let the model know that "bug report" and "defect ticket" might refer to the same concept.

In this tutorial, we use the model text-embedding-3-large from Azure OpenAI.

🍰 Chunking: Breaking Text into Pieces​

📏 Chunk Size​

Your documents are too long to fit directly into the model's input window, so they are divided into chunks.

Chunk SizeProsCons
Small Chunks (200–300 tokens)🎯 High precision
🔍 Matches narrow queries well		🧩 May lose broader context
Large Chunks (800–1000 tokens)📚 Preserve more context
🔁 Fewer retrievals needed		🧊 More noise
🎯 Lower match precision
Chunk Size Tradeoff

Think of it like reading pages of a book — too many pages at once and the meaning blurs. Too few and you lose the plot.

🔄 Chunk Overlap​

Overlap ensures that context near the boundaries of chunks isn't lost.

Chunk Overlap Visual

For example, if two chunks overlap by 100 tokens, the second chunk repeats the last 100 tokens of the first. This helps the model "remember" what came before — improving answers that need continuity.

🎯 Ranking and top-k Retrieval changed​

Once you have all your documents chunked and embedded, you'll want to find the most relevant ones to a query.

This is where ranking comes in:

  • The retriever uses cosine similarity (or a similar metric) to score each chunk.
  • Then it picks the top K chunks (usually 3–5) with the highest scores.
  • These are sent to the LLM for final answer generation.
Getting the top K right can dramatically improve performance!

Top K alone is not a quality target. In Dify 1.16.1, also test the retrieval mode, score threshold, returned source, and behavior for an unrelated query. Exercise 2.1 uses the same Jira fixture to turn these controls into a repeatable benchmark.

🛠️ Step-by-Step: Ingesting Documents​

📝 Manual Upload via Dify UI​

  1. Go to the Knowledge tab in your Dify instance.
    Dify 1.16.1 Knowledge landing page
  2. Click Create Knowledge.
Get your Jira Dataset sample in .txt format:

⬇️ Download Jira Dataset Files

  1. Unzip the dataset and upload the files to the Knowledge Base.
  2. Upload REST_JiraEcosystem_issues.txt and REST_JiraEcosystem_SUMMARY.txt.
  3. Click on Next.

  4. Configure:

    • Chunk size: Start with 2000 characters.
    • Overlap: Try 200 characters (10% of chunk size).
    • Delimiter: Use a blank line (\n\n). The default single newline fragments pretty-printed JSON into tiny field-level chunks and prevents a Jira key, summary, and description from being retrieved together.
    Processed document showing 2,000-character chunks and paragraph statistics

  5. Select High Quality index method and select the text-embedding-3-large embeddings model.

    High Quality vector index using text-embedding-3-large

  6. Click on Save & Process. You can now investigate your document chunks, sizes, content, etc. What do you think?

    Both Jira documents indexed and available

Verify retrieval, not just indexing: open Retrieval Testing and query REST-266 Default Jackson configuration unknown properties. A useful result contains the issue key, summary, and description in the same retrieved context. This is a smoke check; the measured positive and no-match acceptance suite follows in Exercise 2.1.

REST-266 retrieval with key, summary, description, and score

📦 Example Issue (JSON)​

{
  "summary": "Support 'expand' param on GET /rest/api/2/project",
  "description": "The GET /rest/api/2/project endpoint should support 'expand' to allow additional project details.",
  "issueType": "New Feature",
  "priority": "Medium"
}

🧑‍💻 Advanced: Ingesting via API​

Need a working config?

Why is API ingestion better?
  • ✅ Supports more complex documents and formats.
  • 🔁 Enables automation and batch processing.
  • 📦 You can attach metadata (e.g. issue_key) for filtering or analytics.