Admin Guide — Function & Plan
AI Agents and RAG Knowledge Management
How BSM gives an AI agent — starting with Katie — a real, searchable knowledge base instead of relying on general AI guesswork. Covers building the Python retrieval service on the existing server, loading and updating knowledge, connecting it to the live site, and where this is heading next.
v1.0 — Updated June 2026
Index
- 📖 Knowledge Management, the AI Pro Way
- 1 Phase One — Developer Build
- 1a The Engine
- 1b The Knowledge Store
- 1c Retrieval and Answering
- 1c+ Lessons from the real first run
- 1d Connecting to the Live Site
- 1e The Interface
- 1f Database — MariaDB 11.8
- 1g Credentials — checking and changing them
- 1h Editing Python Code Using FTP
- 2 Phase Two — Knowledge Build
- 2a Content Manager's Hints — writing knowledge Katie can actually use
- 3 Appendix — Build Tracker
- 3a Python Learning — line-by-line for PHP devs
⚡ CRITICAL ARCHITECTURE RULES — DO NOT BREAK THESE
1
No third-party hosting vendor. This runs entirely on the same server that already runs BodySleepMind and its database — one new folder, not a new account, not a new vendor, not a monthly bill from anyone else. The deliberate decision was to avoid relying on an external platform (e.g. Render) for this.
2
Knowledge and the retrieval tool are two separate things. The Python tool does the searching. The actual knowledge lives as plain text files in one folder. Updating knowledge never means touching the Python code — it means changing what's in that folder, then telling the tool to re-read it.
3
Per-user data (GOD JSON) is never part of this knowledge base. A user's own tier, seats, or wellness results are not documents to search — they are looked up directly by user ID at the moment Katie answers, the same way every other BSM file reads GOD JSON today. Mixing the two would be slower, staler, and pointless when an exact lookup already exists.
4
This is reached today only on the server itself (127.0.0.1), not the open internet. It must stay this way until Step 8's bridge is deliberately built and tested — opening it to the internet earlier would expose a tool with no login or permission check of any kind.
5
Re-reading the knowledge folder is not automatic. Editing a text file in
knowledge/ does nothing on its own. The tool must be explicitly told to rebuild its index, and then restarted, before it reflects the change. This is the single most common way this kind of tool appears “broken” when it is actually just running on stale knowledge.6
Vectors live in MariaDB 11.8, in a dedicated database, from day one. The knowledge base uses real vector search from day one (see the Glossary below). It lives in its own separate database on the same MariaDB server BSM already runs — not shared with WordPress's own database, and not a separate vendor either. A dedicated database keeps the permission boundary clean: only the Python service ever connects to it; WordPress never gets a login to it at all, and can't reach it even if something went wrong. MariaDB was chosen over a dedicated vector-database vendor for four concrete reasons: no new vendor relationship to manage, the existing full-server backup already covers it for free, BSM's wellness knowledge never has to leave BSM's own infrastructure, and it uses skills already in-house rather than a new product to learn. The database upgrade to 11.8 (confirmed: 11.8.8) was done deliberately early, while nothing depended on it yet — the two tracks (building Katie, upgrading the database) ran in parallel and never blocked each other.
📖 Knowledge Management, the AI Pro Way
What “training” Katie's knowledge actually means, what it can do today, and where it can go. Read this section first — the term-by-term definitions below build directly on it.
Right now, Katie answers wellness questions using whatever a general AI model already happens to know — broad, but not specific to BodySleepMind, and not guaranteed to match what BSM actually says or sells. “Training” Katie's knowledge, in the sense this guide means it, isn't retraining the AI itself — nothing about how Katie thinks gets touched. It's something simpler and more controllable: handing her BSM's own real documents to search before she answers, so her replies are genuinely grounded in what BodySleepMind actually says, not a general guess.
That alone is worth doing, for BSM's own wellness content today. But there's a bigger opportunity sitting one step further out, and it's worth being deliberate about from the start.
✨ Where this can go
Because the knowledge and the tool are kept as two separate things by design (Architecture Rule 2), this mechanism is never limited to BodySleepMind's own content. Any trusted wellness partner's material — a skin-health brand's product guidance, for example — could be added into Katie's knowledge the exact same way: written, submitted, and reviewed through the same Phase Two process described below. No new AI to build. No separate assistant for the partner to run themselves. Just more documents, inside the same system.
That is the actual pitch to a partner: a relatively small amount of document preparation turns into Katie being able to discuss their products credibly, inside a conversation a user already trusts — without the partner needing to build or host any AI of their own at all.
That is the actual pitch to a partner: a relatively small amount of document preparation turns into Katie being able to discuss their products credibly, inside a conversation a user already trusts — without the partner needing to build or host any AI of their own at all.
Worth flagging honestly, not glossed over: once more than one partner's content sits inside the same knowledge base, deciding which knowledge should apply to which conversation becomes its own small design question — not solved by today's build, but not blocked by it either. A good problem to have later, not a reason to wait.
🛠 Two phases, in brief
Phase One, below, is a one-time technical build — reached only through a server terminal. Phase Two, after it, is what that build makes possible: knowledge written, submitted, reviewed, and published entirely by people with subject expertise, with no terminal and no engineering involvement ever again. The full detail for both is laid out in order below.
None of that is magic — it's documents, turned into numbers, matched by meaning. The definitions below explain exactly how, term by term, starting from zero — including the one piece that has nothing to do with AI at all, but that everything else in Phase One depends on.
SSH Terminal
A direct, text-based command line into the server itself — the actual computer the website runs on, underneath WordPress. Not WordPress's admin area, and not FTP. It's how software gets installed directly onto that machine (Python, in this build), the same level of access used to manage the operating system, not just the website sitting on top of it. Likely unfamiliar territory for a developer used to working only inside WordPress or over FTP — worth knowing before Phase One starts, since every single command in it runs here.
Getting to it on this server: inside Plesk, look for SSH Terminal — usually a labelled tool or terminal icon in the left-hand menu or Tools & Settings. Clicking it opens a black command-line window directly in the browser, already connected to the server as root. No separate app or login needed beyond Plesk itself. Every grey code block in Phase One gets typed or pasted into that window, one at a time.
Getting to it on this server: inside Plesk, look for SSH Terminal — usually a labelled tool or terminal icon in the left-hand menu or Tools & Settings. Clicking it opens a black command-line window directly in the browser, already connected to the server as root. No separate app or login needed beyond Plesk itself. Every grey code block in Phase One gets typed or pasted into that window, one at a time.
The single biggest hint in this entire guide — how to actually paste code without it silently corrupting. Several real problems hit during this build (see “Lessons from the real first run,” below) traced back to exactly this. The reliable way:
- Press Ctrl+C in the terminal first, just to land on a clean, fresh prompt — harmless even if nothing was actually running
- Copy the code being given to you, and paste it into a plain text editor first (Notepad, or equivalent) — not straight into the terminal
- Select it again from there, and paste into the terminal using Ctrl+Shift+V — not the normal Ctrl+V
Ctrl+Shift+V is the dedicated “paste as plain text” shortcut in most terminal apps, where plain Ctrl+V is often reserved for something else entirely.
RAG (Retrieval-Augmented Generation)
The name for a simple idea: instead of an AI answering purely from what it learned during training (which can be wrong, outdated, or just not about your specific business), you let it look something up in your own real documents first, then answer using what it found. The AI is doing two jobs in sequence — finding the right paragraph, then writing a normal-sounding answer from it.
Analogy: the difference between an exam answered purely from memory, versus an open-book exam where you're allowed to find the right page first. RAG is the open-book version.
Grounded / Grounding
When an AI's answer is “grounded,” it means the facts in that answer were required to come from a specific, named source — not from whatever the AI separately picked up during its own training on the wider internet. In this guide, “grounded” always means grounded in BSM's own knowledge: the AI is still using its general language skill to write a fluent, natural-sounding sentence, but it has been instructed to pull its actual facts only from the real BSM documents retrieved by the vector search above — not from its own broader, internet-trained general knowledge. That instruction is what keeps Katie's answers consistent with what BSM actually says, instead of drifting into a plausible-sounding but wrong guess (a “hallucination”).
Chunking
The step that happens before anything gets turned into a vector at all: slicing a document into smaller pieces first. A vector search works best on short, focused pieces of text — turning an entire ten-page document into a single number string would only capture roughly “this document, as a whole,” which isn't precise enough to match a specific question. So before any embedding happens, LlamaIndex slices the document into chunks — typically a paragraph or two each, breaking on natural sentence boundaries rather than chopping mid-sentence. Each chunk then gets its own embedding (see below) and becomes its own separate row in the database (see the table further down this guide). One uploaded document very often becomes several rows, not one — it depends entirely on how long it is.
One common mix-up worth clearing up directly: cosine similarity (see below) does not get calculated or stored at this point. Nothing about “how good a match” gets baked into a chunk when it's created. Only its embedding is stored. Cosine similarity only ever gets calculated fresh, on the spot, the moment a real question is asked — comparing that question's own embedding against every stored chunk's embedding, to work out which ones are the closest match.
One common mix-up worth clearing up directly: cosine similarity (see below) does not get calculated or stored at this point. Nothing about “how good a match” gets baked into a chunk when it's created. Only its embedding is stored. Cosine similarity only ever gets calculated fresh, on the spot, the moment a real question is asked — comparing that question's own embedding against every stored chunk's embedding, to work out which ones are the closest match.
Analogy: imagine turning a ten-page wellness document into a stack of index cards instead, one self-contained paragraph or two per card. Each card later gets its own meaning-coordinate written on the back (its embedding) — that happens once, when the card is made. Comparing coordinates to find the best-matching card only happens much later, every time someone actually asks a question.
Embedding
A way of turning a piece of text into a number string that represents its meaning, not its exact words. Two sentences that mean similar things end up with similar-looking number strings, even if they don't share a single word in common.
How the numbers actually get generated: the text is fed into an already-trained AI model built specifically for this one job (called an “embedding model”). That model was trained beforehand by reading an enormous amount of real-world text and learning, statistically, which words and phrases tend to show up in similar situations — for example, learning that “cancel my plan” and “stop my subscription” tend to appear in similar conversations, even though they don't share a word. Once trained, it can take any new sentence and work out where it belongs on that learned map of meaning, and writes that location down as a number string. This is why two totally different phrases can land on similar numbers: the model isn't matching letters at all, it's matching learned context — the same family of AI technology (called a “transformer”) behind tools like ChatGPT and Claude itself.
How the numbers actually get generated: the text is fed into an already-trained AI model built specifically for this one job (called an “embedding model”). That model was trained beforehand by reading an enormous amount of real-world text and learning, statistically, which words and phrases tend to show up in similar situations — for example, learning that “cancel my plan” and “stop my subscription” tend to appear in similar conversations, even though they don't share a word. Once trained, it can take any new sentence and work out where it belongs on that learned map of meaning, and writes that location down as a number string. This is why two totally different phrases can land on similar numbers: the model isn't matching letters at all, it's matching learned context — the same family of AI technology (called a “transformer”) behind tools like ChatGPT and Claude itself.
Analogy: think of every sentence getting a GPS coordinate based on what it means. “How do I cancel my plan” and “I want to stop my subscription” end up as nearby coordinates on this meaning-map, even though the words are different.
Vector
The actual number string produced by an embedding — one single combined string made of hundreds of numbers used together as a single unit, like a multi-part coordinate (for example:
0.12, -0.87, 0.34, ... continuing on for hundreds of numbers). It is not a sequence of separate, unrelated numbers one after another (like 1234, then 1235, then 1236) — it's one single address, just written with hundreds of numbers instead of two or three. Number string = co-ordinates = vector — these three phrases all mean exactly the same thing in this guide. When people say “store the vectors” or “search the vectors,” they mean storing and comparing these meaning-coordinates, not anything more exotic.Vector search
When a user asks Katie a question, that question is also turned into one of these number strings (see Embedding, above, for how that actually happens), and the system finds which stored documents have the closest-matching number string — i.e. the closest meaning, not the closest spelling. That's the actual “search” part of RAG.
Cosine similarity
The actual mathematical method behind “closest-matching” in vector search, above. It works by checking the angle between two vectors, not the distance between them — two vectors pointing in the same direction score as highly similar, regardless of how long either one is. This suits text meaning well, because two pieces of text can say the same thing at very different lengths, and cosine similarity ignores length entirely, judging purely on direction. It's the standard choice for this exact kind of matching, and it's one of the two methods MariaDB's vector feature supports directly — the one this build uses.
Analogy: picture two arrows starting from the same point, like the hands of a clock. Cosine similarity only asks whether they're pointing the same way — not how long either hand is.
Indexing
The one-time (and then repeated) process of reading every document in the knowledge folder, turning each one into its vector, and storing the result somewhere it can be searched later. This is what Step 5 and Step 12 of this guide are actually doing under the hood every time they run.
Vector store / vector database
The actual place those number strings get kept so they can be searched. This can be something as simple as a file sitting on the server, or something as heavyweight as a dedicated database built specifically for this job. It is just storage — a filing system for vectors, nothing more mysterious than that.
Where the vectors live, in this build
Directly inside BodySleepMind's own database, from the very first version — not a separate file, and not a separate vendor. LlamaIndex (see below) connects straight to MariaDB and stores every vector there, using the database's own native vector ability (Architecture Rule 6).
What MariaDB 11.8 actually added
MariaDB (the database BSM already runs) only gained the ability to store and search vectors itself starting at version 11.8. So 11.8 doesn't introduce vectors as a concept — it means the database itself can be the filing system for them, with no separate vendor or separate system needed. The server has been upgraded to 11.8 (confirmed: 11.8.8) specifically so this could be the starting point, not a later addition.
Analogy: rather than keeping vectors in a shoebox and moving them into the filing cabinet later, the filing cabinet drawer was built first — so the papers go straight into it from day one.
Why start here instead of somewhere simpler
A simpler, file-based starting point was the original plan, precisely because it avoided depending on whether MariaDB's vector feature, and a working connection between it and LlamaIndex, would actually be ready and reliable. Both are now confirmed — the database is genuinely on 11.8.8, and the official connector exists and is maintained. With that uncertainty gone, starting directly on the real, scalable storage avoids a migration step later for no remaining benefit.
Other dedicated vector-storage products that exist (for context only — not chosen)
Beyond a database growing its own vector ability (like MariaDB 11.8) or a simple file, there is a whole category of products built purely to store and search vectors, and nothing else: Pinecone, Weaviate, Qdrant, Milvus / Zilliz Cloud, Chroma, Redis Vector Search (RediSearch), and pgvector (an add-on specifically for PostgreSQL). These names are mentioned here only so they make sense if you ever come across them elsewhere — none of them are being used in this build. MariaDB 11.8 was chosen instead, for four concrete reasons: no new vendor relationship to set up and manage, the existing full-server backup already covers it with no extra work, BSM's own wellness knowledge never has to leave BSM's own infrastructure, and it runs on a system the team already knows how to operate rather than a new product to learn.
Why the vectors get their own dedicated database, not WordPress's
A new database,
katie_rag, is created on the same MariaDB server specifically for this — not added as a table inside WordPress's existing database. Only the Python service ever connects to it, with its own narrowly-scoped login. WordPress never gets a password to it at all. Everything WordPress actually needs — the review queue, who submitted what, decline reasons — stays exactly where it already belongs: inside WordPress's own database, the same way every other custom table on this site already works. The two systems coordinate through a plain text file and one web address (see Build Plan Group 5), never through a shared database login.LlamaIndex
The specific software tool doing all of the above — reading the documents, turning them into vectors, storing them, and matching a question against them. Chosen over a similar tool called LangChain because it's built specifically for this exact job and needs less code to do it.
The “bridge” between LlamaIndex and MariaDB
For LlamaIndex to store its vectors inside MariaDB, a small piece of connecting code needs to exist that knows how to talk to both sides. Confirmed: this exists, officially — it's maintained by the LlamaIndex project itself, and MariaDB's own foundation lists it as one of their recognised integrations. It's used directly in
main.py (Build Plan, Group 3). Also confirmed, from its own source code: it builds the vector index with cosine distance automatically, every time — not MariaDB's own default. Nothing left to check on this point.FastAPI & Uvicorn
LlamaIndex (and the embedding technology behind it) is a Python-only tool — WordPress's own PHP code cannot call it directly, the way it calls Stripe's API. So this Python code needs to run as its own small, separate program on the server, reachable over a normal web address, the same way any website is reachable. FastAPI and Uvicorn are the two small tools that make that possible: FastAPI defines what happens when a question arrives at that address; Uvicorn is what actually keeps that little program running and listening in the background, the same way a normal website needs something running to answer visitors.
Virtual environment (“venv”)
A private, separate copy of Python's tools, kept in its own folder, just for this one project. Used so that installing things for Katie's knowledge tool can never accidentally interfere with Python tools the server itself (Plesk) relies on.
Operating system, and what “Ubuntu” means
The operating system is the base software a computer (or server) runs on top of — the foundation everything else, including Plesk and this RAG tool, sits on. Ubuntu is simply the name of the particular operating system this server happens to run (a well-known, free, Linux-based one), and 24.04 is just its version number, the same way a phone might be on “iOS 18.” It is mentioned in this guide only because that specific version (24.04) is the reason Step 1 needs a virtual environment — it doesn't change anything else about how Katie's tool works.
1 — The Engine
A private retrieval service, on BSM's own server
1
Project folder and environment
Done
In plain English: one new, self-contained folder on BSM's own server, with its own private copy of Python — isolated from everything else the server runs.
Runs on the same server that already runs BodySleepMind and its database — one new folder, no new vendor, no new account.
Prerequisite on a fresh server
Ubuntu 24.04 doesn't include Python's
venv module by default — install it first, or the next command fails immediately.
Command line
apt install -y python3.12-venv mkdir -p /var/www/vhosts/bodysleepmind.com/katie-knowledge cd /var/katie-rag python3 -m venv venv source venv/bin/activate
Confirms it worked
Prompt changes to start with
(venv). Re-run source venv/bin/activate at the start of any new session on this project.
2
Install the retrieval tools
Done
In plain English: four pieces of software — one to search the knowledge, one to let it store and search that knowledge directly inside BSM's own database, two to let the website reach it.
LlamaIndex (the retrieval engine — see Glossary), its official MariaDB connector (so vectors are stored in MariaDB 11.8 from day one, not a temporary file — see Glossary: “the bridge between LlamaIndex and MariaDB”), and FastAPI plus Uvicorn (the web layer). Installed into the private environment from Step 1, the standard approach on this server's operating system.
pip install llama-index "llama-index-vector-stores-mariadb>=0.3.0" openai fastapi uvicorn
Version matters here
The MariaDB connector below version 0.3.0 only works with the old 11.6 preview syntax, not 11.8 — the version pin above avoids installing the wrong one by accident.
OpenAI's technology does two jobs here, both using the API key already in active use elsewhere: generating the numbers — turning each saved document, and each question asked, into its vector (see Glossary); and writing the final answer, instructed to draw its facts only from BSM's own retrieved text. That instruction is what “grounded” means (see Glossary), and it's what keeps Katie's answers consistent with what BSM actually says.
2 — The Knowledge Store
Where documents live, separate from the tool that searches them
3
The knowledge folder
Done
In plain English: one folder, holding every document Katie can search, kept entirely separate from per-user data and from the database (Architecture Rule 2 & 3) — and given to WordPress to write into, without WordPress ever needing root.
Lives at /var/www/vhosts/bodysleepmind.com/katie-knowledge/ — deliberately a sibling of httpdocs, not inside it, so it's never reachable as a public webpage, while still sitting inside the one folder boundary this site's PHP is actually allowed to touch (see the open_basedir note below; this is the corrected final location, not the original one). Every document referenced from here on lives in this one place. Created by root over SSH, so it needed one further step before the knowledge manager (Build Plan Group 5) could actually write to it: handing ownership to the exact user WordPress's PHP actually runs as for this site.
Finding the real PHP user — don't guess, verify
File ownership of an existing WordPress file (e.g.
wp-config.php) is not a reliable way to find this — it can show root even when PHP itself runs as someone else entirely. The reliable method: check the operating system's own process list directly, while PHP-FPM is actually running. ps aux | grep php-fpm showed the real worker for this site as chris2, confirmed independently via id chris2 as belonging to the group psacln — a shared group Plesk uses across subscriptions, not one matching the username. These two values are specific to this server and this subscription — a future server will hand you different ones. Re-run this same methodology there; don't copy these names literally.
chown -R chris2:psacln /var/www/vhosts/bodysleepmind.com/katie-knowledge
Confirms it worked
ls -la /var/www/vhosts/bodysleepmind.com/ shows katie-knowledge owned by chris2 psacln instead of root root.
A real obstacle hit using the knowledge manager for real:
Two fixes existed: widen
What actually moved, for a future rebuild: the one line in
open_basedir
The folder was originally created at /var/katie-rag/knowledge/ — correct for the Python service, which has no restriction on what it can read. But the very first real attempt to approve a real submission through knowledge-manager.php failed with “Could not write to the knowledge folder.” The cause, confirmed with a tiny standalone test script (echo ini_get('open_basedir')): Plesk restricts each website's PHP to only read or write inside its own domain folder — here, /var/www/vhosts/bodysleepmind.com/:/tmp/. /var/katie-rag/ sat completely outside that boundary, so PHP was flatly blocked from writing to it, regardless of how correct the file ownership and permissions already were.Two fixes existed: widen
open_basedir in Plesk to explicitly allow the extra path, or move the folder to somewhere already inside the boundary. The folder was moved — the cleaner, permanent fix, with nothing to ever re-check on a future server or PHP upgrade. Only the knowledge folder itself needed to move; main.py, the virtual environment, and the secrets file all stayed exactly where they were, since PHP never touches any of those directly — only the Python service does, over its own internal network call, which open_basedir has no say over at all.What actually moved, for a future rebuild: the one line in
main.py setting KNOWLEDGE_DIR (Step 6), the matching KM_KNOWLEDGE_DIR constant in knowledge-manager.php (Build Plan Group 5), and the existing knowledge files themselves, moved across with mv. Ownership was re-applied at the new location using the exact same chris2:psacln values already confirmed above — that part never changed.
4
Initial knowledge set
Planned
In plain English: the first real documents Katie can search, saved as plain text — the same pattern every future addition follows.
Originally planned as two named reference documents typed directly into the folder over SSH. In practice, the knowledge manager (Build Plan Group 5) was built first, and the real intake path is through it instead — type or paste text, or upload a .txt, PDF, Word, or Excel file, reviewed and approved the same way every future document will be. The first genuinely live content will arrive this way, not via direct terminal authoring.
Pattern, not a one-off
Every future document — BSM's own, or a partner's — follows this same shape: submitted through the knowledge manager, reviewed, approved, and picked up by the rebuild step in Group 5.
3 — Retrieval and Answering
The service that turns documents into searchable meaning, and answers questions from it
5
Secrets file — the OpenAI key and database password
Done
In plain English: the OpenAI key and the database password live in one small, locked-down file of their own — never typed directly into
main.py's own code, the same lesson applied across every other file touched today.One extra package, then one file. python-dotenv lets main.py read both secrets from a separate file at startup.
pip install python-dotenv
In plain English: the two real secrets themselves, typed straight into a file that only root can read — created directly in the terminal, never pasted into chat with an AI assistant, including this one.
Then the file itself:
cat > /var/katie-rag/.env OPENAI_API_KEY=<the real masterKey value> <Ctrl+D> cat >> /var/katie-rag/.env DB_PASSWORD=<the real katie_rag database password> <Ctrl+D> chmod 600 /var/katie-rag/.env
Confirms it worked
cat /var/katie-rag/.env shows both lines present and correct. chmod 600 means only root can read the file at all — appropriate while the service is run manually over SSH as root; revisit if it's ever run as a different system user later.
6
main.py
Done
In plain English: the one file that does the thinking — reads every document, builds the vectors directly inside MariaDB 11.8, answers questions through one web address, and rebuilds itself on demand through a second. Written; not yet started or tested — that's Steps 7–8.
Before this runs
The dedicated
katie_rag database and its chris3 user (Step 16), and the secrets file (Step 5) — both done.
cat > /var/katie-rag/main.py << 'EOF'
import os
import glob
from dotenv import load_dotenv
from fastapi import FastAPI
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, StorageContext
from llama_index.vector_stores.mariadb import MariaDBVectorStore
# Secrets loaded from .env — never hardcoded directly in this file
load_dotenv("/var/katie-rag/.env")
os.environ["OPENAI_API_KEY"] = os.environ.get("OPENAI_API_KEY", "")
DB_PASSWORD = os.environ.get("DB_PASSWORD", "")
KNOWLEDGE_DIR = "/var/www/vhosts/bodysleepmind.com/katie-knowledge"
vector_store = MariaDBVectorStore.from_params(
host="127.0.0.1",
port=3306,
user="chris3",
password=DB_PASSWORD,
database="katie_rag",
table_name="katie_knowledge_vectors",
embed_dim=1536, # OpenAI embedding dimension
)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
query_engine = None
def build_index():
global query_engine
files = glob.glob(os.path.join(KNOWLEDGE_DIR, "*.txt"))
if not files:
query_engine = None
return
docs = SimpleDirectoryReader(KNOWLEDGE_DIR).load_data()
index = VectorStoreIndex.from_documents(docs, storage_context=storage_context)
query_engine = index.as_query_engine()
build_index() # build once when the service starts
app = FastAPI()
@app.get("/ask")
def ask(q: str):
if query_engine is None:
return {"answer": "No knowledge has been loaded yet."}
return {"answer": str(query_engine.query(q))}
@app.get("/reload")
def reload():
build_index()
files_found = len(glob.glob(os.path.join(KNOWLEDGE_DIR, "*.txt")))
return {"status": "reloaded", "files_found": files_found}
EOF
Confirmed, from the connector's own source code: when it creates the table automatically (the default behaviour, used above), it builds the vector index with
DISTANCE=cosine explicitly — not MariaDB's own Euclidean default. The connector's code even comments why: its search queries use VEC_DISTANCE_COSINE, and an index built for a different distance function wouldn't be used. Nothing extra to configure — cosine is what this exact tool does automatically.
The empty-folder case is handled deliberately: if
knowledge/ has no documents yet, /ask answers honestly rather than crashing. /reload always rebuilds the whole index from scratch rather than adding incrementally — the simpler, correct choice at today's scale; revisit only if the knowledge base grows large enough for a full rebuild to become genuinely slow.
7
Run the service
Done
In plain English: switches the service on, listening privately on BSM's own server (Architecture Rule 4).
cd /var/katie-rag nohup uvicorn main:app --host 127.0.0.1 --port 8420 > katie.log 2>&1 &
Always check the log straight after, every time
The shell returns a process number (e.g.
[1] 407815) the instant this runs, whether or not the service actually started successfully a moment later — that number alone is not confirmation. Always follow with cat katie.log. Confirmed working when it shows:
INFO: Started server process INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:8420
8
Verify retrieval directly
Done
In plain English: a direct, real question, confirming the service answers correctly before it's connected to anything else.
Spaces in the question need encoding
This server's
curl rejects a literal space in a URL outright (URL rejected: Malformed input to a URL function), even inside quotes. Use %20 in place of each space.
curl "http://127.0.0.1:8420/ask?q=what%20is%20the%20secret%20test%20phrase"
Pass criteria — actually confirmed
Asked the real planted test phrase from knowledge.txt (see Glossary & Section F), and got back the real, grounded answer:
{"answer":"THE SECRET TEST PHRASE IS: BLUE ELEPHANTS DANCE AT MIDNIGHT."}What's actually inside
katie_knowledge_vectors, and when a row gets createdIn plain English: not one row per document — one row per chunk of a document. And every approval rebuilds far more than just the new file.
The five columns, what each one actually is
| Column | What it actually is |
|---|---|
id | MariaDB's own row number — internal bookkeeping only, not used by any AI logic |
node_id | LlamaIndex's own unique ID for one chunk of a document, not the whole file |
text | The literal text of that one chunk — what actually gets matched, and handed to OpenAI when forming an answer |
metadata | A small JSON blob attached automatically — source filename, file path, that kind of thing |
embedding | vector(1536) — the actual numbers representing that chunk's meaning, compared at query time |
The real lifecycle, step by step
- A submission is Approved in the knowledge manager
- The file is written into the knowledge folder, and
/reloadis called automatically build_index()runsSimpleDirectoryReader(KNOWLEDGE_DIR).load_data()— this reads every.txtfile currently in the folder, not just the one just approvedVectorStoreIndex.from_documents(...)does the genuinely clever part, in three distinct steps — worth separating clearly, since they're easy to blur together:- Chunking — each document is sliced into smaller pieces first (see Glossary: Chunking), typically a paragraph or two each. A short file might end up as one single chunk; a long one becomes several. This happens before anything else.
- Embedding — each individual chunk, on its own, is sent to OpenAI and comes back as its own vector (see Glossary: Embedding) — one chunk in, one set of 1536 numbers out.
- Storing — that chunk's text and its vector are written together as one new row.
Worth knowing, not yet fixed: step 3 reads every file, every single time, and
build_index() never clears the table first. That means every approval likely re-embeds and re-inserts every document already in the knowledge base on top of what's already there, not just the new one — the table can accumulate duplicate rows over time, and every approval re-spends real OpenAI cost on documents that haven't even changed. Checkable directly: Browse katie_knowledge_vectors in phpMyAdmin and look for the same source file appearing more than once. Two real fixes exist if this turns out to matter — clear the table at the start of build_index() before rebuilding, or move to properly incremental updates instead of a full rebuild each time — deliberately not built yet, a genuine open decision rather than an oversight.
Getting Steps 1–8 actually running — the real first attempt
In plain English: none of the above worked on the first try. Every real obstacle hit while actually starting this service for the first time, and the exact fix for each — kept here so the same problem doesn't need re-solving from scratch next time.
Lessons 2 and 3 below were both eventually traced to the same root cause: pasting straight from a chat window into the terminal. See the paste technique flagged in the SSH Terminal glossary entry, above — using it from the start would very likely have prevented both.
1. A planned step can get skipped without anyone noticing
main.py was written out as a command, but a detour into updating this guide happened immediately after, before that command was actually run on the server. The result: uvicorn failed with a vague Could not import module "main". The fix was simple once found — ls -la main.py showed the file plainly didn't exist. Worth a habit: after any step that creates a file, confirm it's really there before moving on, especially after a context-switch away from the terminal.
2. Large pastes into this terminal can silently corrupt
Pasting the entire
main.py content in one go (roughly 50 lines) produced a file with whole sections missing and fragments from different parts of the script mashed together mid-line — not a shell error, a silently broken file that only revealed itself as confusing Python errors later. The fix: rebuild in small chunks (roughly 10–15 lines each), checking wc -l after every single one, rather than trusting one large paste. Slower, but every chunk verifiable on its own.
3. Short pastes can arrive wrapped in stray characters
Occasionally a short command pasted into this terminal arrived as
^[[200~actual command~, and the shell tried to run that whole garbled string, failing with command not found. This is the terminal's own paste-marker leaking through uncleanly. The fix: for short commands, just type them by hand instead of pasting — it happened consistently with paste, never with typing.
4. A vague server error means: ask the language directly, not the framework
uvicorn's own error message for an import failure (Could not import module "main") is a dead end by itself — it hides the real Python exception underneath it, even running in the foreground with no log redirection. The fix that actually revealed the cause: bypass uvicorn entirely and ask Python directly — python3 -c "import main" — which prints the full, real traceback uvicorn was hiding.
5. The real cause, once visible: a stray semicolon in the password
The real traceback showed
Access denied for user 'chris3'@'localhost' (using password: YES) — a wrong-password error, despite the password looking right at a glance. Opening .env directly in nano and reading it character by character showed why: DB_PASSWORD=...rs; — a trailing semicolon had been typed onto the end of the value, a leftover habit from the many PHP files edited earlier the same session, where every line ends in ;. That semicolon was silently treated as part of the password itself. The fix: reset the password fresh in Plesk, then edit the line in nano deleting back to immediately after the = before typing the new value — confirmed visually before saving, not assumed correct.
6. curl on this server rejects literal spaces in a URL
Testing
/ask with a real multi-word question failed with URL rejected: Malformed input to a URL function, even with the whole address inside quotes. The fix: encode each space as %20 in the query string rather than leaving it literal.
None of these were architecture problems — every Architecture Rule and design decision earlier in this guide held up exactly as planned. These were all small, ordinary friction points in actually typing commands into a real terminal, the kind every real build runs into. Worth keeping this list rather than deleting it once solved, since a server rebuild in future will very likely hit the same handful of things again.
Three more real obstacles were hit later in the build, kept inline at the step where they actually happened rather than duplicated here: a regressed API key at Step 9, a strict-comparison bug at Step 10, and a port conflict at Step 14.
Three more real obstacles were hit later in the build, kept inline at the step where they actually happened rather than duplicated here: a regressed API key at Step 9, a strict-comparison bug at Step 10, and a port conflict at Step 14.
4 — Connecting to the Live Site
Linking the service to WordPress and to Katie's conversation
9
WordPress bridge
Done
In plain English: the connection that lets the live website ask the service a question, the same way it already talks to Stripe.
A wp_remote_get() call to 127.0.0.1:8420, internal to the server, never exposed publicly — built directly into triage_backend.php's ?ai=1 handler (Step 10, below).
A real mistake worth recording so it can't happen quietly again
When this file was edited a second time, it was copied fresh from the original uploaded file rather than from the already-corrected version — silently undoing an earlier fix and putting the old, exposed OpenAI key back in as plain text. Caught only because the result was checked carefully before uploading. The lesson: when re-editing any file for a second time in a long session, always build on the most recently corrected version, never re-copy from the original source.
10
Connect Katie's conversation
Done
In plain English: Katie now genuinely checks the real knowledge base before answering a free-form question — proven live, through the real chat widget, not just over curl.
triage_backend.php's ?ai=1 handler no longer reads knowledge.txt directly. It calls /ask first. Two decisions were made deliberately, not by default:
- No silent fallback to the old approach. If the knowledge service is unreachable or has nothing loaded, Katie returns a clear, honest message — "Sorry. I'm just getting my wellness check up. I won't be able to answer your question right now. Back shortly!" — rather than quietly reverting to stuffing the whole of
knowledge.txtinto a fresh OpenAI call. A silent fallback would hide a real outage completely, and cost real money every time it happened; a visible message means a real failure gets noticed and fixed, not buried. - No blending with GOD JSON, for now. The retrieved answer is shown as-is. Deliberately not built as a default or a guess at what might be wanted — if and when Katie's answers should ever draw on a user's own wellness profile too, that's a separate, considered decision to make case by case, not something to bundle in here.
A real bug hit getting this working: a strict type comparison
The first live attempt always returned the fallback message, even though the knowledge service was genuinely running and answering correctly when tested directly. The cause, found by writing a small standalone test script that called the exact same WordPress function in isolation: the response-code check used PHP's strict
=== comparison — wp_remote_retrieve_response_code($rag_response) === 200. WordPress's HTTP functions don't always return a plain integer; depending on the server's transport, the code can come back as the text "200" instead, and "200" === 200 is false in PHP even though the call succeeded. The fix: cast explicitly before comparing — (int) wp_remote_retrieve_response_code($rag_response) === 200 — checking the value, not the type.
Confirmed working, live, through the real widget
Asked Katie directly through the floating chat on the live site: "what is the secret test phrase". Real answer, correct: "THE SECRET TEST PHRASE IS: BLUE ELEPHANTS DANCE AT MIDNIGHT." The full chain, proven from a real visitor's point of view, not just at the API level.
5 — The Interface
The specific build that hands knowledge management to non-developers, permanently — the line between Phase One and Phase Two
11
Submission page
Done
In plain English: the actual page behind stage 2 of the operating model in Phase Two, below — where someone submits new knowledge they've written, uploaded, or is editing.
knowledge-manager.php, a WordPress admin page in the same family as every other admin tool already built for BSM. Built differently from the original plan: rather than a holding folder (e.g. /var/katie-rag/pending/), every submission is saved into a database table, katie_knowledge_queue, that self-installs the first time the page loads. Nothing is written to the live knowledge/ folder until Step 13's approval — the file only needs write access to that one folder, never to a second one, and the table doubles as a genuine audit trail (who submitted what, when, and why anything was declined).
Four ways in, all feeding the same queue: type or paste text directly; upload a .txt file; upload a PDF (text pulled out via pdftotext); upload a Word (.docx) or Excel (.xlsx) file, read directly with no AI step at all, no extra cost, no misread risk. Editing something already live works the same way — the edit goes to the queue, the live file is untouched until it's approved.
Considered and deliberately not built: an image/photo analyser
The same vision capability already reading food labels elsewhere on the site would add real misread risk and real per-use cost, for a case that barely applies here: it earns its value from real-time field capture (a label, no file available at all), and knowledge curation is desk-based, where a real file almost always exists already. Revisit only if that specific situation actually comes up.
Access level
Tracked in Appendix, Section I — the specific roles permitted to submit.
Full source code
The complete, current file is embedded in full after Step 13, below — not just described.
12
Review queue
Done
In plain English: the actual page behind stage 3 of the operating model in Phase Two, below — where a reviewer checks each submission before anything goes live, and a permanent record of every decision either way.
A Review Queue tab inside knowledge-manager.php, listing everything currently pending in katie_knowledge_queue — tagged NEW or EDIT-of-X, with who submitted it, when, and the source type. For each submission, the reviewer chooses:
- Approve — the content is written into the live
knowledge/folder (Group 2) for the first time, the row is marked approved, and Step 13's reload is triggered immediately, automatically - Decline — marked declined, with an optional note on why; the file is never written anywhere Katie can see it
A separate Declined tab, added once it became clear a decline with no visible record afterward was a real gap: every declined item stays visible, with who declined it, when, and the reason — a genuine audit trail, not a second queue, and nothing in it can come back to life by itself.
Access level
Default split currently in the code: submitting requires
edit_posts (any Contributor+), reviewing and declining requires manage_options (Administrators only) — a sensible starting point, not a final decision. Tracked in Appendix, Section I.
13
Reload on approval
Done
In plain English: the moment a submission is approved, the service re-reads its knowledge folder and rebuilds its index — this is what makes stage 4 of the operating model (Phase Two, below) instant, with no separate technical step for anyone.
Built on both sides now. knowledge-manager.php's Approve action calls /reload via wp_remote_get() the moment a reviewer clicks Approve. main.py's /reload address (Step 6) rebuilds the whole index from the knowledge folder and reports back how many files it found. Not yet tested end-to-end — that happens once Step 7 actually starts the service.
This address changes what Katie knows, so it's restricted to internal calls only (Architecture Rule 4) — reachable from WordPress on the same server, never from outside it.
Full source — knowledge-manager.php (Steps 11–13 combined)
Done
In plain English: the actual, complete, currently-live file — not a description of it. Embedded here in full so this guide alone is enough to recreate it, on this server or a future one, without needing to track down the separate file.
knowledge/knowledge-manager.php — complete file
<?php
/*
=============================================================================================
Filename: knowledge/knowledge-manager.php
Description: Katie's Knowledge Admin Interface — Phase Two of the RAG build.
VERSION: v1.0
=============================================================================================
DEVELOPER GUIDE: WHAT THIS FILE DOES
This is the admin page that lets a knowledge admin write, upload, review, and publish
knowledge for Katie — without ever touching SSH. It follows the same "Monolith SPA"
pattern as cheat_master.php: PHP AJAX interceptor on top, scoped CSS, then HTML + vanilla JS.
1. SUBMISSION (4 ways in)
- Type/paste text directly
- Upload a .txt file
- Upload a PDF (text extracted automatically via `pdftotext`)
- Upload a Word (.docx) or Excel (.xlsx) file (read directly, no AI involved)
An image-upload / vision-analysis method was deliberately considered and parked —
see the note above the (removed) handler further down. It earns its value for
real-time field capture (a label, no file available); knowledge curation is
desk-based, where a real file almost always exists already.
Every submission — new or an edit of something already live — is saved into a
database queue table, NOT written to the live knowledge folder. Nothing goes live
until approved.
2. REVIEW QUEUE
Shows every pending item. Approve writes it into the live knowledge folder and
triggers the Python service's /reload address. Decline marks it declined (kept,
not deleted, for an honest audit trail) and it never reaches Katie.
3. WHY A DATABASE TABLE INSTEAD OF A SECOND FOLDER
The original plan used a `/var/katie-rag/pending/` folder. A DB table does the same
job with one real advantage: this page only needs write access to the live
`knowledge/` folder, and only at the moment of approval — not a second folder needing
its own permissions setup. It also gives a genuine audit trail (who submitted what,
who reviewed it, when) for free.
4. ACCESS LEVELS — PLACEHOLDER, NOT YET A FINAL DECISION
Submitting requires 'edit_posts' (any Contributor+). Reviewing/approving requires
'manage_options' (Administrators only). This is a sensible default split, not a
final answer — change the current_user_can() checks below to match whatever BSM
actually wants. Tracked as an open decision in the build guide, Appendix Section I.
5. THE OPENAI KEY
Pulled from getenv('OPENAI_KEY') — the same convention already used in
triage_backend.php. Deliberately NOT a third hardcoded copy of the key.
=============================================================================================
*/
// =========================================================================
// CONFIGURATION
// =========================================================================
define('KM_KNOWLEDGE_DIR', '/var/www/vhosts/bodysleepmind.com/katie-knowledge/');
define('KM_RELOAD_URL', 'http://127.0.0.1:8420/reload'); // Must match whatever Step 12 actually builds
define('KM_TABLE', 'katie_knowledge_queue'); // No wp_ prefix, matching user_cheats / assessment convention
$km_can_submit = current_user_can('edit_posts');
$km_can_review = current_user_can('manage_options');
if (!$km_can_submit) {
echo '<p>You do not have permission to view this page.</p>';
return;
}
// =========================================================================
// SELF-INSTALLING TABLE (runs once, harmless if it already exists)
// =========================================================================
global $wpdb;
$wpdb->query("CREATE TABLE IF NOT EXISTS " . KM_TABLE . " (
id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
action_type VARCHAR(20) NOT NULL,
target_filename VARCHAR(255) NULL,
suggested_filename VARCHAR(255) NULL,
content_text LONGTEXT NOT NULL,
source_type VARCHAR(20) NOT NULL,
ai_meta TEXT NULL,
submitted_by BIGINT UNSIGNED NOT NULL,
submitted_at DATETIME NOT NULL,
status VARCHAR(20) NOT NULL DEFAULT 'pending',
reviewed_by BIGINT UNSIGNED NULL,
reviewed_at DATETIME NULL,
decline_note TEXT NULL
) " . $wpdb->get_charset_collate());
// =========================================================================
// PHP BACKEND: AJAX INTERCEPTOR
// =========================================================================
if (isset($_POST['km_action'])) {
ob_clean();
header('Content-Type: application/json');
$user_id = get_current_user_id();
if (!$user_id) { echo json_encode(['error' => 'You must be logged in.']); exit; }
$action = sanitize_text_field($_POST['km_action']);
// --- Helper: safely resolve a filename inside the knowledge dir, no path traversal ---
function km_safe_path($filename) {
$clean = basename($filename); // strips any ../ or directory parts
return KM_KNOWLEDGE_DIR . $clean;
}
// --- Helper: extract plain text from a .docx file (it's a zip with XML inside) ---
function km_extract_docx_text($path) {
if (!class_exists('ZipArchive')) return ['error' => 'PHP\'s zip extension is not enabled on this server.'];
$zip = new ZipArchive();
if ($zip->open($path) !== true) return ['error' => 'Could not open this file as a .docx (is it actually a Word document?).'];
$xml = $zip->getFromName('word/document.xml');
$zip->close();
if ($xml === false) return ['error' => 'This doesn\'t look like a valid .docx file.'];
// Turn paragraph breaks into newlines before stripping the XML, so the
// result reads as actual paragraphs, not one giant run-on block of text.
$xml = str_replace('</w:p>', "\n", $xml);
$xml = str_replace('</w:tab>', "\t", $xml);
$text = html_entity_decode(strip_tags($xml));
$text = trim(preg_replace('/\n{3,}/', "\n\n", $text));
return ['text' => $text];
}
// --- Helper: extract a readable table from a .xlsx file's first sheet ---
function km_extract_xlsx_text($path) {
if (!class_exists('ZipArchive')) return ['error' => 'PHP\'s zip extension is not enabled on this server.'];
$zip = new ZipArchive();
if ($zip->open($path) !== true) return ['error' => 'Could not open this file as an .xlsx (is it actually an Excel file?).'];
// Excel stores repeated text once in a shared lookup table, and cells just
// reference it by number — so we build that lookup first.
$shared = [];
$shared_xml = $zip->getFromName('xl/sharedStrings.xml');
if ($shared_xml !== false) {
$dom = new DOMDocument();
@$dom->loadXML($shared_xml);
foreach ($dom->getElementsByTagName('si') as $si) { $shared[] = trim($si->textContent); }
}
$sheet_xml = $zip->getFromName('xl/worksheets/sheet1.xml'); // first sheet only, v1 limitation
$zip->close();
if ($sheet_xml === false) return ['error' => 'No readable sheet found in this file.'];
$dom = new DOMDocument();
@$dom->loadXML($sheet_xml);
$lines = [];
foreach ($dom->getElementsByTagName('row') as $row) {
$cells = [];
foreach ($row->getElementsByTagName('c') as $cell) {
$type = $cell->getAttribute('t');
$v_nodes = $cell->getElementsByTagName('v');
$raw = $v_nodes->length ? $v_nodes->item(0)->textContent : '';
$cells[] = ($type === 's' && $raw !== '' && isset($shared[(int)$raw])) ? $shared[(int)$raw] : $raw;
}
if (count(array_filter($cells, fn($c) => trim($c) !== ''))) {
$lines[] = implode(' | ', $cells);
}
}
if (empty($lines)) return ['error' => 'No data found on the first sheet.'];
return ['text' => trim(implode("\n", $lines))];
}
// --- LIST LIVE KNOWLEDGE FILES ---
if ($action === 'list_live') {
$files = [];
foreach (glob(KM_KNOWLEDGE_DIR . '*.txt') as $path) {
$content = file_get_contents($path);
$files[] = [
'filename' => basename($path),
'preview' => mb_substr(trim($content), 0, 140),
'size' => filesize($path),
'modified' => date('Y-m-d H:i', filemtime($path)),
];
}
echo json_encode(['success' => true, 'data' => $files]);
exit;
}
// --- GET ONE LIVE FILE'S FULL CONTENT (for editing) ---
if ($action === 'get_live_file') {
$path = km_safe_path($_POST['filename'] ?? '');
if (!file_exists($path)) { echo json_encode(['error' => 'File not found.']); exit; }
echo json_encode(['success' => true, 'content' => file_get_contents($path)]);
exit;
}
// --- SUBMIT: NEW TEXT (typed/pasted) ---
if ($action === 'submit_text') {
$text = sanitize_textarea_field($_POST['content'] ?? '');
$suggested = sanitize_file_name($_POST['suggested_filename'] ?? '');
if (empty($text)) { echo json_encode(['error' => 'No content provided.']); exit; }
if (empty($suggested)) { $suggested = 'submission-' . time() . '.txt'; }
if (!str_ends_with($suggested, '.txt')) { $suggested .= '.txt'; }
$wpdb->insert(KM_TABLE, [
'action_type' => 'new',
'suggested_filename' => $suggested,
'content_text' => $text,
'source_type' => 'text',
'submitted_by' => $user_id,
'submitted_at' => current_time('mysql'),
'status' => 'pending',
]);
echo json_encode(['success' => true]);
exit;
}
// --- SUBMIT: EDIT OF AN EXISTING LIVE FILE ---
if ($action === 'submit_edit') {
$target = sanitize_file_name($_POST['target_filename'] ?? '');
$text = sanitize_textarea_field($_POST['content'] ?? '');
if (empty($target) || !file_exists(km_safe_path($target))) {
echo json_encode(['error' => 'Original file not found.']); exit;
}
if (empty($text)) { echo json_encode(['error' => 'No content provided.']); exit; }
$wpdb->insert(KM_TABLE, [
'action_type' => 'edit',
'target_filename' => $target,
'content_text' => $text,
'source_type' => 'text',
'submitted_by' => $user_id,
'submitted_at' => current_time('mysql'),
'status' => 'pending',
]);
echo json_encode(['success' => true]);
exit;
}
// --- SUBMIT: UPLOAD A .TXT FILE ---
if ($action === 'upload_txt') {
if (empty($_FILES['file']['tmp_name'])) { echo json_encode(['error' => 'No file received.']); exit; }
$text = file_get_contents($_FILES['file']['tmp_name']);
$suggested = sanitize_file_name($_FILES['file']['name']);
$wpdb->insert(KM_TABLE, [
'action_type' => 'new',
'suggested_filename' => $suggested,
'content_text' => $text,
'source_type' => 'txt_upload',
'submitted_by' => $user_id,
'submitted_at' => current_time('mysql'),
'status' => 'pending',
]);
echo json_encode(['success' => true]);
exit;
}
// --- SUBMIT: UPLOAD A PDF (text extracted via pdftotext) ---
if ($action === 'upload_pdf') {
if (empty($_FILES['file']['tmp_name'])) { echo json_encode(['error' => 'No file received.']); exit; }
// Confirm pdftotext is actually available before relying on it
$which = shell_exec('which pdftotext');
if (empty(trim($which ?? ''))) {
echo json_encode(['error' => 'pdftotext is not installed on this server. Install poppler-utils first.']);
exit;
}
$tmp_path = $_FILES['file']['tmp_name'];
$text = shell_exec('pdftotext ' . escapeshellarg($tmp_path) . ' -');
$text = trim($text ?? '');
if (empty($text)) {
echo json_encode(['error' => 'No text could be extracted — this may be a scanned/image-only PDF.']);
exit;
}
$suggested = sanitize_file_name(pathinfo($_FILES['file']['name'], PATHINFO_FILENAME)) . '.txt';
$wpdb->insert(KM_TABLE, [
'action_type' => 'new',
'suggested_filename' => $suggested,
'content_text' => $text,
'source_type' => 'pdf_upload',
'submitted_by' => $user_id,
'submitted_at' => current_time('mysql'),
'status' => 'pending',
]);
echo json_encode(['success' => true]);
exit;
}
// --- IMAGE UPLOAD: DELIBERATELY NOT BUILT ---
// Considered and parked. The food/cheat analyser's vision capability earns its value
// from real-time field capture (a label in a supermarket, no file available at all).
// Knowledge curation is desk-based and deliberate — a real file almost always exists,
// so .docx/.xlsx/.pdf direct reading covers this need more reliably and at no AI cost.
// Revisit only if a genuine "photo is the only option" case actually shows up.
// --- SUBMIT: UPLOAD A WORD DOCUMENT (.docx, read directly, no AI involved) ---
if ($action === 'upload_docx') {
if (empty($_FILES['file']['tmp_name'])) { echo json_encode(['error' => 'No file received.']); exit; }
$result = km_extract_docx_text($_FILES['file']['tmp_name']);
if (isset($result['error'])) { echo json_encode(['error' => $result['error']]); exit; }
if (empty(trim($result['text']))) { echo json_encode(['error' => 'No text found in this document.']); exit; }
$suggested = sanitize_file_name(pathinfo($_FILES['file']['name'], PATHINFO_FILENAME)) . '.txt';
$wpdb->insert(KM_TABLE, [
'action_type' => 'new',
'suggested_filename' => $suggested,
'content_text' => $result['text'],
'source_type' => 'docx_upload',
'submitted_by' => $user_id,
'submitted_at' => current_time('mysql'),
'status' => 'pending',
]);
echo json_encode(['success' => true]);
exit;
}
// --- SUBMIT: UPLOAD AN EXCEL FILE (.xlsx, read directly, no AI involved) ---
if ($action === 'upload_xlsx') {
if (empty($_FILES['file']['tmp_name'])) { echo json_encode(['error' => 'No file received.']); exit; }
$result = km_extract_xlsx_text($_FILES['file']['tmp_name']);
if (isset($result['error'])) { echo json_encode(['error' => $result['error']]); exit; }
$suggested = sanitize_file_name(pathinfo($_FILES['file']['name'], PATHINFO_FILENAME)) . '.txt';
$wpdb->insert(KM_TABLE, [
'action_type' => 'new',
'suggested_filename' => $suggested,
'content_text' => $result['text'],
'source_type' => 'xlsx_upload',
'submitted_by' => $user_id,
'submitted_at' => current_time('mysql'),
'status' => 'pending',
]);
echo json_encode(['success' => true]);
exit;
}
// --- LIST PENDING QUEUE (review screen) ---
if ($action === 'list_pending') {
if (!$km_can_review) { echo json_encode(['error' => 'Not permitted.']); exit; }
$rows = $wpdb->get_results("SELECT * FROM " . KM_TABLE . " WHERE status = 'pending' ORDER BY submitted_at ASC", ARRAY_A);
foreach ($rows as &$row) {
$user = get_userdata($row['submitted_by']);
$row['submitted_by_name'] = $user ? $user->display_name : 'Unknown';
$row['preview'] = mb_substr(trim($row['content_text']), 0, 200);
}
echo json_encode(['success' => true, 'data' => $rows]);
exit;
}
// --- LIST DECLINED ITEMS (audit trail — nothing here can go live) ---
if ($action === 'list_declined') {
if (!$km_can_review) { echo json_encode(['error' => 'Not permitted.']); exit; }
$rows = $wpdb->get_results("SELECT * FROM " . KM_TABLE . " WHERE status = 'declined' ORDER BY reviewed_at DESC", ARRAY_A);
foreach ($rows as &$row) {
$submitter = get_userdata($row['submitted_by']);
$reviewer = get_userdata($row['reviewed_by']);
$row['submitted_by_name'] = $submitter ? $submitter->display_name : 'Unknown';
$row['reviewed_by_name'] = $reviewer ? $reviewer->display_name : 'Unknown';
$row['preview'] = mb_substr(trim($row['content_text']), 0, 200);
}
echo json_encode(['success' => true, 'data' => $rows]);
exit;
}
// --- APPROVE A PENDING ITEM ---
if ($action === 'approve') {
if (!$km_can_review) { echo json_encode(['error' => 'Not permitted.']); exit; }
$id = intval($_POST['id']);
$row = $wpdb->get_row($wpdb->prepare("SELECT * FROM " . KM_TABLE . " WHERE id = %d AND status = 'pending'", $id), ARRAY_A);
if (!$row) { echo json_encode(['error' => 'Item not found or already reviewed.']); exit; }
$filename = ($row['action_type'] === 'edit') ? $row['target_filename'] : $row['suggested_filename'];
$filename = sanitize_file_name($filename);
if (!str_ends_with($filename, '.txt')) { $filename .= '.txt'; }
$write_path = KM_KNOWLEDGE_DIR . $filename;
$written = file_put_contents($write_path, $row['content_text']);
if ($written === false) {
echo json_encode(['error' => 'Could not write to the knowledge folder — check folder permissions.']);
exit;
}
$wpdb->update(KM_TABLE, [
'status' => 'approved',
'reviewed_by' => $user_id,
'reviewed_at' => current_time('mysql'),
], ['id' => $id]);
// Trigger the live reload — see Build Plan Group 5, Step 12
$reload_result = wp_remote_get(KM_RELOAD_URL, ['timeout' => 15]);
$reload_ok = !is_wp_error($reload_result);
echo json_encode(['success' => true, 'reload_ok' => $reload_ok]);
exit;
}
// --- DECLINE A PENDING ITEM ---
if ($action === 'decline') {
if (!$km_can_review) { echo json_encode(['error' => 'Not permitted.']); exit; }
$id = intval($_POST['id']);
$note = sanitize_textarea_field($_POST['note'] ?? '');
$wpdb->update(KM_TABLE, [
'status' => 'declined',
'reviewed_by' => $user_id,
'reviewed_at' => current_time('mysql'),
'decline_note' => $note,
], ['id' => $id, 'status' => 'pending']);
echo json_encode(['success' => true]);
exit;
}
echo json_encode(['error' => 'Unknown action.']);
exit;
}
?>
<!-- =========================================================================
FRONTEND CSS — scoped to .km- to protect the theme
========================================================================= -->
<style>
.km-wrapper { max-width: 1000px; margin: 30px auto; padding: 0 20px; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; color: #2F3E46; }
.km-header { text-align: center; margin-bottom: 25px; }
.km-header h1 { font-size: 26px; font-weight: 900; margin: 0; }
.km-tabs { display: flex; gap: 10px; justify-content: center; margin-bottom: 25px; }
.km-tab { padding: 10px 20px; border-radius: 20px; background: #f1f5f9; color: #5a6b72; font-weight: 700; font-size: 14px; cursor: pointer; border: none; }
.km-tab.active { background: #2F3E46; color: #fff; }
.km-panel { display: none; }
.km-panel.active { display: block; }
.km-card { background: #fff; border: 1px solid #e5e9ea; border-radius: 12px; padding: 20px; margin-bottom: 15px; box-shadow: 0 2px 10px rgba(0,0,0,0.03); }
.km-card-title { font-weight: 800; font-size: 15px; margin-bottom: 8px; }
.km-preview { font-size: 13px; color: #5a6b72; margin-bottom: 10px; }
.km-meta { font-size: 12px; color: #a1b0b5; margin-bottom: 10px; }
.km-input-methods { display: flex; gap: 10px; margin-bottom: 20px; flex-wrap: wrap; }
.km-method-btn { flex: 1; min-width: 140px; padding: 12px; border-radius: 8px; border: 2px dashed #c7d2d4; background: #faf9f5; text-align: center; font-weight: 700; font-size: 13px; cursor: pointer; color: #5a6b72; }
.km-method-btn.active { border-color: #819b81; background: #f0f5f0; color: #2F3E46; }
.km-hint { font-size: 13px; color: #5a6b72; background: #f8fafc; border-left: 3px solid #819b81; padding: 8px 12px; border-radius: 4px; margin-bottom: 12px; }
.km-textarea { width: 100%; height: 220px; padding: 14px; border: 1px solid #e5e9ea; border-radius: 8px; font-size: 14px; box-sizing: border-box; resize: vertical; }
.km-input { width: 100%; padding: 10px; border: 1px solid #e5e9ea; border-radius: 8px; font-size: 14px; box-sizing: border-box; margin-bottom: 10px; }
.km-btn { background: #819b81; color: #fff; border: none; padding: 12px 20px; border-radius: 8px; font-weight: 700; cursor: pointer; font-size: 14px; }
.km-btn:hover { background: #6d8a6d; }
.km-btn.km-btn-decline { background: #d63638; }
.km-btn-small { font-size: 12px; padding: 6px 14px; border-radius: 6px; border: none; font-weight: 700; cursor: pointer; }
.km-tag { display: inline-block; font-size: 11px; font-weight: 700; padding: 3px 10px; border-radius: 12px; margin-right: 6px; }
.km-tag-new { background: #e0f2fe; color: #0369a1; }
.km-tag-edit { background: #fef3c7; color: #92400e; }
.km-tag-source { background: #f1f5f9; color: #5a6b72; }
.km-status { text-align: center; font-size: 13px; font-weight: 700; height: 20px; margin: 10px 0; }
.km-empty { text-align: center; color: #a1b0b5; padding: 30px; }
</style>
<!-- =========================================================================
FRONTEND HTML
========================================================================= -->
<div class="km-wrapper">
<div class="km-header">
<h1>📚 Katie's Knowledge Manager</h1>
</div>
<div class="km-tabs">
<button class="km-tab active" data-panel="km-panel-existing">Existing Knowledge</button>
<button class="km-tab" data-panel="km-panel-new">Add New</button>
<?php if ($km_can_review): ?>
<button class="km-tab" data-panel="km-panel-review">Review Queue</button>
<button class="km-tab" data-panel="km-panel-declined">Declined</button>
<?php endif; ?>
</div>
<div id="km-status" class="km-status"></div>
<!-- EXISTING KNOWLEDGE -->
<div id="km-panel-existing" class="km-panel active">
<div id="km-existing-list"><p class="km-empty">Loading...</p></div>
</div>
<!-- ADD NEW -->
<div id="km-panel-new" class="km-panel">
<div class="km-input-methods">
<div class="km-method-btn active" data-method="text">✍️ Type / Paste</div>
<div class="km-method-btn" data-method="txt">📝 Upload .txt</div>
<div class="km-method-btn" data-method="pdf">📄 Upload PDF</div>
<div class="km-method-btn" data-method="docx">📘 Upload Word Doc</div>
<div class="km-method-btn" data-method="xlsx">📊 Upload Excel</div>
</div>
<div id="km-method-text" class="km-method-panel">
<input type="text" id="km-new-filename" class="km-input" placeholder="Filename (e.g. apnea-analytics.txt) — optional, we'll suggest one if left blank">
<textarea id="km-new-text" class="km-textarea" placeholder="Write or paste the knowledge here..."></textarea>
<button class="km-btn" onclick="kmSubmitText()">Submit for Review</button>
</div>
<div id="km-method-txt" class="km-method-panel" style="display:none;">
<input type="file" id="km-file-txt" accept=".txt" class="km-input">
<button class="km-btn" onclick="kmUploadFile('txt')">Upload & Submit for Review</button>
</div>
<div id="km-method-pdf" class="km-method-panel" style="display:none;">
<p class="km-hint">Pulls the text straight out of the PDF. Best for PDFs that contain real text, not a scanned image of a page.</p>
<input type="file" id="km-file-pdf" accept=".pdf" class="km-input">
<button class="km-btn" onclick="kmUploadFile('pdf')">Extract Text & Submit for Review</button>
</div>
<div id="km-method-docx" class="km-method-panel" style="display:none;">
<p class="km-hint">For an actual Word document (.docx). Pulls out all the written text, paragraph by paragraph. Tables and formatting are simplified to plain text.</p>
<input type="file" id="km-file-docx" accept=".docx" class="km-input">
<button class="km-btn" onclick="kmUploadFile('docx')">Extract Text & Submit for Review</button>
</div>
<div id="km-method-xlsx" class="km-method-panel" style="display:none;">
<p class="km-hint">For an actual Excel file (.xlsx). Reads the <strong>first sheet only</strong> and turns each row into one line, with columns separated by " | ". Best for simple tables — a product list, an ingredient sheet — not complex multi-sheet workbooks.</p>
<input type="file" id="km-file-xlsx" accept=".xlsx" class="km-input">
<button class="km-btn" onclick="kmUploadFile('xlsx')">Extract Data & Submit for Review</button>
</div>
</div>
<!-- REVIEW QUEUE -->
<?php if ($km_can_review): ?>
<div id="km-panel-review" class="km-panel">
<div id="km-review-list"><p class="km-empty">Loading...</p></div>
</div>
<!-- DECLINED (audit trail only — nothing here can go live) -->
<div id="km-panel-declined" class="km-panel">
<div id="km-declined-list"><p class="km-empty">Loading...</p></div>
</div>
<?php endif; ?>
</div>
<script>
const kmCanReview = <?php echo $km_can_review ? 'true' : 'false'; ?>;
// --- TAB SWITCHING ---
document.querySelectorAll('.km-tab').forEach(tab => {
tab.addEventListener('click', () => {
document.querySelectorAll('.km-tab').forEach(t => t.classList.remove('active'));
document.querySelectorAll('.km-panel').forEach(p => p.classList.remove('active'));
tab.classList.add('active');
document.getElementById(tab.dataset.panel).classList.add('active');
if (tab.dataset.panel === 'km-panel-existing') loadExisting();
if (tab.dataset.panel === 'km-panel-review') loadReview();
if (tab.dataset.panel === 'km-panel-declined') loadDeclined();
});
});
// --- ADD NEW: METHOD SWITCHING ---
document.querySelectorAll('.km-method-btn').forEach(btn => {
btn.addEventListener('click', () => {
document.querySelectorAll('.km-method-btn').forEach(b => b.classList.remove('active'));
document.querySelectorAll('.km-method-panel').forEach(p => p.style.display = 'none');
btn.classList.add('active');
document.getElementById('km-method-' + btn.dataset.method).style.display = 'block';
});
});
function showStatus(msg, isError) {
const el = document.getElementById('km-status');
el.innerText = msg;
el.style.color = isError ? '#d63638' : '#46b450';
setTimeout(() => { el.innerText = ''; }, 4000);
}
async function postAction(action, extraData, isFormData) {
let body;
if (isFormData) {
body = extraData;
body.append('km_action', action);
} else {
body = new URLSearchParams({ km_action: action, ...extraData });
}
const res = await fetch(window.location.href, { method: 'POST', body: body });
return res.json();
}
// --- EXISTING KNOWLEDGE ---
async function loadExisting() {
const data = await postAction('list_live', {});
const container = document.getElementById('km-existing-list');
if (data.error) { container.innerHTML = `<p class="km-empty">${data.error}</p>`; return; }
if (!data.data.length) { container.innerHTML = '<p class="km-empty">No knowledge files yet.</p>'; return; }
container.innerHTML = data.data.map(f => `
<div class="km-card">
<div class="km-card-title">${f.filename}</div>
<div class="km-meta">${f.size} bytes · last modified ${f.modified}</div>
<div class="km-preview">${f.preview}...</div>
<button class="km-btn-small km-btn" onclick="editExisting('${f.filename}')">Edit (sends to review)</button>
</div>
`).join('');
}
async function editExisting(filename) {
const data = await postAction('get_live_file', { filename });
if (data.error) { showStatus(data.error, true); return; }
const newText = prompt('Edit content for ' + filename + ' (your edit goes to review, the live version is untouched until approved):', data.content);
if (newText === null) return;
const result = await postAction('submit_edit', { target_filename: filename, content: newText });
if (result.error) { showStatus(result.error, true); } else { showStatus('Edit submitted for review.'); }
}
// --- ADD NEW: TEXT ---
async function kmSubmitText() {
const content = document.getElementById('km-new-text').value.trim();
const filename = document.getElementById('km-new-filename').value.trim();
if (!content) { showStatus('Write something first.', true); return; }
const result = await postAction('submit_text', { content, suggested_filename: filename });
if (result.error) { showStatus(result.error, true); }
else {
showStatus('Submitted for review.');
document.getElementById('km-new-text').value = '';
document.getElementById('km-new-filename').value = '';
}
}
// --- ADD NEW: FILE UPLOADS (txt / pdf / image) ---
async function kmUploadFile(type) {
const input = document.getElementById('km-file-' + type);
if (!input.files.length) { showStatus('Choose a file first.', true); return; }
const fd = new FormData();
fd.append('file', input.files[0]);
showStatus('Processing...');
const result = await postAction('upload_' + type, fd, true);
if (result.error) { showStatus(result.error, true); }
else { showStatus('Submitted for review.'); input.value = ''; }
}
// --- REVIEW QUEUE ---
async function loadReview() {
if (!kmCanReview) return;
const data = await postAction('list_pending', {});
const container = document.getElementById('km-review-list');
if (data.error) { container.innerHTML = `<p class="km-empty">${data.error}</p>`; return; }
if (!data.data.length) { container.innerHTML = '<p class="km-empty">Nothing pending review.</p>'; return; }
container.innerHTML = data.data.map(row => {
const typeTag = row.action_type === 'edit'
? `<span class="km-tag km-tag-edit">EDIT of ${row.target_filename}</span>`
: `<span class="km-tag km-tag-new">NEW</span>`;
let aiMetaHtml = '';
if (row.ai_meta) {
const meta = JSON.parse(row.ai_meta);
aiMetaHtml = `<div class="km-meta">[AI] Tokens: ${meta.tokens} | Est. Cost: ${meta.cost}</div>`;
}
return `
<div class="km-card">
${typeTag} <span class="km-tag km-tag-source">${row.source_type}</span>
<div class="km-meta">Submitted by ${row.submitted_by_name} on ${row.submitted_at}</div>
${aiMetaHtml}
<div class="km-preview">${row.preview}...</div>
<button class="km-btn-small km-btn" onclick="approveItem(${row.id})">Approve</button>
<button class="km-btn-small km-btn km-btn-decline" onclick="declineItem(${row.id})">Decline</button>
</div>
`;
}).join('');
}
async function approveItem(id) {
const result = await postAction('approve', { id });
if (result.error) { showStatus(result.error, true); }
else {
showStatus(result.reload_ok ? 'Approved and live.' : 'Approved, but the reload call failed — check the service is running.');
loadReview();
}
}
async function declineItem(id) {
const note = prompt('Optional note on why this was declined:') || '';
const result = await postAction('decline', { id, note });
if (result.error) { showStatus(result.error, true); } else { showStatus('Declined.'); loadReview(); }
}
// --- DECLINED (read-only audit trail) ---
async function loadDeclined() {
if (!kmCanReview) return;
const data = await postAction('list_declined', {});
const container = document.getElementById('km-declined-list');
if (data.error) { container.innerHTML = `<p class="km-empty">${data.error}</p>`; return; }
if (!data.data.length) { container.innerHTML = '<p class="km-empty">Nothing declined yet.</p>'; return; }
container.innerHTML = data.data.map(row => {
const typeTag = row.action_type === 'edit'
? `<span class="km-tag km-tag-edit">EDIT of ${row.target_filename}</span>`
: `<span class="km-tag km-tag-new">NEW</span>`;
const noteHtml = row.decline_note
? `<div class="km-meta"><strong>Decline note:</strong> ${row.decline_note}</div>`
: '<div class="km-meta"><em>No note given.</em></div>';
return `
<div class="km-card">
${typeTag} <span class="km-tag km-tag-source">${row.source_type}</span>
<div class="km-meta">Submitted by ${row.submitted_by_name} on ${row.submitted_at}</div>
<div class="km-meta">Declined by ${row.reviewed_by_name} on ${row.reviewed_at}</div>
${noteHtml}
<div class="km-preview">${row.preview}...</div>
</div>
`;
}).join('');
}
// --- INITIAL LOAD ---
document.addEventListener('DOMContentLoaded', loadExisting);
</script>
Keeping this current matters
If this file is edited directly on the server again in future, the copy embedded here goes stale. Worth re-pasting the real file back into this guide whenever a meaningful change is made — the same discipline this whole guide depends on.
14
Resilience — survives a crash or a server restart
Done
In plain English: the service now looks after itself. If it ever crashes, or the server restarts for any routine reason, it comes back on its own — nobody has to notice and manually re-run a command.
Stop the manual process first
Confirm nothing is already using port 8420 before creating the proper service, or the two will collide:
If that shows a process, note its exact PID and stop it:
ss -tlnp | grep 8420
kill <PID>. Confirm the port is genuinely clear (the command above returns nothing) before continuing.
Creates the real systemd service
cat > /etc/systemd/system/katie-rag.service << 'EOF' [Unit] Description=Katie RAG Knowledge Service After=network.target mariadb.service [Service] Type=simple User=root WorkingDirectory=/var/katie-rag ExecStart=/var/katie-rag/venv/bin/uvicorn main:app --host 127.0.0.1 --port 8420 Restart=always RestartSec=5 [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable katie-rag systemctl start katie-rag systemctl status katie-rag --no-pager
Restart=always is the whole point
This single line is what makes it resilient: if the process ever exits for any reason, systemd starts a fresh one automatically, waiting RestartSec=5 between attempts. enable means it also starts itself the moment the server boots, with no manual command needed at all.
Avoiding the terminal's pager entirely
Plain
systemctl status or journalctl can drop into a full-screen viewer that's easy to get stuck inside, especially over a remote terminal. Adding --no-pager to either command prints straight to the screen and returns immediately to a normal prompt — safer by default, every time.
The real test — not "did it start," but "does it survive"
Starting once proves nothing on its own. The test that actually matters: deliberately kill the running process, wait, and confirm a new one took its place on its own.
OLDPID=$(systemctl show -p MainPID katie-rag --value) kill $OLDPID sleep 6 systemctl status katie-rag --no-pager curl "http://127.0.0.1:8420/ask?q=what%20is%20the%20secret%20test%20phrase"
Main PID number, active (running), and a real grounded answer straight afterward — the actual planted test phrase, correct. The whole chain survived a deliberate crash with nobody touching anything by hand.
If
/ask fails immediately after a restart, that's not necessarily a real failure — building the index involves loading several Python libraries, connecting to the database, and calling OpenAI to generate embeddings, which genuinely takes a few real seconds. Wait a little and try again before assuming something's actually broken.
The real obstacle hit getting here: a silent port conflict
The very first attempt at this step failed in a confusing way —
That command's output includes the real PID directly (
systemctl status showed activating (auto-restart) with the restart counter climbing into the dozens. The actual cause, found in journalctl -u katie-rag -n 30 --no-pager: [Errno 98] address already in use. An old manual process from earlier in the day was still genuinely running and still holding port 8420 — every attempt to stop it had silently failed due to mistyping its PID (one digit short, a five-digit number instead of the real six-digit one), so kill kept reporting "no such process" against a PID that had simply never existed, while the real process kept running undisturbed. The fix: ask the kernel directly which process actually holds the port, rather than trust a remembered PID number:
ss -tlnp | grep 8420
pid=...), removing any chance of acting on a wrong or mistyped number.
6 — Database
The vector store from day one, on BSM's own infrastructure, in its own dedicated database — not a later swap
15
MariaDB 11.8 as the vector store, from day one
Done
In plain English: the database itself has been upgraded to the version that can store and search vectors directly, ahead of need, so it's ready before the rest of the build needs it.
Server confirmed on Plesk Obsidian 18.0.78.4, Ubuntu 24.04, running MariaDB 11.8.8, verified directly in Plesk's Database Servers list. A genuine daily automatic full-server backup ran throughout the upgrade. The official LlamaIndex–MariaDB connector is confirmed to exist, and is confirmed — from its own source code — to build its vector index with cosine distance automatically. Nothing outstanding on this point.
16
Create the dedicated
Done
katie_rag databaseIn plain English: a new, separate database, on the same MariaDB server, just for the vectors — with its own login that only the Python service ever uses. WordPress never gets a password to it (see Glossary: "Why the vectors get their own dedicated database").
Created via Plesk's Database Servers tool: database name katie_rag, on localhost:3306 (MariaDB 11.8.8), related to the bodysleepmind.com subscription. A dedicated user, chris3, was created alongside it — specifically without the "access to all databases within the subscription" option, so this login cannot reach WordPress's own database even by mistake. Access control set to local connections only, since nothing outside this server ever needs to reach it — main.py always connects as 127.0.0.1. This is the database referenced in main.py's connection settings (Step 6). The database name, katie_rag, is worth keeping consistent — but the username chris3 is just whatever was typed in on the day; pick anything on a future server, and match it in main.py's code.
7 — Credentials
How to check and change the database password and the OpenAI key, going forward — not a one-time build step, a reference for whenever it's needed
Database credentials —
chris3 on katie_ragIn plain English: the database password isn't something Plesk will show you again once it's set — the one real record of what's currently in use is the secrets file itself.
To check it: the database name and username aren't secret — both are visible directly in main.py's own code (Step 6), and in Plesk's Database Servers list. The password is the only real secret, and Plesk doesn't let you view a password again after it's set — so the actual current value in use is whatever's sitting in the secrets file:
cat /var/katie-rag/.env
To change it: two places need to agree, the database itself and the file main.py reads from.
- In Plesk: Databases → the
chris3user → set a new password (the same Generate-and-copy pattern as creating it the first time, Step 16) - Update the secrets file to match, editing the one line in place rather than recreating the whole file:
Find the
nano /var/katie-rag/.env
DB_PASSWORD=line, replace the value after the=, then save (Ctrl+O, Enter) and exit (Ctrl+X). - Restart the service so it actually picks up the new value — it only reads
.envonce, at startup (Step 7's command, run again)
Confirms it worked
/ask (Step 8) still answers correctly after the restart — if the password and the file disagree, the service fails to connect to the database at startup.
Editing Python Code Using FTP
In plain English: how to safely update Python files forever into the future without using a scary terminal editor, and how to safely lock in a backup before you do.
Editing Python files directly in an SSH terminal (using tools like nano) introduces risks of silent copy-paste corruption. For any future Python edits, using an admin-level SFTP connection (or the visual Plesk File Manager) is the safest and most reliable method.
- Connect: Log into your server using an IONOS Root FTP client (like FileZilla) with your root/admin credentials. This bypasses the normal "website jail" and lets you navigate directly to the
/var/katie-rag/folder. - Create an Instant Backup: Right-click the file you are about to change (e.g.
main.py) and rename it (e.g.main_old.py). If your new code breaks, you just delete the broken file and rename this backup back to its original name to restore the working version instantly. - Edit and Upload: Write or paste your new Python code in a normal text editor (like Notepad) on your own computer, save it, and drag-and-drop it into the FTP window.
- Restart Python for Changes: Python loads code into memory once at startup. Simply uploading a file does nothing on its own. You must force the server to dump the old memory and load your newly uploaded file. (Auto-restart on file changes is deliberately disabled so a half-uploaded FTP file doesn't crash the live agent).
The easiest SSH terminal to use is located at: Plesk > Tools and Settings > Tools & Resources > SSH Terminal.
The exact restart code to type into SSH is:
systemctl restart katie-rag
The OpenAI key — currently "masterKey"
In plain English: the same key is meant to live in exactly two places, kept in sync — WordPress's own constant, and this service's secrets file.
To check it: the live key value itself can't be retrieved from OpenAI's own dashboard once created — only the last few characters are ever shown there again. To see what's actually configured on this server:
- WordPress side — check the
OPENAI_API_KEYconstant inwp-config.php(every PHP file that calls OpenAI reads from this one constant, not its own copy) - Python side —
cat /var/katie-rag/.env, same command as the database password above - Compare the last few characters of each against platform.openai.com/api-keys to confirm which named key is actually in use
To change it: the same "create new, switch over, confirm, then revoke the old one" sequence used when masterKey itself replaced the previous exposed key.
- Generate a new secret key at platform.openai.com/api-keys
- Update
wp-config.php'sOPENAI_API_KEYconstant to the new value — every PHP file referencing it updates automatically, nothing else to touch on the WordPress side - Update the Python side the same way as the database password:
nano /var/katie-rag/.env, replace theOPENAI_API_KEY=line, save and exit - Restart the service so
main.pypicks up the new value - Test everything that depends on it — Katie's chat, the food analysers,
/ask— before revoking the old key in the OpenAI dashboard. Never revoke first and check second.
Never paste the actual key value into a chat with an AI assistant, including this one — type or paste it directly into the terminal, Plesk, or
wp-config.php only.
Anyone with real knowledge of a subject — a wellness specialist, a partner brand, a member of the BSM team — can extend what Katie knows. Writing that knowledge down well is the actual skill involved; nothing about getting it in front of Katie requires writing or running code.
1
Write it. Clear, accurate content, in BSM's voice — the same skill as writing any other piece of customer-facing material.
2
Submit it. Entered directly through a simple submission tool — see the toolkit below.
3
Review it. A second person checks each submission before it goes anywhere near a real conversation, and approves or declines it.
4
It goes live. The moment a submission is approved, Katie has it. No separate technical step, no waiting on anyone else.
5
It gets proven, not assumed. Every addition is checked with a real question before it's trusted in front of an actual user.
For example: BSM wants Katie to properly explain its apnea analytics. A specialist writes up what the score measures and what it means for a user. It's submitted, reviewed, approved — and tested with a direct question before it's trusted. Katie can now discuss apnea analytics accurately, in a conversation a user already trusts.
The same path extends outward, by design: a partner brand's own product knowledge follows this exact process to reach Katie too — their own people, writing what they know, reviewed before it goes live. Nothing about that path is different from BSM's own content going through it.
Content Manager's Hints — Writing Knowledge Katie Can Actually Use
In plain English: a few practical writing habits that genuinely change how well Katie can use what's submitted. None of this requires understanding the technical machinery underneath — but each hint links back to the exact Glossary term explaining why, for anyone curious.
1. Write each paragraph so it stands on its own
Documents get cut into smaller pieces before Katie ever sees them (see Glossary: Chunking). A paragraph that leans on "as mentioned above" or "see the previous section" can lose that connection the moment it's split apart from what it referred to. Write each paragraph as if it might be read completely alone — because mechanically, it often will be.
2. Don't try to guess the user's exact wording
Katie matches by meaning, not by matching words (see Glossary: Embedding, Cosine similarity) — someone can ask "what is an OSA score" and correctly match content that never uses those exact words at all. Write naturally, the way you'd actually explain it to a real person. No need to stuff in alternate phrasings or try to anticipate exact questions.
3. One clear idea per paragraph
If a single paragraph blends several unrelated topics together, the whole thing can get pulled in as one chunk even when only part of it is actually relevant — which can make an answer feel a little "off," even when nothing in it is technically wrong.
4. Say the important thing in words, not just formatting
Bold text, colour, tables, and layout all get stripped away — only the plain words survive. If something is important enough to highlight visually, it's important enough to also say directly in a sentence.
5. Be aware when content overlaps something already live
If two separate documents say slightly different things about the same topic, both can get pulled in together, and Katie's answer can blend them in a way that reads as contradictory. Worth a quick look at Existing Knowledge before submitting something that covers ground already there.
6. The real test, every time: ask it back in your own words
After something is approved, the most reliable check isn't re-reading what was written — it's asking Katie a real question, phrased the way an actual visitor would ask it, not copying the document's own wording. If she answers it correctly, it was good content.
Worth a wry, useful note: this very guide is a textbook example of content that would chunk badly if it were ever submitted as Katie's own knowledge — long cross-references, headings depended on for structure, numbered steps that only make sense read together in order. Good for a person reading start to finish; exactly the wrong shape for a vector search. A real, memorable reminder of exactly why these hints exist.
The Knowledge Admin Toolkit
What a knowledge admin actually has in front of them. Two tools cover Phase Two on day one; the list grows from there without touching the engine underneath, because the format knowledge arrives in and the engine that searches it are kept deliberately separate (Architecture Rule 2).
Included
Text Submission
Write or paste knowledge directly into a simple form. The baseline tool — covers every example in this guide.
Included
Review Queue
Every submission, approved or declined by a second person, before it ever reaches Katie. Declined items stay visible with the reason, in their own tab.
Included
Document Upload (PDF, Word, Excel)
Upload a PDF, Word document, or Excel file directly; text is extracted automatically — no AI step, no extra cost — ready for the same review step.
Considered, not built
Image Analyser
The same vision-analysis capability already reading food labels elsewhere on the site, considered for product labels or printed leaflets here. Deliberately not built: it earns its value from real-time field capture with no file available at all, which knowledge curation almost never is.
Every tool added to this toolkit, present or future, feeds the same review queue and the same knowledge base — new ways in, not a parallel system. Adding a new intake format later is additive, not a rebuild.
Appendix — Build Tracker
A live working record, kept permanently rather than deleted once complete — the same structure used across BSM's build documentation.
📋 Status at a glance — Phase One complete
- Build Plan Group 1, environment (Steps 1–2) — done
- Build Plan Group 2, knowledge folder and permissions (Step 3) — done; initial content (Step 4) arrives via the knowledge manager, ongoing
- Build Plan Group 3, secrets file and
main.py(Steps 5–6) — done - Build Plan Group 3, actually running and testing the service (Steps 7–8) — done, proven end to end with a real grounded answer; real friction along the way documented in full (anchor: Lessons from the real first run)
- Build Plan Group 4, WordPress bridge and Katie's conversation (Steps 9–10) — done, live, proven through the real chat widget (Sections E–F)
- Build Plan Group 5, the knowledge manager (Steps 11–13) — done, built and live as
knowledge-manager.php - Build Plan Group 5, resilience (Step 14) — done, a real systemd service, tested by deliberately killing the running process and confirming it came back on its own (Section G)
- Build Plan Group 6, the MariaDB 11.8 upgrade (Step 15) — done, confirmed 11.8.8 (Section H)
- Build Plan Group 6, the dedicated
katie_ragdatabase (Step 16) — done (Section H) - Genuinely open by choice, not by gap: final confirmation of access-level defaults (Section I), and whether Katie's answers should ever blend with GOD JSON (Section F) — both deliberately deferred, not forgotten
🏗 Section A — Server and environment
- Confirmed: Python 3.12.3 already present on the server
- Confirmed: root terminal access built directly into the existing Plesk panel
- Standard practice on this server's OS: install into a virtual environment, fully isolated from anything else the server runs
📚 Section B — Initial knowledge set
- Two source documents identified: the Product Knowledge Reference and the BSM User Management Guide
- Next action: convert both into plain text for the knowledge folder
- Design note: the feature/tier comparison table reads better in this format as prose — a table's structure doesn't carry over to plain text
🐍 Section C — The service
- Done:
main.pywritten in full (Build Plan Group 3, Step 6), storing vectors directly in MariaDB 11.8 via the official connector, with both/askand/reloadbuilt in from the start, secrets loaded from.envrather than hardcoded - Done: the OpenAI key (masterKey, via the same
.envfile) and the dedicatedkatie_ragdatabase/user (chris3) both exist and are referenced correctly in the code - Done: the service started successfully and answered a real question correctly — the full chain proven end to end for the first time (Steps 7–8)
- The first real run surfaced genuine friction, none of it architectural — a skipped file-creation step, a corrupted large paste, a stray semicolon in the database password, curl rejecting spaces in a URL. Every one diagnosed and fixed; full detail kept in "Lessons from the real first run," right after Step 8, specifically so it doesn't need re-discovering on a future server rebuild
- Rebuilds its full index on every restart and on every
/reloadcall — efficient at today's scale; revisit once the knowledge base is large enough for that to matter - Package version pinned to
llama-index-vector-stores-mariadb>=0.3.0— earlier versions only support the old 11.6 preview syntax
✅ Section D — Verification
- Pass criteria: a sensible answer, genuinely drawn from the saved documents
- A wrong or vague answer points to the source document, not the code
- Done: confirmed for real, using the planted test phrase in
knowledge.txt— asked "what is the secret test phrase," got back the exact real answer, "BLUE ELEPHANTS DANCE AT MIDNIGHT," not a general-knowledge guess (Step 8)
✅ Section E — WordPress bridge
- Done: built directly into
triage_backend.php's?ai=1handler — awp_remote_get()call to127.0.0.1:8420, internal only (Architecture Rule 4) - Decided: access scope is unchanged from before — whoever could already reach the
?ai=1handler still can; no new restriction added or needed - Decided: fallback behaviour is a clear, honest message, never a silent reversion to the old approach (Section F)
- Real bug found and fixed: a strict
===comparison on the HTTP response code silently failed because WordPress can return it as a string, not an integer; fixed by casting to(int)before comparing - Real mistake found and fixed: a second edit of this file was accidentally built from the original uploaded copy instead of the already-corrected one, briefly reintroducing the old hardcoded key; caught before upload, not after
✅ Section F — Katie's conversation
- Done: Katie's chat now genuinely calls
/askfirst, replacing the oldknowledge.txtcontext-stuffing approach entirely in that one handler - Decided: no blending with GOD JSON (Architecture Rule 3) for now — the retrieved answer is shown as-is. Not a default or a guess; a deliberate choice to revisit case by case if it's ever actually wanted
- Decided: no silent fallback to the old context-stuffing approach if the knowledge service is unreachable — that would hide a real outage and cost money every time it happened. Katie returns a clear, honest message instead, with no further OpenAI call at all
- Verification idea, reused successfully: the planted test phrase in
knowledge.txt("BLUE ELEPHANTS DANCE AT MIDNIGHT"), asked through the real live chat widget, not just over curl — confirmed working end to end
✅ Section G — Resilience
- Done: registered as a real systemd service (
/etc/systemd/system/katie-rag.service),enabledso it starts on boot,Restart=alwaysso it restarts itself if it ever crashes - Done: actually tested, not just configured — the running process was deliberately killed, and confirmed a brand new process took its place on its own within seconds, with
/askanswering correctly straight afterward - Real obstacle found and fixed: a port conflict from an old manual process that was never actually stopped, caused by repeatedly mistyping its PID by one digit; every
killattempt silently targeted a PID that never existed, while the real process kept running undisturbed. Fixed by asking the kernel directly (ss -tlnp | grep 8420) for the real PID rather than relying on a remembered number
✅ Section H — MariaDB 11.8
Run ahead of the rest of this build, ahead of need, so the database was ready well before Katie depends on it (Architecture Rule 6).
- Done: server confirmed on MariaDB 11.8.8, verified directly in Plesk's Database Servers list, data checked and sound post-upgrade
- Done: a genuine daily automatic full-server backup remained active throughout, independent of Plesk's own backup-before-upgrade step
- Done: the official LlamaIndex–MariaDB connector confirmed to exist (
llama-index-vector-stores-mariadb, version 0.3.0+), maintained by the LlamaIndex project, recognised by MariaDB's own foundation - Done: confirmed directly from the connector's own source code that it builds its vector index with
DISTANCE=cosineautomatically, every time it sets up the table — not MariaDB's own Euclidean default. Nothing further to configure or check on this point. - Done: the dedicated
katie_ragdatabase created (Step 16) — its own database, its own narrowly-scoped login (chris3), local connections only, used only by the Python service. Not WordPress's database, and WordPress gets no login to it at all (see Glossary: "Why the vectors get their own dedicated database").
🖥 Section I — Submission and review access levels
The submission page, review queue, and reload address are built in Build Plan Group 5, Steps 11–13. A default access split already exists in the code, deliberately flagged there as a placeholder rather than a final decision:
- Current default: submitting requires
edit_posts(any Contributor+) — open, not yet confirmed as the right scope (Step 11) - Current default: reviewing and approving requires
manage_options(Administrators only) — intentionally a smaller, more trusted group than who can submit (Step 12) - Next decision: confirm both defaults are actually what BSM wants, or adjust the
current_user_can()checks inknowledge-manager.phpaccordingly
🐍 Section J — Python Learning (Line-by-line for PHP/JS Devs)
Python uses spaces (indentation) instead of curly brackets {}. When the spaces stop, Python knows the function is over. Here is the main.py script broken down with direct PHP/JS analogies.
BLOCK 1: The "Imports"
In PHP, you use
In PHP, you use
require_once or Composer's use. In Python, this is import.
import os, glob: Loads Python's built-in OS and file-searching tools (identical to PHP'sglob()).from [package] import [tool]: Loads only a specific tool from a package to save memory.FastAPI: The tool that creates the web server and URLs (like Laravel's router).
BLOCK 2: Loading Secrets
load_dotenv(...): Opens the hidden.envfile and loads it into memory.os.environ["OPENAI_API_KEY"] = ...: Python's version of$_ENV. Locks the key into the system environment so AI tools find it automatically.DB_PASSWORD = ...: Saves the password to a variable (Like PHP's$DB_PASSWORD = getenv("DB_PASSWORD");).
BLOCK 3: Strict Rules (System Prompt)
Settings.llm = OpenAI(...): Instantiating the OpenAI class globally (In PHP:$settings->llm = new OpenAI(...);).temperature=0.0: Kills creativity to stop random guessing.
BLOCK 4: Database Connection
vector_store = MariaDBVectorStore...: Creates a database connection object (Like PHP's$db = new PDO(...)).embed_dim=1536: Tells MariaDB the exact array length of an OpenAI vector.
BLOCK 5: The Index Builder Function
query_engine = None: Like$query_engine = null;. Declared outside the function to share it globally.def build_index():: Python's way of writingfunction build_index() {.global query_engine: Exactly like PHP'sglobal $query_engine;.if not files: return: In PHP,if (empty($files)) { return; }.query_engine = index.as_query_engine(node_postprocessors=[SimilarityPostprocessor(similarity_cutoff=0.70)]): The 0.70 mathematical guardrail. Filters out chunks that score lower than 70%.
BLOCK 6: The Web Server Routes (Endpoints)
app = FastAPI(): Starts the web server router.@app.get("/ask"): A decorator. If someone visits/askvia GET, run the function below.def ask(q: str):: Automatically grabs?q=from the URL as a string (Like PHP's$q = $_GET['q'];).return {"answer": ...}: FastAPI automatically converts Python dictionaries into JSON strings (Like PHP'secho json_encode(...); exit;).valid_nodes = [n for n in nodes if n.score >= 0.70]: A "List Comprehension". Python's ultra-clean 1-line version of PHP'sarray_filter().
BLOCK 7: The Reload Endpoint
len(...): Counts the files. Identical to PHP'scount().