Connecting AI to the state of the system (without leaking secrets)
The state of your system lives in different formats, repository, observability, tickets, documentation, and each one needs a different connection path. Getting the path wrong gives a wrong answer; forgetting the security check leaks secrets.
You ask AI why the checkout service has been slow since yesterday. It answers with a technical explanation full of architecture jargon, and made up. AI never opened your observability dashboard or your repository, so it guesses a plausible cause.
Legal asks for the list of open-source library licenses used in the product, cross-referencing the dependency file (structured data) with the license text of each one (running document). AI returns a confident list, but treated the dependency file as text to search for meaning, losing the exact version of each library, and summarized the license text generically without noticing the specific clause for that version. An old library, stuck on a version with a license different from the current one, was classified wrong because AI didn't read the dependency file as the structured data it is. Wrong format, invisible legal risk: the dependency file needs structural reading with exact versions; the license text needs full RAG, not a generic summary.
You ask AI to say whether the engineering team is overloaded, cross-referencing the time-tracking system (structured data) with managers' one-on-one notes (running text in a document). It returns a confident conclusion of "normal load", but treated the time-tracking system as text to search for meaning and ignored the overtime column, and treated the one-on-one notes as a spreadsheet, missing the tone of exhaustion showing up in the sentences. The engineer who most signaled burnout in conversations disappeared from the report because neither source was read the right way. Wrong format, wrong diagnosis: structured data needs schema reasoning; conversation notes need meaning-based search.
You ask AI to point out which bug is most blocking new user activation, cross-referencing the bug board (semi-structured) with the analytics funnel (structured data). It returns a candidate bug, but treated the board as loose text to search for meaning, ignoring the severity field, and treated the analytics funnel as if it needed semantic search instead of direct calculation over conversion rate per step. The bug that really tanked activation was marked high severity on the board, structured data AI never read as structured. Wrong format, wrong prioritization: the bug board needs a structured field combined with text; the funnel needs calculation over structured data.
You ask AI to confirm, for the salesperson, whether the public API supports a specific use case a prospect asked about, cross-referencing the API documentation (running text) with the code repository (structure and dependency). It returns a confident "yes, it supports it", but only read the documentation, which is two months out of date, without checking whether the endpoint still exists in the real repository. The endpoint had been removed in the latest release, a fact only the code, read with structural reasoning, would reveal. Wrong format, wrong promise to the client: documentation needs RAG, but the code is the source of truth for what exists now, and needs structural reading.
You ask AI why the automatic billing process failed for some customers overnight, cross-referencing the error log (structured time series) with the operations runbook (running text). It returns a probable cause, but treated the log as text to search for meaning, losing the exact pattern of failure times, and treated the runbook as if it were a list of events. The real pattern, all failures at exactly 3 AM, matching a database maintenance window, only shows up to whoever reads the log as a structured time series. Wrong format, lost root cause: the structured log needs time series reading; the runbook needs RAG.
You ask AI to confirm that no production log stores CPF in plain text, cross-referencing the logging source code (structure) with logs already stored in the observability system (a large volume of historical data). It returns "compliant", but only read the current code with structural reasoning, without consulting the log history already recorded two years ago, which is a source of completely different format and volume. The real liability, CPF already stored in old logs before the code fix, stayed invisible because the check looked at the wrong source for this type of risk. Wrong format, incomplete audit: code needs structural reasoning; log history needs direct querying of the observability system, not inference over the current code.
You ask AI to find where users abandon the signup flow most, cross-referencing the analytics event (structured data) with the session recording (visual content, not text). It returns a candidate screen, but treated the analytics event as text to search for meaning instead of calculating the abandonment rate per step, and had no way to "read" the session recording, which requires a visual reading path, not text search. The real abandonment step, visible only by watching the recording, stayed out of the diagnosis because the right source was never read the way its format demands. Wrong format, incomplete finding: the analytics event needs calculation over structured data; the session recording needs reading the actual content, not inference on top of the event.
You ask AI whether the platform can handle the projected user growth for next year, cross-referencing capacity documentation (running text) with the real usage dashboard (structured time series). It returns a confident "holds up with room to spare", but treated the usage dashboard as text to search for meaning and lost the real month-over-month growth trend, and treated the capacity document as if it were a table of numbers. The real bottleneck, visible only in the dashboard's growth curve, stayed invisible because neither source was read the right way. Wrong format, wrong investment decision: the structured dashboard needs time series reading; the capacity document needs RAG.
Whoa, notice something: the problem is almost never that AI reasons badly. The problem is that it's answering off the top of its head, having never opened the real state of YOUR system. It's like hiring the best senior engineer in the market, sitting them at your desk, and never giving them access to your repository, your observability dashboard, your ticket board. They'll speak with confidence and get it badly wrong, because they're guessing. Connecting AI to the state of the system is what pulls it out of generic. Except connecting it the wrong way is worse: either it reads the wrong format and gets the diagnosis wrong, or it leaks secrets and credentials that should never have gone out.
The core idea of this lesson. AI only really helps when it can see the real state of your system, and the FORMAT of each source decides the path. A code repository needs reasoning over structure and dependency, not search for similar snippets. Observability and logs need reading structured time series, not meaning-based search over text. A semi-structured ticket combines structured fields with meaning-based search. Architecture documentation, in running text, is RAG territory. And before anything goes out to an external model, there's a security check that isn't optional: secrets, credentials, and customer data in logs never go out without masking first.
01The format of the source decides the path
The first shift is to stop treating "the state of the system" as one single thing. There's no single path to connect AI to what's happening in your environment. What exists is the format of each source, and the format dictates the road.
Think of four different natures that coexist in any technology team. There's the code repository, which isn't loose text: it's structure, folders, modules, a dependency graph between functions and services. There's observability, metrics and logs, which is structured time series: numbers over time, with timestamps, with counts. There's the ticket, which is semi-structured: fixed fields (severity, status, owner) mixed with free-text descriptions. And there's architecture documentation, the runbook, real running text, where the idea lives in sentences and paragraphs.
Each one needs a different tool, and the most common mistake is trying to squeeze all four into the same path, usually traditional RAG, thinking text is text. The @datasciencebrain study hits exactly this note: the data format decides the memory architecture, not the other way around.
02Repository and observability: reasoning over structure, not naive vector search
Here lives the most common mistake of anyone who just learned RAG and tries to apply it everywhere: thinking that throwing the entire repository into a vector index solves it. It doesn't, and in the repository the cost of the mistake is specific: code isn't a collection of similar paragraphs, it's a graph. Function A calls function B, which is imported by service C, which another team depends on without knowing it. If you ask "what breaks if I change this function" and AI only searches "text snippets similar to this function", it finds the obvious places and misses the seven calls hidden in another repository, another service, another import context.
The right path is to give AI the structure: the folder tree, the dependency graph between modules and services, the relevant commit history, and let it reason on top of that, not just search by textual similarity. It's the difference between asking "which paragraphs look similar to this one" and asking "who really depends on this".
Observability has the same problem with a different face. Metrics and logs are time series: numbers, with timestamps, with counts per minute. Asking "is this error log related to the deploy" isn't a question of textual meaning, it's a question of correlation in time: exactly when the error started, exactly when the deploy happened, and the distance between the two is the data that matters. Searching for "similar text" between logs before and after the deploy is the wrong path; looking at the error count per minute, cross-referenced with the exact timestamp, is the right path.
03Documentation and ticket: RAG and meaning-based search
Now the two formats where meaning-based search, real RAG, makes sense. Architecture documentation, the runbook, the technical decision text: that's running text, exactly the territory where searching by meaning works well. You ask "how did the team decide to handle payment timeout" and semantic search finds the right passage, even if the document uses a different word to describe the same thing.
Learn more: why a ticket is neither purely structured nor purely text
A ticket looks like a spreadsheet row at first glance, it has severity, status, owner, date. But the part that matters most for real prioritization tends to be in the free-text description, where the person who opened the ticket explains what's happening in their own words. Treating the ticket only as structured data ignores that account; treating it only as text for semantic search ignores the severity field that comes ready-made and is more reliable than any inference over the text. The right path is always both together: use the structured field to filter and sort (high severity, open for a long time, reopened more than once), and use meaning-based search inside the description to group duplicates and understand the real account behind the number. Ignoring either side is the mistake that makes the right ticket disappear in the middle of the board.
The ticket, then, needs both: the structured field (severity, status, time open) to filter and sort, and meaning-based search in the description to group and understand the account. Treating the ticket only as text loses the reliable signal of the field; treating it only as a spreadsheet loses the account that only exists in natural language.
04Security: secrets and customer data in logs never go out without a check
Everything so far was about getting AI to nail the diagnosis. Now the point that, in technology, can cost a lot more than a wrong diagnosis: what leaves the company when you connect AI to your repository and your production logs.
The moment you give access to these sources, there's a question that needs to come before any other: what's a secret, credential, or customer data in there, and can it go to an external model without masking? Code repositories tend to have API keys, tokens, database connection strings, sometimes all hardcoded in an old file nobody remembers exists. Production error logs tend to carry customer data by accident: CPF, email, card number, all of that can show up in a stack trace nobody thought to mask before logging.
You already saw, in the lesson about connecting AI to the numbers of the financial module, the same gateway that blocks what can't pass and logs what did. Here the principle is identical, just applied to your specific source: before the agent reads the entire repository or queries production logs, the secret needs to be out of the way, whether because the repository already follows good practices of not versioning credentials, or because the log passes through a masking layer before reaching an external model. It's not bureaucracy. It's the difference between "I connected AI to my system" and "I leaked a production credential without noticing".
05Putting it together: from the opaque system to the system AI actually sees
Now you can see the whole picture. Connecting AI to the state of the system is what stops it from answering off the top of its head, it's what makes it stop guessing root cause and start reasoning on top of your repository, your dashboard, your real ticket. But connecting is a double-edged sword: connecting in the wrong format gives a wrong answer that looks right, and connecting without a security check leaks what should never have gone out.
The sequence that protects the team is always the same. First, look at the FORMAT of the source and choose the path: a repository becomes reasoning over structure and dependency, observability becomes time series reading, a ticket becomes a structured field combined with meaning-based search, documentation becomes RAG. Second, before plugging anything into an external model, go through the security check: is there a secret, credential, or customer data in there, does it need masking, does it run locally or in the cloud, is the gateway in the middle. Format first, security always.
Do it now
Pick a real question about your system that AI answered off the top of its head and got wrong, or one you haven't even tried using AI for yet: your real task.
- List 3 sources that would contain the right answer (repository, observability dashboard, ticket board, architecture documentation, production log).
- For each source, classify the FORMAT: structure and dependency (code), time series (metrics and logs), semi-structured (ticket), or running text (documentation).
- Mark the path for each: code leads to reasoning over structure, observability leads to time series reading, ticket leads to structured field plus meaning-based search, documentation leads to RAG.
- Do the security check for each source in one line: is there a secret, credential, or customer data there? Does it need masking? Does it run in the cloud or require a closed environment?
- Point out which of the three sources is the most sensitive and write, in one sentence, what the gateway should block in it.
You've just designed your system's connection to AI through the right path, with the security gate in place. You're ahead of anyone who just pastes the whole log into a chat and hopes for the best.
Practice
1. You want to know what breaks if you change a function in the repository. What's the most appropriate path?
2. Why shouldn't asking 'does yesterday's error log relate to the 9 PM deploy' be treated as meaning-based text search?
3. Before connecting AI to your repository and production logs, what's the correct posture?
For the board
On the diagnosisit speaks with confidence and gets it badly wrong because it never opened your repository, your dashboard, your board.
On formateach source has its own nature. Logs, tickets and code do not connect the same way.
On secretsconnecting is not opening everything. A credential never enters the context, for no convenience at all.
Thanks for the feedback. It helps sharpen the next lesson.