Connecting AI to your product's context (without leaking anything)
Product context lives in three different places, the backlog, analytics, and research, and each one calls for a different technical path. Connecting them wrong scrambles all three; connecting without a security check leaks real user data.
You ask AI why your new feature's activation is low, expecting it to cross-reference the reported-bugs board with the analytics funnel and with what users said in the latest research. It hands back a confident answer, but it scrambled everything: it treated the board as if it needed semantic search, the funnel as if it needed meaning-based search, and the research transcript as a spreadsheet. The real reason for the low activation was in a specific sentence from an interview, which it never actually read as text.
You ask AI what your contribution margin was last quarter. It answers with an impeccable analysis, full of pretty finance jargon. And made up. AI never opened your ERP or your closing figures, so it guesses a plausible number.
You ask AI to review a service contract's penalty clause. It comes back with "the two percent penalty is within the legal limit", confident, polished, ready for the client. Your case's actual limit was different, AI guessed the number and nobody checked the source.
You ask AI which campaign had the best ROI, cross-referencing the paid media dashboard, which is structured, with the brand survey responses, which is free-flowing text. It hands back a pretty number, but it treated the dashboard as text to search for meaning and the survey as a spreadsheet, missing the real pattern that only showed up in the open-ended answers.
You ask AI to cross-reference the turnover rate, which lives structured in the payroll system, with the termination policy, which is free-flowing text in a PDF. It hands back an answer full of HR jargon, but it treated the policy as if it were a spreadsheet and the payroll data as if it needed meaning-based search.
You want to understand why the new feature's activation is below expectations, and you ask AI to cross-reference three sources in one shot: the ticket board in Linear, the activation funnel in Amplitude, and the transcripts from the latest round of interviews in Dovetail. It hands back a confident, well-written answer, but it scrambled all three: it treated the structured funnel as if it needed semantic search, it treated the ticket board as loose text and lost each ticket's status and label, and it treated the interview transcript, free-flowing text full of nuance, as if it were a spreadsheet of numbers. The real reason for the low activation was in a specific sentence a user said in the interview, "I didn't know that button existed", and AI never came close to reading that as actual text. Wrong format, wrong root cause: the funnel calls for schema reasoning, the backlog calls for structured ticket reading, the interview calls for meaning-based search.
You ask AI to explain where the pipeline gets stuck most, cross-referencing the CRM export, which is rows and columns, with the win-loss report in PDF the sales team builds every quarter. It hands back a consultant-style answer, but it treated the CRM as text to search for meaning and the PDF as a plain spreadsheet, missing the conversion-by-stage table.
You ask AI to find where the SLA blows up most, cross-referencing the structured time-per-stage system with the quality report in PDF full of charts. It hands back an answer full of efficiency jargon, but it treated the PDF as plain text, missing the timing table.
You ask AI to list which internal controls are most exposed, cross-referencing the risk matrix, which is rows and columns, with the data-handling policy, which is free-flowing text. It hands back a generic list, but it scrambled both sources, treating the matrix as text and the policy as a spreadsheet.
You ask AI to find which service causes the most incidents, cross-referencing the structured monitoring log with the postmortem report in PDF full of latency charts. It hands back a guess with the air of someone who knows infrastructure, but it treated the log as text to search for meaning and the PDF as a spreadsheet, missing the chart.
You ask AI to find where the user drops off the flow most, cross-referencing the structured funnel data with the research report in PDF full of heatmaps. It hands back a UX cliché, but it treated the funnel as free-flowing text and the PDF as a spreadsheet, missing the heatmap that showed the real friction point.
You ask AI which competitor is gaining market share fastest, combining the win-loss report in PDF, full of charts, with the quarterly sales spreadsheet in one shot. It hands back a pretty answer, but it scrambled both: it treated the chart-filled PDF as plain text and lost the table, and it treated the spreadsheet as if it needed meaning-based search, not the other way around.
Man, notice something: the problem is almost never that AI is bad at analysis. The problem is it's answering off the top of its head, having never actually opened YOUR real product. It's like hiring the best PM on the market, sitting them at your desk, and never giving them access to your Linear, your Amplitude, your Dovetail. They'll talk a good game and get it badly wrong, because they're guessing. Connecting AI to your product's context is exactly what takes it out of the generic. Except connecting it wrong is worse: it either scrambles the formats and gets the diagnosis wrong, or it leaks user data that shouldn't have gone out.
The core idea of this lesson. Your product's context lives in three different natures, and each one calls for its own technical path. The backlog (Linear, Jira) is structured but full of short text and status fields: AI reasons over the fields, it doesn't need heavy vector search. Analytics (Amplitude and similar) is structured event data: AI reasons over the schema, the funnel, the statistics. Research (Dovetail and similar) is free-flowing transcript text: it calls for meaning-based search, real RAG. Mixing the three paths is mistake number one. And before connecting any of them, there's the security question: can this user data leave the company, or does it need anonymizing first?
01Product context lives in three places, and each has its own nature
The first thing that changes everything is to stop treating "my product's context" as a single thing. What exists are three sources of different nature, and the nature dictates the path.
The backlog lives in Linear or Jira: each ticket has structured fields (status, priority, label, assignee) and a short text body describing the problem. Analytics lives in Amplitude or equivalent: events, funnels, cohorts, all in rows and columns, pure structured data. Research lives in Dovetail or equivalent: interview transcripts, field notes, free-flowing text full of nuance, the actual words of a real person. Three natures, three paths.
02Analytics: AI reasons over the schema, it almost never needs vectors
Your Amplitude dashboard is structured data: events with a name, a property, a timestamp, a user. To answer "where does the user drop off the activation funnel most", AI doesn't need to turn every event into a coordinate of meaning. It needs to understand the STRUCTURE: what the events are, what each one means, how to calculate the conversion rate between two steps, which cohort to compare against which.
Instead of searching for similar excerpts, it reasons over the schema, over the statistics (conversion rate, cohort size, median time between events), and over a representative sample. It's cheaper in tokens, more precise, and without the risk of semantic search pulling in the wrong event. Before building anything fancier on top of analytics, ask whether the data isn't already structured enough for AI to simply calculate.
03Backlog: structured tickets, not loose text
Linear or Jira looks, at first glance, like a pile of text. But each ticket carries structure: status, label, priority, sprint, assignee, and only afterward the descriptive body. The common mistake is throwing all of that into meaning-based search and losing exactly the structure that answers the most useful question: "how many bug tickets have been open for more than thirty days under the checkout label?".
That question is a count and filter over structured fields, not semantic search. AI that reads the backlog directly from the Linear or Jira API, respecting the fields, answers this with precision. AI that treats everything as free-flowing text might even get lucky, but it loses the precision the real board has to offer.
04Research: here, real RAG applies
The Dovetail interview transcript is the natural territory of RAG, the same territory as any free-flowing text document. The question "do users mention difficulty finding the export button?" isn't solved by counting any field: it's solved by searching for meaning within what was said, even when the user said "I couldn't get the report out" instead of the word "export".
This is where product's most expensive trap lives: treating the transcript as if it were structured data, summarizing too fast and losing the specific sentence that carried the insight. Research exists precisely to capture the divergence and nuance that no average summarizes well. If you let AI summarize the whole interview into a generic paragraph without letting you go back to the original words, you lost the reason you did the research.
05Security: real user data needs a check before it goes out
Everything up to here was about getting it right. Now the point that, in product, can bring you down: what leaves the company. Interview transcripts carry the words of real people, sometimes with a name, an email, an account detail that identifies the person. Sending that without thinking to a model in the cloud isn't optimization, it's a leak.
The protection layer you apply before connecting anything is simple: anonymize what identifies (swap the name for a code, strip email and account data), decide where the analysis runs (a model in the company's environment versus the public cloud, depending on sensitivity), and keep a record of which source went into which analysis. It's not bureaucracy. It's the difference between "I connected AI to my research" and "I leaked a real user's words to a third party without noticing".
Learn more: why summarizing too early kills the insight
A subtle and common mistake: asking AI to summarize the entire transcript before you've even formed your question. That looks efficient and is the opposite. The summary has already decided, for you, what was important, and it frequently throws away the odd sentence that didn't fit the pattern, which is exactly where the rare insight lives. The safer path is to keep the transcript as a live search source (RAG) and only summarize AFTER you've already asked the right question and seen the original words that support it. Summarize at the end of the investigation, never at the start.
Do it now
Take a real case from your product where you needed an answer that depended on more than one source, your real task or another one (why a feature didn't catch on, where the user gets stuck, what to prioritize).
- List the sources that would have the answer: the ticket board, the analytics dashboard, the research transcript.
- For each source, classify its nature: structured (backlog, analytics) or free-flowing text (research).
- Mark each one's path: backlog and analytics lead to reasoning over fields and schema; research leads to meaning-based search.
- Do the security check: does this source have data that identifies a real user? Does it need anonymizing before any analysis leaves the company?
You just designed the connection of your product context to AI down the right path, with the security precaution in place.
Practice
1. You want to know 'where the user drops off the activation funnel most' from the Amplitude dashboard. What's the best-suited path?
2. Why does treating a research interview's transcript as if it were structured data usually lose the most valuable insight?
For the board
On the three placesanalytics, backlog and research have different natures. Treating all three the same is the common mistake.
On analyticsthe funnel is already structured. The AI reasons over the events, it does not need vector search.
On researchthe interview exists to capture the exception the average does not summarise. Summarising too early throws away the reason you ran it.
Thanks for the feedback. It helps sharpen the next lesson.