From making sense to
making a practice.
The headline for AI is that it makes building faster. That’s true, and it’s not the most interesting part for me. The speed went into how quickly I can see a pattern, test an assumption, or find out an idea is wrong without wasting other people’s time. Plenty of serious people believe AI rots thinking. It made mine sharper, and faster to correct.
First attempts
Great ideas, useless requirements.
More than a year ago I used AI to write requirements. What came back read like it was transformational. Actually, it was useless.
Academic requirements for a utopia; a broad new framework capability for a unified admin system … Lofty, ambiguous and large features with no naming of risks or constraints. Nothing about our actual applications, what our business model would bear, or what my teams could build.
Nobody told me. They were confused and stayed quiet, and I found out the slow way. The conversations I expected never happened. Discussion got less focused than it had been before I brought the documents in.
That’s what the documents cost. The thing that makes a team work is shared understanding and common goals, and none of that output produced any. It only produced confused teammates.
So I stopped. I went back to writing detailed, focused and shorter specs myself. AI became mostly a research and synthesis tool for me, which is what it’s actually great at.
Day-to-day relief
Are we winning?
Then I pointed AI at the operational things that slowed me down every day. In a lean product org, feedback and customer signals are critical to do the job well. Unfortunately, a lot of times it comes to you as a flood of information. At the end of the day I just wanted time to think and make sense of it all.
The first thing I built was a local HTML file that used Jira filters to map planned work against strategic outcomes. The goal was to see whether we were moving. It worked. Kind of. Eventually I abandoned it.
Then a stream of customer complaints turned into a focused escalation effort, and I left every daily standup frustrated. Five delivery teams, executive sponsors, and the customer-facing teams all needed the same picture while each contributed to it from a different place. We reviewed a spreadsheet every day. Every day, we barely got answers and asked the wrong questions.
I built an internal dashboard in an afternoon. It pulled from a Jira filter that consolidated the work and refreshed itself twice a day, so nobody had to maintain it.
The main section was titled “Are we winning?” That’s what the exec sponsor always asked. A graph of tickets logged against tickets completed over time let us all answer it.
Status reporting, intake and triage management had been eating the part of the job I’m actually quite good at: analysis, research and strategic planning. Once something else handled it, I could think about the actual product problems instead.
AI for me
The questions I couldn’t answer.
I kept finding problems. Personal life hacks for planning my family’s 10-person trip to London; a product idea I’m still working on. I’ve kept building.
The most useful thing I built runs my job search. I created three tools to help me evaluate, track and prep. I wrote them because the search itself was drowning me, which was a feeling I had felt before.
Once basic skills were in place to make my job search easier, the LLM started making things up. It invented a connection between me and someone I’d never met. It produced a number I’d never given it. Both would have gone into a conversation with a real person if I hadn’t double checked every response.
So, I matured the process with scaffolding (a term I learned in the process). Every finding in a job description evaluation carries a label now so I know how the assertion is sourced. What the model can do with a finding depends on where that finding came from. Its own guesses can’t eliminate anything. Nothing estimates a number. Every claim about my work has to come from a card of facts I’ve verified myself.
That worked. But I was writing those rules one failure at a time as I experienced them. And in pondering how the responses kept getting off track, I kept hitting questions I couldn’t answer. How would I know if a change actually fixed something, or if I just got a good run? How many runs is enough?
I was doing evaluation by hand without knowing that’s what it was called.
How a finding is sourced decides what it can do. Click to enlarge.
Learning on purpose
Into the sausage-making.
I signed up for a course called AI Evals For Engineers & PMs, taught by Hamel Husain and Shreya Shankar. I started the first week of September and, as of this writing, I’m partway through.
The work is on a mock support agent in a made-up e-commerce product. In the first assignment I judged 10 of its conversations against its written spec and sorted every failure by where the fix belonged: the prompt, the tool code, or the spec itself. The code held every time. The prompt held sometimes.
That distinction is the thing I’d been missing. It was a missing link in how I understood LLMs in products. A rule written into a prompt is a request. A rule written into code is enforced. If a requirement is critical to the product, a prompt is the wrong place to keep it.
One homework scenario had the chat agent processing an account change on its own when the spec said to hand it to a human. Working with LLMs has made me a bit paranoid about how they reach for plausibility not necessarily correctness. So before touching the prompt, I set a rule: after any prompt change, the scenario runs at least three times and has to pass all three. Before the change, the agent escalated 0 of 3 times. I added one sentence to the prompt, and it escalated 3 of 3.
That fixed the escalation. Then a nagging thought showed up: why would even a mock spec route an account change to a support person? That belongs in self-service account management. It wasn’t what the course was teaching, though, so I parked the product question and stayed on the eval.
I question everything now, and this coursework is why. I could already tell when a response was wrong. Now I can usually tell which layer it was wrong in, and the layers don’t behave the same way. One enforces, one asks, one just states an intention and hopes. Most of the argument about whether AI can be trusted skips straight past that, and it’s the basis of the whole question.
I still build tools for efficiency, life hacks and product dreams. Now I’m looking at the structure underneath them. What’s enforced, what’s merely requested, who wrote the standard, and whether anyone is checking that the standard is any good.
Auditing the rule is a different job from owning it. I need to know both.
Where I’m heading
Still building.
The building part has a name now. CMM Labs & Consulting.
Labs is where I build to learn. I pick something I don’t understand, or an idea I want to chase, and learn by doing. More courses are ahead, along with friends and colleagues who know more than I do. Then I publish what worked and what didn’t. Consulting is the other half, and it only sells what I already know how to do well.
The first Labs project is a free tool for product and engineering leaders. It maps what customers touch against the squads who own it, and shows where those two don’t line up. Surfaces with no owner, or maybe two, and showing where platform teams hold surfaces up.
All of it points the same direction: I want to bring these skills and my product judgment to an organization worth the effort.
I’m not nearly done finding out what this changes about my work. I’ll be building in public. More soon.
Visit cmm.callmemandy.com