I’ve been spending time in the trenches connecting these pieces, and I’ve hit some bugs that aren’t just minor annoyances—they are real roadblocks in certain circumstances.

It is important to note that none of these bugs are caused by ADK or Agent Engine. They all seem to be introduced by Gemini Enterprise and how it currently interacts with custom agents.

I have recorded a video in which I walk through some of the issues I have encountered so far:

• Sessions not working as expected: You’d expect a Session ID to persist throughout a chat so you can manage state or store files. In Gemini Enterprise against a custom agent? It resets on every single conversation turn.

• Issues rendering files to users: If your agent tries to render multiple unique PDFs to the user, Gemini Enterprise struggles. It ignores display names and, worse, if you try to preview different files, the UI frequently renders the exact same document for all of them. It’s a UI bug that makes document retrieval frustratingly unreliable.

• Not being able to access user uploaded files in their original format: This is the big one. When a user uploads a file (like a Word doc) to your agent, you probably want the raw binary to process yourself. Gemini Enterprise intercepts it, "helpfully" chunks it into text and images, and hands you the processed scraps.

The source code used in this video can be found at https://github.com/christo-olivier/gemini-enterprise-bugs

The comment was made on a post about how AI coding assistants are getting lots of people, who used to be engineers but have moved into other positions, back into coding.

As you can imagine the comment section was lively and full of opinions. Our industry can be labelled many things but lacking in opinions is not one of them.

The comment in questions was making the claim that, and I quote

“If I tell a chef how to prepare a meal I want, and they prepare it, then I have cooked the meal”.

I was simply dumbstruck reading that. We seem to have slowly been losing our grip on reality.

We all rely on technology to get our work done faster and more efficiently. Nothing wrong with that. But surely we must also realise that if you cannot do the work without AI then you don’t actually know how to do the work.

If we take that comment to its extreme then a non-technical manager, who has told an engineering team what to build, can consider themselves an engineer. The same as anyone that ordered a meal at McDonald’s would be able to call themself a chef or cook.

You might think I am being extreme here but I have seen this shift happen over the last two and a half years. True mastery and skill is being devalued at an alarming rate. Yes AI tooling is extremely useful in certain circumstances, but to wield it wisely you need the skill to know when it is busy messing up.

I don’t blame the commenter, we have been collectively bombarded with this messaging since AI coding tools have hit the industry. However, I cannot help but wonder if we are losing our grip on what skill and mastery mean?

To be fair to Cursor it is not the only culprit in this AI IDE hype arms race, but it and Windsurf are the prominent two that are forks of Visual Studio Code. Herein lies the first issue.

Foundation built on a competitor’s code base

Both Cursor and Windsurf are forks of Visual Studio Code. They are forks because VS Code extensions are limited in their ability to manipulate the user interface. Fair enough, that is a valid reason, but it is also a critical weakness.

VS Code at its core is open source but the version me and you download and install is a distribution built from that open source core. So what we normally download and install is NOT OPEN SOURCE. Yes I am yelling it so that it sinks in because somehow people are still surprised when I point this out.

There is VS Codium which is an open source distribution, but that is limited in the number of extensions that are available. Right here is where you should already start seeing the smoke from the dumpster fire that is “forking” VS Code.

The reason VS Code is so popular is due to all of the extensions that have been built for it. Many of those built by Microsoft, tons of them built by the community. It turns out that those core Microsoft extensions are only licensed to be used with VS Code. Microsoft recently stopped turning a blind eye to the shenanigans of those forks, and started blocking the installation of their extensions in them.

Cursor had to scramble and start to implement open source versions of the extensions that got blocked, so for now things are kind of back to normal. This should however raise serious questions.

Do you think that any corporation would spend resources building something like VS Code simply out of the kindness of their hearts? Do you think that it is a good idea to build your entire business on the back of a code base controlled at its core by a competitor? Not only a competitor but a behemoth who has survived many industry transitions. (Metaphorically the equivalent of the crocodile having been around since the age of the dinosaurs)

The answer here should be a clear NO.

A forced way of working

This is the one that really annoys me. See if your AI coding tools force you into a specific IDE then I get uncomfortable. Not all professionals have the same workflow, not all work requires the same tools. Have we learnt nothing after all of these years?

I know some exceptional SREs that live and die on the hill that is Vim. I am in awe of their knowledge, skill, and the impact they have.

I know exceptional developers that use Jetbrains tooling because that works best for them. Likewise I know others who swear by VS Code, and others that use different tools like Zed.

The point is, people spend a lot of time finding the tooling that works best for their particular niche. It makes them more effective at what they do. The tool should fit around the person and the job.

Having your AI tooling tied into a specific IDE runs counter to this. I am sorry, but not everyone in the world is coding React and NextJS applications. (Something Cursor is exceptionally good at). It is great if Cursor works for you and your workflow. Please don’t force it on everyone.

Yet this is exactly what has started to happen. I have heard multiple first hand accounts of companies buying Cursor licenses for their teams and encouraging people to switch to it.

The shift to coding agents

While the AI IDE wars have been raging, the shift to coding agents has been slowly gathering pace.

Aider (a firm favourite of mine) and Claude Code, were the first prominent ones on the scene, and OpenAI has just released Codex.

These tools are command line tools that run outside of your IDE. This immediately makes them compatible with more people’s workflow.

It is refreshing to not have the tool dictate the IDE I should use. It also opens up ideas of using smaller scale agents in other places in your workflow. For example agents doing initial code review or suggesting performance improvements.

My own experience so far

My own journey with AI IDEs and tools has not been overnight. I have built two products from scratch with the help of them. Both products are non-trivial solutions, each having data pipelines, databases, backend APIs, UIs, and all the infrastructure to deploy and run them.

I started out 18 months ago and the journey has been:

1. VS Code + GitHub Co-Pilot

2. VS Code + Cody (by Sourcegraph)

3. Cursor + Aider

4. VS Code + Augment Code

During that time I always checked back on the tools I used previously to see if they have improved.

Through all of this I have always had the best experience with the tools that did not force me into a single IDE. Ones where I get to choose what works best for me.

If my own experience is anything to go by then choose the tools that allow you the freedom to work how you work best.

Being forced into a single tool is a step backwards for all of us. This is especially the case when that tool can do a Venture Capital rug pull on you at any time, or can themselves be clobbered by the giant on who’s back they have decided to build their fortune.

I have always enjoyed the writing of Steve Yegge. He is clearly a very experienced and accomplished engineer. His writing on the Sourcegraph blog has been great. But his post on 22 March 2025 titled Revenge of the junior developer was the kick up the backside I needed to write again. I recommend you read it before you continue with this post because I will be “Don Quixote-ing” my way into some of the windmills I have picked out of that post.

Buckle up fellow nerds! This one is not for people with weak stomachs. There will be foul language… and I am possibly going to go full Karen… you have been warned!

(If you want to get into the full spirit of this post then may I suggest you pair it with Anarchy in The UK by the Sex Pistols)

Opening credits

For almost two years I have been building LLM based applications and solutions, or “AI” as most people now seem to call it. Not only have I been building solutions that use it heavily to solve problems, but I have also been using AI agents and IDEs for the development of these solutions.

Two years is sufficient time to get so close to the tech that what you see as normal is actually not the norm.

On top of that, if you did not know, I don’t live in Silicon Valley or in a Big Tech “scene”. I have been a tech consultant for a large part of my career and then went rogue and became a freelancer, before going mad and bootstrapping a startup.

So why am I telling you this? Because I want you to know I come from a world where your mouth needs to write cheques that your arse can cash. If you can’t deliver what you say you will deliver you won’t be in business for long. Also, you carry liability for the work you deliver. If you screw up on a project you can and will get sued for damages. We will get back to liability later, just let it sit there in the back of your mind as you keep reading.

Commentary on the “Waves”

The first thing that kicked me square in the “WTF?!!” was the “six waves” presented at the start of the post. Even though we are told that this is coming from “Exaggeration Central”, we are still introduced to the timeline of how chat-based vibe coding is going to be obsolete by Q3 of 2025. Yes, you read that correctly. Vibe coding is evolving, and the evolution is spectacular!

Below is Figure 1: Overlapping waves of AI coding modalities, taken from Steve’s blog post.

The following is a direct quote from the post.

The chart in Figure 1 depicts six overlapping waves of programming: traditional (2022), completions-based (2023), chat-based (2024), coding agents (2025 H1), agent clusters (2025 H2), and agent fleets (2026).

Holy *&^%&^ batman! Forget that these tools need CONSTANT supervision to ensure they don’t go off the rails. Forget that a single chat-based LLM in “Agent Mode” can already generate more code than an experienced developer can review in a timely fashion. Just forget about all of that because what we have to look forward to is “agent clusters” in less than 6 months’ time, and “agent fleets” in 2026. (On a side note, who has money on the job title Admiral of Architecture appearing on a job post soon?)

There is another industry where this sort of “like cures like” thinking is very prevalent. Yes people, I am going there! This sounds like the tech equivalent of homeopathy. Because if Agents cause issues then surely MORE agents would be the cure.

Now if you are thinking “surely Christo has gone mad, or he is now just a coding boomer coming to spoil our fun. This like cures like stuff is just an old man running at windmills”. Well may I present to you the solution proposed to keep those AI Coding Agents in line: (Figure 2: FY26 Org Chart from Steve’s blog post)

I present to you AI Managers! Are you starting to see that homeopathy analogy come to life yet?

Productivity gains

Right, that was a lot of ranting and raving. I think you probably want a break from my soapbox antics, and I need to have another hot beverage (probably one with less caffeine).

Let’s slow down and now look at some of the productivity gains that people claim all of this produces.

The blog post in question states that each new modality in coding brings a 5X to 10X productivity boost. This is also the sort of figures you see thrown around online on LinkedIn and Twitter, sorry I mean X.

What no one is talking about are the details surrounding those claimed productivity gains. Think about it, is this how much faster you can generate code? Is this how much faster your team completes work? Which parts of the work are we talking about? All of it?!!

It matters knowing what the claims apply to. Most of your time as a software engineer is not spent writing code. Yes, that is correct, most of your time is spent gathering requirements, speaking to users, other teams, speaking to your own team, doing code reviews etc. It is not just sitting with your headphones on cranking out code.

Another thing that is very rarely mentioned when these claims are made is the type of environment they are realised in. Is this an indie dev working on their own? Is this a large development team working in a larger organisation? All of this matters.

There are clear signs that indie devs and small teams have a much bigger performance increase than teams in large organisations. You can already guess why that is right? That is correct, it is because small teams that can operate autonomously always move faster than large teams that need a lot of additional coordination to keep things on track and aligned to the overall goal.

The argument can be made that there are tons of productivity gains to be realised in larger teams by simply helping make these teams more autonomous without introducing tooling that can create an avalanche of code each day.

This brings me to my next point, what is the word on the street where people are currently using these tools?

Where the tyre hits the road

Let’s start by mentioning another blog post that made it to the font page of Hacker News. This one is called Why LLM-Powered Programming is More Mech Suit Than Artificial Human by Matthew Sinclair. It strongly aligns with my own experience using these tools daily.

Now Matthew looks to be a seasoned professional with bucket loads of experience through many different roles. What stood out to me was his mentioning of the vigilance required when working with a coding agent, a single coding agent not a cluster or fleet of them. He was specifically talking about Claude Code.

Here are some quotes from his blog post.

With great power comes great responsibility. You must maintain constant awareness when working with AI coding tools—something I learned through several painful lessons.

Claude Code occasionally made bewildering decisions: changing framework code to make tests pass, commenting out whole sections of code and replacing them with hardcoded values to achieve a passing test rather than fixing the underlying problem, or introducing dependencies that weren’t necessary or appropriate.

…

Taking your eyes off the process, even briefly, can lead to trouble. In my case, the backend required three complete rewrites because I looked away at crucial junctures, allowing the AI to go down problematic implementation paths that weren’t apparent until much later.

Let that sink in for a moment. Then read the next quote.

For me, using Claude Code has been a learning exercise in itself. Progress often felt like two steps forward and three back, particularly in the early stages. Generating 20k+ lines of code became relatively straightforward on a daily basis, but knowing when to throw everything away and rebuild from scratch—that took 30 years of experience.

20k+ lines of code generated daily. Stew in that statement for a bit. Can you imagine having to review that amount of code daily?!

Can you ensure that the code will be of good quality, that the agents have not gone off the rails in a moment where you looked away and took a sip from your avolatte? If you say to me “yes” then I am going to call you a liar right here and now. It is simply not possible for a human to review and ensure code quality at that scale each and every day.

So obviously the only way to handle that volume of code is to have machines do that work. But if the machines writing the code can go off the rails, then surely the machines checking them can go off the rails too. See where I am going with this? DO YOU SEE WHERE I AM GOING WITH THIS?!!

Let me illustrate with a picture.

The revenge of Liability

Hello liability my old friend… you have come to talk to us again… because a vision is softly creeping…

It is time we speak the name of the one thing in GenAI no one wants to talk about. L-I-A-B-I-L-I-T-Y

Everyone wants to sell you an agent or a service where you hand over your money, but you keep all the risk and liability. Can you imagine that ever happening when you bring in a freelancer or a consulting firm? HELL NO! Consulting firms and freelancers cannot simply put a little sentence on a screen that says, “People make mistakes so please be aware of it and double check everything”.

No no no sir, we carry liability for our work. Don’t believe me? Think I just made that up? Let me tell you that I have been on projects where the consulting firm I was working for needed a project signed off by its parent company due to the project carrying unlimited risk. Yes, that means that if we did anything that caused damage to the client, they could take us to court for whatever amount of damages we caused. Literally they could sue us and the parent company for any amount even if it means the whole group of companies went bankrupt. (That type of project makes you think twice about spouting BS to a client)

As a freelancer I must have insurance as well for exactly this same reason. Clients demand it because otherwise they are left out of pocket when something bad happens and I am unable to pay the damages from my own assets. Oh, and as a startup we also need insurance with minimum amounts dictated by the clients that we do business with.

Why am I droning on about this? Because it is the answer to adopting the fantastical “Fleet of Agents” scenario.

If you are willing to accept liability for your Fleet of Agents’ output, then I am willing to sign up to that service. Yes, you must accept the same liability a consulting firm or agency accepts when you engage them for work.

We already know that humans will have to trust the AI Managers that oversee the Coding Agents because no human will be able to review all the work produced by a single coding agent in full flow, never mind a Fleet of Agents. So, the seller of this service or tooling needs to be willing to accept liability. If they don’t, then you are clearly not confident in the output of your product.

If we need each human to check every line of code, then those fantastical 10X productivity gains are starting to look more and more out of reach won’t you say?

What about the junior developers

I cannot help but get one last jab in at the ludicrous idea that junior developers will do much better in this new world of agentic coding cluster armadas. Do you think, after everything that I have highlighted, that junior engineers will have the skill and intuition to deliver better outcomes in this new world?

If you do then please reach out, I have a supplement that works for all ailments and has no negative side effects. Big pharma does not want you to know about it…

What we need is even more rigorous training for those in our field that will oversee these tools. The amount of damage that can be done is orders of magnitude more with these tools than when you wrote code by hand. I am inclined to say that those productivity gains of 10X are more realistic as damage multiples in the hands of someone that does not know what they are doing.

Before you get on your high horse and come at me with your posse of agentic agents, I am not anti-junior. Relying only on the experienced developers is not an option because we all end up being a junior in some way shape or form at certain points when we tackle something new. So just because you are experienced in one domain does not mean you are not going to juggle chainsaws like a vibe ninja and lose your limbs when you tackle another domain.

We need to hammer home the fundamentals now more than ever! We need a new appreciation for knowing the fundamentals because only those that know them will be able to call bullshit on the “agent fleet” when it goes off the rails. Everyone else will be vibe coding their way into an episode of The Lazarus Heist. (Go listen to the podcast, all seasons of it. It is amazing!!!)

Wrapping up

I have had a lot of fun just “vibe blogging” this post. No, I did not write a single piece of it with AI, I cranked up the tunes and simply wrote what was on my mind without much of a filter.

Where we are at today in tech feels a bit like one has stumbled into a trance party by accident. There are vibes all around. People making bold claims, while others are just chilling and embracing things one moment at a time.

I cannot help but get angry at some of the ridiculous claims being made. Especially because they are so far removed from my own experience over the last two year, and because I see the damage these claims are already causing.

We have not even touched on issues such as security, data sovereignty, financial feasibility or any of the many other things that factor into this discussion. I might write some more about that at some point.

What I do know is that we need to call bullshit on things loudly and more often than we currently do. This tooling is not magic and making exaggerated claims that read like fan fiction is helping no one in the real world.

I cannot predict the future, but neither can anyone else! What I can do is point out the flawed logic of some of what we are being bombarded with at the present time.

Things could change tomorrow, there could be a breakthrough that completely changes things. When that day comes, we will adapt.

Until then, I will catch you on the flip side.

What type of AI application are we talking about?

There are a ton of different applications for LLMs and generative AI inside software. Some of what I am going to describe will not generalise to all uses of LLMs.

The specific application I am talking about is one where the user interacts with the application through a chat interface as an assistant or partner in thought.

What I have been working on allows someone in a B2B sales team to do some of the following:

• Ask for up to date information about a deal they are working on.

• Request a review of the deal and get insights of how well this deal is going.

• Analyse the 10Q filings and quarterly earnings call transcripts of a company they are interested in and identify topics or information they are interested in.

• Bring back data from web searches about target accounts into the conversation and combine it with internal data from CRM and other sources to ask questions against.

• Ask questions about internal documentation to help them better prepare for upcoming meetings. For example, have the assistant pull out relevant information on how best to position the company’s product against that of a competitor who they know are also targeting this account.

The application is built in such a way that new skills can be added over time.

This type of application is very similar to the interfaces of chat apps such as ChatGPT and Google’s Gemini. However we can also embed it into line of business applications, making it more integrated into the daily lives of our users.

A mental model

LLMs might look like magic at times yet other times like utter nonsense spouting wastes of time. In order to make them useful it helps to have a good mental model of how these applications function.

You see the LLM part of the application is just the piece that handles the text translation from and to the user. Underneath the surface a ton of things are happening to make the LLM look intelligent.

My mental model for these types of applications is that each conversation is an A4 page being filled with user input, data returned from tools, and the LLM’s response based on the conversation on that page so far. Let me try and illustrate this graphically.

You have probably heard the word “context” mentioned a lot in Gen AI conversations. This entire A4 page that we build up serves as the context that we feed to our LLM for it to then generate an answer.

Now depending on how sophisticated you get you can send pieces of the A4 page to different LLMs tasked with specific functions. In these cases the pieces of the A4 page you send to the LLMs will be the context. As you will learn, context (or "cheating" as Steve Yegge puts it) is absolutely king.

When I started out I had the idea that I might need to train my own LLMs to perform certain tasks. You might have also heard others talk about fine tuning models to perform better at certain tasks. I will do my best to give some useful thoughts on this. However for a lot, and I mean the vast majority of what you might be doing, context is going to be your secret sauce.

Context is everything

The majority of AI applications, at least the kind I am describing here, will do some form of RAG. When people think about RAG they normally think about Vector databases. This is however only one form of RAG.

RAG (retrievement augmented generation) is just one part of building up the context that an LLM is operating within. The context of an LLM comprises everything that is fed into the LLM that it operates on to give a response. This includes:

• The prompt

• Chat history (input from the user and response from LLM)

• Data from function/tool calls

• Data fetched from Vector stores

• Data fetched from any source system

The better the information in the context the better the response from the LLM.

Not all parts of the context are visible to the end user. You might hide some of it in the background and only inject it into the prompt that you send to the LLM. This normally gives rise to that magical feeling of “The LLM knows things!”. The application looks magical to the user because they cannot see the big chunk of information the LLM is using to answer their questions. Yet every follow up question gets answered accurately. 

With the increase in input token size for most models it is becoming much easier to build out large amounts of context. Some of these models now have a 1 million token input size which is a vast amount of text. Just keep in mind that throwing everything at the context is not always the best strategy and can lead to performance issues in both duration of LLM calls and the output it generates.

You can get sophisticated with your context strategy where you cherry pick the pieces of chat history and information you pass to the LLM on each conversation turn. This can lead to very good results.

Functions/Tools can be more than meets the eye

Function calling with LLMs can be so much more than what you see in the normal documentation. The documentation typically highlights that you provide the LLM with a list of function definitions and the LLM comes back to you with which function to call and what arguments to pass to the function’s parameters.

This hides the fact that the functions can be the entry point to a much more complex process. For example you could create entire Python packages in which you use additional LLM calls to perform actions all hidden behind an entry point function.

You also do not need to have the LLM choose the arguments for the parameters. You could simply let it choose which function to call and pass through whatever you would like to the function. This can be useful by not having to over engineer your main LLM’s prompt to deal with things like entity extraction for specific function parameters.

An example of this is the Deal Review package we use in our application. The main LLM only decides that the user is looking for a deal review, then we pass the user input with the rest of the desired context to the entry point function of the Deal Review package. The package uses a dedicated LLM call to do entity extraction, looking for the name of a deal or a particular ID in the user input. It then makes the necessary calls out to databases to get the deal information and feeds that into another LLM call that is specifically designed to perform a deal review. Once all of this has completed the resulting text is passed back to the user. This process neatly contains the deal review functionality and allows us to separate out the prompts used. It also allows us to test individual pieces of the application separately making life much easier.

As you can see a function call does not need to be just a simple function call.

Training your own LLM or Fine Tuning a model

Inevitably there will be discussions about training your own LLM or fine tuning a model.

The reality is that training your own LLM is not a trivial undertaking and is prohibitively expensive for most people and companies. We are talking millions of dollars, not to mention the massive amounts of data you need to collect in order to have any chance at producing something good. Training of foundation model LLMs is currently only feasible for very large organisations.

A much more reasonable option is to take an LLM (typically a foundation model) and fine tune it for a specific use case that you might have. This requires you to have a couple of hundred or thousand data points (depending on what you are trying to do), and does not come at the same cost as training an LLM from scratch. The fine tuned model can then be used for that nice use case where a foundation model with good prompt and context is just not giving you what you are after.

Most of the time you will end up with a normal LLM foundation model and the correct context giving you good results. If that fails you will move to a fine tuned model.

User experience

As with all applications the user only has the frontend interaction by which to judge the application.

For AI applications additional thought needs to be put into the user experience. We have all become accustomed to the ChatGPT style interfaces where you are having conversation with what looks like a single agent. It serves as your single point of contact through which you access all the other services, agents and tools. This feels like sitting in a meeting room with a single expert who relays messages to others behind the scenes and then answers you..

We started out in our team with such an interface before one of our team members highlighted how unnatural this is in the real world. We then pivoted to a user interface where the user can directly access the different agents and tools we provide if they choose to. This is the equivalent of being able to address everyone in the meeting room and having a discussion about the information they provide.

As an example the user could ask the Deal Review agent for a review of a specific deal. This agent then pushes a detailed set of deal information and its summary into the chat context. The conversation can then continue and asking other agents questions would result in them having access to this context as well.

All of the above functionality is thanks to some very nice UX design in the front end. We use “@” to allow a user to address an agent. If the agent requires more input from the user, such as to make a choice between a set of values, the UI keeps focus on that agent in the front end until it has all the user input it needs. This makes it an easy and familiar experience since most social media and messaging applications work this way. We found that once users were shown this functionality it was extremely natural to them.

We have also embedded our application within some of the applications used by our users. This has made it easy for them to click on a “AI Chat” icon next to an account which would launch the chat interface with the account’s information preloaded into the context with an account review provided straight away. They can then continue the conversation from that point.

Most of what we have done on the UX side is to observe how our users work normally, then sprinkle our AI application into their normal workflows. This has made ordinary tools seem magical.

Technical components at a high level

The final thing I want to do is give a high level view of the technical components that make up the solution. We started out with a more complicated set of components and over time simplified it. This gave us the ability to focus more on the application’s functionality than keeping an eye on the infrastructure.

Below is a diagram of the main components we currently have.

Front End

Our front end is created using Svelte and makes the assistant available in two ways. One is in a line of business application that each team member logs into to see their metrics for the accounts they are working on. The other is a pure chat interface application akin to ChatGPT and Gemini.

Having this ability to bring the assistant into a line of business application that is used daily has been phenomenal. It allows us to sprinkle some “magic” into the application. Users find it intuitive to click on the assistant button next to things like deals, accounts, graphs etc. to launch a conversation that is pre-populated with the context to make the conversation instantly useful.

Having this great UX for our users is arguably the most important part of the entire application. Without it the tool would just be another tool, instead it now feels like an enhancement to their workflow.

Backend

The backend API is the heart of our application. It is created using FastAPI and contains all of the logic we use to make the assistant work for the user.

We currently expose a set of endpoints from this API. Some of the more interesting ones are:

• Chat - Used to interact with the agent and have the chat loop execute.

• Memory - Used to get, set and delete key value pairs in the memory that is then fed into the context. We have the concept of both chat and user memory. Chat memory only exists for the specific chat while user memory is available across all chats for a user.

• History - Used to get a list of previous chats for a specific user or get the full message history for a specific chat

The tools available to the agent are also part of the backend API. This is currently a python package in the backend codebase but could in future be moved into a separate project of its own that is imported into the backend.

Persistent Storage

We use a combination of Firestore and Google Cloud Storage as our storage layer for the application. We decided on this combination because of the ever increasing input token size for models (Gemini Pro 1.5 Flash can take 1 million input tokens) and the Firestore 1MB size limit on documents. GCS is used for pieces of information that cloud go beyond the 1MB Firestore limit.

GCS might seem an odd choice but the read and write latency proved fast enough in our testing and it vastly simplified our stack.

Sources

Our sources of data range from 3rd party APIs and BigQuery to internal systems. Most of these sources are used by our tools that the agent has access to. Here is a brief overview of what we use:

• BigQuery (with BI Engine enabled) - Used as the source for some specific sets of data that is only available and curated in BigQuery.

• SerpAPI - Used for web searches.

• Vector store - Used to allow searching over all of the organisation’s internal PDFs and documents that serves as its knowledge base.

Final thoughts

I hope this overview has given you some insight into an example of an AI application. It is important to note this is only one example built against specific use cases. Not all AI applications need to be a chat bot, in fact I see more applications for LLMs in workflows and inside regular applications than I do for the pure chat interface type.

While the tech is not perfect it is certainly useful. It is up to us to remove the hype and focus on applying it correctly. Once you cut through the noise you realise that it is just another tool in your toolbox, it does not replace the toolbox.

What I am going to explore in this post is what it is, how it works, and how to achieve this with any LLM regardless of if its API or SDK supports it.

What is function/tool calling with LLMs

LLMs are next token predictors. That’s it. Nothing more, and nothing less. Based on the input (prompt) it will predict the next most likely token, then re-evaluate the original input + predicted token to predict the next token.

This loop runs until the LLM reaches its end where there are no more tokens to predict, or it has reached the number of output tokens it is allowed to produce.

Again, that’s it, nothing more and nothing less. LLMs DO NOT call functions. Regardless of what you have been told, they simply don’t. If your LLM API/SDK or model of choice calls functions for you then there is a layer of software wrapped around it that is taking care of this and invoking the function(s).

When you dig into the documentation for Google’s Vertex AI and that of OpenAI this becomes clear. What is described in both is that the calling client (your software, ChatGPT, or whatever service you are playing with) sends to the LLM a list of the functions that are available, a description of each of them and a description of the parameters each function takes. The LLM will then return a response that states which function to call and which arguments to pass to the parameters of the function based on the prompt it received.

Once you receive this response from the LLM the calling client is responsible for executing the function and then returning the result back to the LLM as part of an updated prompt, now containing the output from the function. The LLM then returns a response by either specifying another function to call or returning a response that can be sent to the user by the calling client.

Here is the diagram from Google Cloud Vertex AI showing this flow:

And here is an extract of the OpenAI API documentation describing exactly the same steps.

The basic sequence of steps for function calling is as follows:

1. Call the model with the user query and a set of functions defined in the functions parameter.

2. The model can choose to call one or more functions; if so, the content will be a stringified JSON object adhering to your custom schema (note: the model may hallucinate parameters).

3. Parse the string into JSON in your code, and call your function with the provided arguments if they exist.

4. Call the model again by appending the function response as a new message, and let the model summarize the results back to the user.

So now you know that there is no magic! This is good old fashioned software development with some LLM magic sprinkled in.

NB! Remember that LLMs will hallucinate, and this also applies to the arguments it specifies for the parameters of the function call. Code accordingly!

How to use any LLM to call functions/tools

So now that we know how it works, what is stopping us from using any other LLM that does not have a function or tool calling capability to perform the same action? The answer is nothing.

Almost all the SDKs and LLM APIs supporting this require you to send the definition of your functions/tools as a JSON Schema object or an OpenAPI Schema. So given that most of the LLMs are trained on loads of JSON we can take advantage of this by passing our list of tools to the LLM as a JSON object and specify a JSON Schema object that we would like to get back.

So, our plan is:

1. Create a python dictionary that contains the name of the function and a description of it and its parameters.

2. Pass this to the LLM with a prompt instructing it to call functions when it requires to.

3. Receive the response and process it, calling any functions it specifies with the arguments it specifies for each parameter.

4. Return the result of a function call to the LLM as an updated prompt.

5. If not function call is received from the LLM but instead a message to pass to the user then we send this output to the user.

6. Profit! (ok not really, but you get the idea :D )

I will be using text-bison on Google Cloud Vertex AI to try the above out. This is a TextGeneration model type that does not have function calling as a capability. I will also be using my Modelsmith python package to helps get a structured response from the LLM.

Lets look at an example. (The code for each screenshot can be found here on GitHub)

The functions

We set up two functions for our LLM. These are very simple functions to help keep the example as short as possible. We include a print statement to show when the function is executed and what arguments were passed to it.

Next up is doing some prep work to provide the LLM with a list of function it can call and creating a dictionary with the function name mapped to the function object. This dictionary is there to help us avoid a bunch of if statements, so is purely for convenience.

I am using inspect.getdoc to get the docstring of the functions and add it to the description. For the parameters I use the Pydantic TypeAdapter class to get the JSON schema of the function’s parameters. This saves us having to build that out ourselves.

The response object

Now that we have our function related work out of the way it is time to turn our attention to the Pydantic model we want to get back from the LLM as a response. In this example it is a combination of either a message (if the LLM has enough information to answer the question) or a function (that needs to be called) and arguments (for us to use when calling the function).

The descriptions on the Pydantic model’s properties are important as this is fed into the LLM’s prompt and helps give additional information of what is expected from the LLM.

The prompt

Moving on we look at the prompt that we will use.

The prompt makes use of jinja templating to add the functions, response model, and finally the context to the prompt.

There is so much that can be said about prompts and “prompt engineering”, but I will leave that for another post. Suffice to say that it is less engineering and more magical trial and error.

The application logic

So, all that is left for us to do is put in some application logic to have the user ask a question and see if we get our functions executed and the question answered based on the data they return.

Let’s break the application logic down according to the main parts.

Lines 1 - 5

Using Modelsmith, we create our instance of Forge specifying the text-bison model and pass it the LlmResponse as our response model, and also passing in our custom prompt.

Lines 7 - 10

We create our prompt_values dictionary and pass in the functions available as JSON. The context is initialised to None as we will only use it to capture the output of a function call when that is required.

Line 11

The user is asked for input which we will use as the input to the prompt sent to the LLM.

Lines 14 to 44

This is the main body of our application loop.

First, we initialise our counter as we are only allowing two iterations of the logic. This is done to ensure that we do not end up in an infinite loop. Remember, the user’s input is sent as part of the prompt and as we have not built any safeguards into this example the wrong user input can cause the application to loop indefinitely. (Hello prompt injection!)

We check to see if we have received context, and we add it to the prompt_values if we did.

Next up we generate our response from the LLM, passing in all the arguments needed.

Based on the response from the LLM we go and retrieve the function to call, if one was specified. You can see the use of our FUNCTION_MAP constant in getting the function to call instead of using if statements. Once a function has been called the response from it is assigned to the context variable which will be added to the prompt_values in the next iteration of the loop and thus make it into the prompt going to the LLM.

If we have a message back from the LLM instead of a function call, then we print that out to the user and we break out of the loop.

Otherwise, we go around once again until we break out of the loop without returning a message from the LLM to the user. If this happens, we print a simple message to ask them to rephrase their question.

The results

Let’s take our toy example for a spin. First up we will ask it for the weather in Amsterdam.

Next up let’s ask for local news in Manchester.

And there you have it. Function calling with a model that does not support it.

Wrap-up and caveats

We looked at function calling and realised it is simply an LLM telling us what function to call with which arguments for the function’s parameters. Hopefully demystifying function/tool calling in the process.

We then built a very basic example script to test out function calling with a model that does not have this capability in its SDK or API.

This was a very basic example to illustrate the concept. While it works it does not mean that it will always work reliably. This is the catch. When it comes to LLMs you are looking for reliability and predictability above all else. You need deterministic behaviour. Models that have function/tool calling as part of their SDK and API are typically trained to perform function calling, while other models are not. Hence, they will give better and more consistent results.

The usual caveats apply here, code defensively when it comes to LLMs. You are placing a non-deterministic element that makes up results inside your application logic and should make sure that it cannot cause problems.

I hope this has been a helpful example to show the concept. As always leave a comment or reach out to me if you would like to discuss this further.

If you missed it earlier, the code for the example script can be found here on GitHub.

Many developers create custom solutions to solve the challenge of settings management. However, the well-known Pydantic package provides a solid implementation for settings management that we can extend when needed.

Using Pydantic's settings management, you can have a standardised solution across all your projects with the added benefit of quickly extending it to sources not supported out of the box.

It is common to use Google Cloud Secret Manager when deploying solutions to Google Cloud. It is unfortunately not supported as standard in Pydantic's settings management tooling. To demonstrate how you can extend Pydantic's settings management to connect to Google Cloud Secret Manager, I have created a GitHub repository with an example and a YouTube video to provide a quick overview. I hope you find this as useful as I have.

GitHub Repo: https://github.com/christo-olivier/pydantic-google-secrets

One thing that has always been a bit difficult to get a clear direction on is how to deploy Streamlit applications to your own cloud platform and secure them. Most of the content out there always deploy public facing apps. Most teams would like to limit these applications to only be accessible to users in their organisation.

I have put together a step-by-step guide on how to do just that on Google Cloud Platform. Get yourself a cup of coffee or tea, I hope you enjoy it.

As you know, I have some experience with Airbyte's open-source version but have yet to use their cloud offering. I have written a few connectors for it and have Airbyte's open-source solution on multiple projects. I have also worked with Fivetran on a few projects. Naturally, I have some opinions on both.

The author of the post correctly states that Airbyte has challenges. The project has many open issues, and most relate to the connectors provided. Now many of these connectors are open-source contributions, and hence there are some rough edges. It also makes it rather challenging to build connectors for many of the targeted SaaS tools, as you need access to them. Something you usually only get when you are paying for them. Few people can pay for them personally as they are tools made for business use.

I am by no means making excuses for Airbyte. I want to highlight why you will find open issues and encounter rough edges. What is good, though, is that when you run into these, you can look at the code and fix things, typically quite quickly. Again, this comes from my experience having done that for clients.

Now Fivetran is a whole other beast. It does a fantastic job of getting your data into your cloud data warehouse. But there are a few gotchas that you should be aware of.

1. It will not load the raw data from the source for you. It will be loading data based on its own data model for that source. These models make sense as they are relational models for otherwise messy nested JSON from the source API. However, it means the data is processed before you receive it and in a structure that won't match the raw data. When you decide to one day move off of Fivetran you have a challenge. You don't have the original raw data, your existing pipelines are using the Fivetran relational model, and thus and thus you cannot simply swap Fivetran out with another tool. You will need to do a fair bit of work to replace it.

2. Fivetran sometimes does not provide all the raw data from a source's API. This has happened to my clients for sources like Shopify and Klaviyo. Sometimes this is a bug, which can take a lot of time to get fixed, and sometimes it is due to an API change stuck somewhere on the Fivetran team's backlog.

3. Security is another part of this that gives me pause. You effectively have to provide a Fivetran account access to your sources and cloud data warehouse for it to write data to it. This means your data flows through a 3rd party's systems whom you need to trust will not be doing nefarious things with that data. Call me paranoid, but there is a big difference between being able to look at the code being used and trusting that you will do the right thing as you specify in your EULA. (Just think about the recent issues with Tesla sharing sensitive images recorded by customer cars)

4. Pricing for Fivetran can be somewhat obscure and work out quite expensive when you start to scale. I say pricing can be obscure because you are charged for the number of records you sync, but you can *negotiate* pricing in certain circumstances with them.

My experience with Fivetran has made me very cautious of closed-source tools for data ingestion. You get an incredibly fast start when you use Fivetran or similar tools, but it comes at a cost. The cost of being in control of your Extract and Load part of your platform and being at the mercy of someone else's backlog, priorities, and pricing.

Does this only apply to Fivetran? No, it certainly does not; it applies to any tool you use to ingest data and includes things such as the SaaS version of Airbyte.

Be vigilant when making your tooling choices. Ensure you understand the price you will pay further down the line for the control you give up. The speed you gain at the start might be dwarfed by the effort needed to change later.

Below is a diagram showing the main components. One of the main goals was to keep the solution as cloud agnostic as possible. It is currently implemented on the Google Cloud Platform, but the GCP components can easily be changed for others on another platform.

The architecture is deceptively simple yet extremely powerful. It has a low cost of ownership and does not need a large engineering staff to operate and maintain.

Let's look at some of the chosen components.

Data Extraction

One of the most time-consuming parts of any data solution is extracting data from the various source systems that feed it. Marketing and Sales teams use many software products, most of which are SaaS solutions.

Instead of writing custom code to pull data from APIs, I chose to use Airbyte instead. More specifically, the open-source version of Airbyte. This version allows you to spin up Airbyte in your cloud environment, keeping all your data under your control. It is an essential consideration for many organisations with a strict security vetting process for new software, especially SaaS tools. There is also Airbyte Cloud if you prefer not to manage your own instance.

Airbyte provides many connectors to SaaS products and other data sources such as databases, Kafka, cloud storage etc. There is also an active open source community contributing to the connectors, with improvements and new connectors coming through almost daily. You can find the current list of supported connectors here.

Building your own connectors is also a straightforward process. You can use any language you want since connectors run as containers on either Docker or Kubernetes. I have created a few custom connectors so far and will be looking to contribute these back to the Airbyte community.

All data from source systems is landed in its raw format in cloud storage which provides a cost-effective historical store of data and does not tie the data into any particular tool for processing.

Data Transformation

The data stored in cloud storage is modelled and transformed into activities and attributes using DBT and Google BigQuery. BigQuery allows for serverless SQL against cloud storage, while DBT provides the data transformation, testing, and documentation framework.

I use a custom DBT macro that allows you to write SQL scripts or Data Definition Language queries instead of the regular SELECT statements DBT recommends. It is used only where necessary, such as where I need to define external tables against cloud storage or where complex SQL is required to generate an attribute.

I cannot overstate the value of DBT as it is an absolute game-changer for analytics engineering.

The solution uses the open-source version of DBT, which we run on our GKE cluster. As with Airbyte, a DBT Cloud option is available if you prefer not to manage it yourself.

BigQuery provides the processing power to turn large amounts of raw data into activities and attributes. The attribute calculations are typically complex analytics queries over large volumes of data. It is difficult to beat the flexibility and value of processing large amounts of structured and semi-structured data with BigQuery, especially for this use case.

Data products built from the activities and attributes can be kept in BigQuery or could be created and stored in a different tool depending on requirements.

Orchestration

For orchestration, I use Google Cloud Composer (Apache Airflow). Airflow provides a lot of operators out of the box that connects to all the components in the solution. With Cloud Composer being a Google-hosted version of Apache Airflow, you avoid having the overhead of managing your Airflow cluster.

It is important to note that Airflow is used purely as an orchestrator. No custom python tasks or applications run on Airflow itself. Custom applications or python scripts are packaged and launched on our GKE cluster.

Ensuring Airflow only performs orchestration makes testing custom scripts and applications easier. It also allows you to move to a different orchestration tool if required with minimal effort.

Flexible Compute

A Google Kubernetes Engine (GKE) Autopilot cluster is used for all custom applications and tasks. GKE Autopilot charges for the duration your containers run and scales up and down to meet demand automatically.

The flexibility and scale it provides and the cost savings of not being charged for idle resources help make the solution cost-effective. There is no "ops" or managing of the cluster; simply vast scalable compute resource you can tap into when needed.

Other Components

There are, of course, other components on the periphery not included in the diagram. These would be for managing Infrastructure as Code (IaC) and CI/CD.

For our Infrastructure as Code, we use the open-source version of Terraform for the same reasons mentioned earlier on other components.

For our CI/CD, we currently make use of Github Actions. The steps can be performed via any other CI/CD pipeline tool, so there is nothing that ties the solution to Github.

Final Thoughts

The architecture of the solution is straightforward by design. It did not start out like this. There was a period when additional components were used, such as Kafka, custom integration code, and apps running on Kubernetes.

Those components only added complexity and maintenance overhead while delivering no real additional business value for the problem being solved.

The guiding principle of "delivering maximum business value from development effort while keeping ownership cost as low as possible" helped guide the solution's architecture to what it is currently.

• How do we prove marketing's impact on sales?

• How does the buyer journey for manufacturing customers on the US east coast differ from those in Germany?

• What were the marketing and sales touch points before we won or lost an opportunity?

• Are we wasting advertising budget targeting accounts that are not in-market?

To answer these day-to-day questions, Marketing Directors and CMOs of B2B companies know they need data, which needs to be sourced from all the different tools and platforms they use. So far, so good.

But once they try to get any insight out of that data, the problems usually start. Their "off-the-shelf" marketing data platform turns out to not be flexible enough for their customised systems and processes.

When it comes to data and insights in marketing, it is easy to be seduced by the promises of simple off-the-shelf solutions. However, each company is unique and requires a solution that adapts to them, not one that forces the company to change to fit the tool.

The solution

For the past 18 months, I have been hard at work building a solution that has helped a client answer the above questions and many more. I named it SquareOne because it is the foundation for marketing insights.

The core concept is to take all the data from your tools and platforms and turn it into an activity stream. The activity stream is then processed into attributes for each account you target.

The following diagram shows this concept in its simplest form.

All activities share a standard set of properties, allowing them to be analysed together. They also have custom properties that enable deeper analysis of specific activity types.

Attributes are derived from the activity stream and are calculated daily. This starts from the chosen start date of the solution. Having attributes calculated for each day across history makes analysing attributes over time fast and easy.

While a straightforward concept, it becomes a powerful platform when combined with non-activity type data such as your customer accounts, opportunities, marketing campaigns etc.

SquareOne allows you to identify activities and attributes that serve your business better than any generic off-the-shelf solution. Once in place, you can build data products and enable automation of processes.

Below are some examples of what a recent client achieved with the platform.

Buyer journey

Identify which activities lead to success in the buyer journey.

Each of the activities taking place during the lifetime of an opportunity can be displayed in chronological order while highlighting stage changes as it moves through the marketing and sales funnel.

This data enables you to identify which activities lead to success or failure for specific industries and regions. You can determine a fingerprint for success, so to say.

Multi-touch attribution model

Customise the attribution model to your business needs

Moving on from the buyer journey analysis, you can construct a rich multi-touch attribution model. This model can be custom to your organisation and use concepts such as activity-specific half-lives and weightings to attribute a percentage of your opportunity's value to each of the activities that took place during the sales cycle.

For a client, the model was designed to assign a higher weighting to activities closer to a buyer journey stage change while giving others further from a stage change a lower weighting.

Using such a model shows the impact specific activities have had on sales in monetary terms. It also allows you to roll activities up to the team level to show the contribution of teams or departments to revenue.

Below is an example of what an analysis for an opportunity can look like.

Automate adding accounts to campaigns

Go from running a limited number of campaigns simultaneously to an unlimited number.

Instead of performing manual tasks to pull a list of accounts you think would be a good fit for a marketing campaign, you can specify which attributes an account should have to be included in a campaign.

Having up-to-date attributes calculated for each target account from the combined systems used across marketing and sales allows you to automate campaign cohort selection.

A client used SquareOne to do this and went from running a dozen concurrent campaigns at any point to running 150 at the last count. The platform produced the cohorts of accounts that needed to be added and removed from each campaign daily.

This allowed the marketing team to get the best value for their marketing budget in the following ways:

• It avoided wasting money serving ads to accounts no longer in-market.

• Accounts that were not part of the initial cohort but later showed strong signals of being in-market were not missed as they were added by the system.

• Campaigns could be better targeted to specific groups of accounts due to a rich set of attributes. This avoided wasting resources on broad catch-all campaigns.

Stop trying to adapt to tools

Contact me to discuss what a marketing data platform looks like that adapts to your business and delivers real value.

www.christoolivier.com

Since python does not have any forced folder structures for applications it is easy for those new to Python to get really confused about it all.

Let's dig into what these files do and how they should be used.

Application folder structure

A lot of the posts I have seen have been for applications in the Data Science space. These were python apps written to be executed from the command line without being installed as a package into your Python environment.

Thus, the majority of these applications would be executed from the command line as follows:

$ python awesome.py

The most common folder structure for these applications would normally follow something that looks like this.

|-- AwesomeApp
    |-- __init__.py  <-- Why is this here?
    |-- __main__.py  <-- What does this do?
    |-- awesome.py 
    |-- readme.md
    |-- requirements.txt
    |-- some_module.py
    |-- package
        |-- __init__.py
        |-- package_module.py

As you can see in the above folder structure it is the __init__.py and __main__.py that normally raised questions for me.

There are a whole bunch of ways you can structure your Python applications depending on their use and how you would like your users to install and run them.

Again, we are looking at the above structure as it is used often when __init__.py is at the root level and not because it is the correct folder structure for your project. The correct structure depends on what type of application your want to create and how you want to distribute it.

If you would like to read more about Python project structures, then have a look at these two links.

The Hitchhiker's Guide to Python - Structuring Your Project

Real Python - Python Application Layouts

__init__.py

So, let's get have a look at __init__.py. As per the Python documentation:

The __init__.py files are required to make Python treat directories containing the file as packages. This prevents directories with a common name, such as string, unintentionally hiding valid modules that occur later on the module search path. In the simplest case, __init__.py can just be an empty file, but it can also execute initialization code for the package or set the __all__ variable, described later.

You can read the full detail here in the official documentation.
Python Documentation - Packages

In our example project structure, the top level __init__.py file would only be valid if AwesomeApp was intended to be imported as a package by other applications. Since this is not the case it is unnecessary to have the top level __init__.py file there. It seems this is a mostly harmless habit some people develop. Try to avoid it if you can as it is not the correct use of __init__.py.

For some additional information around Python packages and the use of __init__.py have a look at the following two links.

Real Python - Python Modules and Packages

The Hitchhiker's Guide to Python - Packages

__main__.py

Now let's move on to __main__.py. This file again has a specific purpose in the context of Python packages. When a package is installed in your Python environment and you use the -m argument of Python to call that package then Python will look for the __main__.py file and execute the code in it.

For example, if AwesomeApp was a package installed into our Python environment (which it is not, but go with me on this one), we could run the following on the command line which will then execute the code in __main__.py:

$ python -m AwesomeApp

Since our current app is not designed to be installed as a Python package it also does not make a lot of sense to have __main__.py in the root level folder.

You can read a bit more about __main__.py from the following links.

Real Python - How to Publish an Open-Source Python Package to PyPI

Python Documentation - Top-level script environment

Conclusion

We have looked at the use of __init__.py and __main__.py in the context of a Python application that is not designed to be installed or used as a Python package but as a standalone application.

In this context the use of both __init__.py and __main__.py can cause some confusion as they are primarily used when creating Python packages.

The folder structure we looked at, which seems common in a lot of blog posts, is also not optimal for developing Python packages or applications that are to be distributed via PyPI or to be installed manually via setuptools etc.

Python packages and distributing applications are complex topics that we will look at in a future blog posts.

The interest in Power BI has never been greater than it is today. It was thus no surprise that I would at some point run into a situation where I needed to orchestrate some actions against Power BI from an Airflow data pipeline. Currently no built-in functionality exist in Airflow to interact with Power BI.

In order to make life easier I have created a Power BI plugin for Apache Airflow. The code is available on Github here. Installation instructions and further detail about the plugin can be found in the repository on Github.

Feedback, requests for additional functionality and pull requests are all welcome.