What an LLM Actually Is, and What It Still Gets Wrong

Around 12 per cent of Australian businesses used AI in the workplace in 2024-25, on the ABS numbers, up from the very low rates of 2021-22 when ChatGPT did not exist. Among large businesses it went from 9 per cent to 35.

So a lot of the country is having this conversation for the first time, and it usually arrives as a specific question. Can this thing fill in our CRM fields on its own? Can it draft the contract?

Answering either one takes a bit of understanding of how a large language model works, and rather more about where it breaks. This is the plain version. No acronyms you have to look up.

What an LLM is, in one paragraph

A large language model is a text prediction system trained on an enormous amount of human language. You give it text, it predicts what comes next, one token at a time, from the patterns in what it was trained on. That is the whole mechanism. It is very good at writing, summarising, classifying and working through problems made of language, and every limit below follows from prediction being all it does.

What LLMs cannot do, and where people go wrong

This is the part that gets glossed over in most AI coverage.

They do not know what is happening right now. Unless the model has a live search tool attached, its knowledge stops at a training cutoff date. Ask about last week and you get an honest "I don't know" or, worse, a confidently wrong answer. If your process depends on current prices, current stock or current law, the model has to be handed that data. It cannot recall what it never saw.

They hallucinate. That is the term for output that reads as completely plausible and is factually wrong. It happens because the model is predicting the most likely next token rather than verifying anything. Where accuracy matters, check the output against a primary source. It is also the reason "draft the contract" is a worse first use case than it looks. A clause that sounds like a clause is exactly what the machine is built to produce.

They do not understand. LLMs pattern match language extremely well. They are not reasoning the way a person reasons, they hold no beliefs, and they keep nothing between conversations unless built to. Text prediction at this standard is remarkable, and it is still text prediction.

They reflect their training data. Biases, gaps and errors in the training data turn up in the output. A model trained mostly on English performs differently in other languages, and one trained mostly on formal writing reads differently from one trained on conversation.

What that means for a process you would actually run

Context is the lever. The model predicts from what it has been given, so what you give it is the biggest thing you control. Clear instructions, the right background, a specific output format. That is why prompt engineering turned out to be a genuine skill rather than a fad.

A general model knows nothing about your business. It has never met your customers, your product or your process. To get output worth reading you put that context in the prompt, connect the model to live data through an AI API, or build a retrieval step that pulls the right record at query time. HubSpot's AI features do the connecting part natively against your CRM.

Human review is not optional for consequential work. Anything that goes to a customer, gets published, or informs a decision that costs money gets a person's eyes on it. LLMs are capable assistants, not autonomous decision makers.

The model matters for the job. Some are better at long reasoning, some are faster and cheaper at volume, some follow precise instructions more reliably. Knowing roughly what each is good at beats defaulting to whichever one you heard about first.

Which answers the two questions from the top. Auto-filling CRM fields is a good fit: high volume, language in and language out, the model can be pointed at the record it is summarising, and a wrong answer is visible and cheap to fix. Drafting a contract fails every one of those tests. Low volume, expensive to get wrong, and the failure mode is text that looks right.

There is now an Australian framework for this

The National AI Centre published its Guidance for AI Adoption in October 2025, an update to the 2024 Voluntary AI Safety Standard. It runs on six essential practices: decide who is accountable, understand the impacts and plan for them, measure and manage the risk, share the essential information, test and monitor, and keep meaningful human control. There is a foundations version for low risk use and a longer implementation one for the harder cases.

Held against the CRM example, most of that is ordinary operational hygiene. Someone owns the feature. You have thought about what happens if it writes rubbish into 400 records. Your team knows the field was filled by a model. Somebody checks a sample each week. If you already pay for support on your portal, that is where the work belongs. Worth a look at the managed HubSpot support models this kind of AI feature sits inside.

The one thing worth remembering

An LLM writes, summarises, classifies and works through language problems well. It does not know what happened today, it can be wrong with total confidence, and it is not a substitute for judgement on anything that matters.

The businesses getting the most out of them are not the ones most excited about AI. They picked one specific high volume job made of language, writing, classifying, summarising, responding, connected a model to it, and put a person in the loop where being wrong would cost something.

So here is the question worth taking to your next planning session. What is the highest volume language job your team does every week, and would anyone notice if the model got it wrong?

If you want to work that out inside your HubSpot setup, give us a nudge. Or join our HubSpot User Group, follow us on Instagram and subscribe on YouTube.