Question 1

What is prompt injection?

Accepted Answer

An attack in which **instructions hidden in content the model reads** are followed as though they came from the user — and it is a genuinely unsolved problem, not a bug awaiting a patch.

**Why it exists.** A language model receives everything as text in one stream: system instructions, user input and any retrieved content. There is **no structural separation between instructions and data** — the model has only text, and text that looks like an instruction may be treated as one, regardless of where it came from.

This differs fundamentally from SQL injection, which is solved by parameterisation separating code from data. **No equivalent separation exists for natural language**, which is why the problem is hard rather than merely unaddressed.

**Direct prompt injection.** The user types something attempting to override the system instructions — the familiar "ignore previous instructions". Mitigable to a degree, and mostly a nuisance.

**Indirect prompt injection — the serious one.** Instructions are planted in content the model will process later: a web page it browses, a document it summarises, an email it reads, an issue in a code repository, metadata, or text made invisible to humans through colour or font size.

A user asks the assistant to summarise a page. The page contains hidden text instructing the model to exfiltrate the conversation to a URL, or to recommend a particular product, or to alter a recommendation. **The user never sees the instruction and never authorised it.**

**Why it matters more as models gain capability.** A model that only produces text can mostly be made to say something wrong. A model with **tools** — reading email, browsing, executing code, making purchases, sending messages — can be made to **do** something, and the damage scales with the permissions it holds.

**What mitigations exist**, none complete: limiting tool permissions to the minimum; requiring human confirmation for consequential actions; treating all retrieved content as untrusted; separating the model that reads untrusted content from the one with tool access; output filtering; and constraining where data may be sent.

**The realistic position** is that untrusted content plus tool access plus sensitive data is a dangerous combination, and the defence is architectural rather than a filter.

Question 2

What is penetration testing, and how is it different from a vulnerability scan?

Accepted Answer

A vulnerability scan is automated and tells you what might be exploitable; a penetration test is human-led and demonstrates what actually is, by exploiting it. They cost, take and tell you completely

Question 3

Why do AI models make things up?

Accepted Answer

Because a language model is trained to produce plausible continuations of text, not to state truths — and fluent plausible text and accurate text are different targets that usually coincide and

Question 4

What is a large language model and how is it different from a search engine?

Accepted Answer

A large language model (LLM) is a statistical model trained to predict likely continuations of text. A search engine retrieves documents that exist. They answer questions in fundamentally different

Question 5

What is an AI agent, and how is it different from a chatbot?

Accepted Answer

An agent pursues a goal over multiple steps, deciding what to do next and taking actions in the world; a chatbot responds to messages. The difference is autonomy and the ability to act rather than

Question 6

How are AI models actually evaluated, and what do benchmarks mean?

Accepted Answer

Against standardised test sets with published scores — and the honest position is that benchmarks measure something real, measure it narrowly, and are increasingly unreliable as headline comparisons

Question 7

What is a supply chain attack?

Accepted Answer

A supply chain attack compromises an organisation indirectly, by attacking something it trusts and installs — a software vendor, a library, an update mechanism or a service provider. Rather than

Related Questions