Can a novel be libellous?
Yes. Calling something fiction is not a defence, and publishers take this seriously enough that legal reads are routine for memoir, and not unknown for novels.
The basic test. Defamation generally requires that a statement identifies a real person, is published, and damages their reputation. The question for fiction is identification: would a reasonable reader who knows the person conclude the character is them? The author's intention is not the test — a disclaimer that characters are fictional does not protect against a portrait that is recognisable.
What makes identification likely: an unusual combination of job, location, physical description, personal history and relationships; a distinctive real incident; retained names or near-names; and small communities where few people fit the description.
The factors that raise the risk:
Real people appearing as characters, particularly living ones in invented scenes.
Memoir and autofiction, where the events are asserted as true and other people are necessarily depicted.
Composite characters, which are frequently more dangerous than people assume, because a composite can be recognisable as one of its sources while containing material that is untrue of them.
Thinly disguised institutions, since organisations can sue in some jurisdictions.
Serious allegations — criminality, dishonesty, professional misconduct, sexual behaviour — which is what most actual claims concern.
The defences vary by jurisdiction and generally include truth, honest opinion, and publication on a matter of public interest. Notably, the dead cannot generally be defamed, though their surviving relatives may have other claims, and privacy and data protection are separate routes that fiction does not avoid either.
Jurisdiction matters enormously. Thresholds, burdens of proof and costs differ substantially between countries, and publication online is widely read.
What publishers actually do: a legal read, changing identifying details, obtaining releases, adding context, and occasionally cutting material. Your contract almost certainly makes you liable to indemnify the publisher, which is the clause worth reading before you sign.
General information, not legal advice — take advice on a specific manuscript.