Aposematic coloration, often called “warning coloration”, is the presence of bright or conspicuous colors or patterns in animals that are toxic, noxious, dangerous, or poisonous to predators. Here’s an example from Wikipedia, the granular poison frog (Oophaga granulifera). Like many dendrobatid frogs, this has a number of poison alkaloids in its skin, and they have been used in Central and South America to tip arrows or darts, which can kill mammals. Any predator that tried to eat one of these would probably be dead, or at least very ill.
My own frog, Atelopus coynei, looks conspicuous too [but see Lou Jost’s comment below], and may be toxic, but I don’t think people know anything about that:

But of course far more animals than amphibians are aposematic. The skunk advertises its toxicity with a pair of conspicuous stripes. Many insects, like ladybugs and some leipidopterans, are also aposematic and toxic, including at least one bird species: see here for a Google image search of aposematic animals.
The colors and patterns, as the name implies, gives their bearers an evolutionary advantage over their presumably camouflaged ancestors, for predators will deliberately avoid the pattern, usually because they’ve learned to recognize and stay away from it because of previous unpleasant experiences. (The avoidance can also be evolved rather than learned, as you’ll see if you think about it. Even if eating one of these kills you, individual predators having less of a propensity to attack the pattern would be favored.) Usually, however, learning is involved.
But to get that advantage, the aposematic species has to be sufficiently numerous to afford predators a chance to learn and then avoid the next aposematic animal. And this creates an evolutionary problem.
We are pretty sure that aposematic species evolved from camouflaged ones. To get the warning coloration started, there have to be mutations in the camouflaged population that produce individuals with bright colors and patterns, at least in incipient form.
And that’s the rub: the first mutant individual stands a higher chance of being attacked and killed than do cryptic individuals. Even if it’s toxic, it may still get killed or injured by being attacked for being a novel, conspicuous creature. So how does the adaptation ever spread through the population from a rare initial state?
Previously, as described in the excellent Nature News & Views summary by Tim Caro below (click to read), we had a few answers:
1.) The trait could evolved by kin selection in gregarious animals. While the first mutant individual might be attacked, it might be part of a group of relatives that share that aposematic mutation. Assuming the predator learns to avoid the pattern after killing or hurting the first individual, it would avoid its similarly-colored kin, and that is a form of kin selection on the color/pattern genes that could make them spread.
2.) The trait could have evolved from a state that was conspicuous but not as conspicuous as the animals above. But this runs into the same problem as #1!
3.) The attacked aposematic mutant could avoid being killed by the predator because it smells or tastes bad, or is injured only slightly. If the predator learns from one experience (and some do), then that individual would henceforth be protected from predation, perhaps giving the mutant color/pattern gene an advantage. This seems somewhat likely, and could be tested by exposing naive predators to aposematic prey.
4.) Predators might avoid novel colors or patterns in general since the hunters have a search image for edible species. As Caro says, there’s some evidence for this, too.
But now, in his summary of the original paper, Caro describes a fifth hypothesis that is described in the Science paper below that. The authors test this interesting hypothesis using phylogenetic data, and it seems to be supported.
Click the original Science paper below to read about the novel hypothesis for the evolution of aposematism. The authors test it in amphibians, but may hold for other creatures as well. You can also find the pdf here , and the reference is at the bottom.
Again, I’ll try to be brief, but may not succeed. The authors’ hypothesis, which is very clever, is that full aposematic coloration may have evolved, at least in amphibians from an earlier state where it wasn’t clearly visible to predators. This could involve the colors/patterns starting their evolution on the BOTTOM (ventral) side of the animal, which wouldn’t draw attention until the animal was attacked, at which point it could flash its pattern and possibly startle the predator (the predator could also learn from a brief encounter that the prey was toxic). And the bottom-colored state could itself be of two types: small patches on the ventral surface (PV) or a fully colored ventral surface (FV). This is in contrast to an animal that is fully colored all over its body.
Once the predator started learning what the color/pattern means from the animals that had it on their belly, then the color could evolve to cover the animal, making it fully aposematic.
But how do you test this hypothesis? Well, you could see if predators learn to avoid toxic amphibians that had color patches painted on their belly, but there are few amphibians that are toxic and lack aposematic coloration. No, the authors tested their hypothesis by doing phylogenetic reconstruction: they used living species and their known family tree to deduce what the color/pattern of the ancestors were. This kind of reconstruction, which makes sense if you have enough data, is increasingly used to study evolution.
And so Loeffler-Henry et al. did a big reconstruction of the evolutionary history of amphibians, many of whom were aposematically colored. They used 1106 species, putting each in one of five evolutionary categories:
species cryptic (camouflaged; “cry” in photo below)
species PV (ventral side partly aposematic)
species FV (ventral side fully aposematic)
species fully aposematic all over its body (“conspicuous” or “con” in photo below)
species polymorphic (some individuals are aposematic, others not). There aren’t many of these, and I won’t go into why they are supposed to exist.
Here’s a photo from the paper showing four of the five states (a polymorphic species isn’t shown):

And here’s the reconstruction of the phylogeny showing the position in the family tree of each of the five states. Click to enlarge:

There’s a pie diagram at each node of the tree showing the probability that that ancestor had one of the five states scored. I won’t go into the methods for deriving probabilities (in truth, I don’t understand them); but her are the salient points:
1.) Ancestors tend to be cryptic (camouflaged; gray dots), with the possible exception of some salamanders. This comports with the evolutionary view that aposematic coloration was not an ancestral condition but evolved as a defensive adaptation to deter predators.
2.) Full aposematism—the orange state—didn’t appear until later in amphibians, and
3.) . . . it did so generally going through an intermediate state of aposematic coloration on the belly (purple and red species)
4.) The preponderance of purple circles earlier than red ones suggests that the condition of full ventral coloration was preceded in time by the evolution of partial ventral coloration: patches of color that could be flashed but are still less conspicuous to predators than fully belly coloration. This suggestion is supported by statistical analysis of the likelihood of the models, but I’ll skip that.
Now this is an analysis of amphibians, but could apply equally well to other species. In fact, many butterflies that have warning coloration have it on their rear wings, which are covered up when they’re resting. It’s only when they fly, or when a predator startles them, that the aposematic coloration is revealed. Here’s an example: an aposematic butterfly from Ray Cannon’s Nature Notes. It’s the common birdwing (Troides helena), known to be very poisonous since the larvae feed on plants containing toxic aristolochic acids.
And here’s a fully aposematic butterfly:

For a long time the evolution of aposematic coloration posed the problem of what evolutionists call an “adaptive valley”: how do you get from one adaptive state (toxic but camouflaged) to a presumably more adapted state (toxic and brightly colored), when the intermediate evolutionary stage (the first mutant individual) was at a disadvantage: mired in an adaptive valley? This could not occur by natural selecction since selection cannot favor the less adapted (here, “less avoided”) individuals.
The authors propose a solution to this: an adaptive valley wasn’t crossed because the intermediate state—ventral coloration—did confer a selective advantage on the first mutant individuals.
The authors end the paper by suggesting that their scenario could apply to many species; and it well could:
. . . macroevolutionary studies on animal coloration should take into account these underappreciated hidden signals, which are both common and widespread across the animal kingdom, to advance our understanding of the evolution of antipredator defenses. Indeed, many animal taxa such as snakes, fishes, and a variety of arthropods (see fig. S12 for example groups) include species that are cryptic, are aposematic, and have hidden conspicuous signals. We therefore encourage follow-up studies in other taxa to evaluate the generality of the stepping-stone hypothesis as a route to aposematism.
_________________
Loeffler, K., C. Kang, and T. N. Sherratt. 2023. Evolutionary transitions from camouflage to aposematism: Hidden signals play a pivotal role. Science 379:1136-1140. DOI: 10.1126/science.ade5156