You may have heard of the idea of a genetic bottleneck. It is when genetic variation is lost by sampling a very small subset of a population. Just by chance, the frequencies of alleles on the subset of the population will be different to those in the whole population, and overall genetic variation is likely to be lost by population sampling.
This is why small population sizes are not good, and species on conservation lists are required to not drop to too small population sizes. The limited genetic variation remaining in a small population poses a risk because it makes the species susceptible to whatever relies on genetic variation for adaptation, such as environmental changes or diseases.
Here is a visualisation of a bottleneck.