The evolutionary transitions of mating systems between outcrossing and self-fertilization are often suggested to associate with the cytological and genomic changes, but the empirical reports are limited in multicellular organisms. Here we used the unicellular zygnematophycean algae, the Closterium peracerosum–strigosum–littorale (C. psl.) complex, to address whether genomic properties such as genome sizes and chromosome numbers are associated with mating system transitions between homothallism (self-fertility) and heterothallism (self-sterility). Phylogenetic analysis revealed the polyphyly of homothallic strains, suggesting multiple transitions between homothallism and heterothallism in the C. psl. complex. Flow cytometry analysis identified a more than 2-fold genome size variation, ranging from 0.53 to 1.42 Gbp, which was positively correlated with chromosome number variation between strains. Although we did not find consistent trends in genome size change and mating system transitions, the mean chromosome sizes tend to be smaller in homothallic strains than in their relative heterothallic strains. This result suggests that homothallic strains possibly have more fragmented chromosomes, which is consistent with the argument that self-fertilizing populations may tolerate more chromosomal rearrangements.