The number of flowers blooming simultaneously on a plant may have profound consequences for reproductive success. Large floral displays often attract more pollinator visits, increasing outcross pollen receipt. However, pollinators frequently probe more flowers in sequence on large displays, potentially increasing self-pollination and reducing pollen export per flower. To better understand how floral display size influences male and female fitness, we manipulated display phenotypes and then used paternity analysis to quantify siring success and selfing rates.

To facilitate unambiguous assignment of paternity, we established four replicate (cloned) arrays of Mimulus ringens, each consisting of genets with unique combinations of homozygous marker genotypes. In each array, we trimmed displays to two, four, eight or 16 flowers. When fruits ripened, we counted the number of seeds per fruit and assigned paternity to 1935 progeny.

Siring success per flower declined sharply with increasing display size, while female success per flower did not vary with display. The rate of self-fertilization increased for large floral displays, but siring losses due to geitonogamous pollen discounting were much greater than siring gains through increased self-fertilization. As display size increased, each additional seed sired through geitonogamous self-pollination was associated with a loss of 9·7 seeds sired through outcrossing.

Although total fitness increased with floral display size, the marginal return on each additional flower declined steadily as display size increased. Therefore, a plant could maximize fitness by producing small displays over a long flowering period, rather than large displays over a brief flowering period.

Floral variation, pollination biology and mating patterns were investigated in sunbird-pollinated Babiana (Iridaceae) species endemic to the Western Cape of South Africa. The group includes several taxa with specialized bird perches and it has been proposed that these function to promote cross-pollination.

Pollinator observations were conducted in 12 populations of five taxa (B. ringens subspp. ringens, australis, B. hirsuta, B. avicularis, B. carminea) and geographic variation in morphological traits investigated in the widespread B. ringens. Experimental pollinations were used to determine the compatibility status, facility for autonomous self-pollination and intensity of pollen limitation in six populations of four taxa. Allozyme markers were employed to investigate mating patterns in four populations of three species.

Sunbirds were the primary pollinators of the five Babiana taxa investigated. Correlated geographical variation in perch size, flower size and stigma–anther separation was evident among B. ringens populations. Experimental pollinations demonstrated that B. ringens and B. avicularis were self-compatible with variation in levels of autonomous self-pollination and weak or no pollen limitation of seed set. In contrast, B. hirsuta was self-incompatible and chronically pollen limited. Estimates of outcrossing rate indicated mixed mating with substantial self-fertilization in all species investigated.

Despite the possession of specialized bird perches in B. ringens and B. avicularis, these structures do not prevent considerable selfing from occurring, probably as a result of autonomous self-pollination. In eastern populations of B. ringens, smaller flowers and reduced herkogamy appear to be associated with a shift to predominant selfing. Relaxed selection on perch function due to increased selfing may explain the increased incidence of apical flowers in some populations.

Sexually dimorphic populations are often located in drier habitats than cosexual populations. Gender plasticity (GP), whereby hermaphrodites alter female and male functions depending on resources, and sex-differential plasticity (SDP) between hermaphrodites and unisexuals are predicted to affect sexual system stability. Here, GP and SDP are evaluated in cosexual and gynodioecious Wurmbea biglandulosa and sub-dioecious and dioecious W. dioica.

GP was evaluated under two resource conditions, compared among sexual systems and assessed as to whether (1) males produced perfect flowers and (2) hermaphrodites altered investment in perfect (female function) and total (male function) flowers. SDP was assessed within sexual systems as differences between sex functions of hermaphrodites vs. unisexuals. Males and hermaphrodites were compared to assess whether size thresholds for female function differed among sexual systems. Plasticity costs were evaluated using correlations between female function and male traits in hermaphrodites, and in W. dioica by comparing hermaphrodite and male regressions between plant size and pollen production.

In dioecious W. dioica no males exhibited GP, whereas 100 % did in gynodioecious and cosexual W. biglandulosa. In sub-dioecious W. dioica, resources affected GP (high, 66 %; low, 42 %). Hermaphrodites in all sexual systems reduced perfect but not total flowers under low resources. Unisexuals were unaffected, demonstrating SDP for female function only. Thresholds for female function were greater in sub-dioecious W. dioica than in W. biglandulosa. Plasticity costs were detected only in sub-dioecious W. dioica.

SDP for female function could assist female establishment in cosexual populations and maintain females in gynodioecious and sub-dioecious populations. Although the absence of male SDP should stabilize sub-dioecy, plasticity costs would render sub-dioecy unstable, favouring canalized males over hermaphrodites. This study highlights the importance of interactions between environmental conditions and hermaphrodite sex expression for the stability of dimorphic sexual systems.

Self-fertilizing taxa are often found at the range margins of their progenitors, where sub-optimal habitats may select for alternative physiological strategies. The extent to which self-fertilization is favoured directly vs. arising indirectly through correlations with other adaptive life history traits is unclear. Trait responses to selection depend on genetic variation and covariation, as well as phenotypic and genetic responses to altered environmental conditions. We tested predictions of the hypothesis that self-fertilization in Mimulus arises through direct selection on physiological and developmental traits that allow seasonal drought escape.

Phenotypic selection on mating system and drought escape traits was estimated in field populations of M. guttatus. In addition, trait phenotype and phenotypic selection were compared between experimental wet and dry soil in two greenhouse populations each of M. guttatus and M. nasutus. Finally, genetic variation and covariation for traits were compared between wet and dry soil treatments in a greenhouse population of M. guttatus.

Consistent with predictions, selection for early flowering was generally stronger than for mating system traits, and selection for early flowering was stronger in dry soil. Inconsistent with predictions, selection for water-use efficiency was largely absent; selection for large flowers was stronger than for drought escape in the field; and most drought escape and mating system traits were not genetically correlated. A positive genetic correlation between flowering time and flower size, which opposed the adaptive contour, emerged only in wet soil, suggesting that variation in water availability may maintain variation in these traits. Plastic responses to soil moisture treatments supported the idea that taxonomic divergence could have been facilitated by plasticity in flowering time and selfing.

The hypothesis that plant mating systems may evolve indirectly via selection on correlated life history characteristics is plausible and warrants increased attention.

The ‘gynodioecy–dioecy pathway’ is considered to be one of the most important evolutionary routes from hermaphroditism to separate sexes (dioecy). Despite a large accumulation of evidence for female seed fertility advantages in gynodioecious species (females and hermaphrodites coexist) in support of the first step in the gynodioecy–dioecy pathway, we still have very little evidence for the second step, i.e. the transition from gynodioecy to dioecy.

We review the literature to evaluate whether basic predictions by theory are supported. To establish whether females' seed fertility advantage and frequencies are sufficient to favour the invasion of males, we review these for species along the gynodioecy–dioecy pathway published in the last 5 years. We then review the empirical evidence for predictions deriving from the second step, i.e. hermaphrodites' male fertility increases with female frequency, selection favours greater male fertility in hermaphrodites in gynodioecious species, and, where males and hermaphrodites coexist with females (subdioecy), males have greater male fertility than hermaphrodites. We review how genetic control and certain ecological features (pollen limitation, selfing, plasticity in sex expression and antagonists) influence the trajectory of a population along the gynodioecy–dioecy pathway.

Females tend to have greater seed fertility advantages over hermaphrodites where the two coexist, and this advantage is positively correlated with female frequency across species, as predicted by theory. A limited number of studies in subdioecious species have demonstrated that males have an advantage over hermaphrodites, as also predicted by theory. However, less evidence exists for phenotypic selection to increase male traits of hermaphrodites or for increasing male function of hermaphrodites in populations with high female frequency. A few key case studies underline the importance of examining multiple components of male fertility and the roles of pollen limitation, selfing and plasticity, when evaluating advantages. We conclude that we do not yet have a full understanding of the transition from gynodioecy to dioecy.