Large brains are thought to result from selection for cognitive benefits, but how enhanced cognition leads to increased fitness remains poorly understood. One explanation is that increased cognitive ability results in improved monitoring and assessment of predator threats. Here, we use male and female guppies (Poecilia reticulata), artificially selected for large and small brain size, to provide an experimental evaluation of this hypothesis. We examined their behavioural response as singletons, pairs or shoals of four towards a model predator. Large-brained females, but not males, spent less time performing predator inspections, an inherently risky behaviour. Video analysis revealed that large-brained females were further away from the model predator when in pairs but that they habituated quickly towards the model when in shoals of four. Males stayed further away from the predator model than females but again we found no brain size effect in males. We conclude that differences in brain size affect the female predator response. Large-brained females might be able to assess risk better or need less sensory information to reach an accurate conclusion. Our results provide experimental support for the general idea that predation pressure is likely to be important for the evolution of brain size in prey species.