Background: Brain size is expected to evolve by a balance between cognitive benefits and energetic costs. Several influential hypotheses have suggested that large brains may be especially beneficial in social contexts. Group living and competition may pose unique cognitive challenges to individuals and favor the evolution of increased cognitive ability. Evidence comes from comparative studies on the link between social complexity and brain morphology, but the strength of empirical support has recently been challenged. In addition, the behavioral mechanisms that would link cognitive ability to sociality are rarely studied. Here we take an alternative approach and investigate experimentally how brain size can relate to the social competence of individuals within species, a problem that so far has remained unresolved. We use the unique guppy brain size selection line model system to evaluate whether large brains are advantageous by allowing individuals to better assess their performance in a social contest situation. Based on theoretical literature, we predict that contest duration should depend on the brain size of the loser, as it is the capitulation of the losing individual that ends the fight. Results: First, we show that studying the movement of competitors during contests allows for precise estimation of the dominance timeline in guppies, even when overt aggression is typically one-sided and delayed. Second, we staged contests between pairs of male that had been artificially selected for large and small relative brain size, with demonstrated differences in cognitive ability. We show that dominance was established much earlier in contests with large-brained losers, whereas the brain size of the winner had no effect. Following our prediction, large-brained individuals gave up more quickly when they were going to lose. Conclusions: These results suggest that large-brained individuals assess their performance in contests better and that social competence indeed can depend on brain size. Conflict resolution may therefore be an important behavioral mechanism behind macro-evolutionary patterns between sociality and brain size. Since conflict is ubiquitous among group-living animals, the possible effects of the social environment on the evolution of cognition may be more broadly applicable than previously thought.