Whether young infants can exploit socio-pragmatic information to interpret new words is a matter of debate. Based on findings and theories from the action interpretation literature, we hypothesized that 12-month-olds should distinguish communicative object-directed actions expressing reference from instrumental object-directed actions indicative of one’s goals, and selectively use the former to identify referents of novel linguistic expressions. This hypothesis was tested across four eye-tracking experiments. Infants watched pairs of unfamiliar objects, one of which was first targeted by either a communicative or an instrumental action and then labeled with a novel word. As predicted, infants fast-mapped the novel words onto the targeted object after pointing (Experiments 1 and 4) but not after grasping (Experiment 2) unless the grasping action was preceded by an ostensive signal (Experiment 3). We also found that whenever infants mapped a novel word onto the object indicated by the action, they tended to map a different novel word onto the distractor object, displaying a mutual exclusivity effect. In sum, communicative actions enabled infants both to rapidly map labels onto highlighted objects and to carry out mutual-exclusivity inferences about other words. This reliance on nonverbal action interpretation in the disambiguation of novel words indicates that socio-pragmatic inferences about reference likely supplement associative and statistical learning mechanisms from the outset of word learning.