In this paper, we outline the different dimensions of sexism by grounding them in their implementation in psychological scales. From the scales, we derive a codebook for sexism in social media which we use to annotate existing and novel datasets. Next, we leverage the annotated datasets to generate adversarial examples, and test the reliability of sexism detection methods.