Deep Learning (DL) is a branch of Artificial Intelligence (AI) that focuses on training deep neural networks. Thanks to their ability to process large amounts of data, these networks have achieved remarkable results across a variety of fields. Despite these successes, DL still faces several limitations that hinder its adoption in real-world scenarios. This thesis addresses three key challenges: reducing the need for supervision, defending against adversarial attacks, and explaining neural network behavior. The first two challenges are tackled through learning from constraints, which incorporates domain knowledge to guide the learning process and enhance model robustness. The third challenge, on the other hand, is addressed using learning of constraints, which helps identify and formalize logical relationships among learned tasks, thereby providing interpretable explanations of the networks' behavior.