When transmitting polarised light through histological brain sections, different types of diattenuation (polarisation-dependent attenuation of light) can be observed: In some brain regions, the light is minimally attenuated when it is polarised parallel to the nerve fibres (referred to as D+), in others, it is maximally attenuated (referred to as D-). The underlying mechanisms of these effects and their relationship to tissue properties were so far unknown. Here, we demonstrate in experimental studies that diattenuation of both types D+ and D- can be observed in brain tissue samples from different species (rodent, monkey, and human) and that the strength and type of diattenuation depend on the nerve fibre orientations. By combining finite-difference time-domain simulations and analytical modelling, we explain the observed diattenuation effects and show that they are caused both by anisotropic absorption (dichroism) and by anisotropic light scattering. Our studies demonstrate that the diattenuation signal depends not only on the nerve fibre orientations but also on other brain tissue properties like tissue homogeneity, fibre size, and myelin sheath thickness. This allows to use the diattenuation signal to distinguish between brain regions with different tissue properties and establishes Diattenuation Imaging as a valuable imaging technique.