Hi Annie, yes, in general, readability suffers when type/text is set on a high contrast back ground. It causes eye fatigue ~ which is why the majority cannot read text set in reverse (white on back) for extended periods of time.
For those sensitive to light/brightness, removing blue light for warmer backgrounds helps protect eye health. If one has other impairments, they will have settings that work best for them. In general, contrast is the issue and too much brightness.