Dark night of the soul

Apropos San Juan’s “Noche oscura,” I find it interesting to see how the phrase “dark night of the soul” has migrated into popular culture for all the wrong reasons. Wikipedia writes “It has become an expression used to describe a phase in a person's spiritual life, a metaphor for a certain loneliness and desolation.” “Dark night of the soul” is a cool phrase that seems to evoke existential despair. Hence, an interview with Christian Bale about the new Batman movie is titled … “Dark Knight of the Soul.”

But in my reading of San Juan’s poem, the dark night is not fearsome; certainly not desolate. The dark night is quiet, absent of distractions and thus enables mystic union. If the soul is lonely it is a necessary, even welcome loneliness. The house is quiet, the soul is "dichosa." How did this essentially positive, optimistic phrase become a metaphor for harrowing angst? Perhaps there’s something in San Juan’s lengthy treatise, which I guess I need to read carefully. Any thoughts?