Well, let's start from the beginning.
It's generally accepted that the purpose of art is to make a statement and/or elicit some sort of (emotional) response from the audience. There are some modern theories which might disagree, but let's try to stick to the basics.
As with all art, then, the question becomes this: "What is the end (i.e., intended result/reaction) of this work?"
Transgressive works are based on the idea that "Sacred cows make the best hamburgers."
By violating some norm, you force your audience to reflect on the norm, what it means for this norm to be broken, and whether the norm was justified in the first place.
As an expected result, transgressive art tends to be inherently offensive (and potentially subject to fraud). This is the basis for the idea that an artist can defecate on a carpet and call it art. However, it's important to remember that not all shit stains on a carpet are the considered product of a conscious artistic decision. Sometimes, the dog just couldn't hold it in.
So, in short, transgressive art is only worthwhile and meaningful as long as the transgression leads the audience to respond in a considered fashion. Unfortunately, the inherently subjective nature of the experience of art and the concept of "naturally-occurring art" blurs the lines considerably. Generally, the conclusion is that art is art only when the objective (that is, unchanging) portion of the experience tends to elicit a consistent response from the target audience.
Of course, that's just the baseline. Advertisements, pornography, literature, etc. all fall into that catch-all category. From that point, you need some criterion for judgement as to what value different art might have (Personally, I use Maslow's hierarchy as a general guideline - the higher up the hierarchy the filled need, the more meritorious and valuable the work).
So yes, I would consider transgressive art acceptable, provided it offers something beyond mere shock value.
As for the actual occurrence of transgressive art that actually comes up here, I'd say there's actually very little, and it generally falls into one of four categories:
A: Dark, brooding, and violent (See: much of Doomish's oeuvre)
B: Anything that might conflict with Hanako's status as the moe equivalent of pink slime (See: Mendacium)
C: Anything that implies that Lilly might be anything other than Mary Poppins with a killer rack (See: Two Hallways)
D: Anything that leaves open the possibility that Jigoro or Nomiya might have an ounce of humanity between the two of them
While there are certainly arguments to be made regarding the quality of some of the works, whenever controversy comes up, it relates more to violation in question (e.g., "[Character] would
never do that!"). I will admit that the examples I've presented deal primarily with character interpretation, and some interpretations are in fact more justified than others, but that ties into the next point:
In order to be transgressive, the transgression must offend the sensibilities and mores of the community.
Seems straightforward enough, but it's the "of the community" that's the clincher. We're talking about this community right here. What does that mean?
The basis of this community is KS itself - a visual novel with (softcore) graphic depictions of sex. Because of the origin, much of the community at least has a vague familiarity with 4chan. It is often affectionally referred to as a "cripple porn game." Two of the most popular jokes involve corprophagy and lemon-flavored sodomy, respectively. This is the sort of community that demonstrates the need for the NSFW warning (You should totally check this out, but, fair warning, you could get fired if people know you looked at it.) In other words, it's pretty hard to shock or offend us in the normal way.
Which, then, brings us tot he specific work that sparked this discussion. Was it transgressive? Not really - the people complaining weren't clutching their pearls and harrumphing about how unorthodox it was - they were saying it wasn't terribly clever or funny, leaned too heavily on in-jokes, wasn't very well-written, and was, to be quite frank, bland and uninspired. For that work to have been "transgressive," it would have to have been a transgression of the rules of writing fiction. And, in all fairness, many authors have done a better job of
that without even trying.