I don't. I did some reading on it before hand because I had that concern, and it doesn't "kill" it imo based on what I found, it does hinder it to an extent though I agree.Do you have quick disconnects on the sway bar? It will totally kill the Tremors off-road abilities.
Given the majority of my driving it's good, if I went to a purposeful event that i knew was big rocks and dips, I'd pull it off. For most off-road though, and on-road, it'll be fine.
What kind of quick disconnect screw would you use?