As a grad student at a STEM school, I work and study alongside students from every corner of the world who have English skills ranging from perfect with an adorable accent (New Zealand!) down to "how did they pass the TOEFL!?".
My school is located a significant distance from where I was previously living. Since I was only staying for a couple of years, I left most of my stuff in storage and moved into (furnished) graduate housing. Grad student housing is generally reserved for non-local students and is probably about 75% international. My first 3 roomates and I were from four continents, so I lived with non-native English speakers as well.
Growing up, verbal communication was fairly low on my list of interests and/or abilities. I tended to mumble, and as a massive geek/bookworm, I had a vocabulary larger than most adults. Even in college, I was accused of using too many big words. Working as a consultant took care of the mumbling in short order. But I have to admit to still using big words. Certain terms have a very specific conotation and I prefer to direct my speaking and writing so that it is exactly what I mean.
Now that I have long conversations with friends with varying English abilities, I've become adept at simplifying my language and using lots of gestures. (gar! more big words!) I've also had lots of practice describing things in a different way if someone stalls out on a particular concept. I tend to emote more when talking to non-native English speakers so that if someone misses the words, they still get what I'm trying to say. I do try to avoid seeming condescending. The folks I talk to are highly intelligent and knowledgeable - we just have to figure out a way to keep our languages from interfering with what we want to say.