Hey, good news
I followed your advice and went to a local supply shop where they seemed to have rubber grommets. They had some variety there, since they were cheap I grabbed a few of different sizes and tried it out at home:
You can see the small one on the right of the connector, that's the original one with some silicon on it. The one plugged to the connector is the one I ended up using, and now is holding perfectly!