This is fast, and only a bit ugly. There are a few problems with it, though. First, you don't have the usual set-theoretic operations like intersection, difference, union, and so on. There's no easy way to find the cardinality of the set, unless you maintain a separate counter. You can fix all these by writing a custom set class, but that would take effort.
What we need, to make a general-purpose set data type, is to use unordered arrays for storage. At least, that's the best option I could come up with that didn't involve making an extension in C. I've written a library to do this, and tested it with Node.js, although it should work on any web browser as well. Here's the repository on GitHub. The code can be very simple: