If your only issue was personality mismatch then I would say that you should still go for dermatology, as I don't believe that stereotypes or personalities should influence your decision of which specialty you choose. Any doctor can set up any practice setting how they want, so it doesn't matter what all the other specialists in that field are like.
However, when you say "I found clinic unbelievably draining despite the relatively short hours I worked" that's a huge red flag and is highly suggestive that you would not enjoy the day to day of dermatology. If you hate it and feel drained as a medical student (where you essentially do next to nothing and are just there to learn), you would be absolutely crushed in residency and eventually in practice when you realize that you have to slog through 30+ patients a day, do all the procedures, type up 30+ notes, actually deal with the difficult patients (anxious, morgellons, body dysmorphia, needy, angry patients) and do callbacks and follow ups for all your biopsies, labs, and medication refills. It's exhausting, even for people who enjoy the field. If you can't handle that as a medical student when you're protected from 99% of the scutwork and slog, I don't see you being happy in residency and beyond. The people who do well in dermatology tend to thrive on (or at least not mind) the breakneck pace and high volume of the specialty.
As has been mentioned before, you can find "immense lifestyle benefits" in a ton of other fields. You need to find something you enjoy or at least don't mind doing for the next 40 years, and dermatology doesn't sound like a match for you. If you want an unbeatable lifestyle and want a slower more relaxed clinic you should definitely consider rheumatology, endocrinology, PM&R, or psychiatry. There are way more options than just those for great lifestyle specialties, but if you're a clinic person and want to have a more relaxed pace while still being a specialist and in high demand, you can't go wrong with any of those fields.