Although the wealth of some of the inhabitants appears to be stratospheric (e.g., sport team owners), most of Del Mar's housing stock was built in the 1970s or early 1980s and thus is not glitzy by 21st Century standards. Several of the better sort of NFL quarterbacks are said to live there. Many of the homes near the beach seem to be inhabited only part time by rich people who own multiple homes. (By the way, security systems are unobtrusive but omnipresent, so don't get any ideas.)
The key that determines Del Mar's sociological make-up is that there is a reef offshore that keeps the surf from getting terribly dangerous, so it's ideal for people looking for a safe beach for their young children or grandchildren. Dogs are free to run off leashes on the northern stretch of Del Mar's beach, just south of the lagoon that leads to the famous horserace track. So, it is about as genteel / respectable as Southern California gets. The Romneys have a house a few miles to the south in La Jolla, and they would fit in in Del Mar: a rich, grandchild-oriented, outdoorsy family.
The public can pay to park and use Del Mar beach, but the daytrippers tend to be people who are sympatico with the locals. So, what are two things you don't see much of on the beach at Del Mar?
1. Pit bulls (instead, lots of retrievers)