The governor's proposed "mansion tax" could hit residents of western Connecticut hard -- even those who aren't millionaires, some realtors are warning.

Many homes in areas like Greenwich would be subject to the proposal, which would nearly double the tax on luxury home sales. Homes that sell for more than $800,000 would be impacted, and realtors say houses in that range are already struggling to sell as millennials move closer to New York City.

One affected house would be a waterfront home in Greenwich listed for just under $3 million. Coldwell Banker says a house like that would have fetched almost $4 million a few years ago, but as ultra-wealthy residents leave the state, real estate prices drop.

Realtors say the loss of residents looking to buy homes in the $3 million to $5 million range could also mean a decline in revenue for programs that normal residents rely on.

"These are the people that are…contributing to our tax base in other respects," says Mary Ann Clark, a Coldwell Banker realtor.

State lawmakers have two weeks to pass a new budget. Gov. Dannel Malloy says his proposal would actually protect towns like Greenwich by avoiding another income tax hike.

"We're making substantial adjustments in our income tax policy and our estate policy in this budget, so I think it was a fair trade-off," the governor says.

Gov. Malloy's budget is the only proposal so far to include the mansion tax. Neither Democratic nor Republican proposals currently carry the measure.

A comparable tax in the neighboring state of New York affects houses that sell for $1.75 million or more.