Trying to buy or rent a home? Good luck finding one – and affording it. Now, a battle is brewing over how to address Connecticut's housing crisis.

Gov. Ned Lamont thinks the answer is in Norwalk. The city's Colonial Village housing complex is expanding. Next year, 69 new homes will be built on an empty lot – thanks in part to $1.8 million in state funding. The new complex even includes a child care center.

"That's just what this facility is all about," said Lamont. "You can be working at Walmart, a local restaurant. You can afford to live in the town where you're working."

Lamont wants to build 6,400 new housing units across the state. To accomplish it, his budget proposal spends $600 million on housing – mostly affordable and workforce units.

But actually building that housing is a challenge, especially in wealthy suburban towns.

In New Canaan, just a few miles from Norwalk, the average home cost $1.9 million in March, according to Realtor.com. Only 3% of the town's housing is considered "affordable."

New Canaan's housing authority just finished Canaan Parish, a glistening affordable housing complex where Section 8 tenants pay as little as $250 per month. First Selectman Kevin Moynihan said he'd like to build more projects, but basic economics make it a challenge.

"The cost of land in New Canaan is so expensive," said Moynihan, a Republican. "Generally, a property down here that goes for nine or 10 million dollars would be a great location, but someone's got to pay that nine to 10 million dollars. If the state gave us money to buy land, that would help."

Private developers are willing to shoulder the cost, but accuse wealthier towns of shutting them out.

"Minimum lot size requirements, setback requirements, and other kinds of exclusionary zoning that require the builder to basically build a larger lawn," Robert Dietz, an economist with the National Association of Home Builders, recently told state lawmakers.

In New Canaan, the town's Planning and Zoning Commission recently rejected a plan to build 20 apartments behind a downtown home because it would interfere with the "consistent rhythm and coherence of the streetscape within the historic district." Officials also raised concerns about the project's height.

Legislators are considering several housing bills this year. Instead of mandates, some proposals offer communities incentives to ease zoning restrictions. But critics believe the incentives are heavy-handed.

"You have to build an onerous density around a train station, and if you don't opt-in, you are de-prioritized for numerous infrastructure spending," said Maria Weingarten with the group CT169Strong, which successfully defeated similar legislation last year.

Weingarten believes more density could backfire, raising average rents for everyone.

"That's the concern here is, that all we're doing is creating market-value rentals, not the real need is creating affordable single-family homes, which is really the American dream," she said.

It's a "dream" that's an unaffordable nightmare for many Connecticut home buyers.