On the rental side, 35% of the rental stock in the US is single-family homes, and single-family vacancy is the lowest in the US among all property types. Single-family rent growth is especially strong in the west at 7% to 8%, said Miller. "Low vacancy is expected for a long time, and rental rates are expected to rise."

Miller also cited stats from CBRE showing that San Diego has among the lowest cap rates on single-family detached homes in the US, and deliveries are expected to increase dramatically in 2017 and 2018.

He also referenced the London Group, saying, "We have a housing shortage of 50,000 to 118,000 units." Most of what gets approved for development is upper-income housing, with very little lower-priced housing in the planning stages.

Next came the panel titled, "Housing Affordability and Its Impact on the Economy," which was eye-opening. Moderator Stephen Doyle, president of Sandy Point Properties, said that San Diego is land constrained, and supply in San Diego can't go farther east than where the San Diego County Water Authority services, further restricting growth in the county. Deborah Ruane, SVP of San Diego Housing Commission, said her organization is trying to convert market-rate housing to affordable housing but has had to ask lenders for more money because construction costs have risen so dramatically. "The government is the driver for housing-cost increases," she said.

In response to this problem, the SDHC produced a report that identifies the key factors slowing down housing growth in San Diego and the efforts that could help growth, from deferring development fees to supporting CEQA reform to stopping the City from requiring commercial space to be built below residential—which requires tools residential developers don't possess. "Leadership will play a huge role in this. We need to set a housing goal" and try to obtain it.

Nathan Moeder, a principal with the London Group, referred to his company's recent white paper addressing housing affordability and the sea change in San Diego of middle-class housing becoming unaffordable. "It's become a case of the haves and the have-nots, and the middle class are the have-nots." The question of how to add more housing units since we're out of vacant land can be answered by San Diego accepting more density, said Moeder. Without it, companies will have difficulty recruiting talent since there will be nowhere for them to live within a reasonable distance from where they work and in desirable neighborhoods. "We need the political will to get things done."

Michael Innis-Thompson, managing director of community lending for MUFG Union Bank, N.A., said one of the main issues that affects home affordability is bad credit, since the cost to borrow with poor credit is high. Still, low-down-payment loans are becoming more popular, and his firm originates these mortgages for sale to the secondary market. "Interest rates will go up to 4.5% in 2016, close to 5.2% in 2017 and 5.7% in 2018," said Innis-Thompson, based on data from the MBA. He said the 10-Year Treasury is more of an indicator of how mortgage interest rates behave than any pending rate increases by the Fed. "Mortgage rates react to how the 10-Year Treasury performs."