Silverstein holds a ground lease with the Port Authority of New York and New Jersey on the WTC site. The trade center is not zoned for residential uses and various legislation in New York and New Jersey would need to be enacted for any changes to be made. Silverstein estimates he spends $300,000 a day on the ground lease.

"Over the last four years, the Mayor and his representatives on the Lower Manhattan Development Corp. have participated at every stage in the development of the World Trade Center master site plan which calls for rebuilding the office space destroyed by terrorists on 9/11," Silverstein said in a statement. "Together with the Governor, the Mayor has frequently urged us to proceed as quickly as possible. I believe that New Yorkers want to see rapid rebuilding and not yet another exercise in planning and re-planning."

At a breakfast meeting held by CB Richard Ellis on Friday, Silverstein said he has "dedicated the rest of my working life to getting this done. Fast forward yourselves to 2010, 2011, what you're going to see is Downtown as a new destination environment. This community will unlike anything New Yorkers have experienced before."