In the mid nineties, there was a brief shining hope for space enthusiasts. The Ballistic Missile Defense Organization (BMDO), descendent of Reagans Strategic Defense Initiative, had a spaceship. Known as the DC-X, for Delta Clipper Experimental, this spaceship could not reach orbit. DCX was a one third scale prototype of a single stage to orbit (SSTO) vehicle. Unlike traditional expendable rockets or even the space shuttle, the full size DCX would take off vertically, travel to orbit, and return in one piece.
This concept for an SSTO vehicle promised to reduce the per pound cost to orbit by several orders of magnitude, because the program did not entail throwing away large parts of the launch vehicle every time it was used. Further, the vehicle was designed using many of the lessons used in the design of large passenger jets, so that it would require much smaller ground crews and less turn around time than the space shuttle.
The McDonnell-Douglas engineers who worked on the project had several firm rules to guide them. Most important was this: no new technology. The DCX project did not require a single item of new technology. No research was needed. Every component of the DCX was off the shelf technology. Only a few components even needed to be custom designed, such as the fuel tanks and the outer skin of the craft. The DCX team took the flight control system directly out of an MD80 passenger jet. (One engineer quipped that the DCX thought it was a airliner with a very unusual flight path.)
For $600 million, the Ballistic Missile Defense Organization (BMDO), successor to Reagan's Strategic Defense Initiative (Star Wars) built a working, flying prototype space ship. By comparison to typical NASA expenditures, even in NASA Director Goldin's "Faster, Better, Cheaper" era, this was chump change. The BMDO flew the DCX over twenty times, each time pushing the boundaries, each time learning more of what they would need to know when they built the first full size prototype. (Some of these flights were even broadcast on CNN.)
However, this happy progress was not to last. With typical government fickleness, the DCX program was transferred to a jealous NASA. NASA crashed the DCX the first time they flew it, and declared that the program was a failure. When the time came for NASA to name a contractor to build an SSTO craft, the contract was awarded to Lockheed's X-33 program. The X-33's success was predicated on the development of novel aerospike engines. At the time that Lockheed was awarded the contract, no one in the world had ever constructed a working aerospike engine. At the time that the X-33 project was cancelled, five years and billions of dollars later, no full-scale working aerospike engine had ever been built.
Why would NASA, which at least theoretically desired an inexpensive to operate, earth to orbit vehicle, pick the X-33 project over the DCX? Especially considering that the DCX program had actually built a working prototype, and did not require the invention of several new technologies to even have a chance of succeeding? We may never know the answer to that question, but the experience of the last decade should suggest something to those who are planning NASA's next moves.
Small programs with clear design goals have a much better chance of success than typical NASA programs. Private industry, given a clear mission and a free hand on how to go about achieving it, can achieve wonders. NASA should issue a clear set of specifications, in much the same way that the military does for new combat aircraft. Industry must build a flying prototype (though perhaps with some seed funding from the government.) One of the prototypes will be chosen, and the winner will get a contract to build production versions of the spacecraft. NASA should not have the opportunity to micromanage development, nor to continually change the specifications. And NASA should be forced to pick one of the working prototypes.
Here is a situation where the government could help the market: primarily by creating a market for SSTO spacecraft. The aerospace industry can justify spending even very large amounts of money designing a spaceship if they know they have a chance of actually selling some once they're done. Boeing spent tens millions of dollars just designing the 777, knowing that they would have a market for them. Boeing would spend at least that much on creating an SSTO, if it were assured that the government would buy them.
One other important qualification should be that the winning company could also sell the SSTO to private industry. FedEx, among other private companies, did research which indicated if an SSTO with sufficiently large cargo were available at around the price of a 747, they could operate several for point to point cargo shipping on earth, and have a good profit margin. (If you can get to orbit, that same vehicle can reach any point on earth in little over 45 minutes. When it absolutely, positively has to be there in an hour...) But FedEx is not going to pony up the development costs, any more than it would for the cargo jets it flies today.
Once we have cheap, regular and frequent access to space, then everything will start to happen - orbital hotels, miracle materials developed in zero-g labs, lunar colonies, the works. But we will only get there if NASA is forced to get out of the way, and focus on what it's good at: research and deep space exploration.