When you stand under the Burj Tower it doesn't look that tall at all. Bizarre. Alone on the flat desert landscape of Dubai, apart from the generic glitzy towers of the Sheikh Zayed Road, it seems slightly abstract, with nothing for your eye to compare it with. It could be 250m. It could be 305m. Or 750m. Who knows? Skyscraper internet chat rooms are ablaze with speculation about its final height. For the most famous fact about the Burj - apart from that it's the first piece of architecture to be both the tallest building and man-made structure in the world - is that its final height is a closely guarded secret until it opens next June, for fear of some rival stealing its thunder with an extra bucket of quick-drying concrete.

As if. Because nothing else comes even close. Burj, still under construction, is in another league. This is literally a step change in the future of skyscrapers. Welcome to the world's first superscraper.

Ever since adverts for the Burj started appearing in the glossiest of mags, inviting anyone with a hedge fund or two to invest in its luxury condos (the majority of the building; the rest will be devoted to a five-star hotel and a few offices), its property company, Emaar, has gone into overdrive to stop leaks and fuel speculation about its eventual height.

When I visit the site I'm not allowed any cameras or recording instruments, and of course I'm not allowed to go to the top. Even its designer, William Baker from the architects SOM (yes, he knows), can tell me only that “it's over 700m,” though most estimates put it at more than 800m, which would make it as tall as four and a half Gherkins, or almost two Empire State Buildings. That's very tall indeed. Taipei 101 in Taiwan, the next tallest building - still officially the world's tallest, until the officials measure the Burj - is a “mere” 508m, including its spire. No, Burj Dubai is a very different specimen indeed, more than half as tall again. If you look at the line-up of recent world's tallest, such as Taipei 101, and its predecessor, Petronas Towers in Kuala Lumpur, they shuffle taller bit by bit. Next to them the Burj shoots impossibly skywards.

Back in his office in London, the black-clad, Missouri-born Baker, who has the air of a kindly maths professor and is widely regarded as the finest skyscraper designer in the biggest skyscraper firm in the business, is giving me a quick masterclass. “You can feel the climate change up there,” he says, tantalisingly. “Usually Dubai's so searingly hot that it's oppressive, but when you step out onto the terraces a few hundred metres up it's almost pleasant. It's so high, so high. I mean you should see it up there on top. It is pretty amazing. You can see the shadow of the building on clouds.”

So how did this step change take place? How come the Burj is so much taller than its rivals? Though they grew out of the first real skyscrapers in late 19th-century Chicago and New York, when structural steel frames and hydraulic lifts upped the ante in man's high-rise dreams, “today's skyscrapers have evolved into different beasts”, Baker says. Back then, what were mostly office towers were essentially grids of structural steel. The trouble was, the higher you got, the heaver the whole thing got, and the fatter the base needed to be to keep it all up.

The first and second generation skyscrapers from the 1870s to the Empire State's peak in the Great Depression could go only so high without becoming unstable. “The main enemy of any skyscraper is wind,” Baker says. “When wind hits an object, whether it's a streetlamp or a mountain, it has to go around it. First it tries one way, then the other, and in doing so creates a pressure differential on either side which causes it to oscillate.

“Once you get to a certain scale, or if the movement and the wind hit a certain frequency, this can get self-reinforcing, so the skyscraper oscillates more and more violently, like a child on a swing, kicking the ground with his feet, getting higher and higher.” That's not good news.

In the 1960s Baker's predecessor at SOM, Fazlur Khan, came up with an evolutionary breakthrough: the tube structure. Khan used the innate structural strength of tubes to push skyscrapers taller than ever before. “You should think of tube skyscrapers not as made up of separate beams,” Baker says, “but as one single beam cantilevered out from the foundations. That was Khan's breakthrough.” A tube skyscraper is essentially one giant tube with windows cut into it, the exterior supporting the whole. Fewer internal columns makes economic sense because rentable area increases, and it creates a lighter, stronger structure that can resist lateral forces such as wind.

The tube system was behind the third generation of skyscrapers in the late 1960s and 1970s: the World Trade Centre in New York (not by SOM) and the John Hancock Tower and Sears Tower, in Chicago, technically speaking a bundle of tubes and, at 442m, the skyscraper that held the title of the world's tallest for two decades after it opened in 1973, so pioneering was Khan's work. Since then others have gone higher, and other structural systems have been tried out, but none has been bettered.

Burj isn't a breakthrough in overall design; the fundamentals are those of Sears. It's essentially the same system of bundled tubes that Khan came up with in the Sixties - a variation on Gothic architecture, a spire tube supported by neighbouring tube “buttresses” in a Y-shape that spirals skywards, each concrete-framed wing propping up the next, and locked together by a central hexagonal core, which stops the building from twisting and contains the service core: lifts, etc.

But Burj is a massive breakthrough in microdesign. What Baker had discovered in his decades of experiments in wind tunnels is that by keeping the fundamental structure the same, but modifying small details, “we could go up and up and up”. To what? “We simply didn't know.”

The theory is a little like streamlining the smallest details on a super-fast car, or making tiny modifications to an Olympic swimmer's costume. “The very window mullions act like the roughened skin of a shark,” Baker says, “creating little eddies or vortices which lubricate the wind round the building and dampening down its effect”. His team even found that changing the plan's spiral from counter clockwise to clockwise made a major difference. By continually modifying its tiny details in the wind tunnel, the building rose higher exponentially.

That publicity stunt concealing the final height was no stunt at all. “Every month we found we could go higher and higher. Even while we were building the thing. In fact we know we could have gone even higher than what we've built. We found that once we got to a certain height, astonishingly all the forces got smaller not bigger.” Alas, by the time he'd discovered this, the building was already a quarter up, and would have required rebuilding a little farther down. But the theory itself was revolutionary.

It owes something to mathematical algorithms - equations used throughout science to predict hitherto unpredictable non-linear behaviour in nature. Baker starts pushing the evolutionary metaphor, talking of “genetic algorithms”, “cross-breeding” and “mutations” in skyscrapers that obey new theories of physics more complex than those discovered by Isaac Newton. “The kilometre-high building is not only within reach,” he says, “it's possible right now. Frank Lloyd Wright's Mile High Tower is possible, too. I'm working on it!” What holds it back isn't engineering, it's the economics. The reason that so many superscraper schemes never come to fruition is usually the business plan. Every time you increase the height, the volume increases by several times more. “That's a lot of space to sell.” Burj mitigates this by keeping the building as slim as possible. But extreme height presents other practical problems, such as how you pump concrete 800m into the sky (super-powered pumps), or how you stop people having to wait half an hour for a lift (the fastest lifts in the world).

Eight years after 9/11, though, does he ever fear hubris? “I was in Chicago at the time, and we were about to start work, that day, on the new Trump Tower that's just been finished. The guys were flying in from New York. It was going to be the world's tallest building. It got shortened a lot after that.”

The skyscraper market took, shall we say, a little dip. But it didn't take long to bounce back. “I think there's something basic in this quest for height. For structural engineers it's always the widest, the tallest, that you're aiming for. We like a shot at immortality.” For everyone else “it's just chasing infinity. I'm sure it's hard-wired into us.” It's the same quest that encourages mountaineers, explorers, space travellers. “You're at the edge, on the precipice. You're always looking round the next corner, or over the next hill.” It takes on a spiritual, abstract quality. The chase.

“It takes a long time for me to breathe out after the building's finished. Months. But it's good for structural engineers to always be a little bit nervous. Though if one thought too long about what one was actually doing you'd never design another thing.”

William Baker is addressing the Architecture Foundation at the Royal College of Physicians, London NW1, on Dec 3 at 7pm. Tickets £10/£8 from 0844 4124313

Editor's choice

• Vote for the Breakthrough Award winner
• David Tennant back as Hamlet after back surgery
• Matt Smith is the new Dr Who