This was presented at Xconomy 6×6: Six Cities, Six Big Tech Ideas in Boston, Massachusetts on December 1, 2011.

Well, I like to do projects that seem insane.

And I guess I’ve been doing that in science for about 35 years (I started young), and in the technology industry for about 30 years. And I thought today I’d talk a little about what it’s like to do insane projects, and a little about my projects and what they mean.

In a first approximation, I’ve worked on 3 big projects in my life.

As you probably know, Mathematica is a language—a system—that gets used all over the place when people want to do algorithmic computation.

Its early core market was in math and education and things. But it’s become kind of the general highest-end tool for algorithmic computation and algorithmic software development. And actually, for various reasons, like this thing called CDF (Computable Document Format) that we invented, I think it’s poised to become a lot more widespread over the next little while.

A New Kind of Science—or NKS as it’s often called—is a field of basic science that I’ve developed.

You know, we tend to think of programs as things that we humans write. But there’s a question in basic science: if you just start constructing tiny programs, say at random, what will they typically do? Well, NKS is about exploring that computational universe of possible programs.

And the big discovery is that it’s pretty exciting out there. Little tiny programs can do incredibly rich and complicated things.

And that seems to be a big secret of nature—connected to how all those systems in physics, and biology, and so on manage to do such complex things.

It also gives us this amazing new source for technology. Instead of mining the physical world for materials we can mine the computational universe for algorithms. Getting all this amazing stuff that us humans would never think of—but which, as we’re finding in practice in our products, is incredibly useful. Sort of a new kind of technology.

Well, one place we’ve used that a lot is in Wolfram|Alpha. I hope most of you have heard of that—and use it.

I suppose the idea of Wolfram|Alpha is a bit like the idea of the Star Trek computer. Take the knowledge of our civilization and make it computable. So we can automatically answer questions as well as experts in every field.

And actually you might have seen Apple’s Siri intelligent assistant on the iPhone 4S which uses Wolfram|Alpha, and makes that Star–Trek-style “talk to your computer” even more direct.

How does it work? How do we compute all those automatic reports? Get all that knowledge in there?

I might like to be able to say: “There’s just this neat trick”. We’re foraging the web with some fancy algorithm or something. But, no, that’s not what we’re doing. We’re actually custom computing responses to every query.

And… I have to say… it’s really hard. It’s an absurdly complex stack of technology and content that makes this possible.

You’ve got to get all those thousands of different domains of data flowing in; you’ve got to curate them. You’ve got to implement all the models and methods and algorithms from every field—now about 15 million lines of Mathematica code. You’ve got to understand the natural language people type in. You’ve got to figure out how to structure and present the reports you generate. You’ve got to inject human experts in at the right points.

You know, all these things are really quite quite hard. And any of them could perfectly well have turned out to be just plain impossible. But the exciting thing is that they didn’t. And now we have this growing Wolfram|Alpha thing, creating what I call “knowledge-based computing”, and making all sorts of things possible.

Well, OK, so how did I get into doing these kinds of projects?

I grew up in England, went to good schools that I always thought were irrelevant, got very interested in physics—and by the time I was a mid-teenager, started publishing all sorts of physics papers, and kind of playing professional physicist.

Very early on—and this was the mid-1970s—I realized that I could automate a bunch of what I was doing, using a computer. And that’s sort of been a constant thing for me: at any given time, figure out the best possible tools, perhaps by building them myself, and then use them.

Well, by the time I was 20 years old, I was starting out as a physics faculty member at Caltech. And I was building a big software system that was a forerunner of Mathematica. And—well, it was a grizzly early IP-meets-universities story—but I ended up deciding I had to start a company around my software system.

Of course I was just a physics kid, so what did I know about starting companies? And indeed I made lots of mistakes. Like not running it myself, and hiring a CEO who was twice my age.

And as far as I was concerned, the company quickly started doing things I thought were stupid and boring. Though in the end, after many trials and tribulations, it did in fact do OK, and finally even went public in the mid-1990s.

But I’d learned a lot, particularly about what not to do in starting a company. And I reinforced that by doing a bunch of strategy consulting for various technology companies, and seeing all sorts of goofy things—where I’d say “I’d never do that if it was my company”.

But meanwhile my day job was doing basic science, and I’d discovered some pretty interesting and fundamental things there… which I thought had lots of implications and applications.

Well, my plan A was to get lots of other people to help work those out. And so I defined this field that I called “complex systems theory”—that pretty much became “complexity theory”.

And I went around to figure out where to start the first research center for this field: the winner was University of Illinois. Started the first journal. And tried to get the whole sort of societal thing going.

Well, it was really slow. And so I decided I needed a plan B.

And that was: build the best possible tools I could, set myself personally up in the best possible way. And forget all those other people. And just have a blast figuring out the science for myself.

And that’s how—25 years ago last month—I started working on Mathematica.

Both because I wanted it myself, and because I knew that the time was ripe to make a very useful product. And this time, of course, I set my company up so I was CEO, and I didn’t take other peoples’ money, and so on.

And basically my plan B worked out really well.

For 5 years I worked on building and launching the first version of Mathematica, and launching the company.

And then I did something crazy: I decided it was time for me to actually use Mathematica, and the situation I was now in, to make progress on the science I wanted to do. I thought it would only take a year—maybe two. It actually took ten and a half.

And so for a decade I was almost a complete hermit, running the company from a distance, and spending every night working on basic science.

Looking back on it, I suppose several things are amazing to me about that time. First, that the company continued to grow, and in fact do quite well. Second, that despite my hopes in a way that this would happen, there was no coup. And third, that after more than a decade, I actually managed to finish my big book about A New Kind of Science, having filled out pretty much precisely my original table of contents, but with vastly vastly richer discoveries than I’d ever imagined I would be able to make.

Well, after I finished the book, it was pretty interesting coming back with full intensity to the company. I got to rethink so many things. And I started really figuring out what was possible given what we’d built.

You know, somehow this has happened to me a lot. I build up some whole intellectual framework, often with technology. And then gradually, over maybe 5 or 10 years, I begin to realize just what that framework makes possible.

Like with Mathematica. Where there’s this big idea of symbolic programming that’s the unified framework that it’s all based on. And that gradually I’ve realized is more and more general and powerful, and can support more and more.

You know, it’s funny how my various projects have interacted.

I started off doing physics and using computers as tools. Then I built my first computer language, thinking about it a bit like physics: how do I take all these computations people want to do, and drill down to the underlying primitives—the “elementary particles” of computation? But then I realized that, gosh, from simple primitives that I make up, I can build all this stuff.

So I start wondering about building up sort of an artificial physics, from my own primitives. And then wondering about the whole computational universe of such things.

Well, then from working on that, I figured out all sorts of deep things. About how our actual physical universe might be put together. About general ways to create technology. And also about things like the nature of intelligence.

And I suppose one might have assumed that that would just be like academic philosophy. But for me it did something very important.

Actually, I have to say there are starting to be important applications of NKS all over the place—in science, software, biotech, the arts… all over. But probably the first really big one is computational knowledge, and Wolfram|Alpha.

What’s the connection?

Well, ever since I was a kid I’d been thinking about how to systematically organize knowledge, and make it computable. Actually, I found recently stuff I did when I was 12 years old, and it’s almost embarrassing how similar some of it looks to modern Wolfram|Alpha.

But I always figured that to make this work, I’d have to figure out the whole general problem of artificial intelligence. And about every decade or so I came back to that, and always thought it was too hard.

But from NKS, and thinking about the nature of intelligence, I realized that I wasn’t thinking about it correctly… and that in some fundamental sense we already had everything we needed for computational knowledge just with computation.

Well, of course, as a practical matter, I also had Mathematica, which was the biggest and strongest tool for actually doing computation.

And I had another thing: I had a company which by now has been consistently profitable for 23 years. Full of an eclectic group of very talented people—now about 700 of them—who were enculturated into doing “impossible” projects.

So in a sense I had this weird combination, building up for 30+ years, that made it conceivable to try doing Wolfram|Alpha.

You know, it’s funny. At some level I think the single most important thing in that project, and in other “impossible” projects is just the simple thing of believing that I can do the project.

Of having the confidence—maybe arrogance—to believe that even though other people have tried and failed, and people think it’s impossible, that we can nevertheless do it. And then being able to lead other people to throw themselves into it too.

It’s always a strange thing, going from nothing to something. Starting with just an idea, and gradually turning it into something real. Inventing along the way all these things that start so small, and gradually become these whole structures.

I always think anyone who’s been involved in such a thing has kind of a glow of confidence that lasts at least a decade—realizing that, yes, with the right effort nothing can turn into something.

For me, I like coming up with ideas, and figuring things out. And I have to say that right now it’s a very exciting time. I think with all the things we’ve built, between Mathematica, and NKS and Wolfram|Alpha—as well as the current state of the technology industry—we have this sort of Cambrian explosion. I’m having more fun and am feeling more productive than at any time in the 30 years I’ve been involved in the technology industry.

I’d love to talk about what we have coming, but I don’t think I have time to do that here right now. But let me say a few more meta things.

You know, I’ve sort of architected my life and my company to achieve one goal: of having the most efficient possible scheme for turning ideas into reality. And it’s great: these days every day I spend most of my time dealing with ideas, and it’s a bad day when I haven’t come up with a few really pretty good ideas or inventions.

I’ve taken a great deal of trouble building up the team at my company. I guess I’m a person who’s very interested in people, so I’ve enjoyed doing that. We have lots of very long-term employees. And for 20 years I’ve done the seemingly crazy thing of being a remote CEO, these days 1000 miles away from company headquarters, just doing conference calls and screen-sharing all day. It’s just so efficient: people distributed and virtual, and used to working that way.

You know, if you build something like Wolfram|Alpha, it’s really scary how many different kinds of expertise you need. From systems to linguistics to content knowledge and so on. Lots of big projects, but thousands of micro-projects too. Where you just need the right internal expertise.

We have this database we call WhoKnowsWhat. And I’m always trawling that to find internal experts. Like a couple of days ago, as part of some big content review for Wolfram|Alpha, there were questions about data we have on—of all things—oil rigs. And I said: surely there must be some in-house people who know about that. And, sure enough, WhoKnowsWhat told me, that yes, there were two people who’d worked in the oil industry in relevant positions. And so what will have happened is that those two people got pulled into some meeting about oil rigs, sorting out Wolfram|Alpha content in that area.

I suppose it’s an important piece of company culture that nobody thinks that’s odd. One of those people is in international sales; one is in graphics development. But everyone’s used to the idea that they can make little contributions to other projects.

These days we’re doing an almost absurd number of projects—actually products. I’m quite proud of how many, for our comparatively small—700 person—operation.

I think we end up being able to do so many for a few reasons.

First, we’ve been sort of obsessive about building very solid things. We’ve steadily built up Mathematica in this very coherent way. Where each piece is as automated to use as possible.

And that means we can sort of just rely on the existing building blocks, and create bigger and bigger blocks, that allow us to build faster and faster.

And in our company—well, we absolutely eat our own dog food. And we’ve used Mathematica to build everything, from our QA systems to release engineering to web analytics to financial reporting, and so on.

And over the years we’ve built terrific systems for all these things. Which, yes, we should be systematically selling to other people too, and probably eventually we will.

Oh, and we’ve also built up all our service groups in house. Whether it’s graphic design, or legal, or localization, or whatever. So when we have a new project, we understand well our shared services, and they’re all fully enculturated and ready to roll.

I guess also with every new project, we form teams in a very personal way. Picking individual people to bring together who we think will be a good fit for the project. Sort of creating an in-house startup. That can achieve some pretty big things, with comparatively modest resources, because it’s building on that whole stack of technology and processes we have.

A few of these startups have actually split off into separate companies, and probably there’ll be more of it, maybe with outside partners.

When I look at what we’re developing, I always try to have a distribution of timescales. From the we-release-a-new-version-of-the-codebase-every-week Wolfram|Alpha development. To things that will take many years to develop.

And, you know, I always have a big supply of ideas.

Often the biggest issue is just when to pursue a particular one. Not too early; not too late. I mean, if Isaac Newton had decided to start an artificial satellite company, that would have been too early. And with NKS there are ideas that are 50, maybe more, years out.

But somehow what seems to happen is that gradually the ambient technology to support different ideas “comes over the horizon”, and that’s when I really launch into working on them.

I’m a big projects person, and I suppose a difficult thing is when to do new things. I mean, the projects I’ve already started will certainly last as long as I do. And they’re all growing and developing, and taking lots of effort.

But in each of these projects, I’ve been lucky enough to grow teams that are really enthusiastic and talented, and can take them forward, with me being involved at only a higher and higher level.

So that I get to do that scary thing with great regularity of saying “I’ve got a new idea”. “Let’s see who can work on it”. “Let’s have a kickoff meeting”. “Who’s going to project manage it”? And then, over the course of months, or maybe years, it turns into reality. And then, for me, I get to use that thing as yet another layer, to build yet more.

It’s been a wonderful 30 years in the technology industry for me. And it’s so exciting now, having systematically built this whole stack, and being able to do so much with it.

Our company and our way of doing things is pretty different from a lot of the industry. And maybe our goals are a bit different: about building these giant things, independent of difficulty.

But I hope that perhaps I’ve been able to share a little bit of what it has been like to build some of those “insane” products of ours.

Thanks.