Technology and Creative Expression

October 1995
by Tod Machover

My deepest goal has always been to create art - or to support anyone creating art - that has a chance of transforming people's lives for the better. This kind of transformation can be something simple (like becoming more stimulated, enthusiastic, or especially aware and sensitive), or something more comprehensive (like a reevaluation of what is fundamentally important in life, and a rejection of the superficial). This may sound a bit naive these days, when most art has become "entertainment," but I have always believed that it is possible to stimulate such reaction, and that one should strive for no less.


I have also always believed - since I was a kid - that the deepest and most meaningful experiences are those that somehow combine and synthesize EVERYTHING that is part of being alive, exclusive of nothing. In this way, I am part of the very American musical tradition of people like Charles Ives, John Cage, Elliott Carter, John Coltrane, and Bob Dylan: people who tried to convey the complexity and richness of life as directly as possible, without oversimplifying it. I grew up with such ideas around me, and with a natural test bed for studying how to put diverse life-views together: my mother is a pianist, humanist, and embodiment of the European fine art and intellectual tradition, who has spent her career developing methods for teaching musical creativity - and love of music - to children (her book on this subject will be released in 1996 by Oxford University Press); my dad is one of the pioneers in computer graphics, an engineer who early understood the ability of images to simplify complex information, and who also is a deep believer in popular culture and the positive, transformative power of technology.


So my work has always tried to come to terms with these paradoxes, striving for unity by using the most diverse sounds and emotions. I have always tried to bring together worlds that don't seem to quite belong: popular and serious, acoustic and electronic, straightforward and mysterious, active and contemplative, humanistic and technological. In fact, ever since I was a kid I have believed that there is more similarity than difference between people, and that it is possible to convey the most serious and significant messages if you just talk to people in a simple, unpatronizing, confident and intelligent way. As our world becomes more complex, with increasingly dazzling amounts of information to absorb and the resurgence of petty misunderstandings and intolerances between people, I find this theme ever more important and essential.


Technology has always been for me a way to exaggerate the differences between things, as well as to create a larger context - a kind of viscous, amniotic liquid - in which diverse ideas and feelings could find a context and a medium in which to communicate. I got to know technology as a kid, visiting my dad's company and watching the early experiments in man-machine interface.


As a teenager (after years as a die-hard classical cellist), I formed a rock band and experimented with tape-recorder-type manipulations of sound (I had also grown up listening to an incredible assortment of music: from Cage to Subotnick to Boulez to Beethoven to Coltrane to Bikel......no pop, no opera, everything else). At Juilliard in the mid-70's, I became interested in computer music and got someone to teach me, a VERY unpopular thing to do at the time, as the 60's synthesizer era was long over, and computers were decidedly un- hip, at least at orchestral-oriented Juilliard. But I learned FORTRAN, wrote music with punch cards, and heard the result of my programs after a week's delay!


At the beginning, my interest in computers was to achieve greater musical complexity, in fact to "hear" instrumental music I had been composing that was unbelievably difficult to perform (like everyone in a different tempo). In 1978, I got invited to Pierre Boulez's IRCAM in Paris, first as a composer-in- residence, and then to stay on as director of musical research.


Being at IRCAM was a technological revelation to me in two ways. First, I arrived there - luckily for me - just at the time when the first real-time digital synthesizers were being tested by Giuseppe de Giugno, the first machines fast enough to give musical results immediately with no wait whatsoever, and I was fortunate to become the kind of musical guinea pig for this project. As a serious performer (cellist and conductor), I had always felt that computer music would not soar until the computer could be manipulated like an instrument, responsive to the subtlest gestural and emotional response, and the questions raised in 1978 about how to make such a thing possible have motivated my work ever since.


Second, I had been trained in the Western classical tradition, capped off at Juilliard, where the ideal to strive for is Beethoven....deafness and all! That is to say that a composer was meant to imagine all musical ideas inside the head, in total perfection, with no input from the outside world; composers who plinked out tunes on the piano were condescended to as sort of tin pan alley types..."real" composers did not NEED the piano. The idea was to finish this perfect, imagined score, write it out briskly a la Mozart in pen and ink, deliver it 10 minutes before the performance to the orchestra, and sit back as the premiere unfolded impeccably, reproducing exactly what one had imagined in the inner ear. No revisions afterwards, of course!!


Now I am really lucky that I DID receive such a rigorous training, and I still do much of my best musical thinking sitting in an armchair at home, eyes closed, dreaming and imaging what I want my music to be. But at IRCAM, for the first time I had access to di Giugno's real-time synthesizers which allowed me to TEST what I had in mind, to sculpt my orchestra with my fingers and hands, to play with my composition as it developed, and -perhaps most importantly - to then change my musical materials (the instruments, sounds, interactions, whatever!) as my ideas developed - simply by programming and reprogramming the computer.


These ideas - "playing" a machine with physical gesture and intuitive action, moving back-and-forth between imagination and materials, and experimenting with an "orchestra" while inventing and composing - remain fundamental changes that computers have brought to music, accessible in 1978 in only the most arcane research institutions, but now available to anyone with a MIDI studio.


Between 1979 and 1982, I composed the first music for real-time digital synthesizer, culminating in a piece called "Fusione Fugace" premiered at the Venice Biennale. From 1982 to 1985, I experimented with greatly expanding the musical palette of available sounds, creating pieces that combined acoustic and electronic instruments seamlessly, attempting to replace traditional harmony with tension and release patterns based on sound color changes, from melody to harmony to spectrum to noise and back.


I came to the MIT Media Lab in Fall 1985, right when it opened, and got active in the MIDI world as suddenly computer music technology became widely available. At this point, I started the Hyperinstrument project. The idea of hyperinstruments was to develop computer systems that could monitor and eventually "understand" every nuance of musical performance, so that the musician's interpretation and feeling could lead to an enhanced and expanded performance. My idea was always to try to capture the most complete and integrated sense of the musician's meaning and intention, rather than to collect a set of unrelated "parameters" from performance which could then be "mapped" to independent features of a synthesizer or automated composition system. I always want the musician to imagine a musical result in its totality, to use highly developed musical skills and talents, and then to have the machine do the work to translate this into a desired and predictable result.


For this reason, my hyperinstruments between 1986 and 1991 were all designed for highly skilled performers who could master all the nuances and subtleties needed to control such systems. I designed instruments, and wrote music, for virtuosi such as Yo-Yo Ma, Kim Kashkashian, the Los Angeles Philharmonic, and various rock and pop musicians. Hyperinstruments were designed for keyboards, guitars, string instruments, conducting and hand gesture, etc. One of main interests in this work was to explore HOW an individual's expressive intentions could be enhanced in a meaningful way, as if the computer was reading one's mind and filling in the blanks in a totally natural way. I have always believed that humans should be in CONTROL of such systems, and that the computer result should reflect our goals and desires, either our conscious ones, or - increasingly - the subtler ones that we may not even be totally aware of!


Since 1991, I have become increasingly interested in developing hyperinstruments that could allow "ordinary" people to participate actively in music-making, or at least in listening. My goal here has been consonant with my beliefs stated at the beginning of these comments: that any normal, intelligent person is capable of far more sensitivity and creativity than she is normally given credit for. This is not a demogogic or naive argument that any person could be a Beethoven if given the proper "tools" - such a statement, all too common in this era of interactivity, is total nonsense.


It is very difficult to create things, and only a few people have the insight and talent to make things that are truly unique and universal. However, everyone is capable of expressing their own unique qualities and special experiences, and doing so enriches life. Also, we have all become passive consumers of art and entertainment, a far cry from the 19th century when all played piano and tried out new scores by sight-reading on Saturday evenings, or even earlier when everyone sang in weekly religious services.


A first goal of interactivity today is to restore this sense of participation and direct experience to audiences, for music and others activities. Music is a great place to start, since it actually has such a long tradition of amateur community music-making, found in cultures around the world.


The goal of our recent hyperinstruments for amateurs is to enable such active music participation to take place in a significant way. We have designed a whole series of such instruments. "Drum-Boy" allows people to make complex rhythm compositions, by having the computer automatically "answer" and suggest music based on what you play, and also by allowing you to "talk" to the computer by using to adjectives to change the music in the way you want. Unlike other similar systems, this one really seems to work, and actually uses these adjectives to analyze any music already playing, and to change it in the desired way immediately....people seem to recognize their adjectives! Yamaha has already integrated main features of Drum-Boy into its new RY-20 series of drum machines.


The next series of amateur hyperinstruments extended similar principles to all aspects of music, including melody, harmony, "texture," and orchestration. Using the "seed music" concept, such systems allow you to give the computer a fragment or "seed" of any music you want, which is then automatically analyzed. The computer then makes up more music - indefinitely - similar to, but different from - what you fed it. The trick is that the system is designed so that the user can then "shape" and "steer" (much like a driving game) the music as it develops, changing its qualities and direction with intuitive commands, from interfaces like videogame joysticks. This kind of system is very interesting as a new kind of gestural instrument, maybe as a kind of "music game," and even as an automated soundtrack to a movie or videogame, where the music must change automatically in response to unpredictable turns in the action.


Another goal of mine has been to make the interaction with such sophisticated systems/instruments as intuitive and natural as possible for the general public. I feel that it is very important now to develop interfaces that rethink the necessity of forcing people to study a particular musical instrument for 10-20 years (so make easy PHYSICAL interfaces), but that do NOT eliminate the necessity of concentration, skill and imagination (so make challenging MENTAL interfaces). Far too many interactive music systems right now are, in my view, kinds of toys - they take no skill to play, make you feel like a "master" after 5 minutes, but then don't allow you to get any better. Such systems are a bad idea because they become boring very fast, but also because they encourage people to be lazy and complacent - feeling that they are doing things that they aren't really doing - rather than more sensitive and aware, the goals stated at the outset here.


With our current hyperinstruments, I am trying to prove that the latter is possible. It means, in my view, simplifying the physical interface while redefining completely the mental and imaginative control that people can have over a music system. To do this, we are trying to improve interfaces, while putting more intelligence and sensitivity in the machine. With my colleague Neil Gershenfeld, a physicist at the Media Lab, we have developed a new technique of magnetic field sensing that gives a very accurate measurement of physical movement with no wires or funny gadgets, the most sensitive technique developed so far for allowing amateur conducting, danse- to-music conversion, etc. We are developing systems that measure the emotion in acoustic signals, allowing anyone's voice to be used to control an interactive environment. And we are moving towards systems which measure not only instantaneous feeling or mood, but have a sense of your goals and intentions, allowing you to sketch out a musical "story," whose details are filled in by your computational assistant.


Such instruments are requiring a totally new definition of the concept of interactivity, and since computer musicians have been working with interactive systems for over ten years now, it is not surprising that we are somewhat in the forefront in helping to define this new field of interactive entertainment. If you compare what is needed now with my "Beethoven-the- ideal-creator" model above, it is clear that what is needed now is a new role for artists where a work is imagined not as a perfect, totally finished creation which is sent out into a world of passive consumers, but rather a conception of creating works and situations which are partially finished, partially opened, where the audience is invited to explore, complete, and maybe totally change what is offered to them. Such an idea implies that listeners should be actively invited to SEARCH for meaning, rather than to accept ready-made answers.


Such an interactive experience is VERY hard to achieve, and I think that most experiments so far fall short of the mark, either offering unsupervised wandering (like the encyclopedia model), egocentric fanzine attention (the "learn-everything-you-never-wanted-to-know-about-the-artist" school), or only superficial interaction (as in remixing a couple of tracks of some rock song). One of my goals this year is to create a whole series of short pieces, each with a different interactive model, to find out what kinds of things work best, and which give the user the most sophisticated sense of interaction.


These separate interactive music pieces, which I sometimes think of as "music games," will form the core of my "Brain Opera" project (for which you will find a project description tomorrow). The idea of the "Brain Opera" is not only to present such a collection of different kinds of interactive musical experiences to the public (ranging from those that encourage listening, to performance, to creation), but also to organize all of them in a coherent, directed whole, a total experience that allows a kind of cohesion to emerge out of the actions of many individuals (the most recent expression of my ideas stated at the outset). Now, I am planning to present the first version of the "Brain Opera" at the new Lincoln Center Festival in Summer 1996, and then on to Japan and Europe in 1997. In addition, the "Brain Opera" will be an official project of the 1996 Internet World's Fair, thus making it available online to people around the world.


As for the "professional" hyperinstruments, I completed my hyperstring trilogy last year, with "Begin Again Again..." for solo hypercello, "Song of Penance" for hyperviola and chamber orchestra, and "Forever and Ever" for hyperviolin and orchestra. I am starting work with Yo-Yo Ma to make a CD- Rom of "Begin Again Again..." for Sony Classical release next year.


I am currently working on a hyper-string quartet for the Kronos Quartet, to be premiered next year. This work will push the hyperstring technology much further (hopefully involving technology which would allow any acoustic string instrument to become a "hyperstring"), and will also explore my theme of contrasting and combining different cultures, using the hyper- quartet as the transformative medium. The quartet will be given a lot of interactive freedom, and we are starting to design an interactive version of the piece for the public where they will be able to explore the piece and use its elements to create original music.


Orchestras have been slow to find ways to integrate technology in a comprehensive way - little budget for experiments, conservative atmosphere. But I am discussing a project with the National Symphony in Washington, D.C., to create an orchestral fanfare which the public can play with and modify during intermission. And I am composing a totally non-technological, acoustic orchestral work for the Halle Orchestra, and conductor Kent Nagano, in Manchester, England.


One of my more fun projects this past year was a short 25-minute opera that I wrote for magicians Penn & Teller, called "Media/Medium". This piece incorporates music, magic, technology, and interactivity into something which is pretty unusual. With my colleagues at the Media Lab, we designed a "sensor chair," using the magnetic field sensors described above. I turned this into a totally new musical instrument that measures all body movements and turns them into sound while you are sitting in the chair. Movements are measured very precisely, and can control anything from very delicate "finger painting with sound" to a wild percussion performance on 400 different drums. During the opera, premiered at MIT in October and now a part of Penn & Teller's touring show, this technological marvel becomes a mysterious means for conjuring up spirits - as mediums did in the 19th century - thus exploring the fine line between magic and technological marvel, between hype and plain truth. This sensor chair instrument has attracted a lot of attention, and discussions are underway with various people, from symphony orchestra managements to major rock stars to furniture companies, on collaborations for the next version!


In closing, I would say that music is at a major crossroads right now. Amazingly, technology - through MIDI first and then "multimedia" - has put music back at the center of attention. In fact, music has developed faster than most other art/technology fields, and this is cause for celebration. There is no doubt in my mind that our main media for expression in the future will be digital - there is no turning back (even though now we are experiencing a kind of shocked pre-millenial reaction to major change).


However, there are many dangers. Music is more a part of our lives than it has ever been - we listen in the car, while we are eating, while we are working, as background to movies, CD-Roms, everywhere. But therein lies a BIG problem: music has become background, and many people have lost the concentration to listen to music without doing something else at the same time! In fact, I think that one big task for composers in our generation is to reinvigorate music - in a sense to reinvent it - so that people are attracted to it for its own virtues, and not just because it subliminally supports something else.


This is connected with another problem. Art and entertainment swing pendulum-like between periods of complexity and periods of simplicity; the most interesting art always comes from periods somewhere in the middle, when the package is as simple as possible but the message is deep and demanding. In my view, the pendulum has been swinging since the mid- 1980's towards a kind of undemanding, simplistic art and entertainment, originally a necessary antidote to the public-ignoring "who cares if you listen" experiments of the 1950's and 1960's, but now turned into a kind of boring pablum, afraid to engage and challenge the audience, and more and more peripheral to people's lives. Pop culture has been much quicker in significantly embracing new technology than "serious" culture, and my fear is that this rift is growing, and fast! - again emphasizing an outgrown distinction between serious and popular - instead of shrinking.


So I think that we must strive now for a music which can stand on its own against the other media - emphasizing its ability to create mysteriously deep emotional and mental experiences, encouraging listeners to "fill in the blanks" - while combining a seriousness of purpose and depth of content with a colloquial and non-elitist expression.


Our interactive technology should encourage this goal, by creating situations which invite people to participate in significant ways through eliminating every unnecessary barrier, while making sure that the experience offered enhances and expands life experience instead of being a mindless drug.


So, the situation is in fact paradoxical. We DO have to find increasing ways to make music a part of everyday life, by building "instruments" of expression into our everyday environment, into furniture, clothing, toys, walls, everything - but must do this so that our lives become richer, more REAL, and not synthetically plastic and "virtual."


My fear is that people will prefer the latter until they are shown that the former is possible.


To demonstrate this possibility - in our instruments, in our environments, in our music - is one of the principal goals of my work for the coming decade.


And while technology is incredibly important in our future arts and entertainment, it is also important to NOT mistake the tool for the meaning. Too many people nowadays see technology itself as the fundamental issue, becoming interested in a project because of some fancy new techniques, or inversely turning away from that same project because of an aversion to technology.


This technology is there to help us deal with deeper human issues - in my case, the establishment of community between very different kinds of peoples, and the encouragement of everyone to grow and be stimulated, to face life with enthusiasm and optimism and generosity. My most fundamental wish for the future is that we do in fact use our technology this way, and that more and more people view it as such, and not as an end in itself.


webmaster@brainop.media.mit.edu