Sound is a powerfully evocative medium, capable of conjuring authentic emotions and unlocking new experiences. This fall, several cross-disciplinary projects at MIT probed the technological and aesthetic limits of sound, resulting in new innovations and perspectives, from motion-sensing headphones that enable joggers to maintain a steady pace, virtual reality technology that enables blind people to experience comic book action, as well as projects that challenge our very relationship with technology.

Sound as political participation

“Sound is by nature a democratic medium,” says Ian Condry, an anthropologist and professor in MIT’s Department of Global Studies and Languages, adding that “sound lets us listen around the margins and to follow multiple voices coming from multiple directions.”

That concept informed this year’s Hacking Arts Hackathon Signature Hack, which Condry helped coordinate. The multi-channel audio installation sampled and abstracted audio excerpts from recent presidential inaugural addresses, then blended them with breathing sounds that the team recorded from a live audience. Building on this soundtrack, two team members acted as event DJs, instructing the audience to hum and breathe in unison, while their phones — controlled by an app created for the hackathon — played additional breathing and humming sounds.

“We wanted to play with multiple streams of speech and audio,” says Adam Haar Horowitz, a second-year master’s student at the MIT Media Lab, and member of the winning team. “Not just the words, which can be divisive, but the texture and pauses between the words.”

A guy walks into a library…

What happens when artificial intelligence decides what’s funny? Sound and democracy played prominently in "The Laughing Room," an installation conceived by a team including author, illustrator, and MIT PhD candidate Jonny Sun and Stephanie Frampton, MIT associate professor of literature, as part of her project called ARTificial Intelligence, a collaboration between MIT Libraries and the Cambridge Public Library.

Funded in part by a Fay Chandler Faculty Creativity Seed Grant from the MIT Center for Art, Science and Technology (CAST), "The Laughing Room" invited public library visitors into a set that evoked a television sitcom living room, where they told stories or jokes that were analyzed by the room’s AI. If the algorithm determined a story was funny, it played a recorded laugh track. "The Laughing Room" — as well as the AI’s algorithmic calculations — were then broadcast on screens in "The Control Room," a companion installation at MIT’s Hayden Library.

While fun for the public, the project also mined more serious issues. “There is a tension in society around technology,” says Sun, “between the things technology allows you to do, like having an algorithm tell you your joke is funny, and the price we pay for that technology, which is usually our privacy.”

Using sound to keep the pace

How can audio augmented reality enhance our quality of life? That challenge was explored by more than 70 students from multiple disciplines who competed in the Bose MIT Challenge in October. The competition, organized by Eran Egozy, professor of the practice in music technology and an MIT graduate who co-founded Harmonix, the company that developed iconic video games Guitar Hero and Rock Band, encourages students to invent real-life applications for Bose AR, a new audio augmented reality technology and platform.

This year’s winning entry adapted the Bose’s motion-sensing AR headphones to enable runners to stay on pace as they train. When the runner accelerates, the music is heard behind them. When their place slows, the music sounds as if it’s ahead of them.

“I’d joined hackathons at my home university,” said Dominic Co, a one-year exchange student in architecture from the University of Hong Kong and member of the three-person winning team. “But there’s such a strong culture of making things here at MIT. And so many opportunities to learn from other people.”

Creating a fuller picture with sound

Sound — and the technology that delivers it — has the capacity to enhance everyone’s quality of life, especially for the 8.4 million Americans without sight. That was the target audience of Project Daredevil, which won the MIT Creative Arts Competition last April.

Daniel Levine, a master’s candidate at the MIT Media Lab, teamed with Matthew Shifrin, a sophomore at the New England Conservatory of Music, to create a virtual-reality system for the blind. The system’s wearable vestibular-stimulating helmet enables the sightless to experience sensations like flying, falling, and acceleration as they listen to an accompanying soundtrack.

Shifrin approached Levine two years ago for help in developing an immersive 3-D experience around the Daredevil comic books — a series whose superhero, like Shifrin, is blind. As a child, Shifrin’s father read Daredevil to him aloud, carefully describing the action in every pane. Project Daredevil has advanced that childhood experience using technology.

“Because of Dan and his engineering expertise, this project has expanded far beyond our initial plan,” says Shifrin. “It’s not just a thing for blind people. Anyone who is into virtual reality and gaming can wear the device.”

A beautiful marriage of art and technology

Another cross-disciplinary partnership in sound and technology that resulted in elegant outcomes this fall is the ongoing partnership between CAST Visiting Artist Jacob Collier and MIT PhD candidate Ben Bloomberg.

Bloomberg, who completed his undergraduate and master’s studies at MIT, studied music and performance design with Tod Machover, the Muriel R. Cooper Professor of Music and Media and director of the Media Lab’s Opera of the Future group. Bloomberg discovered Collier’s music videos online about four years ago; he then wrote the artist to ask whether he needed any help in adapting his video performances to the stage. Fortunately, the answer was yes.

Working closely with Collier, Bloomberg developed a computerized audio/visual performance platform that enables the charismatic composer and performer to move seamlessly from instrument to instrument on stage and sing multiple parts simultaneously. The duo continues to develop and perfect the technology in performance. “It’s like a technological prosthesis,” says Bloomberg, who has worked with dozens of artists, including Bjork and Ariana Grande.

While technology has opened the door to richer sound explorations, Bloomberg firmly places it in an artistic realm. “None of this would make any sense were it not for Jacob’s amazing talent. He pushes me to develop new technologies, or to find new ways to apply existing technology. The goal here isn’t to integrate technology just because we can, but to support the music and further its meaning.”

Explorations in sound continue into 2019 with the innovative annual performance series MIT Sounding. Highlights of the 2018-2019 season include a collaboration with the Boston Modern Orchestra Project in honor of MIT Institute Professor John Harbison’s 80th birthday, the American premiere of the Spider’s Canvas, a virtual 3-D reconstruction of a spider’s web with each strand tuned to a different note, and residencies by two divergent musicians: the Haitian singer and rapper BIC and the innovative American pianist Joel Fan performing works by MIT composers.

MIT has identified a preferred location for the new MIT Stephen A. Schwarzman College of Computing headquarters: the current site of Building 44. The new building, which will require permitting and approvals from the City of Cambridge, will sit in a centralized location that promises to unite the many MIT departments, centers, and labs that integrate computing into their work.

In October, MIT announced a $1 billion commitment to address the global opportunities and challenges presented by the prevalence of computing and the rise of artificial intelligence (AI) — the single largest investment in computing and AI by a U.S. academic institution. At the heart of the initiative is the new college, made possible by a $350 million foundational gift from Mr. Schwarzman, the chairman, CEO and co-founder of Blackstone, a global asset management and financial services firm.

The college aims to: connect advances in computer science and machine learning with advances in MIT’s other academic disciplines; create 50 new faculty positions within the college and jointly with existing academic departments; give MIT’s five schools a shared structure for collaborative education, research, and innovation in computing and artificial intelligence; educate all students to responsibly use and develop computing technologies to address pressing societal and global resource challenges; and focus on public policy and ethical considerations relevant to computing, when applied to human-machine interfaces, autonomous operations, and data analytics.

With those goals in mind, MIT aims to construct a building, large enough to house 50 faculty groups, to replace Building 44, which sits in the center of the Vassar Street block between Main Street and Massachusetts Avenue. Those currently working in Building 44 will be relocated to other buildings on campus.

Scheduled for completion in late 2022, the new building will serve as an interdisciplinary hub for research and innovation in computer science, AI, data science, and related fields that deal with computing advances, including how new computing methods can both address and pose societal challenges. It will stand in close proximity to a cluster of computing- and AI-focused departments, centers, and labs located directly across the street and running up to the intersection of Vassar and Main Streets. All other buildings on campus are about a six-minute walk away.

“You can think of this intersection of Vassar and Main as the ‘entrance to computing,’” says Associate Provost Krystyn Van Vliet, who is responsible for Institute space planning, assignment, and renovation under the direction of the Building Committee, which is chaired by MIT Provost Marty Schmidt and Executive Vice President and Treasurer Israel Ruiz. Van Vliet also oversees MIT’s industrial engagement efforts, including MIT’s Office of Corporate Relations and the Technology Licensing Office.

“The building is intended as a convening space for everyone working to create and shape computing — not just computer scientists, but people who have expertise in the humanities and arts, or science, or architecture and urban planning, or business, or engineering,” Schmidt adds.

Everyone currently located in Building 44 will be moved to their new campus locations by late summer of 2019. Demolition is scheduled to begin in the fall.

While a final design is still months away, a key planned feature for the building will be “convening spaces,” which will include areas set for interdisciplinary seminars and conferences, and potentially an “open office” concept that promotes mixing and mingling. “You can imagine a graduate student from the humanities and a postdoc from EECS working on a project together,” says Dean of the School of Engineering Anantha P. Chandrakasan, the Vannevar Bush Professor of Electrical Engineering and Computer Science. “Such a building can serve as a place for broad community collaboration and research.”

The centralized location is key to the college’s interdisciplinary mission. Building 44 sits directly across the street from Building 38, which houses the Department of Electrical Engineering and Computer Science; the Stata Center, which the Computer Science and Artificial Intelligence Laboratory (CSAIL) calls home; and the Research Laboratory of Electronics in Building 36.

Down the road, on the corner of Main Street, stands the Koch Institute for Integrative Cancer Research and the Broad Institute of MIT and Harvard, both of which incorporate computer science and AI into cancer and medical research. Buildings behind the headquarters on Main Street, in the area known as “Technology Square,” contain many biological engineering, nanotechnology, and biophysics labs.

The new building will also neighbor — and possibly connect to — Building 46, which houses the Department of Brain and Cognitive Sciences, the Picower Institute for Learning and Memory, and the McGovern Institute for Brain Research. “When you think about the work of connecting human intelligence and machine intelligence through computing — which can be physically connected to a building where people are working on understanding human intelligence and cognition — that’s exciting,” Van Vliet says.

The building could thus help “activate” Vassar Street, she adds, because buildings along the street are somewhat visually closed off to the public. The new building, she says, could include windows with displays that visually highlight the research conducted behind the walls, like peering into the labs along the MIT halls.

“Right now, when you walk down Vassar Street, people don’t know what’s happening inside most of these buildings,” she says. “By activation, we mean there’s more community interaction and pedestrian traffic, and more visible displays that draw the public into campus and make them aware of what’s going on at MIT. It will help us show the breadth of MIT’s activities all the way down Vassar Street, for both the growing MIT community and our neighbors.”

A series of launch events for the MIT Schwarzman College of Computing is planned for late February 2019. The search for the college’s dean is ongoing.

Machine learning algorithms now underlie much of the software we use, helping to personalize our news feeds and finish our thoughts before we’re done typing. But as artificial intelligence becomes further embedded in daily life, expectations have risen. Before autonomous systems fully gain our confidence, we need to know they are reliable in most situations and can withstand outside interference; in engineering terms, that they are robust. We also need to understand the reasoning behind their decisions; that they are interpretable.

Aleksander Madry, an associate professor of computer science at MIT and a lead faculty member of the Computer Science and Artificial Intelligence Lab (CSAIL)’s Trustworthy AI initiative, compares AI to a sharp knife, a useful but potentially-hazardous tool that society must learn to weild properly. Madry recently spoke at MIT’s Symposium on Robust, Interpretable AI, an event co-sponsored by the MIT Quest for Intelligence and CSAIL, and held Nov. 20 in Singleton Auditorium. The symposium was designed to showcase new MIT work in the area of building guarantees into AI, which has almost become a branch of machine learning in its own right. Six faculty members spoke about their research, 40 students presented posters, and Madry opened the symposium with a talk the aptly titled, “Robustness and Interpretability.” We spoke with Madry, a leader in this emerging field, about some of the key ideas raised during the event.

Q: AI owes much of its recent progress to deep learning, a branch of machine learning that has significantly improved the ability of algorithms to pick out patterns in text, images and sounds, giving us automated assistants like Siri and Alexa, among other things. But deep learning systems remain vulnerable in surprising ways: stumbling when they encounter slightly unfamiliar examples in the real world or when a malicious attacker feeds it subtly-altered images. How are you and others trying to make AI more robust?

A: Until recently, AI researchers focused simply on getting machine-learning algorithms to accomplish basic tasks. Achieving even average-case performance was a major challenge. Now that performance has improved, attention has shifted to the next hurdle: improving the worst-case performance. Most of my research is focused on meeting this challenge. Specifically, I work on developing next-generation machine-learning systems that will be reliable and secure enough for mission-critical applications like self-driving cars and software that filters malicious content. We’re currently building tools to train object-recognition systems to identify what’s happening in a scene or picture, even if the images fed to the model have been manipulated. We are also studying the limits of systems that offer security and reliability guarantees. How much reliability and security can we build into machine-learning models, and what other features might we need to sacrifice to get there?

My colleague Luca Daniel, who also spoke, is working on an important aspect of this problem: developing a way to measure the resilience of a deep learning system in key situations. Decisions made by deep learning systems have major consequences, and thus it’s essential that end-users be able to measure the reliability of each of the model’s outputs. Another way to make a system more robust is during the training process. In her talk, “Robustness in GANs and in Black-box Optimization,” Stefanie Jegelka showed how the learner in a generative adversarial network, or GAN, can be made to withstand manipulations to its input, leading to much better performance. 

Q: The neural networks that power deep learning seem to learn almost effortlessly: Feed them enough data and they can outperform humans at many tasks. And yet, we’ve also seen how easily they can fail, with at least three widely publicized cases of self-driving cars crashing and killing someone. AI applications in health care are not yet under the same level of scrutiny but the stakes are just as high. David Sontag focused his talk on the often life-or-death consequences when an AI system lacks robustness. What are some of the red flags when training an AI on patient medical records and other observational data?

A: This goes back to the nature of guarantees and the underlying assumptions that we build into our models. We often assume that our training datasets are representative of the real-world data we test our models on — an assumption that tends to be too optimistic. Sontag gave two examples of flawed assumptions baked into the training process that could lead an AI to give the wrong diagnosis or recommend a harmful treatment. The first focused on a massive database of patient X-rays released last year by the National Institutes of Health. The dataset was expected to bring big improvements to the automated diagnosis of lung disease until a skeptical radiologist took a closer look and found widespread errors in the scans’ diagnostic labels. An AI trained on chest scans with a lot of incorrect labels is going to have a hard time generating accurate diagnoses. 

A second problem Sontag cited is the failure to correct for gaps and irregularities in the data due to system glitches or changes in how hospitals and health care providers report patient data. For example, a major disaster could limit the amount of data available for emergency room patients. If a machine-learning model failed to take that shift into account its predictions would not be very reliable.

Q: You’ve covered some of the techniques for making AI more reliable and secure. What about interpretability? What makes neural networks so hard to interpret, and how are engineers developing ways to peer beneath the hood?

A: Understanding neural-network predictions is notoriously difficult. Each prediction arises from a web of decisions made by hundreds to thousands of individual nodes. We are trying to develop new methods to make this process more transparent. In the field of computer vision one of the pioneers is Antonio Torralba, director of The Quest. In his talk, he demonstrated a new tool developed in his lab that highlights the features that a neural network is focusing on as it interprets a scene. The tool lets you identify the nodes in the network responsible for recognizing, say, a door, from a set of windows or a stand of trees. Visualizing the object-recognition process allows software developers to get a more fine-grained understanding of how the network learns. 

Another way to achieve interpretability is to precisely define the properties that make the model understandable, and then train the model to find that type of solution. Tommi Jaakkola showed in his talk, “Interpretability and Functional Transparency,” that models can be trained to be linear or have other desired qualities locally while maintaining the network’s overall flexibility. Explanations are needed at different levels of resolution much as they are in interpreting physical phenomena. Of course, there’s a cost to building guarantees into machine-learning systems — this is a theme that carried through all the talks. But those guarantees are necessary and not insurmountable. The beauty of human intelligence is that while we can’t perform most tasks perfectly, as a machine might, we have the ability and flexibility to learn in a remarkable range of environments. 

Small imperfections in a wine glass or tiny creases in a contact lens can be tricky to make out, even in good light. In almost total darkness, images of such transparent features or objects are nearly impossible to decipher. But now, engineers at MIT have developed a technique that can reveal these “invisible” objects, in the dark.

In a study published today in Physical Review Letters, the researchers reconstructed transparent objects from images of those objects, taken in almost pitch-black conditions. They did this using a “deep neural network,” a machine-learning technique that involves training a computer to associate certain inputs with specific outputs — in this case, dark, grainy images of transparent objects and the objects themselves.

The team trained a computer to recognize more than 10,000 transparent glass-like etchings, based on extremely grainy images of those patterns. The images were taken in very low lighting conditions, with about one photon per pixel — far less light than a camera would register in a dark, sealed room. They then showed the computer a new grainy image, not included in the training data, and found that it learned to reconstruct the transparent object that the darkness had obscured.

The results demonstrate that deep neural networks may be used to illuminate transparent features such as biological tissues and cells, in images taken with very little light.

“In the lab, if you blast biological cells with light, you burn them, and there is nothing left to image,” says George Barbastathis, professor of mechanical engineering at MIT. “When it comes to X-ray imaging, if you expose a patient to X-rays, you increase the danger they may get cancer. What we’re doing here is, you can get the same image quality, but with a lower exposure to the patient. And in biology, you can reduce the damage to biological specimens when you want to sample them.”

Barbastathis’ co-authors on the paper are lead author Alexandre Goy, Kwabena Arthur, and Shuai Li.

Deep dark learning

Neural networks are computational schemes that are designed to loosely emulate the way the brain’s neurons work together to process complex data inputs. A neural network works by performing successive “layers” of mathematical manipulations. Each computational layer calculates the probability for a given output, based on an initial input. For instance, given an image of a dog, a neural network may identify features reminiscent first of an animal, then more specifically a dog, and ultimately, a beagle. A “deep” neural network encompasses many, much more detailed layers of computation between input and output.

A researcher can “train” such a network to perform computations faster and more accurately, by feeding it hundreds or thousands of images, not just of dogs, but other animals, objects, and people, along with the correct label for each image. Given enough data to learn from, the neural network should be able to correctly classify completely new images.

Deep neural networks have been widely applied in the field of computer vision and image recognition, and recently, Barbastathis and others developed neural networks to reconstruct transparent objects in images taken with plenty of light. Now his team is the first to use deep neural networks in experiments to reveal invisible objects in images taken in the dark.

“Invisible objects can be revealed in different ways, but it usually requires you to use ample light,” Barbastathis says. “What we’re doing now is visualizing the invisible objects, in the dark. So it’s like two difficulties combined. And yet we can still do the same amount of revelation.”

The law of light

The team consulted a database of 10,000 integrated circuits (IC), each of which is etched with a different intricate pattern of horizontal and vertical bars.

“When we look with the naked eye, we don’t see much — they each look like a transparent piece of glass,” Goy says. “But there are actually very fine and shallow structures that still have an effect on light.”

Instead of etching each of the 10,000 patterns onto as many glass slides, the researchers used a “phase spatial light modulator,” an instrument that displays the pattern on a single glass slide in a way that recreates the same optical effect that an actual etched slide would have.

The researchers set up an experiment in which they pointed a camera at a small aluminum frame containing the light modulator. They then used the device to reproduce each of the 10,000 IC patterns from the database. The researchers covered the entire experiment so it was shielded from light, and then used the light modulator to rapidly rotate through each pattern, similarly to a slide carousel. They took images of each transparent pattern, in near total darkness, producing “salt-and-pepper” images that resembled little more than static on a television screen.

The team developed a deep neural network to identify transparent patterns from dark images, then fed the network each of the 10,000 grainy photographs taken by the camera, along with their corresponding patterns, or what the researchers called “ground-truths.”

“You tell the computer, ‘If I put this in, you get this out,’” Goy says. “You do this 10,000 times, and after the training, you hope that if you give it a new input, it can tell you what it sees.”

“It’s a little worse than a baby,” Barbastathis quips. “Usually babies learn a bit faster.”

The researchers set their camera to take images slightly out of focus. As counterintuitive as it seems, this actually works to bring a transparent object into focus. Or, more precisely, defocusing provides some evidence, in the form of ripples in the detected light, that a transparent object may be present. Such ripples are a visual flag that a neural network can detect as a first sign that an object is somewhere in an image’s graininess.

But defocusing also creates blur, which can muddy a neural network’s computations. To deal with this, the researchers incorporated into the neural network a law in physics that describes the behavior of light, and how it creates a blurring effect when a camera is defocused.

“What we know is the physical law of light propagation between the sample and the camera,” Barbastathis says. “It’s better to include this knowledge in the model, so the neural network doesn’t waste time learning something that we already know.”

Sharper image

After training the neural network on 10,000 images of different IC patterns, the team created a completely new pattern, not included in the original training set. When they took an image of the pattern, again in darkness, and fed this image into the neural network, they compared the patterns that the neural network reconstructed, both with and without the physical law embedded in the network.

They found that both methods reconstructed the original transparent pattern reasonably well, but the “physics-informed reconstruction” produced a sharper, more accurate image. What’s more, this reconstructed pattern, from an image taken in near total darkness, was more defined than a physics-informed reconstruction of the same pattern, imaged in light that was more than 1,000 times brighter.

The team repeated their experiments with a totally new dataset, consisting of more than 10,000 images of more general and varied objects, including people, places, and animals. After training, the researchers fed the neural network a completely new image, taken in the dark, of a transparent etching of a scene with gondolas docked at a pier. Again, they found that the physics-informed reconstruction produced a more accurate image of the original, compared to reproductions without the physical law embedded.

“We have shown that deep learning can reveal invisible objects in the dark,” Goy says. “This result is of practical importance for medical imaging to lower the exposure of the patient to harmful radiation, and for astronomical imaging.”

This research was supported, in part, by the Intelligence Advanced Research Projects Activity and Singapore’s National Research Foundation. 

“The Laughing Room,” an interactive art installation by author, illustrator, and MIT graduate student Jonathan "Jonny" Sun, looks like a typical living room: couches, armchairs, coffee table, soft lighting. This cozy scene, however, sits in a glass-enclosed space, flanked by bright lights and a microphone, with a bank of laptops and a video camera positioned across the room. People wander in, take a seat, begin chatting. After a pause in the conversation, a riot of canned laughter rings out, prompting genuine giggles from the group.

Presented at the Cambridge Public Library in Cambridge, Massachusetts, Nov. 16-18, "The Laughing Room" was an artificially intelligent room programmed to play an audio laugh track whenever participants said something that its algorithm deemed funny. Sun, who is currently on leave from his PhD program within the MIT Department of Urban Studies and Planning, is an affiliate at the Berkman Klein Center for Internet and Society at Harvard University, and creative researcher at the metaLAB at Harvard, created the project to explore the increasingly social and cultural roles of technology in public and private spaces, users’ agency within and dependence on such technology, and the issues of privacy raised by these systems. The installations were presented as part of ARTificial Intelligence, an ongoing program led by MIT associate professor of literature Stephanie Frampton that fosters public dialogue about the emerging ethical and social implications of artificial intelligence (AI) through art and design.

Setting the scene

“Cambridge is the birthplace of artificial intelligence, and this installation gives us an opportunity to think about the new roles that AI is playing in our lives every day,” said Frampton. “It was important to us to set the installations in the Cambridge Public Library and MIT Libraries, where they could spark an open conversation at the intersections of art and science.”

“I wanted the installation to resemble a sitcom set from the 1980s–a private, familial space,” said Sun. “I wanted to explore how AI is changing our conception of private space, with things like the Amazon Echo or Google Home, where you’re aware of this third party listening.”

"The Control Room," a companion installation located in Hayden Library at MIT, displayed a live stream of the action in "The Laughing Room," while another monitor showed the algorithm evaluating people’s speech in real time. Live streams were also shared online via YouTube and Periscope. “It’s an extension of the sitcom metaphor, the idea that people are watching,” said Sun. The artist was interested to see how people would act, knowing they had an audience. Would they perform for the algorithm? Sun likened it to Twitter users trying to craft the perfect tweet so it will go viral.

Programming funny

“Almost all machine learning starts from a dataset,” said Hannah Davis, an artist, musician, and programmer who collaborated with Sun to create the installation’s algorithm. She described the process at an “Artists Talk Back” event held Saturday, Nov. 17, at Hayden Library. The panel discussion included Davis; Sun; Frampton; collaborator Christopher Sun, research assistant Nikhil Dharmaraj, Reinhard Engels, manager of technology and innovation at Cambridge Public Library, Mark Szarko, librarian at MIT Libraries, and Sarah Newman, creative researcher at the metaLAB. The panel was moderated by metaLAB founder and director Jeffrey Schnapp.

Davis explained how, to train the algorithm, she scraped stand-up comedy routines from YouTube, selecting performances by women and people of color to avoid programming misogyny and racism into how the AI identified humor. “It determines what is the setup to the joke and what shouldn’t be laughed at, and what is the punchline and what should be laughed at,” said Davis. Depending on how likely something is to be a punchline, the laugh track plays at different intensities.

Fake laughs, real connections

Sun acknowledged that the reactions from "The Laughing Room" participants have been mixed: “Half of the people came out saying ‘that was really fun,’” he said. “The other half said ‘that was really creepy.’”

That was the impression shared by Colin Murphy, a student at Tufts University who heard about the project from following Sun on Twitter: “This idea that you are the spectacle of an art piece, that was really weird.”

“It didn’t seem like it was following any kind of structure,” added Henry Scott, who was visiting from Georgia. “I felt like it wasn’t laughing at jokes, but that it was laughing at us. The AI seems mean.”

While many found the experience of "The Laughing Room" uncanny, for others it was intimate, joyous, even magical.

“There’s a laughter that comes naturally after the laugh track that was interesting to me, how it can bring out the humanness,” said Newman at the panel discussion. “The work does that more than I expected it to.”

Frampton noted how the installation’s setup also prompted unexpected connections: “It enabled strangers to have conversations with each other that wouldn’t have happened without someone listening.”

Continuing his sitcom metaphor, Sun described these first installations as a “pilot,” and is looking forward to presenting future versions of "The Laughing Room." He and his collaborators will keep tweaking the algorithm, using different data sources, and building on what they’ve learned through these installations. "The Laughing Room" will be on display in the MIT Wiesner Student Art Gallery in May 2019, and the team is planning further events at MIT, Harvard, and Cambridge Public Library throughout the coming year.

“This has been an extraordinary collaboration and shown us how much interest there is in this kind of programming and how much energy can come from using the libraries in new ways,” said Frampton.

"The Laughing Room" and "The Control Room" were funded by the metaLAB (at) Harvard, the MIT De Florez Fund for Humor, the Council of the Arts at MIT, and the MIT Center For Art, Science and Technology and presented in partnership with the Cambridge Public Library and the MIT Libraries.

The empty frames hanging inside the Isabella Stewart Gardner Museum serve as a tangible reminder of the world’s biggest unsolved art heist. While the original masterpieces may never be recovered, a team from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) might be able to help, with a new system aimed at designing reproductions of paintings.

RePaint uses a combination of 3-D printing and deep learning to authentically recreate favorite paintings — regardless of different lighting conditions or placement. RePaint could be used to remake artwork for a home, protect originals from wear and tear in museums, or even help companies create prints and postcards of historical pieces.

“If you just reproduce the color of a painting as it looks in the gallery, it might look different in your home,” says Changil Kim, one of the authors on a new paper about the system, which will be presented at ACM SIGGRAPH Asia in December. “Our system works under any lighting condition, which shows a far greater color reproduction capability than almost any other previous work.”

To test RePaint, the team reproduced a number of oil paintings created by an artist collaborator. The team found that RePaint was more than four times more accurate than state-of-the-art physical models at creating the exact color shades for different artworks.

At this time the reproductions are only about the size of a business card, due to the time-costly nature of printing. In the future the team expects that more advanced, commercial 3-D printers could help with making larger paintings more efficiently.

While 2-D printers are most commonly used for reproducing paintings, they have a fixed set of just four inks (cyan, magenta, yellow, and black). The researchers, however, found a better way to capture a fuller spectrum of Degas and Dali. They used a special technique they call “color-contoning,” which involves using a 3-D printer and 10 different transparent inks stacked in very thin layers, much like the wafers and chocolate in a Kit-Kat bar. They combined their method with a decades-old technique called half-toning, where an image is created by lots of little colored dots rather than continuous tones. Combining these, the team says, better captured the nuances of the colors.

With a larger color scope to work with, the question of what inks to use for which paintings still remained. Instead of using more laborious physical approaches, the team trained a deep-learning model to predict the optimal stack of different inks. Once the system had a handle on that, they fed in images of paintings and used the model to determine what colors should be used in what particular areas for specific paintings.

Despite the progress so far, the team says they have a few improvements to make before they can whip up a dazzling duplicate of “Starry Night.” For example, mechanical engineer Mike Foshey said they couldn’t completely reproduce certain colors like cobalt blue due to a limited ink library. In the future they plan to expand this library, as well as create a painting-specific algorithm for selecting inks, he says. They also can hope to achieve better detail to account for aspects like surface texture and reflection, so that they can achieve specific effects such as glossy and matte finishes.

“The value of fine art has rapidly increased in recent years, so there’s an increased tendency for it to be locked up in warehouses away from the public eye,” says Foshey. “We’re building the technology to reverse this trend, and to create inexpensive and accurate reproductions that can be enjoyed by all.”

Kim and Foshey worked on the system alongside lead author Liang Shi; MIT professor Wojciech Matusik; former MIT postdoc Vahid Babaei, now Group Leader at Max Planck Institute of Informatics; Princeton University computer science professor Szymon Rusinkiewicz; and former MIT postdoc Pitchaya Sitthi-Amorn, who is now a lecturer at Chulalongkorn University in Bangkok, Thailand.

This work is supported in part by the National Science Foundation.

Virtual reality (VR) technologies are having a growing impact on people's everyday lives. Sanjay Sarma, vice president for open learning, and D. Fox Harrell, professor of digital media and artificial intelligence in the Comparative Media Studies Program and the Computer Science and Artificial Intelligence Laboratory, have combined their efforts to launch MIT Open Learning’s new initiative, the MIT Center for Advanced Virtuality. The new initiative will help determine how MIT can use a group of technologies including virtual and mixed reality (collectively called extended reality or XR) to better serve human needs through artful innovation of virtual experiences, on-campus and beyond.

Harrell’s research explores the relationship between imagination and computation and involves developing new forms of computational narrative, gaming, social media, and related digital media based in computer science, cognitive science, and digital media arts. Harrell announced the creation of the center, which he will direct, in his remarks at this month’s  “Human-Computer Interaction Salon and Mixer,” as part of the Computational Cultures Initiative, sponsored by the School of Humanities, Arts, and Social Sciences.

“The center’s mission is to pioneer innovative experiences using technologies of virtuality,” Harrell said. “Such technologies, ranging from Virtual Reality (VR) to Mixed Reality (MR) and beyond, all use computing to construct imaginative experiences atop our physical world. We endeavor to design and understand how these systems impact how we now communicate, express, learn, play, and work.”

The center — “MIT Virtuality” for short — will bring faculty, researchers, and VR professionals together to create new models for the deployment of impactful XR learning. The center will focus on creation, research, and innovation, through its Studio, Lab, Salon, and Hub functionalities.

The Studio, Harrell explained, will bring professionals and faculty together to innovate new uses for XR, while the Lab will investigate the impacts of these technologies, focusing on learning, simulation and cognition. The Salon and Hub will focus on capacity building and resource sharing, pairing students and experts with resources that will help expand VR technologies across MIT.

Sarma is ethusiastic about the new venture. “AR and VR are new ways of seeing and experiencing, and will be a key tool in changing and improving how we learn,” he said. “This center will advance fundamental research and application of virtual technologies in teaching, training and work.”  

MIT Virtuality aims to enhance the production, research, and innovation capacity of VR at MIT, while investigating the social and ethical impacts of technologies as they are being innovated. Anyone interested in more information is encouraged to visit MIT Virtuality online.

For years, the tech industry followed a move-fast-and-break-things approach, and few people seemed to mind as a wave of astonishing new tools for communicating and navigating the world appeared on the market.

Now, amid rising concerns about the spread of fake news, the misuse of personal data, and the potential for machine-learning algorithms to discriminate at scale, people are taking stock of what the industry broke. Into this moment of reckoning come three MIT students, Irene Chen, Leilani Gilpin, and Harini Suresh, who are the founders of the new MIT AI Ethics Reading Group.

All three are graduate students in the Department of Electrical Engineering and Computer Science (EECS) who had done stints in Silicon Valley, where they saw firsthand how technology developed with good intentions could go horribly wrong.

“AI is so cool,” said Chen during a chat in Lobby 7 on a recent morning. “It’s so powerful. But sometimes it scares me.” 

The founders had debated the promise and perils of AI in class and among friends, but their push to reach a wider audience came in September, at a Google-sponsored fairness in machine learning workshop in Cambridge. There, an MIT professor floated the idea of an ethics forum and put the three women in touch. 

Then when MIT announced plans last month to create the MIT Stephen A. Schwarzman College of Computing, they launched the MIT AI Ethics Reading Group. Amid the enthusiasm following the Schwarzman announcement, more than 60 people turned up to their first meeting. 

One was Sacha Ghebali, a master’s student at the MIT Sloan School of Management. He had taken a required ethics course in his finance program at MIT and was eager to learn more.

“We’re building tools that have a lot of leverage,” he says. “If you don’t build them properly, you can do a lot of harm. You need to be constantly thinking about ethics.”

On a recent night, Ghebali was among those returning for a second night of discussion. They gathered around a stack of pizza boxes in an empty classroom as Gilpin kicked off the meeting by recapping the fatal crash last spring in which a self-driving Uber struck a pedestrian. Who should be liable, Gilpin asked, the engineer who programmed the car or the person behind the wheel?

A lively debate followed. The students then broke into small groups as the conversation shifted to how ethics should be taught: either as a stand-alone course, or integrated throughout the curriculum. They considered two models: Harvard, which embeds philosophy and moral reasoning into its computer science classes, and Santa Clara University, in Silicon Valley, which offers a case study-based module on ethics within its introductory data science courses. 

Reactions in the room were mixed.

“It’s hard to teach ethics in a CS class so maybe there should be separate classes,” one student offered. Others thought ethics should be integrated at each level of technical training. 

“When you learn to code, you learn a design process,” said Natalie Lao, an EECS graduate student helping to develop AI courses for K-12 students. “If you include ethics into your design practice you learn to internalize ethical programming as part of your work flow.”

The students also debated whether stakeholders beyond the end-user should be considered. “I was never taught when I’m building something to talk to all the people it will effect,” Suresh told the group. “That could be really useful.”

How the Institute should teach ethics in the MIT Schwarzman College of Computing era remains unclear, says Abelson, the Class of 1922 Professor of Computer Science and Electrical Engineering who helped start the group and was at both meetings. “This is really just the beginning,” he says. “Five years ago, we weren’t even talking about people shutting down the steering wheel of your car.”

As AI continues to evolve, questions of safety and fairness will remain a foremost concern. In their research at MIT, the founders of the ethics reading group are simultaneously developing tools to address the dilemmas raised in the group. 

Gilpin is creating the methodologies and tools to help self-driving cars and other autonomous machines explain themselves. For these machines to be truly safe and widely trusted, she says, they need to be able to interpret their actions and learn from their mistakes. 

Suresh is developing algorithms that make it easier for people to use data responsibly. In a summer internship with Google, she looked at how algorithms trained on Google News and other text-based datasets pick up on certain features to learn biased associations. Identifying sources of bias in the data pipeline, she says, is key to avoiding more serious problems in downstream applications. 

Chen, formerly a data scientist and chief of staff at DropBox, develops machine learning tools for health care. In a new paper, Why Is My Classifier Discriminatory, she argues that the fairness of AI predictions should be measured and corrected by collecting more data, not just by tweaking the model. She presents her paper next month at the world’s largest machine-learning conference, Neural Information Processing Systems.

“So many of the problems at Dropbox, and now in my research at MIT, are completely new,” she says. “There isn't a playbook. Part of the fun and challenge of working on AI is that you're making it up as you go.”

The AI-ethics group holds its last two meeting of the semester on Nov. 28 and Dec. 12.

Cognitive flexibility — the brain’s ability to switch between different rules or action plans depending on the context — is key to many of our everyday activities. For example, imagine you’re driving on a highway at 65 miles per hour. When you exit onto a local street, you realize that the situation has changed and you need to slow down.

When we move between different contexts like this, our brain holds multiple sets of rules in mind so that it can switch to the appropriate one when necessary. These neural representations of task rules are maintained in the prefrontal cortex, the part of the brain responsible for planning action.

A new study from MIT has found that a region of the thalamus is key to the process of switching between the rules required for different contexts. This region, called the mediodorsal thalamus, suppresses representations that are not currently needed. That suppression also protects the representations as a short-term memory that can be reactivated when needed.

“It seems like a way to toggle between irrelevant and relevant contexts, and one advantage is that it protects the currently irrelevant representations from being overwritten,” says Michael Halassa, an assistant professor of brain and cognitive sciences and a member of MIT’s McGovern Institute for Brain Research.

Halassa is the senior author of the paper, which appears in the Nov. 19 issue of Nature Neuroscience. The paper’s first author is former MIT graduate student Rajeev Rikhye, who is now a postdoc in Halassa’s lab. Aditya Gilra, a postdoc at the University of Bonn, is also an author.

Changing the rules

Previous studies have found that the prefrontal cortex is essential for cognitive flexibility, and that a part of the thalamus called the mediodorsal thalamus also contributes to this ability. In a 2017 study published in Nature, Halassa and his colleagues showed that the mediodorsal thalamus helps the prefrontal cortex to keep a thought in mind by temporarily strengthening the neuronal connections in the prefrontal cortex that encode that particular thought.

In the new study, Halassa wanted to further investigate the relationship between the mediodorsal thalamus and the prefrontal cortex. To do that, he created a task in which mice learn to switch back and forth between two different contexts — one in which they must follow visual instructions and one in which they must follow auditory instructions.

In each trial, the mice are given both a visual target (flash of light to the right or left) and an auditory target (a tone that sweeps from high to low pitch, or vice versa). These targets offer conflicting instructions. One tells the mouse to go to the right to get a reward; the other tells it to go left. Before each trial begins, the mice are given a cue that tells them whether to follow the visual or auditory target.

“The only way for the animal to solve the task is to keep the cue in mind over the entire delay, until the targets are given,” Halassa says.

The researchers found that thalamic input is necessary for the mice to successfully switch from one context to another. When they suppressed the mediodorsal thalamus during the cuing period of a series of trials in which the context did not change, there was no effect on performance. However, if they suppressed the mediodorsal thalamus during the switch to a different context, it took the mice much longer to switch.

By recording from neurons of the prefrontal cortex, the researchers found that when the mediodorsal thalamus was suppressed, the representation of the old context in the prefrontal cortex could not be turned off, making it much harder to switch to the new context.

In addition to helping the brain switch between contexts, this process also appears to help maintain the neural representation of the context that is not currently being used, so that it doesn’t get overwritten, Halassa says. This allows it to be activated again when needed. The mice could maintain these representations over hundreds of trials, but the next day, they had to relearn the rules associated with each context.

Sabine Kastner, a professor of psychology at the Princeton Neuroscience Institute, described the study as a major leap forward in the field of cognitive neuroscience.

“This is a tour-de-force from beginning to end, starting with a sophisticated behavioral design, state-of-the-art methods including causal manipulations, exciting empirical results that point to cell-type specific differences and interactions in functionality between thalamus and cortex, and a computational approach that links the neuroscience results to the field of artificial intelligence,” says Kastner, who was not involved in the research.

Multitasking AI

The findings could help guide the development of better artificial intelligence algorithms, Halassa says. The human brain is very good at learning many different kinds of tasks — singing, walking, talking, etc. However, neural networks (a type of artificial intelligence based on interconnected nodes similar to neurons) usually are good at learning only one thing. These networks are subject to a phenomenon called “catastrophic forgetting” — when they try to learn a new task, previous tasks become overwritten.

Halassa and his colleagues now hope to apply their findings to improve neural networks’ ability to store previously learned tasks while learning to perform new ones.

The research was funded by the National Institutes of Health, the Brain and Behavior Foundation, the Klingenstein Foundation, the Pew Foundation, the Simons Foundation, the Human Frontiers Science Program, and the German Ministry of Education.

To most of us, a 3-D-printed turtle just looks like a turtle; four legs, patterned skin, and a shell. But if you show it to a particular computer in a certain way, that object’s not a turtle — it’s a gun.

Objects or images that can fool artificial intelligence like this are called adversarial examples. Jessy Lin, a senior double-majoring in computer science and electrical engineering and in philosophy, believes that they’re a serious problem, with the potential to trip up AI systems involved in driverless cars, facial recognition, or other applications. She and several other MIT students have formed a research group called LabSix, which creates examples of these AI adversaries in real-world settings — such as the turtle identified as a rifle — to show that they are legitimate concerns.

Lin is also working on a project called Sajal, which is a system that could allow refugees to give their medical records to doctors via a QR code. This “mobile health passport” for refugees was born out of VHacks, a hackathon organized by the Vatican, where Lin worked with a team of people she’d met only a week before. The theme was to build something for social good — a guiding principle for Lin since her days as a hackathon-frequenting high school student.

“It’s kind of a value I’ve always had,” Lin says. “Trying to be thoughtful about, one, the impact that the technology that we put out into the world has, and, two, how to make the best use of our skills as computer scientists and engineers to do something good.”

Clearer thinking through philosophy

AI is one of Lin’s key interests in computer science, and she’s currently working in the Computational Cognitive Science group of Professor Josh Tenenbaum, which develops computational models of how humans and machines learn. The knowledge she’s gained through her other major, philosophy, relates more closely this work than it might seem, she says.

“There are a lot of ideas in [AI and language-learning] that tie into ideas from philosophy,” she says. “How the mind works, how we reason about things in the world, what concepts are. There are all these really interesting abstract ideas that I feel like … studying philosophy surprisingly has helped me think about better.”

Lin says she didn’t know a lot about philosophy coming into college. She liked the first class she took, during her first year, so she took another one, and another — before she knew it, she was hooked. It started out as a minor; this past spring, she declared it as a major.

“It helped me structure my thoughts about the world in general, and think more clearly about all kinds of things,” she says.

Through an interdisciplinary class on ethics and AI ethics, Lin realized the importance of incorporating perspectives from people who don’t work in computer science. Rather than writing those perspectives off, she wants to be someone inside the tech field who considers issues from a humanities perspective and listens to what people in other disciplines have to say.

Teaching computers to talk

Computers don’t learn languages the way that humans do — at least, not yet. Through her work in the Tenenbaum lab, Lin is trying to change that.

According to one hypothesis, when humans hear words, we figure out what they are by first saying them to ourselves in our heads. Some computer models aim to recreate this process, including recapitulating the individual sounds in a word. These “generative” models do capture some aspects of human language learning, but they have other drawbacks that make them impractical for use with real-world speech.

On the other hand, AI systems known as neural networks, which are trained on huge sets of data, have shown great success with speech recognition. Through several projects, Lin has been working on combining the strengths of both types of models, to better understand, for example, how children learn language even at a very young age.

Ultimately, Lin says, this line of research could contribute to the development of machines that can speak in a more flexible, human way.

Hackathons and other pastimes

Lin first discovered her passion for computer science at Great Neck North High School on Long Island, New York, where she loved staying up all night to create computer programs during hackathons. (More recently, Lin has played a key role in HackMIT, one of the Institute’s flagship hackathons. Among other activities, she helped organize the event from 2015 to 2017, and in 2016 was the director of corporate relations and sponsorship.) It was also during high school that she began to attend MIT Splash, a program hosted on campus offering a variety of classes for K-12 students.

“I was one of those people that always had this dream to come to MIT,” she says.

Lin says her parents and her two sisters have played a big role in supporting those dreams. However, her knack for artificial intelligence doesn’t seem to be genetic.

“My mom has her own business, and my dad is a lawyer, so … who knows where computer science came out of that?” she says, laughing.

In recent years, Lin has put her computer science skills to use in a variety of ways. While in high school, she interned at both New York University and Columbia University. During Independent Activities Period in 2018, she worked on security for Fidex, a friend’s cryptocurrency exchange startup. The following summer she interned at Google Research NYC on the natural language understanding team, where she worked on developing memory mechanisms that allow a machine to have a longer-term memory. For instance, a system would remember not only the last few phrases it read in a book, but a character from several chapters back. Lin now serves as a campus ambassador for Sequoia Capital, supporting entrepreneurship on campus.

She currently lives in East Campus, where she enjoys the “very vibrant dorm culture.” Students there organize building projects for each first-year orientation — when Lin arrived, they built a roller coaster. She’s helped with the building in the years since, including a geodesic dome that was taller than she is. Outside of class and building projects, she also enjoys photography.

Ultimately, Lin’s goal is to use her computer science skills to benefit the world. About her future after MIT, she says, “I think it could look something like trying to figure out how we can design AI that is increasingly intelligent but interacts with humans better.”