Programming with Examples to Develop Data-Intensive User Interfaces
7/15/2016 IDE PX IEEE Computer 49(7)
Abstract
The programming-with-examples (PwE) workflow lets developers create interactive applications with the help of example data. It takes a general programming environment and adds dedicated user interfaces for visualizing and managing the data. This lets both programmers and users understand applications and configure them to meet their needs.
Figure 1. Two workflows for developing data-intensive interactive systems. (a) Programming by example (PbE). (b) Programming with examples (PwE). Unlike PbE systems, PwE systems allow the representation of example data and enable explicit programming. IDE: integrated development environment.
- Jun Kato, Takeo Igarashi, Masataka Goto, "Programming with Examples to Develop Data-Intensive User Interfaces", In Computer 49(7). pp.34-42, Jul. 2016. Special Issue on 21st User Interfaces. BibTeX
What is (and what is not) "Programming with Examples"?
It is...
- It is the workflow of programmers that makes use of example data during the development of applications.
- It is often required for the development of modern interactive systems that have become more and more data-intensive.
- Example target applications include IoT, robot control, gesture recognition, image processing, and animation, and others that require intensive parameter tuning such as machine learning.
- Programmers have often developed their own tools for the purpose such as tools for visualizing recorded sensor data. We argue that such features should be built into IDEs to provide fluid programming experience.
It is NOT...
- It is different from typical PbE systems that encapsulate and hide the implementation of programs from the user. It requires explicit programming. (See Figure 1 above.)
- It is slightly different from the traditional definition of "Programming with Example" that appears in Brad Myers' survey paper on PbE systems [CHI '86]. It indicated the omission of inferences in PbE systems and did not discuss data representations.
- It is different from Joel Brandt's "Example-Centric Programming" that helps programmers to benefit from existing example code on the web. Examples in his work denote the use of APIs (the logic of the programs) and examples in PwE denote data used in the programs.
Which IDEs support "Programming with Examples"?
We have designed IDEs that support the PwE workflow. Some of them are implemented as WIDEs and you can try them out! At this moment, those with Live Tuning interfaces are publicly available. See TextAlive at http://textalive.jp and f3.js at http://f3js.org.
Picode [CHI '13] supports handling robot/human posture information with the code editor that shows inline photos representing the posture data.
DejaVu [UIST '12] supports creating interacitve camera-based apps with the timeline and canvas interfaces for built-in record-and-replay and re-evaluation feature.
VisionSketch [GI '13] implements graphical annotation-based code completion where methods are suggested based on sketched shapes on input images.
TextAlive [CHI '15] supports authoring Kinetic Typography videos of lyrics text synchronized with songs equipped with Live Tuning interfaces for end-users.
There are of course excellent prior work and other ongoing work that supports the PwE workflow.
Programming with Examples in conventional IDEs
Conventional CUI and GUI applications do not involve continuous and complex data handling. As a result, their development environments typically support simple text-based or symbol-based representations of example data.
- Most PbE systems hide the program implementation from the user to require no programming knowledge, but exceptions exist. Pygmalion lets programmers kick-start building a program by giving concrete input data instead of writing abstract program code.
- The Subtext IDE [OOPSLA '05] lets programmers specify test cases by writing example input data next to program code in a code editor.
Programming with Examples in IDEs for data-intensive applications
To develop data-intensive applications, more intuitive (typically graphical) representations of example data are required other than text-based representations.
- d.tools [UIST '06] is a visual authoring environment for prototyping physical computing devices. It is capable of recording a video of the user's interaction with the physical device, logs interaction events, and play it in a video player-like interface.
- Gestalt [UIST '10] provides integrated support for implementation and analysis in machine learning algorithms and influences most of our projects. It comes with rich graphical representations of example data collected for the application development.
- Timelapse [UIST '13] and its subsequent work are built into the developer tool of WebKit project that captures and replays web application executions with the Timeline interface.
- Gneiss [CHI '15] and its related work extend a spreadsheet-based user interface for handling continuous or structured data retrieved from the web.
Please contact us to add your project here.
What are the design implications for future IDEs?
While we would strongly recommend you to read the article (p.40-41) for the detailed discussion, here are some lessons learned in case you are in a hurry.
- Intuitive representations of example data are important. Researchers have proposed many intuitive interfaces for manipulating data through GUIs; we can learn from such interfaces to integrate data manipulation components into IDEs. Although the article focused on inherently visual example data, example data could be sounds, haptic sensations, tastes, and smells. We foresee that future development environments could exploit human sensory organs other than the eyes, including the ears, skin, tongue, and nose. A research question for you: how would you debug virtual reality applications with various kinds of sensations?
- Collaborative application authoring is the key in the age of big data. Collaboration among people with diverse backgrounds is essential for developing successful user interfaces for data-intensive applications in the age of big data. The amount and variations of data far surpass what programmers alone can manage and edit. One potential answer to this implication is Live Tuning that expands benefits of Live Programming to end-users.
- Borders between the development and runtime environments will be eliminated. Programmers will ship programs together with the development environments so that nonprogrammers can customize them by updating the example data. In such environments, representations of the example data serve as the boundary that defines what can be easily customized and what cannot. We believe that user interface design will become a process to create applications with such a boundary that defines the appropriate flexibility.
Revision history
- 7/15/2016
- Upload the initial revision.