User Interface Toolkits and Adaptive Interfaces

From CS2610 Fall 2017
Jump to: navigation, search

Slides

slides


Readings


Related Resources


  • Proximity Toolkit is a catalyst for developing applications that make use of spatial information, and relations between objects in space.
  • Direct manipulation vs. interface agents, Ben Shneiderman, Pattie Maes, ACM interactions, Volume 4 Issue 6, Nov./Dec. 1997 (original debate happened in CHI 1997)

Reading Critiques

Jonathan Albert 17:04:28 9/19/2017

UI Tools: This paper catalogues factors that contribute to the success and downfall of various interfaces along with examples of designs that work well and those that do not. It then recognizes upcoming trends and offers suggestions for what may solve future problems. The paper's recognition that "there is a clear need for new interaction paradigms that minimize the amount of attention demanded" is significant. Current trends are in the opposite direction, unfortunately. Facility of switching to a task on a different device or about a different subject is minimized, with each new app attempting to monopolize the user's attention. While corporations benefit from increased usage statistics, global productivity is hindered. In this vein, the authors' suggestion that devices "should be cognizant of multiple people" can be applied to devices being aware of other devices. With the advent of smart watches, smart phones, tablets, and laptops, single users may have several devices in close proximity--and their locus of attention restricts them to using one at a time. While, the disparate variety of vendors makes total synchronization difficult, some devices should have some built-in "attention harmonizers." For instance, smart watches should not display notifications when the phone to which they are linked is unlocked and can detect the user's face. ---- Mixed UI: This paper focuses on techniques to make systems and UIs more intelligent with regards to offering services at appropriate times. The author details the workings of a component of Microsoft's Outlook program, showing how these techniques were incorporated in it. The discussion about probabilities with regards to a user's goal and desires was intriguing. However, I think not enough hard detail was provided--e.g., equations or statistics. As it stands, this paper seems as though it leans away from academics into the realm of advertisement. Principles are stated that can be applied by the diligent, but any concrete detail seems to be purposely hidden. The paper should seem less like a veiled attempt at boasting about trade secrets for it to be taken more seriously. Nevertheless, I have never seen a Bayesian approach applied to launching a dialog before reading this paper. The idea of learning a user's behavior to determine when they might be receptive to a particular feature corroborates with today's other reading's discussion about a user's attention. While on-screen genies or talking paper clips have rightly disappeared into obscurity, the principles behind an adaptive user interface could be promising. Studying how this dynamism works with user virtuosity may yield interesting results, considering how computer-related virtuosity generally develops as a user masters a relatively static interface.

Kadie Clancy 14:07:47 9/20/2017

Past Present and Future of User Interface Software Tools: This paper examines the successes and failures of past user interface tools, which is as useful study as almost all applications today were built using some form of these tools. User interface tools prove crucial to the iterative design process, as less code is required to create an interface, leading to faster prototyping. The authors outline themes that seemed to have an impact on the success of a tool based on their evaluation. These themes included points like predictability of the system, path of least resistance, and how well the tools adapted to the moving target of interface design. The authors also speculate what consequences emerging trends may have on future interfaces and, in turn, what effects they will have on future interface software tools. It is interesting that even in 1999 (when this draft of the paper was published), the authors identified the need for new types of interfaces catered to drastically different devices than the desktop computers to address trends like ubiquitous computing and recognition based interfaces, which are standard interface types today. To address these trends, the authors identify the need for tools to aid with new issues like collaboration among bluetooth connected devices, and the need for replaceable interfaces for the same applications to allow rendering on different devices. Similarly, the authors also speculate on the emergence of recognition based systems and how they will require feedback to allow the user to monitor and correct actions. The paper stresses the importance of investigating past ideas and technologies for inspiration in the creation of successful products in the future. Principles of Mixed Initiate User Interfaces: Mixed-Initiative user interfaces are interfaces that enable users and intelligent agents to collaborate effectively while also allowing for the direct manipulation of objects. Interfaces utilizing agents typically suffer from key problems including poor guessing about the goals of the user, poor timing of actions, and insufficient regard to the costs and benefits of automated actions. These issues lead Horvitz to outline the critical factors for effective integration of direct manipulation with automated services. These factors include principles like considering the focus of the user’s attention, providing automated services with genuine value, and providing dialog to resolve key uncertainties. In essence, a mixed-initiative interface should act as a benign personal assistant by being helpful to user tasks while dealing with the uncertainty about a user’s goals. These principles and methods were illustrated with examples from the LookOut system, which is an overlay of automated scheduling services on Microsoft Outlook (a direct-manipulation scheduling and messaging system). The LookOut system has several aspects that utilize the principles for effective interaction. For example, LookOut can be configured to operate in several automated-assistance modalities, one employing a hands-free mode with TTS and automated speech recognition. Lookout also infers whether an email is calander relevant or not based on a linear SVM classifier, helping to address the problem of user goal uncertainty. The system determines to take automated action or not based on a probabilistic user model, which aids in the consideration of user’s attention. The creators of Lookout also identified a sigmoid relationship between message size and preferred time for deferring services which can be generalized to construct a model of attention. Lookout also continues to learn from users beyond the pre trained probabilistic and timing models by periodically training on real-time email examples, user response, and the timing of that response. This paper is important as it identifies a set of principles for designing Mixed-initiative interfaces that address systematic problems with the use of automated agents, and then provides practical examples of these principles in the Lookout system. This paper also provides framework to manage the uncertainties that an agent may face, which will be helpful for future designers who are considering developing interfaces of this type.

Xingtian Dong 16:56:03 9/20/2017

1. Reading critique for ‘Past, Present and Future of User Interface Software Tools’ I think this paper is very useful, especially for the indicators to evaluate tools. Low threshold and high ceiling should be a principle for many tools. In the successful examples which the author provided, most of them have low threshold and high ceiling. Although some tools are designed to be easy to use, their unpredictability make them not desirable to use. So multiple indicators should be considered. And the future prospects and visions that the author brought is very impressive, it can be a good example to predict future trends based on current techniques. And the indicators to evaluate tools is also very helpful for designing interfaces. And with the improvement of technology, more indicators should be taken into consideration. The examples which the author listed are also helpful, the author analyzed why some example are successful and why others failed. They are good example for us to analyze interfaces and todays’ tools. 2. Reading critique for ‘Principles of Mixed-Initiative User Interfaces’ I think there two points useful in this paper. First, the author brought out critical factors for the effective integration of automated services with direct manipulation. And the author uses ‘LookOut’ as example to show how he dealt some of the problems. The most interesting part is that the author found a way to calculate how much should a tool take autonomous action to assist the user with an action given observed evidence. He converted a vague concept to a graphical that can be analyzed mathematically. It inspires us how to convert a concept to mathematical problem.

Tahereh Arabghalizi 17:25:53 9/20/2017

Past, Present and Future of User Interface Software Tools: In this paper authors state that innovation in tools draw innovation in user interface design because it only makes sense to develop tools when you know for what kinds of interfaces you are building them. They believe that new tools in the future will be organized for providing a rich context of information about the user, the devices, and the application’s state instead of events. This will enable end-user programming, recognition-based user interfaces and the data sharing needed for pervasive computing. They recommend that tools aim to have a low threshold so they are easy to use, but still provide a high ceiling. Threshold refers to how easy it is to learn to use a program, and ceiling refers to the limit of what is possible to achieve by learning it. The authors also denote that predictability is very important to programmers and should not be sacrificed to make the tools smarter. The predictions of this paper about ubiquitous computing seem very interesting because nowadays most companies try make such tools, for example Google products like Google Doc, Google Data Studio, etc. ------------------------------------------------------------------------------------------------ Principles of Mixed-Initiative User Interface: In this paper the authors discussed the key challenges for building mixed-initiative user interfaces that enable users and intelligent agents to collaborate efficiently. They presented a set of principles for designing mixed-initiative user interfaces that address problems using agents that can guess about users’ needs. They also address methods to manage the uncertainties that agents might have about users’ objectives. They focus on methods and principles with an examples about the Lookout system which is a system prototype for aiding users with calendar organizing and scheduling. The ideas of this paper are used in mail applications nowadays that increase emails with hypertext links to schedule actions. The authors believe that continuing efforts to handle problems in design of mixed-initiative user interfaces can provide basic improvements in HCI. I think although this work was innovative at its time but social agent concept which is mentioned in this paper is no longer used nowadays.

MuneebAlvi 18:40:43 9/20/2017

Critique of Past, Present and Future of User Interface Software Tools Summary: This paper emphasizes the needs of good user interface design tools in 1999. It also predicts the future of interfaces and thus the need to drastically change the interface design tools. The paper mentions interface design tools which use scripting to add the needed complexity to the interface. I made an app for iOS and Android using the Unity engine and I suppose I never realized the depth that the interface building tool offered me in the Unity IDE. The IDE allowed me to places interface objects but it would also try to line them up in order to make the interface more appealing. Also, I was able to modify the object's properties very easily through various different methods (direct interaction with the object vs entering property values in a property chart) and I was able to add complexity to the objects through scripting. I never thought that many modern concepts were actually popular in the field over 17 years ago. I am also very amazed at how well the concepts in this paper still hold up to today's standards and how close the predictions were for modern user interfaces on a variety of devices. The paper predicted that modern interfaces would be dramatically different due to the variety of devices that would support them. While I agree with this in terms of mobile devices, I believe that many of the fundamental principles in this paper still hold for the modern desktop. Critique of Principles of Mixed-Initiative User Interfaces Summary: This paper shows an example of an interface that allows both direct manipulation of the objects and automated assistance by agents. The paper also shows scenarios where agents may or may not be useful. The principles of this paper transfer to many popular services that we use today such as Siri or modern email. The paper mentions having animated characters to provide a better social model to interact with customers but I believe that modern systems such as Siri or Google do a great job of trying to speak as naturally as possible in order to match the social expectations of the user. Other modern tools analyze incoming text messages and try to either schedule appointments or open a map app to show the location of an upcoming event. On my iPhone, if I have boarding passes, the phone pulls them up automatically near my flight times. I think this is a relatively clear prediction model since users have a much higher percentage of actually taking the flight

Ahmed Magooda 19:47:17 9/20/2017

Past, Present and Future of User Interface Software Tools This paper tackles the idea of user interface toolkits or software tools, the paper started describing the state of user interfaces and how most user interfaces are based on desktop, keyboard and mouse paradigm and the paper then goes by evaluating tools over multiple aspects like (threshold, ceiling, path of least resistance, predictability and moving target). The paper introduces the importance of user interface toolkits and software tools as they make the programmers job much easier and at the same time can be used as leverage to achieve the difficult goal of maintaining consistency. The paper then goes on to introduce some of the successful toolkits and software while also provide some that couldn't last for too long due to either high ceiling or the moving target issue. I think the paper is interesting and provides some predictions that actually happen to be true nowadays. ---------------------------------------------------------------------------------------- Principles of Mixed-Initiative User Interfaces In this paper the authors are discussing mixing two of the basic user interface paradigms (1- enhance capability of manipulating objects; 2- automated interface agents) in order to make a better user experience, the authors then go on by introducing some factors that governs the quality of such mixture which are (Provide mechanisms to refine results, Minimizing poor guesses, Considering uncertainty, etc...). The authors introduced and example tool as an example of the mixing approach, the tool is called LookOut. In this tool the agent tries to automate the process of filling calender by various meetings and appointments through automatic parsing and understanding of the emails. The tool uses svm which as a linear classifier to differentiate between mails that should be represented in to the user in a calender and mails that should be just neglected. The also keeps learning from the user behaviour so that it can enhance it predictions and tailor its models to the user preferences. I think a positive point to the paper is providing an example to illustrate each of the aspects it illustrates, along side providing some analysis of the tool behaviour and models.

Xiaoting Li 20:45:18 9/20/2017

1. Principles of Mixed-Initiative User Interfaces: In this paper, the author exemplifies how to mix automation with direct manipulation with the design of a Microsoft product named Lookout. The Lookout system uses SVM and probability analysis to continually learning and regenerating models to manage the uncertainty that the system may have about users’ goals. The highlight of this product is that it has the capability for life-long learning. It uses machine learning techniques to keep training the system so as to improve the performance of the system. Even though the version of the system at that time was trained initially on small volume of data (approximately 1000 messages), the system has the capability to keep learning based on new data from users’ everyday interactions with it. This can help incrementally reduce the uncertainties that the system has about users’ goals. The Lookout system mainly focuses on scheduling based on text in English. The future work can try to add similar functions in other languages. 2. Past, Present and Future of User Interface Software Tools: In this paper, the authors point out several promising changes in some areas and the state that changes in these areas will require support from user interface software tools. The authors first give examples of the existing user interface software tools and point out which are successful tools and which are unsuccessful ones, and what we can learn from these tools for our future design. Then, the authors give their predictions for the future of user interface tools. Their predictions give us significant implications of the requirements for the future design. When the authors mention the varying input and output capabilities of the future design, it reminds me of the use of HTML5 nowadays. HTML5 enables developers to create responsive web pages, which adjust the same web content based on the size of screen. With HTML5, users can enjoy web content on different platforms.

Sanchayan Sarkar 20:55:54 9/20/2017

CRITIQUE 1 (Past, Present and Future of User Interface Software Tools)------- In this paper, the authors narrates a history of strengths and weaknesses of past user-interface software tools and the challenges that software tools need to overcome to meet the prospects of future user-interface designs. One of the merits of this paper is that it delineates a series of themes for evaluating tools like thresholding, ceiling , path of least resistance , predictability, moving targets and judges how the interface software tools of the past fare against these themes. Few of the successes which are of interest are overlapping windows, interactive graphical tools and scripting languages. The author does well in illustrating why these things worked. For example, the overlapping windows does well in managing both computer and cognitive resources and thereby enhance ease. Interactive graphic tools like Visual Basic help in reducing the difficulty of coding thereby reducing the threshold that the programmer needs to learn. On the other hand scripting languages with interface builders like VC# with a C# base can maintain a fair tradeoff between threshold and ceiling (complexity and functionality). The paper also provides a reasoning of failures like in case of formal language tools, which despite providing high level of interaction did not get world-wide acceptance because of Direct Manipulation Interfaces (a case of moving targets). This is analogous to the fall of Nokia non-touch devices which even though were powerful just couldn’t matchup to the directness of the touch phones. Another thing to notice is that most of the software tool features like automatic techniques, high level abstraction in case of UI management systems have failed mostly because of lack of access to low level interacting features and its’ inability to generate predictable, low threshold software tools. So the authors state that key themes like “Modularity”, “Consistency”, “Predictability”, “Low-Threshold” are something that future UI software tools need to look after. Another useful feature of this paper is its’ vast discussion on future challenges like “Ubiquitous Computing”, where multiple interfaces of have to communicate with each other and with various scales of input and output modalities. This is so relevant in case of “Skype”, where a desktop, a laptop and phone can do a conference call sitting thousands of miles apart. New UI software tools must facilitate options such that other than the usual themes, issues like “Security”, ”Synchronization” and “output delivery” could be better handled. Further issues would be creating tools for 3D environments as the “rendering” complexity needs to be accounted for. Another issue which I liked was taking into account the “skill and dexterity of the users”. Since machines are an everyday commodity now, varied age groups use it. New software tools must be aware of this. This is why the paper is so important as it brushes through all the critical issues and aspects that the next generation of research must direct in order to create tools that provide designers opportunities to create next generation User interfaces.******-----------------------------------------------------------------------------------------------*****CRITIQUE 2 (Principles of Mixed Initiative User Interface)---- The main purpose of this paper is to show the principles of integrating direct manipulation user interaction with that of automatic reasoning. The author uses the example of “LookOut”: a scheduling system to demonstrate the principles of tradeoff in a mixed-initiative user interface. The basic goal of this paper is to show where should a system employ automatic reasoning and where it should hold back to ensure direct manipulation. One of the great features about this paper is demonstrating most of the principles using a real system; “Look Out”. For example, one of the principle is for a system to scope the precision of automation when it comes to deal with larger uncertainties. This is done in “LookOut” where when the system is unable to determine the exact date of the schedule, it reduces its’ precision scope by showing the user a “week” for possible schedule. Also principles like employing socially appropriate behavior is done using “social agent modality”; ensuring dialogues to occur when it faces a lot of uncertainty is also aptly demonstrated. Another interesting feature about ‘LookOut’ is the principle of learning user goals. This is done by assigning probabilistic likelihoods of user intentions as “calendar relevant” and using a pre-trained support vector classifier to classify the messages based on these likelihoods. The paper also gives a brief decision model on the action vs inaction issue by basing it on the probability that an action or inaction will occur when the goal has or has not been determined by the set of evidences (“P(G/E)”). Figure 4 explains it perfectly. What is important is the threshold probability beyond which an action should occur. It is interesting that this threshold changes when dialog-goal is modelled using this graph. Also the relationship between the two thresh-hold is good in determining the likelihoods beyond which the system should give into automatic action or hold back for direct manipulation. This paper also layouts new direction of research in mixed-initiative user interfaces. In particular, it states temporal modelling between outcomes and messages can also have factors like ambiguity, complexity, etc. and new research must focus on that. Also, in case of continuous learning of user’s past actions to predict future outcomes can be largely expanded by introducing advanced machine learning algorithms. This paper is of 1999 and as of 2017, we know a successful AI system (Siri) of Apple which have done this. Siri learns from user actions and offers seamless tradeoffs between automatic actions and direct manipulation through prompts and internet searches. Also, modules of speech recognition and perception analysis, that is basic in ‘LookOut’, is quite advanced in Siri. Hence, in retrospect, one can understand the importance of this paper and how even in modern times, the principles of mixed-initiative user interfaces remain solid.

Spencer Gray 21:18:56 9/20/2017

In the first paper, Past Present, and Future of User Interface Software Tools, the authors present a survey of the history, the modern, and the future tools that software developers take advantage of in order to build interactive user interfaces. From the past and present software tools, the authors create a set of themes that they believe should shape the future interfaces tools that we build and implement. I found this to be a very interesting and useful approach to this problem. This approach is widely applicable to all types of problems. In many cases, there is no need to "reinvent the wheel". We should learn from the successes and failures of previous tools in order to shape our future tools. If I were to rewrite this paper today, I could improve on it just because this paper was written 18 years ago. Since it is outdated, there are a lot of themes, such as the Internet of Things, that could be analyzed to discover new themes. That being said, all of the themes discussed in this paper are still applicable, which results in this paper being a significant addition to the HCI literature. In the second paper, Principles of Mixed-Initiative User Interfaces, the authors describe the pitfalls of designing a system that relies too much on direct manipluation and designing a system that relies too much on automation. Both of these things can be over done. Instead, the authors show the virtues of creating an interface that uses a mix of both. Their example is the LookOut system that was created for an email manager to add events to a calendar. I liked the approach that these authors used. First, they explained why the problem exists and why it is relevant to fix it. Next, they identified 12 factors that a solution should take into account. Lastly, they showed how these factors were successfully applied to a system. If I had to rewrite this paper today, one part I would change is how the SVM text classifier was trained. They trained it with 500 relevant messages and 500 irrelevant messages. This ratio of relevant to irrelevant messages does not reflect real life. Instead, more work should have been done to analyze the ratio of relevant to irrelevant messages. The context of the email account would be important in this. For personal use, it could be that more than half the messages will be irrelevant or spam. For a work account, less than half would probably be irrelevant.

Yuhuan Jiang 0:14:09 9/21/2017

Paper Critiques for 09/21/2017 == Past, Present and Future of User Interface Software Tools == This paper is a survey of user interface software tools that are used by developers to design and implement the user interface (hereafter, UI). Not only do the authors summarize the existing tools, they also extract a set of themes that can be used to guide future work. The authors categorize different UI software tools by themes, e.g., the parts of the user interface that are addressed, threshold and ceiling, path of least resistance, etc. The paper first discussed the successfully tools, such as window managers, event languages, and interactive graphical interface builders, and OOP. Then, the paper summarized some promising approaches, such as UIMS, formal language for UI building, and constraint-based layout. The authors identified problems of threshold and ceiling effects in UI tools. The path of least resistance offered by a tool is also an issue. To relate to today’s technologies, one cannot overlook the interface builder in Apple’s Xcode. Much of the UI development is automated. For example, the layout of each component (buttons, text boxes, …) in the screen are automatically computed using a few relative position constraints. This is an excellent implementation of one of the “promising approaches” mentioned in the paper. == Principles of Mixed-Initiative User Interfaces == This paper is a review of the challenges and opportunities for building mixed-initiative user interfaces. A mixed-initiative user interface is defined as one in which users collaborate with intelligent agents. The authors first used some principles that developers can follow to design mixed-initiative UIs. The critical factors for the effective integration of an intelligent, automated service with direct manipulation interfaces are: developing significant value-added automation, considering uncertainty about a user’s goals, a user’s attention int he timing services, and inferring the ideal action, … The authors presented a demo system named LookOut for testing. To show how to model user’s goals, LookOut infers the user’s goals using SVM to classify the user’s emails as “scheduling/review a calendar” vs. “not scheduling/review a calendar”. The idea of modeling user’s uncertainty is an interesting one. In addition to the actions introduced by the paper, I think there is a richer space of user’s intentions that will rely on a more powerful probabilistic framework (such as graphical models) for inference.

Charles Smith 0:40:12 9/21/2017

On: principles of mixed-initiative user interfaces The author of this paper describes their design of an intelligent input system that goes beyond the usual interaction, but uses smart techniques to reduce how much a user is required to enter. Specifically, the author describes the implementation of lookout. As we all know from today’s computing, the era of animated characters helping us is long gone. These were not found to be favorable by many user. But why? My speculation is that these loud and in your face implementations were replaced with quiet background tasks that take the idea that if you do things right, people won’t be sure you’ve done anything at all. Taking a more positive look, the idea of predicting what a user wants to do, and performing some, or all, of that task automatically can drastically increase performance. This could work possibly better than any input device ever could. Creative prediction can be used in basically any application. On: Past, Present, and Future The author of this paper takes a look at what worked, and what had issues, with interfaces at the time and makes predictions of the ‘future’. While interesting to see the light that the past sees the present in, some of the ideas are a bit dated. The author talks about interconnected devices and computers in mounted in things, but this is already the reality. I believe it would be more relevant to look at a more modern paper that takes the same look as this one did.

Mingzhi Yu 0:44:07 9/21/2017

For the first paper, the mixed-initiative UI is overall more intelligent by using agent, a new kind of automated service, to help user. The author gives an example of Lookout scheduling and meeting system which combined with system’s own analysis and user’s directly manipulate. The system has the ability to analyze the email and looking for a goal specific content to calendaring and scheduling. At the same time, the system also give user the authority to direct or edit the valuable decisions when under uncertainty. The challenge of this UI is how to efficiently collaborate the user and agent, which comes down to the timing of service, consideration of uncertainty and many other elements. Overall, author believe that mixed-initiative user face is valuable to human-computer interaction if it is continuing being used. The second paper" Past, Present and Future of User Interface Software Tools" discusses that given the fact that vast verity of user interface tools has been developed to accommodate different software uses, the user interface design will change thoroughly in near future due to the rise of ubiquitous computing, recognition-based user interface, 3D technologies and others. The author analyzed the past and present interface tools, technology and their constrains. Given the introduction of ubiquitous computing, recognition-based user interface, 3D technologies and others, the author then provided the evidence why they will become the trend. The author also gives a expectation that future user interface tools will focus more on the user, the device and state and will maintain predictability. Author also listed out the further issues for future tools and recommend tools shall have low threshold and high ceiling.

Krithika Ganesh 2:23:35 9/21/2017

Past, Present and Future of User Interface Design: The paper sets the stage by first explaining the themes that can be used to evaluate the interface tools, moves on what to what tools could make it to success and the tools which failed, articulating what can be learned out of the failed tools so that there is scope for improvement while designing the interface tools in the future. The author also enunciates that the interface tool should be designed such that it should have “low threshold” and “high ceiling”. This grabbed my attention and made me look out for such tools, finding FabLab to be one such tool. The author states that one should provide replaceable interface, based on the user requirements, which is true as most old people find it difficult to use standard smart phones. What are old people looking for? : simple interface, large screen and something easy to carry about. He also addresses the ‘moving target problem’, around the time of 90s, disks kept flash at bay, but finally, the flash cost-per-bit dropped below that of DRAM, and the flash won the smartphone market, which led to flash prices dropping faster than disks. The author mentions that drastic change is required for designing user interface by the rise of “ubiquitous computing”, which is so relevant today. Most of the Internet of Things (IOT) devices are based on Ubiquitous Computing like Apple Watch, Amazon Echo Speaker, Amazon EchoDot and Fitbit. Also, I don’t completely agree that any project is a complete failure. According to the ‘long nose of innovation’ I believe that some of the past failed projects may be successful in the coming years. Principles of Mixed Initiative User Interfaces: This paper reviews the challenges and opportunities for building mixed initiative user interfaces combining – 1) interfaces that enable users to directly manipulate interfaces and 2) intelligent agents which provide automated services. Taking an example of the LookOut project, the author elaborates on the concepts of: Value added service, Multiple interaction modalities, handling invocation failures, inferring beliefs about a user’s goal, how to convert belief to action, expected utility and threshold probabilities, user attention and timing of service. It is mentioned that the LookOut was trained with 1000 messages. I was curious to know details of the test data and also what kind of messages were used to train the LookOut data which seems to have been missing in the paper. Also, were 1000 messages good enough a number for training data set? It would be interesting if the paper bench-marked how well LookOut performed quantitatively (graphs) than just implicitly stating. What excites me is that, Lookout guesses about the automated scheduling services as well as estimates the best time to invoke the services. To my surprise, Lookout is no more in market, Microsoft pulled the plug on it in 2007 and a replacement I found was Lookeen. What is even mysterious is that I did not find any article describing the reason behind the downfall of Lookout.

Ronian Zhang 6:57:29 9/21/2017

Past, Present and Future of User Interface Software Tools: This paper gives the evaluation methods for tools, researches the success and failure of the past interface tools, explores predictions and observations on future tools, and discusses a little bit on OS side. It gives a strong point: user interfaces should be changing a lot in the near future and what might not be applied in the past might make a different on the future interface develop tools. The paper is visionary: the consideration of the change of diversity really happened: different size of the devices have various interfaces now, new interface developing tools is easier to use (component & coding are combined, barely programming only tool exist now), limited form of constraint only have application on publishing, recognition-based input (siri), react for continuous behavior. But some are remain unsolved: threshold & ceiling dilemma, problems of various resolution, data sharing & syn issues. By looking back, there are lessons to be learned and is possible to find solutions from the abandon methods.————————————————————————————— Principles of Mixed-Initiative User Interfaces: this paper shows us a way to combine automated actions and usr’s direct manipulation into the evaluation system to make comprehensive improvements on building (mixed-initiative) ui. It also gives us an example by applying the method into a system for scheduling and meeting management. The former studies among other researchers mainly focus on separate studies, but not combining them together. The critical factors of integration: value-add from direct manipulation, infer and exploit the uncertainty, usr’s attention, balance between costs and benefits, efficient dialog, direct invoke and terminate, minimize cost, using wider precision, improve the analysis, social expectation, recent event memory, and accumulate data to improve the model. By giving instance example after definition, the paper appears to be clear and convincing. It tries to take as much info from user (email content & user reaction)as possible, find balance if the uncertainty is too wild, and gives various strategies including manually switching between modes and auto-decision from evaluating benefit and cost. It’s a user friendly software that tries to minimum disturbance if failed. It theoretically analysis the thresholds for generating autonomous services (dialog or action) to manage usr’s uncertainties of goals and attention. I think it’s a strong, inspiring paper with mature implementation of how to enable users and agents working collaboratively. Some of the features can still be seen from today’s apple & google calendars and the threshold evaluation method maybe (just guess) the foundations of (or could be applied to) today’s voice assistant.

Ruochen Liu 8:43:29 9/21/2017

1. Past, Present and Future of User Interface Software Tools:This paper, written in 1999, introduces the past, present and future of the software tools which are used to design user interface. The new software tools, which provide information about the user, the devices and the application’s state, make end-user programming, recognition-based user interfaces and the data sharing possible. Some future issues are also presented in the paper. The skill and dexterity level, which is an important property of the targeted users, may change as time passes by. There will be more old adults using the future interactive systems. The gap between demand and supply may be filled by putting more efforts into the research of future user groups and the specific interaction technology. As mentioned in the end of the paper, the opportunities are hide in the challenges. Facing these challenges and fixing the problems will bring human beings a promising future of user interface tools. 2. Principles of Mixed-Initiative User Interfaces: Since developing new metaphors and tools that enhance users’ abilities to directly manipulate objects and directly developing interface agents that that provide automation had caused a heated debate. The main purpose of this paper is to review principles that allow engineers to enhance human-computer interaction and bring out the key ideas about Lookout system for scheduling and meeting management. Among all the principles, “considering uncertainty about a user’s goals” and “considering the status of a user’s attention in the timing of services” are the two principles that interest me the most. The mind of a user may change quickly, the uncertainty of a user’s goals must be put into account while designing interfaces. If not, the user experience may become worse facing different users. Also, in the timing of interaction services, considering the status of a user’s attention is an important point. By calculate the costs and benefits of the delay of an action, the agents can get a specific delay time which can help the users to concentrate on the interaction. By introducing LookOut system, both the challenges and potential opportunities are presented for improving human-computer interaction based on the combination of reasoning machinery and direct manipulation. Personally, I believe the design of mixed-initiative user interfaces has a promising future.

Amanda Crawford 8:50:38 9/21/2017

Past, Present, and Future of User Interface Software Tools, Brad Myers, Scott E. Hudson, Randy Pausch, ACM Transactions on Computer-Human Interaction, March 2000, pp. 3 - 28. In this paper, Myers, Hudson, and Pausch examines the current state of user interface tools, their strengths and weaknesses, and the tools needed for building interfaces for the future of highly connected and multi-modal ubiquitous devices. He draws out themes from past and future tools: user interface aspects addressed, threshold and ceiling, path of least resistance, predictability, and irrelevant targets. He points out that the stabilization of operating systems has been both a blessing and a curse. Stabilization has allowed the development of highly specialized user interface tools and devices, but has confined further development to suite desktop based, mouse-keyboard based architectures. The influx in diversity of devices has diversified the needs of creating interfaces that is responsive to different screen sizes and input interactions. In addition, the future of the network will allow devices to connect and collaborate on shared interface agents and contents. The development of windows managers, toolkits, and event languages allow expert designers to rapidly develop powerful interactive user interfaces. These developments proved to be highly predictable, took a path of least resistance, providing a low threshold and high ceiling outcome. Many of the tools that seemed to miss the target, unpredictable, took the path of most resistance, and valued high threshold and low ceiling, such as the UIMS did not last too long. To debate this, I believe that these tools are now becoming useful as the ever growing field of programmers and knowledge is expanding. We can see these in the flourishing of virtualization and the need for such system architectures. The researchers proposed several principles in which we can see manifest in our time today such as: responsible application and web designs, environment aware software, continuous event sequencing for recognition based gestures, user centric designing, and paradigms for developing connected devices. Principles of Mixed-Initiative User Interfaces, Eric Horvitz, CHI 1999: ACM Conference on Human Factors in Computing Systems, pp. 159-166 Horovitz gives an in depth narrative on the principles that guided the construction of an innovative and semi automactic scheduling tool called the LookOut. The LookOut tool provided automated calendar event scheduling services by calculating the user's goal and inferring which action to take. In addition, the tool included societal constructs that included the dialog between the user and service and considered the user's attention based on a custom tailored model. The lookout tool went a step further to include speech to text capabilities, in which, allowed the tool to embody an anthropromorhoic persona and ease the path of resistance. Horovitz's principles seem to synchronize with Myers', Hudson's, and Pausch's future vision, in particular of developing recognition based designs. In addition, this research combined the power of machine learning and artificial intelligence to reintroduce conversation based metaphors in estimating the accuracy of taking certain actions and remedying actions that were concluded to be wrong or unwanted. This research grew beyond creating a tool to complete a task, it went a step further and designed a tool that could predict a user's next task and automate the process.

Akhil Yendluri 9:00:43 9/21/2017

Past, Present and Future of User Interface Software Tools This paper analyses all user interface models developed and studies why they succeeded or failed. The paper introspects on the reasons behind this and tires to extract themes which can be useful for future work. The authors use threshold, ceiling, resistance and predictability as some of the important aspects on which they judged User Interfaces. The ideal interface is one that has low threshold, high ceiling, low resistance and high predictability to make the user comfortable at developing User Interfaces for multiple devices without bothering too much on the rapid technological changes. The paper also shows us how research is an important aspect behind the development of every User Interface developed. The authors use "GUI" from seminal research at Stanford, "OLE and Java Beans" from CMU as some of the examples. The papers also explain how rapid change in technology affected the development of some promising User Interface Management Systems. The paper then discussed on how the low threshold and high ceiling can have a useful impact on developers. It makes them easy to apply powerful approaches while making them accessible and understandable. The authors also mention how Ubiquitous computing can affect the modern UI as the requirements further evolve with mobile devices becoming a commonplace. Finally the paper talks about 3D technology and how user interface design for it is still in its nascent stage. All of the authors theories have been true looking back. Principles of Mixed-Initiative User Interfaces The paper aims at coupling the use of Direct Manipulating techniques with automated services. The author explains how researchers have concentrated on developing either Direct Manipulation technique or Automated Service separately without considering the advantages of having them work together. The author lists 12 principles for the effective integration of automated services with direct manipulation. The paper talk about how LookOut on Microsoft Outlook had tried to effectively combine both the techniques and also analyses the impact on users. LookOut scans the mail body of the user to find days, dates and common phrases to automatically schedule them into the users calendar. It uses automated text classification, Bayesian text classifier and Support Vector Machine analysis.

Mehrnoosh Raoufi 9:01:29 9/21/2017

Past, Present, and Future of User Interface Software Tools: This paper represents past user interface software tools, their features and how they had influenced the software development. In the first part, the author introduces different tools and materials that have been exploited in building user interface software so far. For instance, It refers to event languages, scripting languages, object-oriented programing, interactive graphical tools, Hypertext and etc. It explains each of them thoroughly and also talks about their success and failures. The explanations are illustrative enough and easy to understand. After overviewing the history of user interface tools, the author turns a light on future prospects. As the paper dates back to 90s, many of its predictions about future have already taken place in today's advanced user interface tools. The paper has adequately envisioned future trends and demands in this area. For example, about how inputs and outputs may vary in the future from the desktop interface and recognition-based user interface.However, it may not be right about various keyboards that PDAs may have as most of them are touchscreen today. In overall, it is worth reading this paper to comprehend the procedure user interface tools have gone through over the time.---------------------------------------------------------------------------------------------------------------------------------- Principles of Mixed-Initiative User Interfaces: This paper talks about the integration of two main methods in user interface designs. One is the direct manipulation of interface user and the other one is automated services. Then it goes through 12 principles to describe the development of this mixed-initiative user interface design. It discusses main challenges such as guessing users' needs and goals and considering user's attention in the timing of services. Then, the author introduces the LookOut software in which these principles are integrated.​ ​​‌