Team Edit

Johnny Zhou and Garrett Eastham

Introduction Edit

JavaFX ( is Sun's upcoming framework for rapidly developing Rich Internet Applications (RIA) that include rich media content such as embedded video and audio. It will provide similar mechanisms as Adobe Flash, Adobe AIR, and Microsoft Silverlight. The front page of the JavaFX website contains a nice dashboard-like set of widgets that allows for the dynamic swapping of these widgets between the background and foreground.


JavaFX Widgets

What is the overall structure of the interaction? Edit

The front dashboard consists of multiple widgets organized such that there is one and only one in focus in the middle foreground of the mashup, and the rest of the widgets sit transparently as faded smaller windows in the background. Every widget, including the one in the foreground, can be dragged around via a drag-and-drop interaction. If a background widget is dragged, it will get swapped to the foreground and the existing foreground widget will swap to the location where the user dropped the previous background widget. The foreground widget can be drag-and-dropped anywhere on screen, where it will simply move to and sit. Each widget can have embedded rich content, such as movies, images, and audio. These media objects are capable of being played directly inside the widget (it's not simply a link to the content). When a widget swaps to the background, a static image is replaced in lieu of any video content in that widget. Dragging and dropping these widgets appears to have significant latency, as it takes a very noticeable amount of time to re-render all of the rich content inside the widget as it is being dragged around the screen. Finally, each widget can also have multiple pages of content, managed by a tab-like swapping interface. Overall, however, this interaction provides a nice, clean, and tangible interface for viewing and manipulating multiple objects containing various kinds of rich content.

What is the role of Javascript (event handlers, AJAX requests, etc.)? Edit

JavaScript plays a huge role in this interaction. In fact, the JavaScript even registers the event handlers for each of the draggable divs.

  • Registers event handlers for the draggables inside the JavaScript
    • onclick, onmousedown, onmouseup, onmousemove
    • addEvent('click', function(e){...})
    • new Drag.Move(box);
  • Initializes the layout of the widgets on page load
    • CSS and HTML define each widget top to bottom on the page statically
    • The JavaScript repositions each widget, sets one to be in focus and registers the event handlers
  • Handles the mouse events
    • Mouse click to start playing a video
    • Drag-and-Drop to move around widgets and to swap them to and from the foreground
  • Determines the largest paragraph amongst the set of widgets and sets the height of all widgets to accommodate that size for consistency
  • Shrinks and grows each widget box as they are swapped from the foreground to the background and vice versa
  • When an in-focus widget that is currently playing a video is transferred to the background, the JavaScript replaces the video with a static image
  • Handles the shrinking and growing transition animation
  • Handles changing opacity of widgets as they move to the background
  • Set movies ready to play as a widget moves into focus
  • Tabs-like swapping of multiple pages of content in each widget

Another view of rearranged widgets

What parts of the interaction happen in the browser and what parts require server interaction? Edit

The interaction of the widgets with each other is completely handled by the browser and JavaScript/CSS manipulation. The rich media content inside the widgets, however, is pulled on demand from the server. For example, a video embedded inside a widget will be streamed to the client when Play is clicked, much like YouTube.

If data is fetched dynamically from the server, what is the granularity of those patches? Are there any special techniques for hiding latency, such as prefetching? Edit

When the embedded rich content is fetched from the server, it is streamed to the client. For content that can be "played", such as video and audio, the data will be buffered at first so as to prevent the playback from reaching the end of the currently available stream, and thus stopping and/or having the appearance of choppiness.

Are there things about the facilities of the Web that made this interaction either easy or hard? Edit

The native JavaScript capabilities allowed for relatively easy handling of this interaction. The use of the MooTools framework greatly reduced the amount of code needed to create the interaction (by providing built in element modification routines in response to a "drag" movement). The framework also provides useful libraries for creating the sliding effects that expand and blend the sidebar with the draggable widget display, which marks an excellent use of existing third-party tools to reduce development time.

Events are well-defined and easy to use (onmouseclick, onmousedown, onmouseup, onmousemove). However, the fact that JavaScript cannot run multi-threaded tasks hinders this interaction somewhat. For example, it may be difficult to handle multiple instances of these interactions simultaneously, such as having two pairs of widgets swap with each other.

Did the interaction have to contort itself to get around problems with the Web? Edit

It appears as though nothing special or hack-ish had to be done to implement this interaction. Assuming that one does not expect the individual "widgets" to be running dynamically after sending them to the background (this would require multi-threading to achieve), then this interaction was easy enough to build. The document-object model is still not a sufficient substitute to the parallel processing environment of an actual OS; however, this implementation proves that keeping the limitations of the current Web in mind can easily help rethinking necessary interactions.

The native features of JavaScript and HTML divs and spans were sufficient to support these dynamics. Contrary to some popular belief, JavaScript is a very powerful language.

Are there improvements to the Web that would have made this interaction easier to implement, or that would have allowed a more powerful interaction? Edit

Multithreaded JavaScript could potentially make this interaction more dynamic and rich, such as having multiple widgets interacting with each other simultaneously and having multiple rich media content play at the same time. Perhaps browsers can also update their rendering engines to better support dynamically moving rich content with minimal latency.

Are the techniques used in this interaction relevant for a broader class of interactions? Edit

This type of dashboard-esque interaction seems to be very popular in present times. JavaFX is somewhat similar to Apple's Cover Flow interface and enterprise portal websites built on top of software such as IBM WebSphere and BEA WebLogic. This kind of interface gives the sense of more real estate and a more tangible mechanism of interaction. The user is able to manipulate the widgets in a very natural manner via drag-and-drop, and the ability to see many of these widgets at the same time gives the user a sense of more screen space and utility.

One of the main features of this interaction is its naturalness. Moving widgets around involve intuitive transitions and in a sense, it brings us back to the childhood days of playing with blocks and manipulating them in random ways. From an HCI perspective, the more natural an interface is, the better, and I believe that more and more web interactions will start to leverage this idea to further enhance the usability, intuitiveness and naturalness of their interfaces.

Another interesting interaction phenomena results from the haphazard arrangement of the widgets after interacting with them. The simple fact that the widgets did not rearrange themselves into their pre-set order creates a somewhat cluttered desktop; however, it also endears the user to the application because they subtley feel as if they've impacted a landscape, leaving their digital footprint. While it might seem trivial, storing a user's input data - whether or not that data is intentional or unintentional - creates a valuable way to build user loyalty and create a more engaging digital interaction.

Areas for Potential Improvement Edit

  • Video and audio content shouldn't reset when the widget is swapped to the background
  • Keyboard manipulation for the widget that is in focus
  • Option of moving background widgets without having them swapped into focus
  • Some kind of tangible response when switching tabs within a widget (e.g. sliding animation)
  • A more 3D look to the widget blocks as opposed to 2.5D
  • Option to resize the widgets
  • Ability to minimize (or close) widgets and to open up new widgets from a list

Ad blocker interference detected!

Wikia is a free-to-use site that makes money from advertising. We have a modified experience for viewers using ad blockers

Wikia is not accessible if you’ve made further modifications. Remove the custom ad blocker rule(s) and the page will load as expected.