Rethinking User Interfaces: Beyond the DOM's Limitations

In this article I use the term "DOM based UI" to refer to the common practice of using a Retained Mode Graphical User Interface (RMGUI) in web development. This is in contrast to Immediate Mode Graphical User Interfaces (IMGUI) which are more common in video games and specific graphical applications.

After extensive web programming over the last decade, I've stepped back to question the efficacy of our current methods in building user interfaces. Can we achieve truly futuristic UIs with existing technologies, or is a paradigm shift required? The animated transitions seen in sci-fi films, suggest a level of fluidity and dynamism that seems unattainable with today's DOM-based UIs. Is this merely a failure of imagination or a fundamental technical limitation?

For a concrete example of what I am talking about take the kinds of animations in the user interfaces of movies like star wars etc. Many argue that the UIs in those movies are "not funcitonal" and simple / functional UIs "won out" against these sorts of futuristic looking interfaces. But I wonder if that is more of an excuse of a technical limitaiton. But here is a simple example of a kind of animation that you might want to have in a user interface that is not possible with a DOM based UI.

Try clicking on that button. The animation that plays goes across the full area, but it is smooth the whole time. This is only possible or easy to implement because I am re-rendeirng the button on every frame.

This aspect of "futuristic" looking interfaces is that they often perform animated transitions which change the entire layout of the interface in a smooth way. This is nearly impossible to do in a Retained Mode User Interface.

UX outside of the DOM

Why did the web DOM take over UX in the first place? There are a lot of other articles that opine on why the web and Electron have seemingly won out in the user interface space so let me just summarize some of the reasons

• Users can view a web app without installing anything on their machine
• In the case of an Electron based app it can be created in a platform independent way with relative ease if there is already a web version
• A lot of the "gotcha" consdierations eg. about accessability or interaction with text have already been solved by the platform and thus allow developers to focus on the unique business value their app provides rather than those kinds of technical details

However there are some good exampels of where Retained Mode has not been used but the interfaces are still highly successful. Video games are the prime example. A lot of video game console interfaces do not use a DOM, but are highly functional despite that. How about one that renders a lot of text and requires heavy user interaction with that text? Google Docs is the prime example. Google Docs actually only uses the DOM for rendering buttons etc, but the actual document iself is rendered on a canvas (including the right click menu etc).

An Easy example to see how bad full layout transitions are in RMGUI just try to resize the window of any electron application. In those cases it has to do a full re-layout computation across all the widgets in the tree (or nodes in the DOM) and then re-render them. This is a very slow process and is noticable to the user, the framerate will almost always dip in such a scenario.

Where to go from here?

I suggest to anyone who has spent the last decade building applications with React to take a step back and try to build a somewhat complex interface with an immediate mode graphics. There are many approaches that you can take to that depending on your comfort level with various languages etc, so I won't be overly perscriptive, but just to say that trying it will illuminate a whole class of things in your React programming that you wouldn't have considered otherwise (both limitations as well as different kinds of bugs that come up, some categories of bugs that are eliminated entirely etc).

Google themselves actually have a half-way solution to the problem with Flutter. Flutter renders the graphics outside of the DOM, but it actually is still a RMGUI. I haven't tested the performance of Flutter on layout changes, it does seem better than the DOM, but I imagine it will still be somewhat limited in the kinds of transitions that can be done.

Concerns?

Probalby the biggest concern regarding non-document based UX is accessability of text for screen readers etc. But this seems like something that ought to be solvable or at least it would be odd if this were the main sticking point preventing going to the next generation of interface design. Eg. I'm sure that it does not take an army of engineers at Google to implement basic accessability features.