UC Berkeley economists Emmanuel Saez and Gabriel Zucman analyzed Senator Elizabeth Warren’s proposal for a wealth tax, and Fernando Hoces de la Guardia from the Berkeley Initiative for Transparency in the Social Sciences (BITSS) wanted to turn their work into an open policy analysis (OPA). I got involved because they needed someone to make an interactive visualization that would allow users to explore different wealth tax proposals. Check it out here.
I learned a lot from the experience and wanted to share some lessons learned.
Up until this point I had only made Shiny apps for my individual use or for the use of a few collaborators who were carefully “trained” on the expected inputs. These apps were functional, but brittle. If a user provided unexpected input, the app had no capability to save face; the app would just crash. The apps were also were run locally, or on the free version of shinyapps.io with no real worries about heavy traffic. Making a Shiny app that was going to be more publically facing came with its own set of challenges.
Unexpected User Behavior
It was fairly straightforward to create an app that had the desired functionality as long as users behaved as expected. However what if the tax brackets were entered in the wrong order? What if there were duplicates in the brackets? What if nonsensical values were entered such as a tax evasion rate below zero or above 100 or a tax rate below zero? It’s not ideal for the app to crash if given unexpected input while giving users no feedback about what went wrong.
My first instinct was just to throw a bunch of checks into every main function (LOTS of
else chunks). However this strategy at one point led to a visible lag in the app’s reactivity. For the final version of the app, we saved time by allowing the user to enter whatever they wanted and postpone dealing with issues until we needed to update the plot and calculations (signaled by a click of a button). At this point we reordered the brackets, broke ties in bracket values, and updated tax rates and evasion to be within the realm of plausibility when the user clicks “update”. We also updated the displayed values to match our fixes, so the user sees that the calculated values and plots are made under conditions different from what they entered.
There are some unexpected user behaviors that we still don’t react to. We allow users to enter non-monotonic tax rates, and the plot and calculations respect this (even though in practice, this would be a weird proposed tax scheme). The app does not immediately crash if you enter a non-numeric entry, but if you don’t catch your mistake before clicking the “update” button, the app will crash. This was mainly a decision based on time constraints rather than being something truly un-fixable.
I didn’t do much on the actual deployment side. Katie Donnelly Moran, Clancy Green, and Akcan Balkir worked to make this happen through the use of AWS to allow for some control in the case of high traffic to the app. This guide was helpful. However, this was the first time I used Binder. Binder “allows you to create custom computing environments that can be shared and used by many remote users” and can handle moderate traffic. For the purposes of this project, this means that a user can go straight to the Shiny app or step through the dynamic document without installing R and RStudio on their own computers. This was much easier to set up than I anticipated. I just followed this example. Also, shout out to Lindsey Heagy who gave me a crash course in how Binder, BinderHub, and JupyterHub work together behind the scenes.
Because I built the app and knew the internal structure, I would, without thinking, avoid doing things that would cause the app to crash, thereby making my testing ineffectual. However, Katie, Emmanuel, and Gabriel were great at finding bugs. I cannot count the number of times that I thought the app was ready and then they would within ten minutes have a list of things that needed to be fixed.
Some of these fixes involved logical errors on my part that they were able to easily identify by using their domain knowledge. I was learning the economics on the fly, and sometimes I misinterpreted how certian values should be calculated.
Most fixes involved anticipating and being resilient to unexpected user behavior. Sometimes unexpected user behavior would crash the app, but other times, unexpected user behavior would violate the assumptions I was making in order to make the required calculations. Nothing would officially crash, but the returned calculations would not be accurate. These errors were more pernicious and harder for me to detect since my expectations for what the output would look like were not appropriately tailored to the underlying economics. A number that would surprise Katie, would not jump out at me as obviously wrong.
Another change that I would not necessarily have thought of on my own was making the switch from using sliders to text boxes. Sliders allow more control of the inputs (users can only pick from allowable options), so I started with those to make my life easier. However, it became clear in testing that the sliders can get annoying if you have a very specific and detailed wealth tax plan in mind to test out. Text boxes allow for more freedom.
Major take-away: It takes a village to really put a Shiny app through its paces.
And now for some technical bits…
Even though reactivity is often what you want in an interactive visualization, there were times where reactivity caused some headaches.
When the reactivity is too fast
Suppose a user is typing in a value: “.” on their way to “.5” or “50” on their way to “500”. If the app tried to start calculating right away, it could run into issues with “.” not having a numeric interpretation or “50” being tied with another bracket. Even if there were reactive fixes to ties, this could be a problem. We might bump “50” up to “60” to break a tie, but the user really just wants to be allowed to finish typing to “500”. As users typed it was also distracting to see the plot keep jumping around, recalculating after every keystroke. Requiring a button click before starting calculations and plots (
eventReative) was a good fix.
req() was also a lifesaver! This wrapper makes sure that the particular value is available before continuing with the calculation. This was handy when we created new brackets on the fly. It would take a split second to create the new object needed for the calculation, so the app needed to know to pause in the calculation until the value was populated.
When the reactivity is too slow
When we updateded the user interface using things like
renderUI, it would take some time for the new values to kick in, affecting the calculations and plots.
req() was helpful for some of this, but there were still some sticking points. For example, one thing that I couldn’t easily get around was that the plot does not automatically reflect the tie updates. Instead, I had to add a conditional warning so that the user would click “update” again to reflect these changes (I’m open to less clunkier ways to do this. Share your wisdom!).
I severely underestimated how much time this project would take. Getting the original mock up with four fixed brackets took a few hours. The rest took weeks. The majority of the extra time spent on the app didn’t even come from adding features to allow users to tune more parameters of a tax plan.
The biggest time commitment came from unexpected user and reactivity behavior. These changes were hardest to make because I had to completely reorganize the inner workings of the app to accomodate certain behaviors.
Dealing with these structural changes was definitely one of those times where I would stare at something for hours, give up and take a break, then come back the next day and fix it in ten minutes. Moral: take breaks!
The good news is that I effectively used git branches to work on these major overhauls which was a good skill to practice. By using branches, others on the team could use the current version of the app to do other tests while I was breaking things on another branch. This also made it easier to start over again if an approach in a different branch wasn’t working out.
The making of this app was certainly an adventure! I learned a lot about my Shiny instincts (the good and the bad) and how to better foritfy an app to withstand unexpected user behavior (plus some economics along the way).
Thank you again to the whole team.
If you are a Shiny afficianado, feel free to dig into the source code and streamline things. Pull requests are welcome. Similarly, if you find a bug, please file an issue.
P.S. Alvin Chang (@alv9n) also made a cool, interactive exploration of the wealth tax.