Solving mutually recursive elements with lazy loading with hooks and dynamic components

The goal

You have a recursive data structure you need to display in a way that matches the data hierarchy.

For example:

A file system, folders with arbitrary depth, and files.
A tree of comments.
A recipe that contains other recipes.

There are different ways to address this in Fulcro, depending on the concrete use case. Next, we will explore a particular one.

The concrete problem

You have recipes containing line items, which further contain either an ingredient or a sub-recipe. This nesting can go on arbitrarily. Think pizza with sauce → the sauce has sub-components. Sub-components, in turn, might have final ingredients (salt, oil, pepper). Line items also hold the element’s quantity qty and unit of measure uom. In other words, we have two mutually recursive elements - a recipe and its line item.

In this particular case, we also want to lazyload the sub-recipes so we don’t load the entire tree up front. That further complicates the usage of EQL recursive queries. (Though it could be achieved by leveraging a depth limit on the recursive query.)

Let’s look at the UI we want and the data we have:

Figure 1. The desired UI

And the data:

Figure 2. The data model in our SQL database

Thus we get the data tree below:

{:recipe/name "Pancake Batter" :recipe/id "1"
 :recipe/line-items
 [{:rli/id "10" ; (1)
   :rli/qty 4, :rli/uom "piece"
   :rli/ingredient {:ingredient/name "Eggs", :ingredient/id "101"}}
  {:rli/id "11"
   :rli/qty 0.25, :rli/uom "kilo"
   :rli/ingredient {:ingredient/name "Flour", :ingredient/id "102"}}
  {:rli/id "12"
   :rli/qty 0.5, :rli/uom "litre"
   :rli/sub-recipe
   {:recipe/name "Yogurt" :recipe/id "2"
    :recipe/line-items
    [{:rli/id "13"
      :rli/qty 1, :rli/uom "litre"
      :rli/ingredient {:ingredient/name "Milk", :ingredient/id "103"}}
     {:rli/id "14"
      :rli/qty 10, :rli/uom "gram"
      :rli/ingredient {:ingredient/name "Yogurt Starter", :ingredient/id "104"}}]}}]}

:rli/ stands for :recipe-line-item/, for space efficiency

Exploring possible solutions

The standard solution to loading data for recursive components is to use a recursive query in EQL. It can load either the entire recursive tree or a particular number of levels. The EQL solution is the most straightforward and covers 95% of all cases.

Tip	A recursive EQL query for e.g. folders would look like this: `[:folder/id :folder/name :folder/children …]` (the `…` indicates the self-recursion).

However, some cases are just too complex to solve with EQL recursive queries. This approach requires that the same entity be present at each level of traversal, i.e. we have a single component that may contain itself. That is not the case here, where we have Recipe containing RecipeLineItem, containing Recipe … .

Option A: Flatten the data to get a self-recursive element

We could change our data model so that a recipe contains directly sub-recipes, for example by separating and storing separately the qty and uom data and the sub-recipes themselves:

{:recipe/name "Pancake Batter" :recipe/id "1"
 :recipe/line-items [{:rli/qty 4 :rli/uom "piece"
                      :rli/ingredient {:ingredient/name "Eggs", :ingredient/id "101"}}
                     {:rli/qty 0.5 :rli/uom "litre" ; (1)
                      :rli/sub-recipe [:recipe/id "2"]}]
 :recipe/sub-recipes [{:recipe/id "2" ; (2)
                       :recipe/name "Yogurt"
                       :recipe/line-items [...]}]}

The line-item information (uom, qty) for Yogurt
The Yogurt sub-recipe itself

Then inside the component you would join the two pieces of data together, at render time.

It might not be elegant, but it gets the job done.

Option B: Use a dynamic, hook-based component to load the sub-recipe

In this case, we have also the additional requirement of lazy loading of sub-recipes. We can solve both problems by using a dynamic component, i.e. one that is not part of the query tree but gets its data independently, via the use-component hook.

Tip	Dynamic use-component components provide great flexibility, at the cost of increased complexity and decreased transparency (since the UI isn’t anymore a pure function of data, passed from the parent).

What we will do is load a Recipe that has a RecipeLineItem that refers to a Recipe (just the ID and name, avoiding the problematic recursive line items). We then swap out the reference for a dynamically loaded entity when the user wants to see the details of the sub-recipe.

The solution

We will adopt solution B, using a dynamic hooks component.

Below we can see what the component instances will look like, for a recipe with three line items, where 1. is a collapsed sub-recipe, 2. is an expanded sub-recipe, and 3. is an ingredient:

Figure 3. The UI components

Notice that the RecipeReference component in its collapsed state only displays the name of the sub-recipe, while in its expanded state it renders a DynamicRecipe component with the complete sub-recipe.

Below, on the left, we can see the "render tree" of components with their queries (actually, query fragments). The solid head arrows ▶ show both query inclusion and component nesting. On the right, we can see the corresponding data tree, and how it matches the query fragments (the hollow head arrows >).

Note: The queries have been abbreviated to save space. Thus, for instance, {:recipe/line-items R.L.I.} should be read as {:recipe/line-items (comp/get-query RecipeLineItem)}.

Figure 4. The render tree and corresponding data tree

We now understand the structure of our components and the data tree, and can explore how the data is loaded:

Figure 5. Data loading

We can see that the initial df/load! only loads the id and name of the Yogurt sub-recipe. When the user expands it, it triggers an additional load, to fetch the remaining data of the recipe.

Tip

There could be a small complication with the on-demand loading of the sub-recipe. The reason is that the use-component hook is slightly tricky to use with something that has yet to be loaded, since it needs some data to exist in the client DB. I.e. <entity> <id> must not be undefined/nil, it must be at least an empty map (or rather, a map with an id). We could initialize the DynamicRecipe component via :initial-params together with specifying :initial-state for the Recipe so that the hook could then make an "empty" but properly identified placeholder in the database to hook up to while the load runs. I.e., use-component with :initial-params generates a {:recipe/id n} map and normalizes it (which puts the ident in place for finding it as well). Then the load gets issued. When the details arrive, you’ve got the full recipe. Fortunately, in our case this is not required, because the parent recipe-line-item has already loaded the name and ID of the recipe in question (just not its line items).

Insights

Power and downsides of use-component

If a static query doesn’t work for your recursive component needs, then the options are dynamic queries or breaking out of the standard Fulcro model by leveraging a use-component hook. Recursive queries work in 95% of cases, dynamic queries works in a few more, and use-component covers the rest.

You could ask, if I can do everything with use-component, then why not just supply that and drop the components and static queries altogether?

Because the static query mechanism offers a few advantages:

Possibility to define a global initial data model locally, component by component, via its :initial-state, instead of doing it at some remote place or using component-local state
Side-effects combined with rendering (i.e., what most React hooks do) cause problems:
- Hot code reload doesn’t work cleanly since components aren’t pure functions anymore
- Controlling the data lifecycle becomes more difficult because it is tied to when things are mounted as opposed to compositional logic outside of rendering.
- Refactoring UI breaks things (E.g., a side effect A did accidentally got coupled to B. Move A and B stops working)
Easier to trace what is going on in the data model because it is more straightforward - each component gets its data only from its parent, and you can see what is the data tree going into the Root component
The query itself becomes a valuable source of information (which we leverage e.g., for Dynamic Routing)

Complete code

Finally, let’s look at some code! First of all, a few highlights. See the complete code listing at the bottom.

DynamicRecipe

(defsc DynamicRecipe [this {:recipe/keys [id]}]
  {:use-hooks? true} ; (1)
  (let [recipe (hooks/use-component (comp/any->app this) Recipe  ; (2)
                                    {:initialize? false :ident [:recipe/id id]} ; (3)
                                     #_ ; this below would have been necessary if
                                    ; DB - :recipe/id - <id> didn't exist already
                                    {:initialize?    true
                                     :initial-params {:recipe/id id}
                                     :keep-existing? true})]
    ;; Load could be hooked into "expand" mutation to remove side-effect from UI logic
    (hooks/use-lifecycle (fn [] (df/load! this [:recipe/id id] Recipe))) ; (4)
    (when recipe
      (ui-recipe recipe))))

DynamicRecipe is a hooks-based component (1), which "fetches" data from the client DB via hooks/use-component (2). Notice that we need to tell Fulcro its :ident (3) since the component has a dynamic one. In this case, we already know that there is some data for the ident in the client DB (namely id, name), so we do not need to initialize it. Alternatively, we could leave :initialize? true (the default) but would then need to provide the :recipe/id via :initial-params, and perhaps also include :keep-existing? true. Though in this particular case, our initialization would only be overriding :name, and we are going to re-load it at once, so it doesn’t really matter.

As the component is mounted in the DOM, its use-lifecycle will fire a df/load! (4) to load the remaining data. If we used a true remote server and a slower network, you could see that the expanded Recipe only has initially id, name, until the rest of the data arrives.

RecipeReference

(defsc RecipeReference [this {:ui/keys     [expand?]
                              :recipe/keys [id name]}]
  {:ident :recipe/id
   :query [:ui/expand? :recipe/id :recipe/name]}
  (if expand?
    (ui-dynamic-recipe {:recipe/id id})
    (dom/span {:style {:cursor "pointer"}
               :onClick (fn [] (m/toggle! this :ui/expand?))}
              " ▶ " (dom/span {:style {:textDecoration "underline"}} name))))

RecipeReference displays either the name of the recipe when collapsed, or renders a DynamicRecipe when expanded.

Finally, here is the complete code listing:

Example 1. dynamic-recursion-cards

(ns dynamic-recursion-cards 
  (:require
   [cljs.core.async :as async]
   [com.fulcrologic.fulcro.components :as comp :refer [defsc]]
   [com.fulcrologic.fulcro.data-fetch :as df]
   [com.fulcrologic.fulcro.dom :as dom]
   [com.fulcrologic.fulcro.mutations :as m]
   [com.fulcrologic.fulcro.networking.mock-server-remote :refer [mock-http-server]]
   [com.fulcrologic.fulcro.react.hooks :as hooks]
   [com.wsscode.pathom3.connect.indexes :as pci]
   [com.wsscode.pathom3.connect.operation :as pco]
   [com.wsscode.pathom3.interface.async.eql :as p.a.eql]
   [nubank.workspaces.card-types.fulcro3 :as ct.fulcro]
   [nubank.workspaces.core :as ws]))

(defonce pretend-server-database
  (atom
    {:recipe/id           {1 {:recipe/id         1
                              :recipe/name       "Pancake Batter"
                              :recipe/line-items [{:recipe-line-item/id 10}
                                                  {:recipe-line-item/id 11}
                                                  {:recipe-line-item/id 12}]}
                           2 {:recipe/id         2
                              :recipe/name       "Yogurt"
                              :recipe/line-items [{:recipe-line-item/id 13}
                                                  {:recipe-line-item/id 14}]}}
     :ingredient/id       {101 {:ingredient/id   101
                                :ingredient/name "Eggs"}
                           102 {:ingredient/id   102
                                :ingredient/name "Flour"}
                           103 {:ingredient/id   103
                                :ingredient/name "Milk"}
                           104 {:ingredient/id   104
                                :ingredient/name "Yogurt Starter"}}
     :recipe-line-item/id {10 {:recipe-line-item/id     10
                               :recipe-line-item/qty    4
                               :recipe-line-item/uom    :piece
                               :recipe-line-item/ingredient {:ingredient/id 101}}
                           11 {:recipe-line-item/id     11
                               :recipe-line-item/qty    0.25
                               :recipe-line-item/uom    :kilo
                               :recipe-line-item/ingredient {:ingredient/id 102}}
                           12 {:recipe-line-item/id     12
                               :recipe-line-item/qty    0.5
                               :recipe-line-item/uom    :litre
                               :recipe-line-item/sub-recipe {:recipe/id 2}}
                           13 {:recipe-line-item/id     13
                               :recipe-line-item/qty    1
                               :recipe-line-item/uom    :litre
                               :recipe-line-item/ingredient {:ingredient/id 103}}
                           14 {:recipe-line-item/id     14
                               :recipe-line-item/qty    10
                               :recipe-line-item/uom    :gram
                               :recipe-line-item/ingredient {:ingredient/id 104}}}}))

(pco/defresolver recipe-resolver [_ {:recipe/keys [id]}]
  {::pco/input  [:recipe/id]
   ::pco/output [:recipe/name {:recipe/line-items [:recipe-line-item/id]}]}
  (get-in @pretend-server-database [:recipe/id id]))

(pco/defresolver recipe-line-item-resolver [_ {:recipe-line-item/keys [id]}]
  {::pco/input  [:recipe-line-item/id]
   ::pco/output [:recipe-line-item/qty
                 :recipe-line-item/uom
                 {:recipe-line-item/ingredient [:ingredient/id]}
                 {:recipe-line-item/sub-recipe [:recipe/id]}]}
  (when-let [it (get-in @pretend-server-database [:recipe-line-item/id id])]
    (merge
     ;; Provide default values so that Pathom won't complain about missing-attribute:
     {:recipe-line-item/ingredient nil
      :recipe-line-item/sub-recipe nil}
     it)))

(pco/defresolver ingredient-resolver [_ {:ingredient/keys [id]}]
  {::pco/input  [:ingredient/id]
   ::pco/output [:ingredient/name]}
  (get-in @pretend-server-database [:ingredient/id id]))

(pco/defresolver all-recipe-resolver [_ _]
  {::pco/output [{:recipe/all [:recipe/id]}]}
  (let [ids (keys (get @pretend-server-database :recipe/id))]
    {:recipe/all (mapv (fn [id] {:recipe/id id}) ids)}))

(def resolvers [recipe-resolver recipe-line-item-resolver ingredient-resolver
                all-recipe-resolver])

(def default-env
  (-> {:com.wsscode.pathom3.error/lenient-mode? true}
      #_(p.plugin/register pbip/mutation-resolve-params) ; not needed here
      (pci/register resolvers)))

(defn process-eql [eql]
  (let [ch (async/promise-chan)]
    (-> (p.a.eql/process default-env eql)
        (.then #(async/go (async/>! ch %))))
    ch))


(declare Recipe ui-recipe)

(defsc Ingredient [_this {:ingredient/keys [name]}]
  {:ident :ingredient/id
   :query [:ingredient/id :ingredient/name]}
  (dom/span " " (str name)))

(def ui-ingredient (comp/factory Ingredient {:keyfn :ingredient/id}))

;; tag::dynamic-recipe[]
(defsc DynamicRecipe [this {:recipe/keys [id]}]
  {:use-hooks? true} ; (1)
  (let [recipe (hooks/use-component (comp/any->app this) Recipe  ; (2)
                                    {:initialize? false :ident [:recipe/id id]} ; (3)
                                     #_ ; this below would have been necessary if
                                    ; DB - :recipe/id - <id> didn't exist already
                                    {:initialize?    true
                                     :initial-params {:recipe/id id}
                                     :keep-existing? true})]
    ;; Load could be hooked into "expand" mutation to remove side-effect from UI logic
    (hooks/use-lifecycle (fn [] (df/load! this [:recipe/id id] Recipe))) ; (4)
    (when recipe
      (ui-recipe recipe))))
;; end::dynamic-recipe[]

(def ui-dynamic-recipe (comp/factory DynamicRecipe))

;; tag::recipe-ref[]
(defsc RecipeReference [this {:ui/keys     [expand?]
                              :recipe/keys [id name]}]
  {:ident :recipe/id
   :query [:ui/expand? :recipe/id :recipe/name]}
  (if expand?
    (ui-dynamic-recipe {:recipe/id id})
    (dom/span {:style {:cursor "pointer"}
               :onClick (fn [] (m/toggle! this :ui/expand?))}
              " ▶ " (dom/span {:style {:textDecoration "underline"}} name))))
;; end::recipe-ref[]

(def ui-recipe-reference (comp/factory RecipeReference {:keyfn :recipe/id}))

(defsc RecipeLineItem [_this {:recipe-line-item/keys [qty uom ingredient sub-recipe]}]
  {:ident :recipe-line-item/id
   :query [:recipe-line-item/id
           :recipe-line-item/qty
           :recipe-line-item/uom
           {:recipe-line-item/ingredient (comp/get-query Ingredient)}
           {:recipe-line-item/sub-recipe (comp/get-query RecipeReference)}]} 
  (dom/ul (str qty " " uom)
    (if (not-empty ingredient)
      (ui-ingredient ingredient)
      (ui-recipe-reference sub-recipe))))

(def ui-line-item (comp/factory RecipeLineItem {:keyfn :recipe-line-item/id}))

(defsc Recipe [_this {:recipe/keys [name line-items]}]
  {:initial-state (fn [{:recipe/keys [id]}] {:recipe/id id}) ; needed for dynamic use
   :ident         :recipe/id
   :query         [:recipe/id
                   :recipe/name
                   {:recipe/line-items (comp/get-query RecipeLineItem)}]}
  (dom/div
    (dom/span {:style {:textDecoration "underline"}}  (str name))
    (dom/ul
      (mapv ui-line-item line-items))))

(def ui-recipe (comp/factory Recipe {:keyfn :recipe/id}))

(defsc RecipeList [this {:recipe-list/keys [recipes]}]
  {:query         [{:recipe-list/recipes (comp/get-query Recipe)}]
   :ident         (fn [] [:component/id ::RecipeList])
   :initial-state {:recipe-list/recipes []}}
  (dom/div {}
    (when-not (seq recipes)
      (dom/button {:onClick (fn []
                              (df/load this :recipe/all Recipe 
                                       {:target [:component/id 
                                                 ::RecipeList 
                                                 :recipe-list/recipes]}))}
                  "Load all recipes"))
    (dom/ul
      (mapv ui-recipe recipes))))

(ws/defcard dynamic-recursive-entity-card
  (ct.fulcro/fulcro-card
    {::ct.fulcro/wrap-root? true
     ::ct.fulcro/root       RecipeList
     ::ct.fulcro/app        (let [remote      (mock-http-server {:parser process-eql})]
                              {:remotes {:remote remote}})}))