There’s a really simple/basic version of that in a few of the example patches. Like @tremblap pick finder one, which I think merges everything else into two components. @jamesbradbury also did something similar in the CV processing thread a while back.
There’s probably a point of diminishing returns if you plan on breaking things down to then build them back up, though the decomposition and reconstitution operate on different paradigms/algorithms.