Making a Web Application Framework

I never thought I'd be creating a framework, but here we are. Though I have to say it probably sounds more grandiose than it is. I'm not trying to reinvent the wheel. Rather, I intend to encapsulate what I'm already doing in my apps in a way that's easier to use. So it'll definitely be built on top of Vue, TailwindCSS, and Soukai.

But I have to start this task with a confession. I've already been working on this for a while 🙈️. Whilst doing the previous task, it was so boring that I just started to tinker on this "on the side". You may not know this, but I work 4 days a week at Moodle, and I allocate 1 day a week to "side-project work"; which is what I write about in this website. In a way, you could say I have a normal 5 days workweek; it's just that I'm self-employed 1 day a week (and I don't have a salary xD). But that means that I still have free weekends and afternoons where I do "life stuff" (not related to programming). However, as things go, sometimes I end up coding anyways. That's usually the same as my side-project work, but during the last month that has felt like a chore and I didn't feel like working on it. So you could say this has been my side-side-project 😅️.

Anyhow, I'm focusing my full attention on this now!

I started making apps "seriously" about 4 years ago (I've talked about this in a recent talk I gave at FOSDEM). During this process, I've been mostly learning about the Solid Protocol and how to make apps the best they can be (according to my tastes, of course). At the beginning, there was a lot of things that changed from one app to the next, but now I feel like I've converged on a certain architecture. The trouble is that I'd like to update my earlier apps, but I don't want to make a complete rewrite. So that's the basic idea and motivation for this task.

My ultimate goal with all of this is being able to focus on the UX of the app itself, rather than thinking about Solid or how to organize my code. I want to avoid accidental complexity complication through conceptual compression. The funny thing is that I was already doing that, back in the days when I was using Laravel. But when I started working on Solid Apps, I lost that. I regained the data layer with Soukai, and I want to create this framework for everything else.

So today, it's official. I'm working on Aerogel!

I haven't reached any important milestones yet, but it's been a month since I started working on this so I thought I'll give a short update on how things are going.

One of the new things I'm trying with this project is using a mono-repo. I have seen this in other frameworks, so I thought I'd give it a try and so far it's working great. I could have used some tools such as Lerna or Turborepo, but I started with npm's built-in in workspaces and so far that's enough.

I also tried using Vitest, and it's also very nice. It's been very easy to get started with since it works mostly like Jest, but mocking has been far easier and everything seems faster. There are also some neat features such as Testing Types that are missing from Jest. It also lives up to the promise of working with the same Vite config in my apps, but for library packages I'm using Rollup so the configuration isn't unified yet. Although I'm giving serious consideration to using Vite for bundling libraries as well. I'll explore that at some point.

Other than that, there isn't much else worth sharing. But if you're curious, I have set up a playground with some of the basic features I've been working on. There is also links to the source for each page, so if you're curious that should give you an idea of what working with the framework will look like.

Funnily enough, I still haven't added any Solid-specific functionality; that will come from Soukai and extracting the Authenticator pattern I'm using in my apps.

Given that we're in summer now and I'll be working less than usual, I don't expect to have any relevant updates for a couple of months. That's also the reason why I'm not thinking of my appetite or setting any deadlines yet. But I'll probably start thinking about that in September/October.

Summer is over (kind of), and I'm back in the saddle.

I did continue working on this during my stay in Japan, but I wasn't trying to get anything done. And to be honest, that's been my mindset for this task since I started it. But I think it's time to start setting some goals into my sight.

Some of the things I've been doing were productive though. For example, I finally added Offline First functionality to the framework, and it was easy to extract from Umai. Other things I've been doing haven't been so fun; in particular all the jazz with bundling, versioning, and taming Node's dependency hell. There's been a lot of talk about AIs taking our jobs as programmers, though I don't think that's happening any time soon. But if anything actually happens, I hope that is AIs taking care of all of this mess. One can only hope.

Anyhow, to wrap up this exploration phase I've prepared a demo that I think encompasses what the framework is all about. If you still weren't sure what the heck I'm trying to do, this should make it clear. And I thought the best way to do the demo was on video. You can find it in my Youtube channel: Aerogel Sneak Peek.

The process of making the video was in itself interesting, because it's the first time I do something like this. It's not like I'm going to become a Youtuber or anything, but I always enjoy watching videos of developers going through code. So I thought it would be fun to give it a shot myself. I'm not 100% happy with the result, but I also think the point of doing this is to get it done quickly, so it's fine that it turned out a bit rough. If you're curious about the tooling, I used OBS for recording the video/camera, and Olive for editing. Honestly, it was a lot easier than I expected. And if I want to improve anything about this in the future, it's more related with my presentation skills than video editing or recording. Special shoutout to Better Dev Screencasts though. Simon is awesome, and even though I didn't spend almost any time on my setup, it was still useful.

Now that that's done, my next goal will be to start writing real apps using the framework. There are still many things I haven't finished in the framework itself, but it's better to start using it rather than waiting for it to be perfect. And the best place to get started is rewriting Ramen. Ideally, the UI will stay the same, because one of my goals with this framework is to empower bespoke UIs. We'll see how that goes!

It is done! We have a full-fledged* Solid App built with AerogelJS: Ramen!

(* Well, maybe saying "full-fledged" is stretching it a bit, it's a very simple app 😅️)

It has taken a bit longer than I expected for a couple of reasons, but overall I'm quite happy with the end result. If you're curious, I suggest that you go straight to the source code, because the UI is exactly the same as before.

One of the reasons for the delay has been yak shaving. Or maybe said differently, over engineering. I've talked about this before (it's actually a recurring theme of this journal), but I think I've got a new twist on it. I've realized that the source of all my misfortunes is my tendency to DCDD (Dream Code Driven Development, I just made that up).

For example, let's take a look at one of the integration tests for Ramen:

it('Teaches Ramen', () => {
    // Arrange
    cy.intercept('PATCH', cssPodUrl('/cookbook/juns-ramen')).as('learnRamen');

    cy.createSolidDocument('/settings/privateTypeIndex', 'privateTypeIndex.ttl');
    cy.updateSolidDocument('/settings/privateTypeIndex', 'register-cookbook.sparql', { cookbookId: '#cookbook' });
    cy.updateSolidDocument('/profile/card', 'register-type-index.sparql');
    cy.createSolidContainer('/cookbook/', 'Cookbook');

    cy.ariaInput('Login url').clear().type(`${cssUrl()}{enter}`);
    cy.cssLogin();

    // Act
    cy.see('You don\'t know how to make Ramen');
    cy.matchImageSnapshot();
    cy.press('Teach me');

    // Assert
    cy.see('You know how to make Ramen!');
    cy.see(`Your Ramen recipe is at ${cssPodUrl('/cookbook/juns-ramen#it')}`);

    cy.get('@learnRamen').its('response.statusCode').should('eq', 201);
    cy.fixture('learn-ramen.sparql').then((sparql) => {
        cy.get('@learnRamen').its('request.body').should('be.sparql', sparql);
    });
});

This test is quite short, but there are a lot of things going on: Snapshot Testing, Accessibility Testing, Solid CRUD Testing, etc. It is indeed encapsulating a lot of complexity, but you wouldn't say that the code itself is complex. This is what I mean with conceptual compression. And this is the type of code I consider good code.

The problem, of course, is that reaching that point takes a lot of effort. This is now real code, but getting there was not straightforward. And I wonder about the alternatives. Surfacing all that complexity to the end user? Doing less? I'm in favour of the latter, but certainly not the former. Although I realize there is such a thing as overdoing it, and I have a tendency to compress too early. This is something I've known for a while, but I recently read a blog post that may finally get me to improve: Live with it for a while.

So yeah, my over engineering afflictions are still ongoing. But I have a renewed resolve to alleviate them by doing less and living with "bad code" for a while.

However, this time I cannot attribute all the delays to my over engineering. It seems like I've also started to feel a bit of maintenance burden. Nothing like the author of that post, but for a couple of weeks it seemed like I wasn't making any progress. And after some reflection, I realized I had spent too much time doing reactive work; things people asked of me or caused by some external dependency.

Not that it was all bad, though, many good things came out of those: giving feedback for rdfjs.dev, giving feedback for soukai-solid-utils, participating in the Solid Practitioners group, etc. Although I probably spent more time on some of them than I should (of my own volition by the way, I'm not saying that anybody coerced me into contributing). But there was one particularly nasty task of rewriting all my CSS testing helpers because the account management architecture changed completely in 7.0.0.

I suppose those are the cost of doing business though, and I'm aware this is nothing compared to what popular repository maintainers must go through. For the most part, I still feel like no one is looking, but I get some interactions from time to time. And I'm still torn on whether that's a good thing or a bad thing. One the one hand, I like the fact that I can be left to my own devices and just enjoy the work that I do. On the other hand, it would be nice that my work is useful to more people. But as Naval says, it's better to be anonymous and rich than to be poor and famous. And being a popular open source maintainer certainly takes you down the path of "poor and famous". So I'm fine being anonymous and well off for the time being.

Now that I'm done with the fundamentals of the framework, and I have proved that I can make a "real app" with it, the next step is to actually start doing something useful. I have two possibilities at the moment: migrating Solid Focus (which I'll probably rewrite from scratch) or migrating Media Kraken and implement TV Shows tracking. As much as I'd like to do the latter, it's more likely that I do the former, because Solid Focus was my first Solid App and it is the one in more need of a facelift.

Whatever I decide, I won't have any updates until January so I hope you enjoy the end of the year and have a chance to spend it with the ones you love. As for me, I can't wait to do my end of year ritual and I'm looking forward to yet another lap around the sun in this hunk of stone.

Hello, 2024!

I'm back from the holiday break, and I'm ready to get back on the horse. Some years, I've used this season to advance on my side-projects as well. But this time I made a pledge of not going near a keyboard (for programming purposes). So I've spend it mostly watching movies and TV shows, playing games, and visiting the land of the Yule Lads.

Right before the break though, I made an end of year gift for Solid developers: 🎁 cypress-solid. You may want to check that out!

Anyhow, now that I'm done with all of new year's chores, I'm ready to get on with this task. And after some thought, I've decided that revamping Solid Focus will be the next and final step. Admitedly, that will be cheating a bit, because I won't be releasing the framework after that (as in making an official announcement that it's "ready"). But I've had enough of multi-year tasks, so I'll tackle the branding and documentation of the framework in a separate task. I will consider the framework's first draft done after I have a real app built with it.

Something I've noticed looking back at 2023 is that I haven't been doing Shape Up cycles as of late. But I found them very useful for getting things done when tasks were running too long, so that's how I'll approach this revamp.

I'm kicking off a cycle today, and it should be done by March 1st. The scope for this cycle is to rebuild Solid Focus, from scratch, using AerogelJS. And yes, I mean a feature-complete rebuild. It took me about 7 months to build the previous version, but I'm a lot more experienced with Solid now, and the point of the framework is that it should allow me to be faster. So this should be a good stress-test. The framework is not finished, however, specially the offline-first approach which I'll want to add and is novel to Solid Focus. Because of that, I also don't want to be too optimistic, and I'm leaving some things out of the scope. In particular, I won't be tackling any of the data migration or branding improvements. Both of which I'll definitely look into before re-releasing the revamped app.

This week was the end of the cycle, and TLDR this is how things stand:

Looking back at the evolution of the chart, it started really well because in a couple of weeks I had most things from the "Workspaces & Lists" and "Tasks CRUD" scopes done. But then came Solid! I thought it'd be easy, because I had already done all the CRDT stuff in Umai. But I didn't anticipate how much the different data structure would affect it.

Umai's data structure is quite straight-forward; there is a container with Recipes and that's it. But Solid Focus has a hierarchy of nested containers, with Tasks within. So it was a bit of a hassle to adapt, but the good news is that all of this should be a lot more robust now. In fact, I'm pretty sure it'll help Umai's interoperability as well, because it'll be able to support multiple Recipe containers.

All in all, though, I'm happy with the outcome of these 6 weeks. I don't think I have complicated things necessarily, and I still managed to create some new abstractions with elegant APIs. If you want to learn more about the current status and my development workflow, I published a new video in my Youtube channel: Rebuilding Solid Focus with Aerogel | Basic functionality.

I'm really enjoying making these videos, so expect more. I'm aware the quality could be better, but I don't want to spend a lot of time on that at the moment. Although this time I think I actually did it too quickly, because just after publishing I realized there were a couple of cool things I hadn't mentioned 😅. Such as Route Model bindings, computed Model refs, global getters, etc. On the flip side, that made it a shorter video so it probably makes it more appealing to some people.

Now, having done that, I think the foundations are laid out. Although there is certainly a lot more left: UI/UX, data migration and interoperability, advanced features, etc. I'm still optimistic that it shouldn't take too long, but at the end of the month I'll be afk for a couple of weeks; so I'm refraining from starting a new cycle now. I'll use the cooldown period to clean up a couple on things, and hopefully by next month I'll be kicking off the next (and final?) cycle for the rebuild.