Making a Web Application Framework

I never thought I'd be creating a framework, but here we are. Though I have to say it probably sounds more grandiose than it is. I'm not trying to reinvent the wheel. Rather, I intend to encapsulate what I'm already doing in my apps in a way that's easier to use. So it'll definitely be built on top of Vue, TailwindCSS, and Soukai.

But I have to start this task with a confession. I've already been working on this for a while 🙈️. Whilst doing the previous task, it was so boring that I just started to tinker on this "on the side". You may not know this, but I work 4 days a week at Moodle, and I allocate 1 day a week to "side-project work"; which is what I write about in this website. In a way, you could say I have a normal 5 days workweek; it's just that I'm self-employed 1 day a week (and I don't have a salary xD). But that means that I still have free weekends and afternoons where I do "life stuff" (not related to programming). However, as things go, sometimes I end up coding anyways. That's usually the same as my side-project work, but during the last month that has felt like a chore and I didn't feel like working on it. So you could say this has been my side-side-project 😅️.

Anyhow, I'm focusing my full attention on this now!

I started making apps "seriously" about 4 years ago (I've talked about this in a recent talk I gave at FOSDEM). During this process, I've been mostly learning about the Solid Protocol and how to make apps the best they can be (according to my tastes, of course). At the beginning, there was a lot of things that changed from one app to the next, but now I feel like I've converged on a certain architecture. The trouble is that I'd like to update my earlier apps, but I don't want to make a complete rewrite. So that's the basic idea and motivation for this task.

My ultimate goal with all of this is being able to focus on the UX of the app itself, rather than thinking about Solid or how to organize my code. I want to avoid accidental complexity complication through conceptual compression. The funny thing is that I was already doing that, back in the days when I was using Laravel. But when I started working on Solid Apps, I lost that. I regained the data layer with Soukai, and I want to create this framework for everything else.

So today, it's official. I'm working on Aerogel!

I haven't reached any important milestones yet, but it's been a month since I started working on this so I thought I'll give a short update on how things are going.

One of the new things I'm trying with this project is using a mono-repo. I have seen this in other frameworks, so I thought I'd give it a try and so far it's working great. I could have used some tools such as Lerna or Turborepo, but I started with npm's built-in in workspaces and so far that's enough.

I also tried using Vitest, and it's also very nice. It's been very easy to get started with since it works mostly like Jest, but mocking has been far easier and everything seems faster. There are also some neat features such as Testing Types that are missing from Jest. It also lives up to the promise of working with the same Vite config in my apps, but for library packages I'm using Rollup so the configuration isn't unified yet. Although I'm giving serious consideration to using Vite for bundling libraries as well. I'll explore that at some point.

Other than that, there isn't much else worth sharing. But if you're curious, I have set up a playground with some of the basic features I've been working on. There is also links to the source for each page, so if you're curious that should give you an idea of what working with the framework will look like.

Funnily enough, I still haven't added any Solid-specific functionality; that will come from Soukai and extracting the Authenticator pattern I'm using in my apps.

Given that we're in summer now and I'll be working less than usual, I don't expect to have any relevant updates for a couple of months. That's also the reason why I'm not thinking of my appetite or setting any deadlines yet. But I'll probably start thinking about that in September/October.

Summer is over (kind of), and I'm back in the saddle.

I did continue working on this during my stay in Japan, but I wasn't trying to get anything done. And to be honest, that's been my mindset for this task since I started it. But I think it's time to start setting some goals into my sight.

Some of the things I've been doing were productive though. For example, I finally added Offline First functionality to the framework, and it was easy to extract from Umai. Other things I've been doing haven't been so fun; in particular all the jazz with bundling, versioning, and taming Node's dependency hell. There's been a lot of talk about AIs taking our jobs as programmers, though I don't think that's happening any time soon. But if anything actually happens, I hope that is AIs taking care of all of this mess. One can only hope.

Anyhow, to wrap up this exploration phase I've prepared a demo that I think encompasses what the framework is all about. If you still weren't sure what the heck I'm trying to do, this should make it clear. And I thought the best way to do the demo was on video. You can find it in my Youtube channel: Aerogel Sneak Peek.

The process of making the video was in itself interesting, because it's the first time I do something like this. It's not like I'm going to become a Youtuber or anything, but I always enjoy watching videos of developers going through code. So I thought it would be fun to give it a shot myself. I'm not 100% happy with the result, but I also think the point of doing this is to get it done quickly, so it's fine that it turned out a bit rough. If you're curious about the tooling, I used OBS for recording the video/camera, and Olive for editing. Honestly, it was a lot easier than I expected. And if I want to improve anything about this in the future, it's more related with my presentation skills than video editing or recording. Special shoutout to Better Dev Screencasts though. Simon is awesome, and even though I didn't spend almost any time on my setup, it was still useful.

Now that that's done, my next goal will be to start writing real apps using the framework. There are still many things I haven't finished in the framework itself, but it's better to start using it rather than waiting for it to be perfect. And the best place to get started is rewriting Ramen. Ideally, the UI will stay the same, because one of my goals with this framework is to empower bespoke UIs. We'll see how that goes!

It is done! We have a full-fledged* Solid App built with AerogelJS: Ramen!

(* Well, maybe saying "full-fledged" is stretching it a bit, it's a very simple app 😅️)

It has taken a bit longer than I expected for a couple of reasons, but overall I'm quite happy with the end result. If you're curious, I suggest that you go straight to the source code, because the UI is exactly the same as before.

One of the reasons for the delay has been yak shaving. Or maybe said differently, over engineering. I've talked about this before (it's actually a recurring theme of this journal), but I think I've got a new twist on it. I've realized that the source of all my misfortunes is my tendency to DCDD (Dream Code Driven Development, I just made that up).

For example, let's take a look at one of the integration tests for Ramen:

it('Teaches Ramen', () => {
    // Arrange
    cy.intercept('PATCH', cssPodUrl('/cookbook/juns-ramen')).as('learnRamen');

    cy.createSolidDocument('/settings/privateTypeIndex', 'privateTypeIndex.ttl');
    cy.updateSolidDocument('/settings/privateTypeIndex', 'register-cookbook.sparql', { cookbookId: '#cookbook' });
    cy.updateSolidDocument('/profile/card', 'register-type-index.sparql');
    cy.createSolidContainer('/cookbook/', 'Cookbook');

    cy.ariaInput('Login url').clear().type(`${cssUrl()}{enter}`);
    cy.cssLogin();

    // Act
    cy.see('You don\'t know how to make Ramen');
    cy.matchImageSnapshot();
    cy.press('Teach me');

    // Assert
    cy.see('You know how to make Ramen!');
    cy.see(`Your Ramen recipe is at ${cssPodUrl('/cookbook/juns-ramen#it')}`);

    cy.get('@learnRamen').its('response.statusCode').should('eq', 201);
    cy.fixture('learn-ramen.sparql').then((sparql) => {
        cy.get('@learnRamen').its('request.body').should('be.sparql', sparql);
    });
});

This test is quite short, but there are a lot of things going on: Snapshot Testing, Accessibility Testing, Solid CRUD Testing, etc. It is indeed encapsulating a lot of complexity, but you wouldn't say that the code itself is complex. This is what I mean with conceptual compression. And this is the type of code I consider good code.

The problem, of course, is that reaching that point takes a lot of effort. This is now real code, but getting there was not straightforward. And I wonder about the alternatives. Surfacing all that complexity to the end user? Doing less? I'm in favour of the latter, but certainly not the former. Although I realize there is such a thing as overdoing it, and I have a tendency to compress too early. This is something I've known for a while, but I recently read a blog post that may finally get me to improve: Live with it for a while.

So yeah, my over engineering afflictions are still ongoing. But I have a renewed resolve to alleviate them by doing less and living with "bad code" for a while.

However, this time I cannot attribute all the delays to my over engineering. It seems like I've also started to feel a bit of maintenance burden. Nothing like the author of that post, but for a couple of weeks it seemed like I wasn't making any progress. And after some reflection, I realized I had spent too much time doing reactive work; things people asked of me or caused by some external dependency.

Not that it was all bad, though, many good things came out of those: giving feedback for rdfjs.dev, giving feedback for soukai-solid-utils, participating in the Solid Practitioners group, etc. Although I probably spent more time on some of them than I should (of my own volition by the way, I'm not saying that anybody coerced me into contributing). But there was one particularly nasty task of rewriting all my CSS testing helpers because the account management architecture changed completely in 7.0.0.

I suppose those are the cost of doing business though, and I'm aware this is nothing compared to what popular repository maintainers must go through. For the most part, I still feel like no one is looking, but I get some interactions from time to time. And I'm still torn on whether that's a good thing or a bad thing. One the one hand, I like the fact that I can be left to my own devices and just enjoy the work that I do. On the other hand, it would be nice that my work is useful to more people. But as Naval says, it's better to be anonymous and rich than to be poor and famous. And being a popular open source maintainer certainly takes you down the path of "poor and famous". So I'm fine being anonymous and well off for the time being.

Now that I'm done with the fundamentals of the framework, and I have proved that I can make a "real app" with it, the next step is to actually start doing something useful. I have two possibilities at the moment: migrating Solid Focus (which I'll probably rewrite from scratch) or migrating Media Kraken and implement TV Shows tracking. As much as I'd like to do the latter, it's more likely that I do the former, because Solid Focus was my first Solid App and it is the one in more need of a facelift.

Whatever I decide, I won't have any updates until January so I hope you enjoy the end of the year and have a chance to spend it with the ones you love. As for me, I can't wait to do my end of year ritual and I'm looking forward to yet another lap around the sun in this hunk of stone.

Hello, 2024!

I'm back from the holiday break, and I'm ready to get back on the horse. Some years, I've used this season to advance on my side-projects as well. But this time I made a pledge of not going near a keyboard (for programming purposes). So I've spend it mostly watching movies and TV shows, playing games, and visiting the land of the Yule Lads.

Right before the break though, I made an end of year gift for Solid developers: 🎁 cypress-solid. You may want to check that out!

Anyhow, now that I'm done with all of new year's chores, I'm ready to get on with this task. And after some thought, I've decided that revamping Solid Focus will be the next and final step. Admitedly, that will be cheating a bit, because I won't be releasing the framework after that (as in making an official announcement that it's "ready"). But I've had enough of multi-year tasks, so I'll tackle the branding and documentation of the framework in a separate task. I will consider the framework's first draft done after I have a real app built with it.

Something I've noticed looking back at 2023 is that I haven't been doing Shape Up cycles as of late. But I found them very useful for getting things done when tasks were running too long, so that's how I'll approach this revamp.

I'm kicking off a cycle today, and it should be done by March 1st. The scope for this cycle is to rebuild Solid Focus, from scratch, using AerogelJS. And yes, I mean a feature-complete rebuild. It took me about 7 months to build the previous version, but I'm a lot more experienced with Solid now, and the point of the framework is that it should allow me to be faster. So this should be a good stress-test. The framework is not finished, however, specially the offline-first approach which I'll want to add and is novel to Solid Focus. Because of that, I also don't want to be too optimistic, and I'm leaving some things out of the scope. In particular, I won't be tackling any of the data migration or branding improvements. Both of which I'll definitely look into before re-releasing the revamped app.

This week was the end of the cycle, and TLDR this is how things stand:

Looking back at the evolution of the chart, it started really well because in a couple of weeks I had most things from the "Workspaces & Lists" and "Tasks CRUD" scopes done. But then came Solid! I thought it'd be easy, because I had already done all the CRDT stuff in Umai. But I didn't anticipate how much the different data structure would affect it.

Umai's data structure is quite straight-forward; there is a container with Recipes and that's it. But Solid Focus has a hierarchy of nested containers, with Tasks within. So it was a bit of a hassle to adapt, but the good news is that all of this should be a lot more robust now. In fact, I'm pretty sure it'll help Umai's interoperability as well, because it'll be able to support multiple Recipe containers.

All in all, though, I'm happy with the outcome of these 6 weeks. I don't think I have complicated things necessarily, and I still managed to create some new abstractions with elegant APIs. If you want to learn more about the current status and my development workflow, I published a new video in my Youtube channel: Rebuilding Solid Focus with Aerogel | Basic functionality.

I'm really enjoying making these videos, so expect more. I'm aware the quality could be better, but I don't want to spend a lot of time on that at the moment. Although this time I think I actually did it too quickly, because just after publishing I realized there were a couple of cool things I hadn't mentioned 😅. Such as Route Model bindings, computed Model refs, global getters, etc. On the flip side, that made it a shorter video so it probably makes it more appealing to some people.

Now, having done that, I think the foundations are laid out. Although there is certainly a lot more left: UI/UX, data migration and interoperability, advanced features, etc. I'm still optimistic that it shouldn't take too long, but at the end of the month I'll be afk for a couple of weeks; so I'm refraining from starting a new cycle now. I'll use the cooldown period to clean up a couple on things, and hopefully by next month I'll be kicking off the next (and final?) cycle for the rebuild.

I've been leaving this task on the side for some weeks, but today I'm back. And I intend to be almost finished with the rebuilt by the end of the cycle (more on that later).

Now, it's not like I've been doing nothing. In fact, I've been tinkering with the UI a lot; and I decided on the style I'm going to use. But it wasn't easy to get there, because I've been struggling with an issue that has been on the back of my mind for a while.

Basically, I really like the idea of doorless apps. That's one of the reasons why mine don't have intricate landing pages. People can start using them right away, without even creating an account, so I haven't spent any time trying to convince anyone. The barrier to entry is so low, I figured, that everyone landing on the apps would just start using them. However, I've always been aware of how uninviting that is. There are some doorless apps like excalidraw that make it immediately obvious. But that's not the case with the current design of my apps.

So when I started thinking on the landing page for this one, I had two options. Either go full doorless and drop people on some empty state, or make a "real" landing page. As fate would have it, I started taking Jack McDade's Radical Design Course in the middle of making this decision. And after some experiments, I've decided to make a real landing page. I'm very excited to share it with everyone, but I don't think I'll show it until I release the final version 🤫. For now, I'll only say that I've been using Fooocus and Penpot, and it will be radically different from any of my other apps.

If I don't add some structure to this, though, this process of tinkering with UI can drag on indefinitely. So today I'm kicking of the next cycle. It will end on June 28th, and my idea is to have the app completely finished by then. However, there are some things I'll still need to do before releasing it, such as migrating data from the legacy version. Because of that, this doesn't mean that I'll release the app by then. But it should take me much closer to the finish line.

Ok, so the cycle finished yesterday and this is where things stand:

As always, I haven't finished everything; but I can say I am mostly done with UI! There are still a couple of screens I have to tweak, but it's mostly a chore at this point. I would consider the branding and style of the app completed. Furthermore, I've also implemented a couple of new features that weren't available in the old version of the app, such as global search and custom workspace colors.

So, what is left, and what are the next steps?

Even though most of the main work is done, I still have to work on some edge-cases. For example, in Umai I let users choose where they want to store their recipes, and I wanted to do the same in Focus. But it's more tricky, because each workspace and list is a separate container. This also complicates the synchronization, I haven't completely finished the secret landing page, and I haven't even started looking at migrating data from the legacy app 😱. Finally, I also have some important news I'll talk about next month that will probably have an impact on this task.

With all that, I've realized the release is farther away than I thought... So I'll do something in between. Similar to what I did with the Umai Beta, I'll release a beta of the new version for early feedback and testing. But this time, I think I'll be more aggresive on the "beta" thing, and I don't think I'll call it "production ready" as I did for Umai. Still, I'll probably use it for production myself, but I won't recommend it.

And that about covers my plans for the next month. You should hear from me again by the end of July / early August, and I'm very excited to finally share what I've been working on in 2024 :).

It's only been two weeks since my last update, but I'm already done with the beta! TLDR, you can peek at the new version in focus.noeldemartin.com.

Turns out it didn't need as much work as I thought, even though it required chasing a bug that involved debugging and Android device over Wifi and dealing with proxies 😅. Also, it's not like I'm done with bundling woes (I'm not sure I'm ever getting rid of those :/). But it's ready now!

I haven't been working on this rebuild for too long, but for some reason it feels very similar to when I pre-released Umai. I guess it's just the thrill of releasing new stuff :). This time, though, I have some things up my sleeve before the final release.

Even though it's "just a rebuild", I'm very excited about this one. Not only because I feel it's better in terms of functionality (I understand Solid a lot better now!), I also redesigned the UI from scratch. In the previous version, I used Vuetify; and I didn't think much about the design at all. But this time I have, and the result is a lot less cluttered and clean. I worry a bit that all my apps end up looking the same, but I think I've managed to give them a unique personality for the most part.

Now that the beta is out the door, the question remains when will I release the final version. You can learn more about what's left in the Beta Feedback issue (and leave some feedback!), but the short answer is that I don't know. I'm posting a life update in my blog early next month, but honestly I don't think I'll be releasing the app this year 😱. You'll learn why in the blog post, but that leaves plenty of time to live with the beta myself, and make sure that it's ready once I finally release it to the world.

Anyways, if you want to learn more about the new version, I also uploaded a new video to Youtube with some more technical details and new Aerogel features: Rebuilding Solid Focus with Aerogel | Routing and UI.