Skip to content

Implementing a Recipes Manager using Solid

(Ongoing)

I'm still on a path to rebuild my online experiences using Solid, and next up is an alternative to Evernote Food. It has been dead for years, so I've kept my cooking recipes in Evernote and the experience has been subpar. It's time to make it better.

In contrast with my previous Solid apps, this one will have sharing functionality from the beginning - because recipes are made for sharing! One of the things I'm looking forward to explore is Solid panes, or how to customize the display of a resource in its public url. I'll also make sure that this app works in different PODs, because NSS is not the only option anymore. I'll test with NSS, CSS, ESS and PSS.

And this will be the first time I use Vue 3 in an app, so a lot of things I'm excited for in this task!

Activity

Task started

I have been tinkering with many projects under development, so it's been a rough couple of weeks but I'm ready to share some progress now.

First I'll start with Vue 3 and Vite. I was really excited to start using those, because I have been hearing about them for a while but I hadn't used them myself. At first it was great, and I think they are the future of my development workflow. But soon, things started to break down. To their credit, it's not exactly their fault.

Vite uses esbuild to compile the dependencies during development, and while it makes it really fast, it also means that the libraries have to be compatible with ES modules. There are ways to work around that limitation, but I spend way too much time trying to make things work and I decided to leave it there. To make things worse, Vite uses Rollup for production builds. While I do prefer Rollup over Webpack, having two build systems increases the possibility for headaches. In the end, I wasn't even sure if the problem was that the packages were not compatible with ES modules, what I know is that using Webpack all my problems were solved. I hope some time from now this is not the case, because I really liked Vite. But at the moment, I don't think it's worth it if you're going to use third party libraries.

Vue 3, on the other hand, has been a better experience. The only thing I haven't been able to use is the script setup sugar because it doesn't work with Vetur. But I am using the Composition API and everything else has been a smooth sailing.

In other news, something I'm starting to do with this project is using my own domain to serve applications. I'm still hosting them on github pages, but other than removing the domain dependency with github this also helps with sandboxing. I have been serving my apps under noeldemartin.github.io/{app-name}, and they are all sharing things like localStorage which is not a good thing. So you can now find my latest finished Solid App at https://ramen.noeldemartin.com.

Wait, what? No, I haven't finished this task yet! Let me explain.

One of the things I want to achieve with this task is to test my apps with multiple server implementations. This is the perfect chance because the codebase is really simple, but as it continues evolving it will become too complex to use as a compatibility tool. So I started making some jokes, one thing lead to another, and I decided to make a separate application just for this purpose.

You can learn what it does in the project's README. In a nutshell, it checks if you've got a Ramen recipe in your POD and if you don't you can create it. I was really tempted to use Mr Jägger's Ramen recipe (click through this link at your own risk), but I ended up using Jun's Ramen recipe.

I have tested this app with the 4 servers I mentioned for this task, and it only works properly with 2 of them (more details about that also in the README). The two where it doesn't work are still under development, so that's to be expected. But it's been helpful to tinker with them anyways, because I've learned some things I wasn't clear about before. Now I understand the difference between identity providers and POD providers (I learned about it in this issue). This lead me to use a new strategy for logging users into my apps. In my other apps, I just send the url that users introduce to the authentication library, but some times that may not work (for example, if they are writing their WebId and the identity provider is hosted on a different server). In this app, I am reading their profile and trying to search for solid:oidcIssuer or infer it doing some requests. Most of the time this will result in some unnecessary requests, but it only happens on login and all of them should be very small.

There are still some rough edges with authentication, but some of them are not in my hands so for now I'll leave them as things to improve. Something I don't like but I don't think I'll be able to avoid is using two authentication libraries: @inrupt/solid-client-authn-browser for new servers that support DPoP authentication and solid-auth-client for old servers that don't. I have been able to avoid increasing the bundle size by code-splitting both libraries and only loading the one that's used (although I found some problems with the latest version of solid-auth-client). But I still haven't found a foolproof way to know if a server doesn't support DPoP authentication, so I'm relying on heuristics and hardcoded domains for now.

And, long as it was, that's it for today's update. With this I think I've covered most of the boring part about this task: authenticating and working with different servers. Now I can get into the fun part: building a recipes manager. For now I am calling it "Umai".

On my last update I mentioned that I'd start working on the recipes manager, but some things have come up and I continued working on the authentication workflow. I was already uneasy about hardcoding domains in the code, but this backfired sooner than I expected. After seeing this and having other conversations, I decided to abandon the strategy of using heuristics. However, asking people what authentication method to use is not a good idea either. So I ended up with a halfway approach. Instead of showing a dropdown with authentication methods, I use DPoP by default, and I've hidden the dropdown behind a "Can't log in?" button. With this approach, most people should log in without any friction and only those experiencing problems will see anything about authentication methods. I also refactored the implementation to decouple authentication libraries from the rest of the codebase. Now it's easy to add any authentication library by extending a class, and I've added solid-auth-fetcher to the list of options for testing purposes.

So, I deployed this new approach to Ramen and that was it. I wasn't planning on adding it to Media Kraken yet, because there is still an issue with Inrupt's library that makes UX sub-optimal. However, I was invited to present at Solid World! This meant that new people would probably check it out, and it would be a shame if they try it with servers that only support DPoP. So I've gone ahead and added it to Media Kraken as well. Now I should be able to forget about authentication for a while. I also found that I'm relying on something that is not part of the spec, but I can't do anything about it.

Also in preparation for Solid World (albeit tangentially), I decided to take a stab at sharing public data with Media Kraken Viewer. I'll explore this a lot more with recipes, but it can hopefully be useful for some people already. If you want to see an example in action, here's a list with 25 of my favorite movies.

So yeah, that was Media Kraken and Solid World. I'm very happy to have presented, and I met more people that have been using my apps and reading this journal. If you want to watch the recording, you can find it here.

Now, back to the recipes manager. Although I've been doing other things, I also did some preparations for it. I started reading Shape Up again, and I want to go back to applying the methodology to my side projects (we all know how that ended last time). I realize one of my mistakes is that I only used the concept of appetite, without doing any of the actual shaping or looking at rabbit holes. This time, I've been doing that and I've written a "proper" pitch. It's a bit pointless because I'm selling it to myself, but I found the process useful. I only included 4 goals for this upcoming cycle, and I've listed a lot of things that are out of scope and rabbit holes.

First, I've made a simple diagram of the initial version of the app. This is the main deliberable, everything else can be dropped or reduced with the scope hammer. I know the diagram is stupidly simple, and that's the point.

Breadboard Diagram

Second, I've decided to create my own ontology to augment the Recipe type on schema.org. I'll talk more about this when it is done, but essentially the schema.org vocabulary is very limited and I want more nuance in my data. But it's also more popular than whatever I make, so I'll try to mixing both in a way that other applications can understand for interoperability.

Third, given that this is my 3rd Solid app, I'll start creating a framework. It's not my intention to announce this as a new project, it's just a way to extract code to avoid repeating myself. But it's very likely that I'll eventually clean it up for others to use. I recently discovered Vitesse, and it seems very nice so I'll probably draw a lot of inspiration from it. You may have noticed that it's built on top of Vite, and I said in my last update that I wouldn't use it. Well, as part of the shaping process I've given it another chance and I decided to give it another chance. If any library doesn't work with Vite, I'll just compile it with webpack and load the chunk dynamically. Not a great solution, but it doesn't seem like I'll be able to resist the temptation. It's already bad enough that I haven't used Laravel in months because I don't write backend code anymore.

And fourth, I'll bring Soukai up to speed with some things I've learned in the last months. Tinkering with Vite and creating my utils library taught me a lot. One of the things I'm excited to solve is an issue I've had since its inception.

This time I'll follow the recommended 6-week cycle length, and I'm not tracking hours like I did last time. I'll start next week, so this should be done by March 22nd. Hopefully, I'll write some updates before that!

It's been 4 weeks since the cycle started, and here's how the Hill Chart looks like (yes, I even started using Basecamp!):

Hill Chart

You'll notice that I've only gotten over the hill in one scope: Soukai Next Generation. It may look disappointing, but to be frank I'm happy with the results (which I'll get into in a second). I think this time I'm finally doing Shape Up rightish. One of the ideas I was more excited about from the book was "fixed time, variable scope". Specially given my situation; I'm only working one day a week on Solid. So time is more precious than usual.

Now, let's get into the meat of Soukai Next Generation. Like always, I'm wondering whether the work I'm doing is even worth it or I'm shaving yaks. 90% of what I've been doing these 4 weeks has been purely focused on developer experience, or said differently, "not necessary". But I'm enjoying it and I'm happy with the progress, so it's worth it in my book.

The first thing I did was migrating to Rollup and ESLint (from Webpack and TSLint). This seemed pretty useless at first, but later on it paid off when I started working in Viteland. I started doing this by hand in each project, but it became cumbersone quickly so I ended up extracting the build configurations into their own repository. Other than upgrading the tooling, I also learned a bit more about all the flavors of javascript. I identified 3 environments where my libraries can be used: Other libraries or apps (will normally use the ESModules build), scripts and tests (will normally use the CommonJS build) and plain HTML (will include the UMD build as a script tag in a page). The latter use-case is the only one that I don't use myself, but I'll still document it because I think it's interesting for getting started without a build step. During this upgrade, I also understood the polyfills I'm actually shipping with the libs and the environments I'm supporting (before this I think webpack was using babel under the hood but I'm not even sure).

Something else I did was improving TypeScript inference in models. It's still not as good as I'd like, but I've removed a couple of annoyances and improved many things. So that's been a big win. I'll document all of this more in depth when I release the next version (which I already know will not happen during this cycle). Here are some tests about the new inference in case you're curious.

As you can see in the Hill Chart, I've also advanced on using Vite and implementing the app. Thanks to the previous work, Soukai now works out of the box with both Vite and Webpack. As I suspected, Vite is a joy to work with once you don't have issues with external libraries. I've even been using a couple of plugins to do some magic stuff and it's working great. I know it won't always be like this, because some library will eventually fail to compile with Vite (I already know this happens with some of the authentication libraries I'm using). But I also have a plan for that, I'll bundle problematic libs with webpack as I did in this issue.

The current status of the app is very basic, but I'd say the foundations are laid out and that's why it's halfway uphill. It's not usable at all, but if you're curious about the code now it's actually a great moment to take a look because there is so little.

The custom vocab thing is something I didn't even start, and I think I'll drop it for this cycle. The nice thing about Shape Up is that this doesn't mean that I'll pick it up on the next cycle, because it's important to keep the slate clean. So it'll compete against all the other things I want to do in my next betting table.

Today is the end of the cycle, and this is where things are at:

Hill Chart

In the end, I didn't finish v0.0.1 as I'd budgeted, that's why that scope is not all the way down the hill. But I almost got there, the only thing that's missing is configuring servings. I'm happy with the results, because the part I was interested in was laying out the foundations. And now I can say that Umai exists! It's definetly not finished, definetly not production-ready, but it exists. From now on, I "only" need to improve it.

The work on Soukai Next Generation has moved a bit, but in reality it hasn't changed much since my last update. I had done a lof of refactors and improvements, but I hadn't actually used them anywhere. Now that I'm using them in Umai, I'm more confident that the scope is further downhill.

The part that's changed the most since last time is the framework. I'm writing what I'd consider framework code in a different folder, and that's what I'll eventually extract into a package to use across applications. While doing this, I've created a couple of new patterns that I'm very happy with (and borrowed some from my favourite frameworks, like Laravel's Facades). If you look around the source code without going into the framework folder, you'll see there is almost no boilerplate. Most of the code is specific to this application. I'm enjoying a lot working on the framework, and I'm itching to document it and share it with others. But I'll resist the urge, at least until I finish Umai.

These two, Soukai Next Generation and the framework, come together during development in a big way. So far I've been able to work on the app without being bothered about any Solid specifics. Foregoing a POD was already possible using a browser engine from Soukai, and now that I've incorporated the Authenticators pattern I introduced in Ramen, I don't have to use an identity provider either. Soukai isn't perfect, and the framework isn't even a thing. But this is promising because I can see a day where I'll be working on apps without thinking of them as "Solid apps". And that's my end goal, Solid should just be a given.

Going back to planning though, this is where the Shape Up methodology breaks down for me. Seeing this cycle from a pure Shape Up perspective, it's a failure because the only "finished" thing I have is Umai v0.0.1. But to really consider it finished I would need to tag the v0.0.1 version in github, release new versions of all the related libraries (soukai, soukai-solid, utils, scripts), and all that goes with that (like documentation). All of that should have been part of this cycle, but I'll leave it for later. That's ok, though. Nobody is forcing me to be a purist. And I'd be surprised if the guys at Basecamp followed it to a T during the development of HEY. So I'm making the most out of it, and this time I can say it's been really useful because I got something done and I've dropped the vocabs thing that could have become a rabbit-hole (may come back in the future though!).

For the upcoming weeks, I'll use the cooldown to do some chores in other projects and shape the work for the next cycle. I'm still not sure if the next cycle will be the one where I release the app, but it probably won't. There are many things I still want to do that I consider essential.

Since my last update, I've been thinking on what to work on next. Something that's been nagging me for a while is that the authentication workflow I have in Media Kraken is not great. I realized it when I prepared the demo for my Solid World presentation.

The basic idea is that a user would start using the app with browser storage, so far so good. But then, when they want to move to Solid, they have to download a json, log out, log in, and import the json. That isn't so bad, but it's definitely more cumbersome than it should (and for some people it'll be a barrier). It's also annoying that even though the app works mostly offline, opening the app takes ages in mobile, because it reads the entire movies container at launch (and I have 1671 movies in my collection at this point). This was probably solved after NSS#1460 was closed, but I'm still using an old version of NSS (and my mobile phone is also quite old). But I'm in no rush to upgrade, after all that's how I notice these things.

Thinking how to improve this situation I brushed up on some ideas I had on my backlog. And now I am convinced that offline-first is the solution to these problems. In Umai, users will be using browser storage by default, and they'll be able to add synchronisation backends (for example, Solid). Reading about this topic, I came across some interesting concepts like CRDTs (and some funny encounters, like leap seconds). In particular, I enjoyed a lot an article about Local-first software.

So now it's almost certain that the next cycle will be focused on making Umai offline-first. But I don't think I'm ready to start yet, so I'll spend some indefinite ammount of time (hopefully not too much) shaping the work. I've already started some of the shaping process, and I opened a couple of posts on the Solid forum to get feedback. If you're interested in participating, check out these posts:

Hi again!

It's been only a week since I updated this task, but I decided to go ahead and kickstart the next cycle. The conversations about authentication and CRDTs are still ongoing, but I didn't get any strong arguments against what I had in mind, so I'll start working on it. There are some concerns that I haven't resolved, but I think the best way to proceed is to start working on the real thing.

As part of the shaping process, I implemented a proof of concept using a Solid POD to sync changes across devices:

There is a lot of hard-coded parts, that's why I didn't push this code to github. But I am very pleased with the result.

So, for this cycle I'll focus on the following:

  • Offline-first: This may seem small, but there are actually a lot of things to do here. My intention is to be done with the data layer after this, so it should take the main focus.

  • Recipe ingredients: This is important to see that the offline-first approach works for more complex structures, like lists.

  • Interoperability: Like last time with the vocabs, this is sort of a wildcard goal. So maybe I won't do anything, but the idea is to be compatible with other apps who would interact with the POD storing the recipes.

The cycle should be done by May 31st.

You may notice that this cycle still doesn't have many features in scope. My idea right now is to use this cycle to get all the non-UI stuff finished, and focus the next cycle on UI, branding, and adding more features. Although I cannot tell for sure because that's the whole point of the Shape Up methodology.

So yeah, the first version of the app won't be finished at least until July. But that's ok, I'm in no rush.

PS: Google recently started tracking people using Google Chrome, even if your site doesn't use Google services. You can prevent it by adding an HTTP header, as I did with my sites. But I cannot do it in my apps because they are hosted using Github Pages, so I'm considering giving render.com a try. I'll report on the experience in a future update.

Today is the end of the cycle I mentioned in my previous update, and here's the current status of the project:

Hillchart 2021/05/30

Yeah, I have barely advanced in a month and a half. Turns out that building the proof of concept for real was more work than I expected. But the real problem have been yaks.

This is not the first time that this happens, but I intended to avoid it by following the Shape Up methodology. At the time, when I got stuck building Media Kraken, I referred to these yaks as \"rabbit holes\". The interesting point is that this is mentioned explicitly in the Shape Up book, and I thought I was guarding against it by doing the proof of concept during the cooldown. Who could have guessed that it would take me almost 6 weeks (working 1 day a week!) to clean up something that I put together in a couple of hours :/.

But as I mentioned, the real problem here have been yaks. However, as I argued the last time, I'm in no rush and I enjoy the process so \"It'll be done when it's done\" :). To be honest, I don't think this will happen too often, because it isn't every day that I embark into a new paradigm like offline-first or CRDTs. And I still think the Shape Up methodology is useful, so I'll consider this cycle done and go into cooldown. Maybe the next cycle will come out of the leftovers from this one. But maybe not, and that's the point.

So, what did I actually do all this time? Inspired by this tweet by Anthony Fu, here's my own Yak Map:

Offline-first Yak Map

As you can see, there were more things going on than I anticipated. To be frank, the Cypress tests and jest/chai plugins were completely optional. But I'm sure I'll use those for a lot of my projects going forward. Something cool I've been doing here is that I'm using the community-server in my CI pipeline, so the integration tests are now running against a real Solid server. This is particularly useful here given that Umai uses two engines at the same time, one local (for offline-first) and one against a Solid POD (to sync on the background). You can find the tests here if you want to see some code.

I found this concept of a Yak Map really interesting, and it speaks to what I have been doing for a while. Here's the Yak Map with my latest projects:

Projects Yak Map

What I really want to do are apps, but I ended up developing a bunch of libraries and packages. Which is cool, it's also my own fault because I have an acute case of the NIH syndrome.

PS: I have also been translating Penny to Catalan and Spanish! Penny is a POD browser for Solid developers, so check it out and maybe you can contribute your own translations (there isn't a lot of text). It was interesting to see how to translate a Solid app, because there are some Solid terms that I had only come across in English. This has also been useful because I'll definitely localize my apps at some point. But for now, that'll remain a hairy Yak.

It's been a while since my last update, and the main culprit has been a blog post I've been writing that ended up way longer than expected. So long, in fact, that I decided to implement an animated table of contents in my website to make it more palatable (which I also added to task comments!). Still, the estimated read time is 40 minutes so yeah... Sorry. I didn't have time to write a shorter one. You can look forward to reading (or ignoring) it next week.

Now, other than writing and working on my website, I've also kept hammering at some Solid stuff. In particular, I have been thinking how to model lists in Solid. That's where I left off in the last cycle, in order to add a list of ingredients to a recipe.

This is how you'd go about doing an "unordered list":

<#ramen>
    a schema:Recipe ;
    schema:name "Ramen" ;
    schema:recipeIngredient "Broth", "Noodles" .

That's pretty standard, basically you'd have multiple values for a property. That's what Soukai's FieldType.Array does. However, the order is not guaranteed to be kept the same, specially when you add or remove values. I've been investigating, and turns out there is already a common pattern to do this, using an rdf:List. This is how you'd write it in turtle:

<#ramen>
    a schema:Recipe ;
    schema:name "Ramen" ;
    schema:recipeIngredient ( "Broth" "Noodles" ) .

Which actually means:

<#ramen>
    a schema:Recipe ;
    schema:name "Ramen" ;
    schema:recipeIngredient _:b0 .

_:b0
    a rdf:List ;
    rdf:first "Broth" ;
    rdf:rest _:b1 .

_:b1
    a rdf:List ;
    rdf:first "Noodles" ;
    rdf:rest rdf:nil .

But I have some existential doubts about doing this. If you look at the recipeIngredient specification, you'll notice that it only accepts Text values. Or, said differently, the range of the schema:recipeIngredient property is Text. In that case, isn't an rdf:List an invalid value for this property? There are two ways to look at it. The first one is about correctness, and I would say that's incorrect because of what I just mentioned. I think we should strive to create data that is modeled properly. The second one is interoperability, which I think is super important. If there are other applications out there working with recipes, developers should be able to make it interoperable with mine just by reading what's on the vocabulary definition. They shouldn't even aim for interoperability, it should just happen if good practices and proper modeling are followed.

But, I do have my doubts about this because it could potentially be a common pattern. Having sorted data is certainly very common, and I think forcing each developer who wants sorted data to define their own vocabulary would be detrimental for interoperability. It would be nice if every property definition had rdf:List implied in their range (although I'm not sure how you'd go about specifying that the items on that list should be of a given type, because the range of the rdf:first property is rdfs:Resource).

So yeah, I'm not sure about this. In any case, seeing all the complexity I found, I will probably conform with the schema.org vocab completely without doing any shenanigans. I've also played with the idea of having my own vocab to extend schema:Recipe, but for the time being I think I'll avoid it. I've already been working on this app long enough, so it'd be nice if it starts converging :). Don't get excited though, there are still some months to go.

Now, with that said, here's what I'll be working on for the next cycle. It is going to be a weird one, because I have some holidays in the middle, and I'm always more busy (in my personal life) in summer. These are the things I'll look into:

  • Ingredients & instructions: I already started working on ingredients last cycle, but I wasn't finished. This time, I'll do instructions as well (which can be sorted using schema.org's vocab!).

  • Offline first: I'd say this is done for the most part, but I have to review a couple of things to see that logging in and out of the app works properly and doesn't mess up the local data.

  • Dependencies bundling: I'm still struggling with adapting some libraries to Vite, although there have been some recent improvements that look very promising (I still haven't tried them though). And that is not limited to external dependencies, my own soukai-solid library has some issues with Stream polyfills. So I'm going to explore some alternatives (like bundling compiled dependencies, crazy as that sounds).

Given the aforementioned summer schedule, this will be a shorter cycle and it should finish by August 16th.