My Face

Implementing a Media Tracker using Solid

(Ongoing)

I've recently started using my Solid Task Manager in production. While I don't consider it finished, it's at a point where I can use it without adding more features. One of the things I want to do in my Path is to build the tools to have control in my digital life. I got tired of being at the mercy of others, so I don't want to use anything that doesn't embrace Autonomous Data. And the next step is to implement a Media Tracker.

At the moment, I am using TViso for tracking TV Shows and Movies, MyAnimeList for Manga and nothing for Books. I am not happy with either of those services, but given the vendor lock-in I can't easily switch to alternatives. This makes a great opportunity to continue expanding my toolbox.

My appetite for this task is of 40 hours with 5 weeks in real-time. Keeping this in mind, I'll scope this first version only to tracking Movies.

Activity

Task started

I've started working on this and you can find the code here. I tried to find a good name for this app but nothing came to mind, for now I am calling it media-tracker. I'm not happy with it, so it'll probably change before I'm done with the task.

Other than naming, something I've come across that I'd like to improve is the scaffolding. I'm starting to duplicate a lot of code across projects, so I'll probably release some packages with utilities. One will be plain javascript utilities, for example I have a wrapper for localStorage, Array utilities, Object utilities, etc. Many of these are inspired by Laravel helpers. And another package will contain application scaffolding for Autonomous Data apps using Vue and Soukai.

But of course, not everything was useless. Every time that I create a project from scratch, it's a good opportunity to see if I can improve something in my setup. This time, I learned more about how I am using service workers in my PWAs. Vue cli comes with Workbox installed out of the box, but I've been having some issues pushing updates to clients. I suspect that this has something to do with the default strategy. It seems that apps use precaching with the default configuration, which acts similar to the CacheFirst strategy. The problem with this would be that new code won't reach the clients until the cache is invalidated. So in my new configuration I am using StaleWhileRevalidate for index.html and all other assets can keep using CacheFirst because they are versioned. The truth is that I still have to learn more about this, but for now I'd say it's good enough (and it's not the main goal of this task).

Finally, other than all the house keeping, I've also started working on application specifics. When it comes to Solid, something that continues being a problem for me is finding fitting RDF definitions. But this time, I was lucky because schema.org already supports Movies. It also supports watch actions, so it doesn't seem like modeling will be a problem this time. But there is something new in this app that I'll need to investigate. In solid-focus, I am using one document per resource. But for this app, I think it's a better approach to keep actions in the same document as the movie they are referencing. Actually my initial idea was to have a simple date attribute to indicate when a movie had been watched, but after looking into schema.org I decided to use their approach. This also makes me ponder if actions should be resources at all, which doesn't make a lot of sense to me. So I'll have to investigate more on this.

It's taken me a long time to post this update because I've been busy attending FOSDEM, preparing a talk on Solid and as I'll explain later, solving some technical challenges. On the flip side I've implemented an RSS feed for these updates, maybe that's where you are reading this :).

As I mentioned on my last update, I've been pondering how to model watch actions in this application. Given that I needed to solve a new use-case, I decided to revisit the Solid docs and I'm glad to say they've improved. I also noticed I am neglecting setting up the Type Registry in my applications, that's something I'll have to fix at some point. The conclusion I reached after learning how others are handling similar use-cases is that I should create a new relationship in soukai-solid. As I suspected, watch actions should not be LDP Resources, although they'll be RDF Resources (because all rdf entities are). I hope this doesn't get confusing, basically LDP Resources are documents (they have their own url), and any other entities that don't map to a document will be plain RDF Resources. In modeling this I've called this non-document entities Embedded Resources, and the new relationship can declared using embeds and isEmbeddedBy. Since they are within an existing document they can't have their own url. I've solved this by using the url fragment (which I think is a standard practice in Solid). For example, if I have a movie with uuid 12345 at https://my-pod.com/movies/12345 the identifier for a watch action with uuid abc on that movie would be https://my-pod.com/movies/12345#abc.

Other than the reasons I've mentioned at the beginning, this has also taken me a couple of weeks in real-time because library development and app development are completely different. When I'm developing library code (soukai and soukai-solid), I have to be careful not to break existing functionality. I also need to make sure that every new feature is congruent with the library. But I think it's a worthy effort because that will allow me to make applications faster (and I plan to make many :D). This may cause this task to go beyond my initial appetite, but I consider that this library work is outside the scope of the task anyways.

I have finally come up with a name that I like for the project: Media Kraken. It's likely to change at some point, because I've seen that it's already used by others. But it'll do as a codename for a while, at least until I start caring about marketing for this project (which may never happen). And yes, brace for the "release the kraken" jokes because they'll be numerous.

Something else I've done is start using the Type Registry like I mentioned in the last update. This has been particularly easy because I've been able to use the new non-document entities feature that I implemented last time. I have created a TypeRegistration model and that's it, so the extra work I did last week has already paid off.

And finally, I have also started integrating the application with 3rd parties. Given that it's an application that will allow browsing movies, the data catalog has to come from somewhere. I have to say that I am both surprised and not surprised at how hard it's been to solve this. What I was thinking at first is that it'd be easy to use some API from imdb (just looking at the name gives the impression that it must be queriable). But it's surprisingly closed, the only thing they provide is downloading some files that are updated every day. It is not very convenient, specially given that I am making an application that lives in the frontend with no backend. So yeah, I've been looking for alternatives and there isn't any real open database for this. I shouldn't be surprised because we are in the age of data and the ones who have it don't want to give it away. This is not the first time that I've faced a similar situation, so it's surprising because that's not how I think the world should work but it's not surprising because it's consistent in how things have been so far.

What many do at this point is start scraping. Again, this is not feasible in this application for a variety of reasons. Let's put aside the ethical, moral and legal implications. Since the application lives in the browser, and CORS is a thing, it's not possible to do scraping without recurring to some sort of proxy. I could have gone down that path, but I'm not convinced that it is a good solution in the long run. So what I ended up doing is using the closest thing I could find to an open database, and that is tmdb.org. I don't consider this 100% open because it requires an API key. This wouldn't be so bad if I had a backend, but I don't. And I cannot ship the API key in the frontend because it would be exposed. So I ended up creating an AWS Lambda that proxies calls to the api, only in order to keep the API key away from the frontend. This is obviously not ideal, specially since it isn't possible to limit AWS usage. But I haven't been able to come up with a better solution, if anyone has it I'm all ears.

What I plan to do at some point is allow users to configure their own proxies. But we all know nobody will do it, unless they have to. And that will happen when the AWS Lambda is close to start incurring costs and I shut it down. I don't think that'll happen anytime soon, so for now I won't worry about this.

Now that I've integrated with a data provider, the app is starting to be functional. But not visually, it sucks, so after a couple more features I'll probably be done with the first version and finish this task by implementing a decent UI.

Today's update is not about Solid, because this week I've been working exclusively on the UI. And the funny thing is, I've only finished the header and the logo! But it's been a fun week.

I'm always looking forward to reaching Flow when I'm working. This is a psychological state that is reached when you're performing a task that you enjoy and it isn't too easy nor too difficult. And that's exactly what designing the header and the logo has been like. I reckon I've spent too much time on this, and I've blown up the appetite budget at this point. This week I've also spent more time working on side-projects than usual (~20 hours, and I usually spend ~10). But in a way that's the point of Flow, that you lose track of time.

After thinking about this I've reached the same conclusion I did a while ago: this is a side-project and doing the work is not my only goal, I also want to learn and explore. But it's important to balance both, that's why I find these reflections useful.

So, what have I actually learned and explored this week? First, let me show you the results:

Mobile layout:

Mobile layout

Desktop layout:

Desktop layout

What you see in these two screens is the same html, styled using CSS responsive utilities. And yes, that includes the animations too! I've done similar things in the past but not with so many interactions. This time I've decided to use Tailwind CSS without any component framework, and I've also been exploring Tailwind UI that was released this week.

My opinion on Tailwind UI is not great so far, and it pains me to say this because I love Tailwind. It's been really useful for inspiration and to learn some things, but it hasn't been copy & paste as "advertised" (although they admit that you may need to adapt it to your project). One of the worst things has been AlpineJS. It isn't that I don't like it, in fact I didn't know it and it seems nice. But having to adapt it to Vue hasn't been straightforward. I also started with a fairly similar approach to their sample code but I ended up redoing almost everything. I suppose this just means that Tailwind UI is not for me.

If you haven't tried Tailwind please don't be taken back by what I said, Tailwind is awesome and if you haven't used it I encourage you to do so. In the process of exploring Tailwind UI I've also upgraded to version 1.2.0, and I've started using the new transition utilities. Which got me into the rabbit hole that ended with all these animations.

Some weeks ago I said that I wasn't completely happy with the approach I had taken to interact with the TMDB API. I recently found a forum discussion where Travis, TMDB's founder, gives green light to exposing the API key in the frontend. I don't think that's a good approach, but if he says it's ok I guess it is. This is probably one of those situations where theory is one thing and practice is another. This is theoretically a security issue, but in practice nobody is exploiting it.

Other than this, the past 3 weeks I've continued to work mostly on UI. I took a detour to implement a TailwindCSS Colors Generator, but other than that I've implemented search and movies management.

Before getting into the details, here's how it looks at the moment:

This may look deceptively simple given the amount of time it's taken me, roughly 30 hours. But there are some nuances to keep in mind.

It cannot be understated how different it is using a UI framework like Vuetify (as I did with Solid Focus) or using plain CSS. I am using TailwindCSS which isn't exactly plain CSS, but it is essentially the same. The fact is that doing it from scratch takes a lot more time. Not only because it's more difficult, you are also missing building blocks that you'd take for granted such as modals and snackbars. Any simple feature that you are developing can become cumbersome when you realize you need a modal or a snackbar.

On the other hand, it's also more rewarding and more fun. I'm also building reusable components for upcoming projects, so in a sense I may be creating my own UI framework. But the important aspect is the flexibility I have with this approach. Sure, I could have done any of the things I'll explain with other frameworks. But creating these interactions is not only a matter of implementation. They are the result of an exploration process, and using this approach allows me to explore without the constraints (and assumptions) that frameworks inherently have. What I'm doing here is not only implementing a spec, I'm constantly refactoring code and UI.

The first thing I want to highlight is the animation that takes place when a movie is marked as "watched" and is, literally, sent to your collection. You may not have noticed that, so I encourage you to look again. When a movie disappears from the grid, it shrinks and is sent towards the "My Collection" link (which is where you have to click if you want to find the movie again). I know it's a very small detail, and if most people didn't notice it's arguable how useful it is. But that's the kind of thing I appreciate, the little details. And it's also super fun to work on this kind of stuff. If you're wondering how I achieved this, it was using a combination of Vue list transitions and a custom JS script.

Something else that was interesting to work on is the button that marks movies as watched. This cannot be appreciated in the video, but that element is actually a button when the movie is pending and it becomes a div once the movie is watched (so, after clicking it). With the magic of Vue and Tailwind combined, this is seamless and cannot be perceived visually. Which is the point. This was achieved using Vue's dynamic component and some advanced attribute bindings:

Vue:

<component
    :is="movie.watched ? 'div' : 'button'"
    class="badge absolute top-0 right-0 -mt-1 w-10 h-10 flex items-center justify-center"
    style="margin-right:-.7rem"
    v-bind="movie.watched ? { class: 'watched' } : { type: 'button' }"
    @click="movie.pending && markWatched()"
>
    <BaseIcon name="bookmark" class="background absolute inset-0 w-10 h-10" />
    <BaseIcon v-if="movie.pending" name="time" class="icon-pending text-blue-600 w-4 h-4 z-10" />
    <BaseIcon name="checkmark" class="icon-watched text-green-600 w-4 h-4 z-10" />
</component>

TailwindCSS:

.badge {
    .background { @apply text-blue-300; }
    .icon-watched { @apply hidden; }

    &:hover, &.watched {
        .background { @apply text-green-300; }
        .icon-watched { @apply block; }
        .icon-pending { @apply hidden; }
    }
}

At this point, I feel like the UI is almost finished. I have gone way past the apetite budget, and I realize this happened because of my nitpicking with the graphic part. But I'm actually confortable working like this, as I explored in a blog post called Order vs Chaos.

Today I want to write a short update on how things are going. I thought by now I'd be almost finished, but turns out I just found something important to improve and that'll probably delay the release even more. I still expect it to happen shortly though, in about 2-3 weeks.

The past two weeks I've been finishing the UI and the only thing that's missing now is the initial loading screen. I've been doing some tinkering with data fetching, and I believe I'll be able to make it really fast (compared with solid-focus which is kind of slow at the moment). I'll have 1500 movies in my account, and that's what I've been using for testing locally.

Something else I've done is deploying the app using github pages (don't use it yet because there'll be breaking changes for sure!). I had some issues with routing that should be solved now. The problem was that some of the application routes, for example /collection, lead to a 404 github page. The reason for that is that github expects to have an html file in every route and the application is a Vue SPA using vue-router. I'm surprised that I didn't find many resources on how to solve this, but I ended up doing a simple script to handle that.

I've also set up a CI testing environment using github actions. If I hadn't done this before that's because I was in exploration mode, and the app is now starting to become stable enough for a first release. I normally use a TDDish approach to development, but I do 0 tests when I'm exploring or tinkering with new concepts. The same applies to documentation.

Something else interesting I've been doing is a markdown component that allows me to simplify the generation of text-based app content. This may be a bit overkill, but I've enjoyed doing it and it allows me to do things like defining modals entirely with markdown and having some nice interactive import logs.

When I was almost done with this task, I went into a couple of new rabbit holes. None of them were essential for the release, I could have pushed through and released anyways. But at this point I'm embracing the "It'll be done when it's done" craftsman mindset. I am logging how much time I'm dedicating to each part and I'll post a summary of what I've spent my time doing when the task is done.

Still, I'm am not abandoning lessons learned from the shape up methodology. Something I applied recently is the circuit breaker. I did not get into these rabbit holes without betting first. And the second one was one hour shy of getting cancelled.

Rabbithole #1: Lazy Elements Loading

The first rabbithole I went into was "paginating" the movie collection. Yeah, that's in quotes because that's what I thought I'd be doing. When I started testing the application with a dataset of 1000+ movies, I realized how slow it was. This was to be expected because I was rendering all the movies in a single page, images and all.

My first instinct was to paginate the results, and I implemented a version with that. But I was not happy with the result. I also experimented with infinite scroll, but I didn't like it either because it took ages to reach the bottom. After some more tinkering I recalled a blog post on how Google Photos implemented their image browser. I am ashamed to admit that I use Google Photos, although it's in my list of things to replace with autonomous data alternatives. But you can't argue against the quality of the product. Inspired by that post and Google Photo's UX, I implemented a solution where the full scroll height is rendered but elements aren't displayed until they appear on screen. This is possible thanks to the Intersection Observer API and chunking the results.

Rabbithote #2: Web Workers & IndexedDB

The second rabbit hole came about looking at the responsiveness and speed of the initial loading. Once the app is loaded it works well, but the initial loading is excruciating. This is also a problem in Solid Focus, but it's accentuated in this application because the dataset is bigger.

There are multiple reasons for this. Two important ones are network requests and parsing semantic data. Looking into this, I reached the conclusion that it can be improved by caching more data in the browser. I was already doing something similar in offline mode, with a local storage engine. But exploring other improvements I found two browser APIs I hadn't been using: IndexedDB and Web Workers.

Those two play very well together, so I spend some time rewriting different parts of the stack to support web workers and I created a new IndexedDBEngine in Soukai. Although I'm not completely finished with this, and the reason is...

Rabbithole #3: JsonLD serialization

SolidModel currently serializes models to "friendly human-readable json". This is something that SolidEngine already knows, and it translates the attributes to a linked data format. The reason why I followed this approach is that other engines, such as LocalStorageEngine, don't know anything about Solid and they'll treat serialized models as normal objects.

My goal was that exported models would look understandable to humans, but in hindsight that was a mistake. JsonLD is a standard format and even though if it isn't the most human-readable thing, it's close enough. The cost of not serializing to JsonLD is that semantic data will be lost. This hinders the ability to export and import data, which has become apparent with Media Kraken because I'm implementing these capabilities from the start.

I embarqued on a crusade to rewrite SolidModel to serialize to JsonLD. And I say crusade because this involves refactoring multiple parts of the stack, and it'll definitely cause breaking changes. But I'm confident that it'll be better in the long run.

This one is so core that I don't really have a bet for it, it'll take as long as it needs to. I've already sorted out some of the core changes, and I've replaced the library I was using for interacting with semantic data with n3 and jsonld-streaming-parser. This should also reduce the bundle size and improve performance.