It's getting hard to keep up with all the recent posts on indie web aggregators (a good problem to have!). I've observed that I always get excited about seeing one such tool -> try to make it a habit to visit those aggregators occasionally -> forget about them -> rinse and repeat.
A great execution on the problem of discovery within the blogosphere is Wander, which was posted here a few days ago - https://qqrl.tk/item?id=47422759. I discover great, small websites not because I found them in any aggregator's list, but more often via someone whose opinions I trust recommended a particular post to me. Wander executes on that idea rather brilliantly.
I've found myself going to my Wander console[^1] more regularly now and I keep finding great folks to follow.
Unfortunately I still haven't had a chance to make a blog post about this, which I really must do. But I can give some quick hints. Anyone reading this, feel free to reach out & I can try to answer questions, and they might help my blog post too.
I started off with a meta-search calling out to Brave / Mojeek / Marginalia, and the basics of that are something that you can ask an AI to make for you as a one-file PHP script. I still think this is a good place to start, because you'll quickly find "okay, I can replace my everyday search engine with this". Once you're dogfooding your engine every day, you'll notice all the rough points you want to improve.
Once you've got an array of objects with Title, URL, Description, and splitting the URL into domain, TLD, subdomain, path, file extension... there's a lot of ranking you can apply just to those. Honestly, a lot of my "ranking" has just been applying increased rankings to domains that I visit most often. I have an array of about 600 domains that it applies ranking boosts to. You can try experimenting with your re-ranking there, before even starting to build your own index.
As for building your own (small, personal) index, the technical details are not as difficult as you'd think. An SQLite database file, that your PHP file reads, will take you a long way... especially if you enable FTS5 indexing. I only did that last week, and I should have done that at the beginning. Search times are 10ms, and not just on my personally curated index of 80,000 pages... I just added a 2nd database with 1.3 Million entries from DMOZ (the old Mozilla Directory), and it's still only about 10ms. My search engine now feels super fast when it gets results from my database. And when it finds zero results, it automatically falls back to the metasearch.
At 1.3 Million entries, the two databases are only about 550MB total. It's running on a shared hosting account and apparently they're not worried - but it's only available to me, so I'm only hitting it maybe 50 times a day maximum. I'll move it onto a VPS eventually, but every time I think "this must be using up too many resources", I find I'm thinking too small by at least a factor of 10x.
For getting started with PHP & SQLite, I found this blog post helpful - but at this point, your AI can vibe code the entire thing for you:
I've been using Stirling-PDF as my go-to solution for any pdf needs and have never needed any other service. Open source gold standard for any pdf needs: https://github.com/Stirling-Tools/Stirling-PDF
I loved this book, and honestly it's one of the best books I read in the past decade. There does exist an eBook, because I read it on Kindle and I remember I used to wonder how amazing it would be to read it in hard copy. Well, it's been almost 3 years now - time for a re-read in Hardcover!
I hate having to go to Google for everything as much as the next guy, but I just can't bring myself to use OSM over Google Maps. GMpas are just.. better in every case.
I’m Danish, OSM is updated a lot faster than gmaps. Obviously google is better at routing and traffic, but routing doesn’t work if a road is closed for construction and google doesn’t know.
Also a bit unfortunate that a software like this has to exist in the first place.
reply