Have any of you done this? What has your experience been like? Have you found a way to make this work with multi-player? What storage provider has the best API for this?
OpenDAL was on HN recently and would be a pretty decent abstraction to use for this: https://github.com/apache/incubator-opendal
I made exactly this: https://github.com/mickael-kerjean/filestash The public API enables programmatical access to S3, SFTP, FTP, GIT, WebDav, Samba, Local FS, NFS, Backblaze, Storj, Artifactory, .... and more. Under the hood, each storage implements this interface:
type IBackend interface {
Ls(path string) ([]os.FileInfo, error)
Cat(path string) (io.ReadCloser, error)
Mkdir(path string) error
Rm(path string) error
Mv(from string, to string) error
Save(path string, file io.Reader) error
Touch(path string) error
}
fun fact: it was born out of a reflection from the infamous top comment of the Dropbox launch on HN
The remoteStorage protocol is one: https://remotestorage.io/. Sadly, it seems remoteStorage never took off. The first version of the spec was published in 2011, and the reference implementation reached version 1.0 in 2017, but if you look at the list of known apps that support it (https://remotestorage.io/apps/), there are few active projects.
Of course, having users manage a storage bucket and its key is a bit tedious. Browsers could generate a per-app bucket and provide the bucket key to the app, all with user permission (like when a site asks for location permissions).
These buckets can be shared between apps too. As long as apps can read the same data formats, data portability is stellar.
It would be great to have an ecosystem of apps that all use compatible buckets/formats.
But this is unlikely to happen bc this sort of transparency and data portability isn't currently a big demand by users / governments. Privacy activists have gotten a great number of wins over the past decade. We need to make progress in many more computing issues.
I also really like the https://sandstorm.io approach which goes a little farther beyond
Yes, using a cloud storage provider might work instead of running your own database...
> Have you found a way to make this work with multi-player?
... But this is where you will run into serious problems.
"Cloud drives" aren't designed for highly concurrent multi-user access. That is when you need an ACID-compliant database; (or otherwise understand this issue well enough that you know what you're doing.)
In our case, our API was based around light usage of files: If two people edited the same file at the same time, we couldn't resolve the conflict automatically, because these were files: We had no knowledge of the files' schemas.
Likewise, our APIs weren't designed for "live" high-performance usage. In our case, if it took under two minutes for an update to sync, we were happy. That's an eternity in a multi-player game.
In general, if your application is going to use a cloud drive as its data storage, you should target light usage of storage needs: IE, the user's data is generally private, the user is generally generating light changes, multi-user usage doesn't require heavy "live" data.
IE, this would work great for a password manager that allows sharing passwords via the cloud drive's existing sharing system. It would also work great for an online image editor, sheet music editor, anything where you could envision a desktop app that edits files.
I love diagrams.net (draw.io) though so I just appreciate that it exists
- Having the entire product be reliant Google is just too big a risk. Google has changed apis, and restricted access to third parties.
- Lots of added complexity around search, pagination, etc.
- Due to the last point, all the files need to be tracked in an external database. This adds a requirement to keep everything in sync.
- Now that the files are not stored in Google Drive, I’m working on adding an option to support auto backups to Google Drive.
A while back I looked into the feasibility of accessing various APIs from the browser directly, and I ran into a dead end. Everything I found has CORS restrictions that prevent this from working. If anybody knows of anything, I'd love to take a look at it.
I don't know what your project is, but on any given day I'm switching between three laptops (one OSX, one Linux, one Windows), an iPad, and my iPhone. Localstorage is really great for prototyping but doesn't sync between machines. Export is ok, but sync is awesome. Plus having access to the raw data, for me, can be really beneficial for integrating with local tools. For example, I believe I once had a Python script that could parse out specific attributes out of a Drawio document; I wish I could remember more details from that.
I've been all over browser filesystem access too (you could e.g. use syncthing to sync from the filesystem) and it's just too clunky for users to use still.
Also tried building a chrome extension in an attempt to use the sync storage that's available to extensions (used to sync your settings across devices), but it's crippled by being maxed out at only 100kb.
- http://scripting.com/2015/06/25/dropboxCouldBeKingOfTheOnepa... ('Dropbox could be king of the one-page app', 2015)
- http://scripting.com/2023/08/05/130423.html ('Identity as a product', 2023)
Technology-wise it is just a Java app running on my raspberry and using Google's libraries for Drive [2] to create the spreadsheets and the actual Sheet API [3] to modify content. Authentication was a bit hard to figure out mostly because of the poor documentation. Each query to these APIs takes about one second, so it is definitely not suitable if you require low latency on the client's side or lots of concurrent accesses to the same spreadsheet. Not sure about pricing since my bot is not so popular and I am still in the free tier (:
[1] https://t.me/gsheet_notes_bot
[2] https://developers.google.com/drive/api/guides/about-sdk
[3] https://developers.google.com/sheets/api/guides/concepts
I use GPTs with it to save ideas / thoughts / notes, but also to parse URLs with lists like "best burgers" and GPTs will just add details to the sheets.
Sheets is GREAT for personal / small group use, but if you're a massive company you'll probably run into limitations quickly (5 API calls/s; 200k cells; monthly limits) etc. but I've never even gotten close.
The huge advantage is that most people already have some kind of syncing solution set up, and they can just use that. This is especially important for corporate environments where IT has to evaluate what's allowed and what isn't.
It comes with some drawbacks. Your code needs to be resilient enough to deal with inconsistent syncs, you have to use file system observation, stuff might be delayed etc.
But on the plus side, it seems to work very reliably, I've not gotten any complaints yet, and if something does go wrong savvy users can just fix it themselves. It's just a bunch of JSON files on disk, after all.
Most sync tools offer a way to share folders with other users, so if you do it right you get multiplayer for free.
As a user, I would love this kind of integration because it means data portability is built into the product.
My app is a Mac app, so it just reads/writes JSON files to disk in a folder that the user selects. The user selects a folder in iCloud drive, Dropbox, etc.
There is nothing special about my code, except that it is written with the expectation that some service modifies the files on disk in the background. Every JSON file has a uuid, so I can track renames of the files. But for the most part it's just a lot of handwritten code that converts JSON into objects and vice versa.
If you have a suitably robust "middle ware" layer this should be doable with any type 1 storage back end.
If you want to save your game progress, you just login with OneDrive.
I previously made a single-page EPUB reader using a similar technique with GDrive: https://h5reader.azurewebsites.net/ (that one does not work with 3rd party cookies disabled though because it uses iframe to login instead of redirects, and I never bothered to update it).
My experience with Drive OAuth is sketchy Colab notebooks asking me to authenticate with Drive, but in such a way that I'm apparently granting access to everything inside of it. I've never been sure if that's because of limitations on the OAuth API, the Colab client specifically, or just sloppy developers.
Is it user-privacy focused? Product cost-savings focused? Something else?
The privacy discussion would be interesting. On one hand, users "own" their data in the sense that they can delete it themselves at any time, but you also allow a product (hopefully limited) access to your own storage provider.
Dropbox has fallen so far in my eyes over the past ~decade. It used to be rock solid file storage, a USB drive in the cloud, now it's so busy trying to be everything to everyone and failing at most things. The macOS is hot garbage, literally one of the worst pieces of software I've had to work with. It's buggy, doesn't give useful info, gets stuck syncing, and has alerts that you can't act on. My mom's laptop yells at her that some files might not be openable and she should upgrade to their new file provider API. Ok fine, let's do that, oh wait you can't because you have a personal and a business account, try again later. It's been this way for a while now (like a year+).
I have Maestral on my laptop but honestly I just need a tiny push to leave Dropbox completely after being a paying customer for over a decade.
When you see a comment that's [dead] but shouldn't be, you can vouch for it. This is in the FAQ: https://news.ycombinator.com/newsfaq.html#cvouch.