Our notebooks are kind of a mix between Notion and Jupyter with extra features, like the ability to schedule notebooks, turn them into dashboards and apps, and write SQL queries whose results turn into data frames automatically.
We're building better notebooks because we think they're a great idea poorly executed - for three reasons.
The first problem with notebooks is that they're difficult to share. Non-technical people don't want to download docker containers and install Python libraries to see what the data team is doing. Then, the data team takes screenshots of their work and pastes them somewhere else. The issue with this approach is that the data gets stale, and the output is not interactive, so it's difficult to get feedback and iterate.
The second problem with notebooks is that they get too messy too quickly. One morning you have 10 blocks, and by the end of the day you have 192, but only six of them are meant to be seen by other people and the rest is just you jiu-jitsuing with the data. Consequently, even if non-technical people could see your work, they'd have a hard time figuring out what's happening.
The third problem with notebooks is that it takes too much work to do simple things like when you want to query a database. In that case, you need to have the database credentials in your machine, and you have to write a bunch of wrapper code. Calling APIs, plotting simple graphs, and adding interactive components is equally as annoying.
We're solving the sharing problem by bringing notebooks to the cloud (so you can schedule them) and using CRDTs to manage their state. Whenever you edit a Briefer notebook, we reconcile that using Yjs, and then propagate it to everyone else who's editing that notebook. Regarding compute instances, each workspace gets its own, and we provision them on demand.
By the way, we manage the execution state of your blocks using Yjs too, which makes it much more stable and responsive across all clients. When you click "run", for example, we don't immediately send a request to run the block. Instead, we change the block's state to "execution-requested". Then, the observers in the back-end react to the change and update the block's state with the results. In other words, the front end and the server communicate with each other through the notebook's state. One side updates the state, and the other reacts.
To solve the "messiness" issue, we've invested a lot of time in designing clean notebooks and allowing you to organize blocks in ways that make them more presentable. In Briefer, you can group multiple blocks into tabs and decide which blocks appear in the published version of your notebook. That way, you can hide all that data wrangling and focus on results. We also know that a vertical format is not always the best way to display results, so you can use your notebook's outputs to build dashboards too.
Last but not least, we're reducing friction in simple tasks like plotting graphs and querying databases. In Briefer, you can plot graphs using a click-through interface, and if you need to plot something more intricate like a Sankey chart there's also a built-in AI assistant that you can prompt to generate code. When it comes to queries, we turn their results into dataframes automatically, and you can use SQL to query uploaded files too (we use DuckDB for that).
We're really happy we get to show this to you all, thank you for reading about it! Please let us know your thoughts and questions in the comments.
You're going in the right direction! for the near future, I suggest you to add a cli, connectors, simple symbolic calculation on notebooks (take some inspiration from calca http://calca.io/examples), offline support and fully encrypted namespaces.
That will put you miles away from everyone else in the field :)
You're probably busy with the launch, but you can contact me if you wanna discuss the list above,
Abraços
Would love to hear more about what you mean regarding the CLI and connectors. Can you give me a few examples of what these would look like?
By the way, I didn't know Calca. Looks neat, I'll definitely have a look.
1. product managers will try to make things dead easy, by setting up simple excel formulas & charts. Taking screenshots etc
2. data engineers will do more technical but opensourced solutions like superset & dbt
3. corporates usually tends to build the end UI themselves. I remember sth that (maybe?) Pinterest opensourced.
I deeply understand how hard it is to build a notebook like this. And there are indeed some problems lying in the collaboration and UX.
It's just so hard to propose those features, given the problem is usually the data itself and a highly competitive market. Popsql does not thrive as I remembered.
Still, I think that we can get to a place where everyone uses the same tool to collaborate on data matters, like a "Retool for data/BI". At a high-level, that's the direction we're going, and we're starting with notebooks and dashboards.
But who knows? Back in 90s no one knows they need an iPhone. Wish all the luck for your journey onwards.
Honestly, I’d love to start trialing this as our dashboard creation to the platform today.
On the downside, after setting up an account and creating a couple notebooks, my experience has been unusable slow. While this is likely growing pains as there’s no cost as a user for compute resources, it’s also a nonstarter for us to actually use as an organization. What’s the roadmap to make this more usable?
Lastly, adopting a new platform like this without a data export or local deployment option gives me pause. What can I expect for the longevity of what we create - pieces that become central to business decisions? This is especially true considering how much upfront training will inevitably go into getting started.
On a surface level, we're doing a lot of work on UX. We intend the UI to be more presentable and easier to navigate through so that technical users find it easier to use while non-technical users find it more approachable and can more easily find what they're looking for.
In terms of capabilities:
1. We want to merge all data-related features in one place, including BI. That's why you can create and publish dashboards too. It's also why we've been putting a lot of effort into making the no-code visualizations flexible and powerful.
2. You'll eventually be able to connect your own computer to Briefer. That way, you can use your AWS/Azure/GCP credits and your own GPUs if you have already paid for them.
3. There will be a way for you to manage notebooks and dashboards as code if you want to. We love the idea of versioning your notebooks, internal data apps, and dashboards, but we think current tools can't do it well.
There's also a huge change coming up in about 2 months. It'll make it really clear how we're different from Hex, but unfortunately I can't talk about that yet!
After a while they realized there were a bunch of other cool features and started engaging with those too.
The AI also helps folks do queries without having to explore the schema for a while.
But Jupyter notebooks can run anywhere including local. Do you support that (via a local install or by relaying a connection to local python install)?
Or would all Python code have to run on your cloud servers (and thus not have access to things like high-end GPUs and such)?
We do not support running notebooks locally right now, but offline access is coming soon.
With regards to running code on high-end GPUs: currently, we manually allocate GPUs to our customers when they need that. Still, we do want to eventually allow people to connect to their cloud providers to spin-up compute instances there.
I teach a lot of Python and data science (pandas, Polars, scikit learn, XGBoost...) in Jupyter.
(I also teach a bunch of software engineering best practices to folks who claim they don't want to be "software engineers".)
My experience is that with training, many of these issues go away. Just saw this again firsthand at a client last week.
My current take is that many focus on making the code newbie friendly. I think they should level up and start writing code for a professional audience, using software techniques that a professional would use. Newbies won't like this code.
I get a lot of flak on social when I post this from naysayers, but the overwhelming response from my students and those who have read my books indicates there might be something to this.
I do believe we should bring software best practices to the data world, not only regarding code design but regarding infrastructure and tooling too (like versioning everything).
Still, I get where they're coming from. A software engineer would also be frustrated if they had to learn everything a data scientist knows (probably even more).
I think the tooling itself can solve this issue by encouraging best practices though.
People need to know how to write functions to write professional software, not just put a soup together of notebook cells.
Disagree with Matt even that they won't like this code -- it is just learning basic stuff like file structure and python environments, see how I approach it in the first two chapters running stuff from REPL here https://crimede-coder.com/blogposts/2023/EarlyReleasePython.
https://www.amazon.com/hapi-js-Action-Matt-Harrison-ebook/dp...
And it's some random kindle book about some JavaScript library I never heard of.
But I'm pretty sure you are the Matt Harrison I follow on twitter:
Does the dashboard feature support any dynamic field selectors (similar to "variables" in Grafana)?
That’s possible. For it to work you’d have to create a “dropdown block” and select the “dynamic” option in its settings. Then, you will be able to select a dataframe and the column from that dataframe you want to use as dropdown options.
By the way, every query automatically becomes a dataframe so you can use either a query result’s column or a raw Python dataframe.
As a side note, I’m putting together detailed product documentation this week too and I’ll make sure to include it there.
The products are similar in the sense that both are cloud-based notebooks, but we have different approaches to building them.
I answered a question about Hex below, and the general outline applies here. To summarize:
1. We want to go beyond just notebooks. We want to centralize all data tasks in Briefer, including traditional BI so we’re building specific features for that, like dashboarding and everything that comes with it.
2. We’ll enable people to use their own compute for the notebooks.
3. We want to allow people to manage notebooks, dashboards, and apps “as code” so they can version it whenever necessary.
We unfortunately didn’t finish items 2 and 3 yet because we’ve only started working on Briefer six months ago, but we’re fast and we’ll get there soon.
Also, we’re working on something new that will make the distinction clearer, but I can’t talk about that yet.
One question : how do you manage database credentials ? That's a question I've had a lot working on my project so I'm pretty sure customers will have the same for you.
Especially given the latest Snowflake security issues.
We do a lot of things to keep the credentials safe that go beyond just putting everything behind a VPN and setting pods' SECCOMP profiles. That obviously includes encrypting credentials and putting in place tight access controls both to the credentials themselves and the encryption keys.
Also, even though we're not SOC2 compliant (yet) we do run pen tests.
Would you consider Briefer to be a competitor to https://count.co/ ?
(Certainly, if you do, your pricing is a helluva lot more appealing...)
It's not as much of a direct competitor because Count's approach seems more like "Miro for Data" while ours is more like "Notion for Data".
Still, I do think people could end up comparing both our tool and theirs when evaluating a few types use cases.
For some replace LoB use cases just storing the data directly and letting users query it would be an advantage.
From my brief look at the site, the product seems to allow users to query and display data only. Can we edit the data at all?
It looks like this product targets teams and not well-suited to individuals (ie private note taking app with database like Notion). I did not see any way to create new data or edit existing data, or if you even have a mobile app.
The first 2 problems you mentioned did not apply to me at all, while the 3rd problem has been my wishlist for Notion for a long time
We've already seen hacky solutions starting to be replaced and that makes me quite happy.
I share github links to my notebooks with my PMs
Question: I am in charge of financials for our condo building. The other people in charge are older than me and only know excel. How do you suggest I think about wanting to use better tools to present analysis while also making sure that everyone else can understand the work and modify?
In Briefer you can hide the blocks you don't want others to see, like the ones in which you're just manipulating data. Instead, you can show only the results so that your analysis is really easy to understand.
You can also display your analyses as dashboards or build small data apps with inputs and dropdowns for them to use.
Still, I wouldn't recommend completely non-technical users to create content within Briefer - only to read it and interact with the apps/dashboards there.
Just minor bug; on /auth/signup the zebra background pattern is not on the entire page on my vertical display (1080x1920). You might want to fix this before release.
1. Do you intend Briefer to be a jupyter replacement or a metabase replacement?
2. Do you have any plans to open source briefer?
Regarding your questions:
1. Briefer is definitely a Jupyter replacement and it can be a Metabase replacement. Most of our users with BI use cases start using Briefer for more advanced types of dashboards and analyses, and they eventually end up slowly replacing their traditional BI with Briefer.
2. That's indeed something that's crossed our minds before. We're not open-sourcing it right now though.
I'll stick to Livebook for the most part but know some companies that might be interested.
Would you consider an option for self-hosting?
Regarding self-hosting: Yes, we have discussed that with other potential customers.
If you're interested in a self-hosted offering, please reach out to me at lucas.costa [at] briefer [dot] cloud and I'll help you out there and walk you through what that would look like.
How do you do your AI? Is it just a local model?
We want the free tier to be so good it becomes the new notebook standard - and then we'll only charge for features that companies need, like Slack integrations or a large number of seats.
Regarding AI, we currently go with GPT-4o by default, but you can change it on the settings panel. Currently, we have only enabled GPT-4o and Mistral though, but we want to add more - it's just not our priority right now because the OpenAI model is just incredibly good.
You can see the response here: https://news.ycombinator.com/item?id=41055495
Thanks for reporting it, it helps us understand what people want!
So far, there seems to be an even split between the people that always go for the AI and the people that write everything themselves.
Personally, I use the AI whenever I need to write anything other than `SELECT * FROM xyz WHERE`. It just saves me so much time.
I haven't yet had time to implement it because I was working on the core pieces of functionality, but as a dark mode user I want to get to that soon.