Code-GUI bidirectional editing via LSP
187 points
14 hours ago
| 22 comments
| jamesbvaughan.com
| HN
mikewarot
10 hours ago
[-]
This is close to the functionality that Borland Delphi had back in the 1990s. The pascal language and the design of their GUI toolkit were a really good impedance match, so you could freely switch between GUI design and editing the text version of it.

Doing so for languages like C++, was a sea of boilerplate that you couldn't touch, which is why I never moved away from Pascal. Similar fragility was evident in WxPython and it's builder.

I'm glad to see that LLMs can provide a match for less well suited Language/GUI pairs. We all deserve to get that kind of productivity.

reply
lmz
2 hours ago
[-]
Delphi's form files weren't pascal. The gui-sync was in my memory just editing the class definition when adding components / event handlers.
reply
jasonjmcghee
10 hours ago
[-]
Hey nice - this stuff is so much fun to me. I've worked a number of experiments like this too, especially related to live coding. Love seeing it in the wild.

I built a small project where you can live-code Love2D. The running program updates in real time (no saving needed) and see all values update in real-time, via LSP.

https://github.com/jasonjmcghee/livelove

And also added the same kind of interactivity like a number slider and color picker that replace text inline, like yours (though via vs code extension: https://gist.github.com/jasonjmcghee/17a404bbf15918fda29cf69...)

Here's another experiment where I made it so you could "drag and drop" to choose a position for something by manipulating the editor / replacing a computed position with a static one, on keypress.

https://clj.social/@jason/113550406525463981

There's so much cool stuff you can do here.

reply
jamesbvaughan
10 hours ago
[-]
That's cool! I suspected that someone must have done something like this before.
reply
madmod
10 hours ago
[-]
I love your project. I am making something with LSP and your code was a great example of what is possible.
reply
jasonjmcghee
10 hours ago
[-]
Glad it was useful! LSP is definitely under-utilized. It's great for the "malleable software" / small tools for yourself kind of stuff especially.
reply
KRAKRISMOTT
9 hours ago
[-]
Isn't autocad lisp powered?
reply
necovek
11 hours ago
[-]
This is a great idea: I'd never think of using LSP for this!

As a software developer, I always get frustrated when I am doing some graphical work and struggle to neatly parametrize whatever I am drawing (wooden cabinets and furniture, room layouts, installation plans...) and switch between coding where that makes most sense and GUI where it doesn't.

The best I've gotten was FreeCAD with Python bindings (I've got a couple of small libraries to build out components for me), but while you can use your own editor, the experience is not very seamless.

And then I start imagining tools like the one here, but obviously doing it just right for me (balancing the level of coding or GUI work).

reply
ttoinou
7 hours ago
[-]
I think we’re a lot of developers who are frustrated by constantly needing to choose between code or GUI… both have their use case, but I feel like there must be a system that combines both.
reply
krebby
3 hours ago
[-]
How about a visual programming language? Plenty of 3D and CAD software uses a VPL for procedural design, which helps a ton to bring out the benefits of both
reply
adastra22
1 hour ago
[-]
Any examples?
reply
baobun
1 hour ago
[-]
Let's lobby for adding UI semantics to MCP. Generating ad-hoc GUIs seems like a major and common enough feature to warrant it.
reply
ogoffart
9 hours ago
[-]
For Slint [https://slint.dev], a (native) GUI Toolkit, I've also developed a LSP server that do live preview and editing. You can try it online at https://slintpad.com : if you click on the toolbar button to enable the right panel, you can edit the properties from the UI, and this is all done through the LSP and can be integrated in any editor that supports it.
reply
czei002
16 minutes ago
[-]
cool! how does this work? e.g. how do you know which UI element matches which text element, do you track it while rendering? How do you propagate changes in the UI? do you update the text and then re-render the whole UI?
reply
rao-v
11 hours ago
[-]
Isn’t the python based build123d the current best CAD in code solution? The problem with OpenSCAD is that it cannot export solid geometry, just a final mesh.

More broadly, I was genuinely shocked to realize, when I was playing with it, that there is no cross CAD file format that captures even simple design concepts like “this hole is aligned to the center of this plate” or even “this is a 2mm fillet”. STEP (the file format) mostly just captures final geometry.

I think CAD people just … redesign the part again if they need to move from say Fusion 360 to FreeCAD or whatever. How do they live like that?!

reply
mk_stjames
9 hours ago
[-]
It's really hard to explain if you don't know how CAD kernels produce final BREP shapes via their process trees, but, try give an example- something like "this is a 2mm fillet" requires a 'fillet solver' that is deeply ingrained in the kernel. It isn't something that would ever be portable, say, from Fusion360 to CATIA natively because those are completely different kernels with different ways of 'solving' the model (and not just fillets, but, everything).

That is why STEP containing the final BREP manifold solid is the standard interchange that it is - it is a final representation of the solved output that IS portable, and anything else is... difficult.

reply
rao-v
3 hours ago
[-]
This is helpful but surely we can save a consistent description of the process step and the input parameters alongside the final geometry in the same file format?
reply
ethan_smith
6 hours ago
[-]
The interoperability problem stems from CAD kernels using proprietary B-rep representations and constraint solvers, with STEP's AP242 standard attempting to address this by including Product Manufacturing Information (PMI) and semantic annotations, though adoption remains fragmented.
reply
rao-v
3 hours ago
[-]
I’d have assumed that somebody had a format for just the basic operations … but it looks like even that is just being considered
reply
rjsw
10 hours ago
[-]
STEP is capable of capturing what you describe, it is down to the "user group" of customers of the CAD vendors to ask for it to be implemented by each CAD system in terms of what they import and export.

We put in for some funding for the next edition of STEP AP242 for me to be able to work more closely with the user group to improve this area.

reply
rao-v
7 hours ago
[-]
That's awesome! Yeah it seems strange to me that we don't have a standard way of tracking how a model is built up and parameterized (basically Fusion 360's history mode). I'd naively assumed that this was a solved problem in the CAD world - given the whole point of parametric CAD is to be able to easily tweak the distance between two points and have the whole model update.
reply
fwip
10 hours ago
[-]
I don't think they change programs very often, and only very rarely in the middle of a project.
reply
rao-v
3 hours ago
[-]
I think this maybe why!

Of course it’s great for vendor lock-in.

But given how many companies need to work with diverse suppliers, there must be a whole bunch of re-creating models happening. There is no chance that everybody is using the same CAD tool

reply
williamcotton
10 hours ago
[-]
While it doesn’t use an LSP (which is a great idea!), this project of mine does bidirectional mapping between a GUI and a shell pipeline:

Screenshots and GIFs for the explanation!

https://github.com/williamcotton/guish

reply
harrylepotter
6 hours ago
[-]
We do quite a lot of this using webviews in vscode over at Darwinium. Similar process that involves postMessages between the UI container and the host containing the LSP. Biggest challenge I had with this was figuring out how to map a diagnostic from the language server (which is inherently range-based) to a UI component that would display it - it involves a fairly intricate pubsub system and mapping UI components explicitly to parts of a tree. Would love to share it eventually.
reply
rtcode_io
10 hours ago
[-]
See https://new.rt.ht for bidi i⇄o sync demos!

The PDF ones are especially fun!

The whole playground is built for bidi sync!

reply
low_tech_punk
9 hours ago
[-]
Wait, what is this?! It's really cool
reply
junon
8 hours ago
[-]
I'm about to set off on this journey too with a code-first PCB editing suite for myself to work on. Seems like this is a good piece of prior art to reference as I need to build similar tooling. Thanks for posting!
reply
taeric
10 hours ago
[-]
Reminds me of the excitement I got on seeing https://www.youtube.com/watch?v=4tyTgyzUJqM. I still get excited about trying something like this again pretty much every time I see it. Have yet to really get any traction on anything. :(

A lot of folks had fun watching Minecraft built using a live code session, if I recall.

reply
jesse__
2 hours ago
[-]
It's concerning to me that the LSP idea is .. a thing. Casey Muratori observed years ago that it's just a way worse way of doing libraries. Like, you're introducing HTTP where there could just be a function call into a DLL/SO. What's the benefit there? Just make vim/emacs/$editor speak some native protocol and be done with it. Then you GUI is just welded directly into the running editor process.. right??

There's no security risk there that wasn't present before as far as I can tell because you were already planning on running the LSP on your local machine..

reply
chamomeal
2 hours ago
[-]
I’ve definitely wondered about this. Like why does the language-understanding standard have to be a server specifically? Just cause it’s a good architecture to build around?

I think I’ve heard that vscode has benefitted hugely from it starting out with a client-server architecture from the start, since it started as a browser based editor. Things like editing code directly on servers via ssh or in containers is easy for vscode cause its client-server all the way down.

Vscode and LSP are both Microsoft products, maybe Microsoft has been pushing the client server thing?

reply
dodomodo
1 hour ago
[-]
there are many practical benefits:

1. naturally async

2. each server and the editor itself can be written in its own language and runtime easily

3. servers can just crash or be killed because of oom errors and your editor won't be affected

4. in a lot of languages it is easier to write a server then to call/export c abi

5. the editor can run in a browser and connect to a remote server

6. you can have a remote central server

all of those things are done in practice

reply
ukuina
2 hours ago
[-]
You're going to love MCP.
reply
whattheheckheck
2 hours ago
[-]
What is up with mcp... feels like CORBA
reply
disqard
2 hours ago
[-]
It's like if CORBA and COBOL had a baby :P

Seriously though, it does combine the "RPC IDL" aspect of the former, with the "Use English" aspect of the latter.

reply
WillAdams
10 hours ago
[-]
I've suggested this sort of interactivity for (Open)PythonSCAD:

https://lists.openscad.org/empathy/thread/GAX4QYYRUC3CEH572I...

The devil is in the details though, and I worry about the UI becoming cluttered and unmanageable.

reply
kennykartman
8 hours ago
[-]
I know basically nothing about CAD, but I know thet fornjot exists and wondered if it might be useful to your purposes, in case you don't know about that.

But well, the project is very cool and I love the idea of using LSP for something more!

reply
FloatArtifact
6 hours ago
[-]
This is really interesting being able to do bi-directional editing. This is desperately needed for accessibility software to get at underlying text. I would use it for semantic editing of code by voice. However, getting at GUI elements would be amazing.
reply
SwiftyBug
10 hours ago
[-]
This is awesome! If someone created a Interface Builder on top of that, I feel like I might even go back to making iOS apps. Xcode makes the whole experience so terrible that it sucks the joy out of it.
reply
nico
9 hours ago
[-]
Very cool concept. Would love to see something like this for web development. Where making changes in the browser could be persisted seamlessly to code
reply
pshc
8 hours ago
[-]
This stuff is a dream of mine, editors in different modes that can collaborate seamlessly. Keep up the good work!
reply
love2read
11 hours ago
[-]
Related: https://chromewebstore.google.com/detail/ghosttext/godiecgff...

> Write in the browser with your text editor.

reply
low_tech_punk
10 hours ago
[-]
So if we replace LSP with MCP, the UI might control the code in interesting ways!
reply
EGreg
7 hours ago
[-]
Whoa … you can use various unicode characters in titles on HN?
reply
jamesbvaughan
6 hours ago
[-]
I was surprised by this too! I actually did a search for that character in order to check before posting this: https://hn.algolia.com/?q=%E2%87%84

I was a bit concerned that HN's algorithm might down-weight posts with non-ASCII characters in titles to discourage people from trying to attract attention with them, but it seems like it's fine?

reply
bitwize
10 hours ago
[-]
See also: zoo.dev, started by an acquaintance of mine.
reply
pjmlp
9 hours ago
[-]
I guess it is about time for the usual Lisp, Smalltalk, Cedar and Oberon remarks.

Why do cool ideas take so much time to be embraced in mainstream?

Retoric question, naturally they weren't VC friendly with exponential growth capitalising user acquisition. /s

reply