> Most of our examples were built entirely on the iPad, using Inkbase’s interface. Sketchy math was not. Much of the code that runs it ... was written on a laptop.
> Building larger, more technical software systems in Inkbase becomes extremely difficult for many reasons, from the poor ergonomics of typing with an on-screen keyboard
Nobody has really solved the ergonomics problem of being able to type on a keyboard and also sketch, and have the entire system be portable and friction free.
Decent handwriting recognition would help here - not as a full substitute, but as a lightweight alternative.
Though I cannot imagine it working within the confines of the keyboard HID API.
With non-defective keyboards (and non-spazzy hands), the key presses on a keyboard are non-probabilistic; that's not the case with handwriting, where a large amount of the information regarding how to interpret text comes from the surrounding characters - including those that follow.
You'd need to deal with the shifting probabilities of text input, and without introducing user noticable latency or triggering an excess of events.
It doesn't sound impossible, but it wouldn't be easy either.
Even ipad handwriting recognition has weak points. For example, a q by itself will almost always turn into a 9. I wonder if PDAs had it right after all and we need to adapt our characters to make them more easily understood by machines.
My handwriting, because of a life of typing (this year is my 40th year of 10 finger touch typing; I was 8 when I got my certificate, so I basically skipped writing), is horrible and only I can read it. The iPad cannot, at all.
You remind me of a Windows CE PDA I had in the late 90s through early 00s [0] that had a pen stylus and consistently amazed me that it could flawlessly interpret my scrawly scribbles perfectly - I'm hard to please but it always amazed me and I felt in awe.
My experience with every hand-writing recognition facility since has been it has got worse, more opinionated, and is probably tied too tightly to grammar and spell checking - rather like what I call "destructive texting" when predictive typing on mobile devices constantly auto-replaces words after I've typed them without me noticing until a message is sent!
[0] I still have it although not powered up in a couple of years!
Out of context, a q and a 9 might be hard to distinguish; but all you need (in context) to tell them apart is where they appear vertically in the line that they're on.
Seems an obvious if orthogonal solution might be to substitute typing with voice input designed in a way that avoids the keyboard non-optionally sliding up to cover half the screen whenever text input is selected. Recognition accuracy is pretty high these days, for me at least, plus it's only getting better each year. And transcription rate is equal to or better than typing speed.
> Nobody has really solved the ergonomics problem of being able to type on a keyboard and also sketch, and have the entire system be portable and friction free.
Writing on Obsidian and sketching on excalidraw (both with mouse and stylus) and inserting the resulting image via MD is a nice workflow for me. I like it.
+1 for that workflow. This way the writing part is done in markdown, and the diagram part in Excalidraw (best tool for the job and all). If you open in split view (screen real estate permitting) you can even do both at the same time with some light context switching.
I spent two years preparing notes and delivering lectures 4-6x a week using a sketchpad.
1. If I put the pad on the right of the keyboard, then I can type normally, but I am writing at an awkward angle. If I put the pad right in front of me, then the keyboard is too far from me, and can't type fluidly. There is this continuous tension about where to place them.
2. Pads use bluetooth or wire to connect. If I use bluetooth, then my headphones must be wired. So there is a tradeoff there. Headphone are necessary for meetings.
3. There are the non-screen sketchpads, and the screen ones. The non-screen ones require a lot of hand-eye coordination (because you have to look at your monitor while drawing on the pad). They are not as stress-free as paper. The screen ones would help a lot more, but now you have two screens showing the same thing. The pad screen to draw things and move them around with your hand/pen, and the monitor to show you what you type. Kinda stupid.
Well, why not use just a ipad-like-tablet and both draw on it and type with a keyboard. Because drawing requires the surface to be flat, and keyboard require the screen to be verticalish so you can easily see what you are typing.
4. The software is still bad. There is still no native cross-platform sketchpad application with collaboration features built in, that will let me conduct a three hour online research meeting with dozens of pages of math. Browsers will inevitably crash, and are just slow.
> 4. The software is still bad. There is still no native cross-platform sketchpad application with collaboration features built in, that will let me conduct a three hour online research meeting with dozens of pages of math. Browsers will inevitably crash, and are just slow.
> 2. Pads use bluetooth or wire to connect. If I use bluetooth, then my headphones must be wired. So there is a tradeoff there. Headphone are necessary for meetings.
Recently entered the iOS/ipadOS world and still having a lot of trouble adapting, but are you suggesting that the ipad can only connect to one bluetooth accessory at a time? Or that the bluetooth connection become unstable when too many devices are connected?
If you mean graphics tablets like the Wacom displays with the pens, no. It works fine with parametric graphics or CAD, for example, so long as I stick to simple dimensions so one hand is on the numpad while the other holds the pen but the second I need full words - like variable names in complex equations, renaming groups or layers, fuzzy searching for a brush or color by name, etc. - putting the pen back down (without losing it!) becomes too cumbersome. It either serves to interrupt the creative process or get in my way when I know what I want and just want to do it as quickly as possible.
It gets worse since touch screens are actually pretty imprecise so I end up using a 3D mouse for viewport manipulation and a regular mouse for precise navigation (nested menus!) on top of the pen and keyboard. At least one of the input devices is considered lost at any given time.
> Building larger, more technical software systems in Inkbase becomes extremely difficult for many reasons, from the poor ergonomics of typing with an on-screen keyboard
Nobody has really solved the ergonomics problem of being able to type on a keyboard and also sketch, and have the entire system be portable and friction free.