Quick update: - Looks like 6 new families (with 11 new learners in total) joined since this post - thank you @geoffbrown2014 if it was you that spread the word (I still have my site set to no-follow, so that's my best guess) - I'm still working on fixing bug #3 (need to build a script to try to reproduce it) - I have a feature-request from one of these new users asking for more voice-to-text for non-readers which i'll start working on once that bug if fixed

Hi Eric, I'm a homeschooling dad with a 5th and a 6th grader. I made an account and will have my children check it out in the morning and report back.

Uh-oh.. Just caught 3 new errors in Sentry - guessing you may have encountered some broken stuff in the app - very sorry about that - I've gone in and fixed 2 of them (still working on the 3rd) in case the bugs were fully blocking..

Oh great! Definitely let me know if you run into any trouble or could use some help - I'd be happy to jump on a zoom call if that might be of benefit.

We had a chance to work through some of the lessons. Overall very solid foundation I would say. My son's critique was that it was a little confusing when it came to answer format and that he felt the lessons kind of jumped around. Other than that he found it 'OK'. He's 10 so that means you didn't mess up too bad. lol.

We did have a few issues.

1. The hint on the equation simplification section describes the format of the simplified equation as 6 x l. (actual number varies 2 x l x 3) But the only answer that is accepted is in the format is 6l, sans the 'x'.

2. I found the hints' font coloring very hard to read on my old eyes. Also, we did not find out the desired input methods until fairly far along. The SHIFT-8 etc was new to my son and kind of obscure given how soft the font was. Maybe a heads up of the conventions at the beginning?

3. In the word questions section, sometimes the amount being compared to determine the total amount of cages vs squirrels was the exact same. Actual examples -> 2 cages and 2 squirrels vs 2 cages and 2 squirrels. I assuming the desired question would have compared 2 cages and 3 squirrels vs 3 cages and 2 squirrels. (Although that doesn't make sense either.)

4. The spoken word button is a great idea, but the voice is almost incomprehensible to a 10 year old. I know you didn't create the voice, but if there is another easier to understand voice that would help.

5. My son was initially confused by going some distance in a negative amount as a representation for reverse.

6. My son is very competitive and he enjoyed the speed challenge, but there was no reward or signification of accomplishment at the end of the challenge. He wanted to know how well he did.

6. There are some minor grammar errors.

My impression was that the layout and design was on par with other online offerings that we use daily. I actually liked the lower distraction level. The site and UX were predictable, easy to use and did not add to the stress level of the student. I'll have him do some more over the next few days. Keep up the good work!

Thank you so much for the excellent feedback!

You've definitely encountered some (fortunately non-crashing) bugs here that I need to look at and fix (the 22 versus 22 for instance needs to have different numbers to make sense in a lesson on operand-order equivalence for multiplication), but seems like there are also quite a few product/design tweaks I should make as well.

For the lessons jumping around --> that's actually intentional & very important for learning (though from your feedback I can see I really need to make it much clearer why so that learners don't experience it as aimless) - question-variation achieves 2 things - it pushes what the learned just covered out of they working memory (so that when they get the question again in a few moments, they need to fully reconstruct the idea in their minds), while at the same time having the learner review lessons they've learned recently (for new users, however, since I don't have any recent lessons for them yet, I'm including questions thy've likely already mastered given their level, which may need to rethink as it could be adding to that potential of aimlessness learners might have).

Thanks again, and definitely let me know if you encounter any other issues.

Are you aware of the work on “programmed teaching” by B.F. Skinner and Siegfried Engelmann (Direct Instruction)?

I was not, but looking now (only skimmed a few articles so far) - it's so interesting that the essential underlying idea for machine-assisted-learning was already laid out clearly by over 65 years ago, and yet, such systems don't appear to be all that widespread despite having been proven to increase learning - makes me wonder what the real blockers are..

As for Skinner's core belief that positive reinforcement is a more effective long-term motivator for learning, I couldn't agree more, and have seen it time and time again when user-testing with kids.

I think we forget as adults just how much mental energy is needed to approach a new concept or idea. And the energy level of kids can wane very quickly when they're discouraged (never mind punishment, even just 2-3 wrong questions in a row with encouraging parents present can sap them completely). I think we also forget how earth-shattering it can be to get a question wrong - I've seen kids go from answering questions correctly no problem, to completely doubting whether they still understand even the basics of the questions they were previously getting correct.

I've found it's crucial to help build back up their confidence whenever this happens. Part of the benefit of Grokkoli is that it will immediately try to get to the core of the learner's confusion towards a given question, usually by asking troubleshooting questions that start with the simplest/easiest components of a question and build back up to the original. And a big part of why this works is that it restores kids confidence very quickly at that moment.

Once their energy is back up, kids are primed to learn, and they want to keep going. Grokkoli checks in with learners after a certain number of questions (a function of their grade-level) to see if they want to take a break. So far, I'm seeing kids opt to keep answering questions 3-5 times per session.

Englemann's ideas seem a little more complex (not as easily distillable) but every bit as interesting - I need to read more before getting back to you.

Thank you for sharing, and definitely let me know if there's anything else you recommend I look into.

Exactly! I am as baffled as you are these techniques are not being used.

(That’s actually incorrect. These techniques are being used. Just not for teaching. Instead, they are being used by companies like Facebook. Positive reinforcement is the backbone of much of our current attention economy, and has generated trillions in wealth.)

I have been studying this topic quite extensively a few years back. There must be many pages of notes and highlights somewhere. But I am on my phone right now you cannot access them.

But your project instantly reminded me of Skinner and of Direct Instruction. Having a program. Going one step at a time. Validating that learning has indeed occurred. And that the responsibility lies never with the learner but always with the program.

I believe I did find sources where Engelmann explained these principles in a quite simple, understandable fashion. I have to do research on my computer to find those.

This is what I found now:

https://pastebin.com/raw/wWMkyzQ7

They are also some videos on YouTube of BF Skinner demonstrating his teaching machines in the 1960s. Just think what is possible today, where we have so much better devices, perpetual connectivity and programming tools that anyone can learn within a couple of weeks.

Another figure I stumbled upon during these studies was Michel Thomas. The guy was famous for teaching people a new language over a weekend. With no homework, that is. I believe he taught quite a few celebrities. Wikipedia will be able to tell you more. What Wikipedia doesn't tell you is that his system it's also very close relative to you direct instruction, i.e. another direct ancestor of Skinner's work on positive reinforcement.

Here are some of my notes:

https://pastebin.com/raw/6iQqe6rx

Wow thank you for compiling this! You'll see from some of the other comments that I have quite a bit of work cut out for me in fixing bugs and issues with the NUX ;P - but I will take a look as soon as I can.

And you're absolutely right re Facebook & the rest of the world doing positive reinforcement well (casinos and game-designers are what immediately come to mind for me).

Recent NYT article on Engelmann: https://www.nytimes.com/2021/09/03/opinion/kids-reading-spel...