I used to work at Disney Research. This is a great example of the kinds of problems that get explored there. The constraints are totally different than the normal consumer-focused engineering you find elsewhere, because the creative conceits are simultaneously a challenge and a get-out-of-jail-free card.
The impossible parts of problems could be solved by visual sleight of hand or some artistic trick, leaving you open to explore and solve some really interesting application specific problems with technology. The much, much harder part is making any of this research work robust and compelling enough to actually show in the parks.
I was part of the pipeline at Disney that looked into operationalizing some of the ideas coming out of Research. I agree that I can't think of any place quite like it. It should also be worth noting that Disney Research is not the same as Imagineering. Research works across all kinds of problems and not just those in the parks.
I was at Disney for close to a decade, and I don't think I could've gotten a better education in how engineering acts in service of the experience the user is having. Disney is really a master of this. Maybe Apple comes close, but I wouldn't know.
That must have been an incredible experience! Do you have any projects (that you’re allowed to share) that you’re particularly proud of, or that particularly inspired you from your time there?
You don't happen to know how someone could get access to some amazing research, do you? They released Autoconnect, and it blew my mind. I needed it just yesterday, but I couldn't find it.
https://la.disneyresearch.com/publication/autoconnect/
It’s kind of funny that it gets explored less in real world applications. Like, if that robot was somehow an automated vacuum cleaner, or if someone made a vacuum bot with wall-e “emotions”, I’d buy that over its high-end (and it would probably need to be high-end to justify cost) competition in a heartbeat.
Of course the “easier” solution was to put those funky eye stickers that move on ours. (Sorry I’m not sure what they are called, jiggly eyes?) My daughter also painted little hearths on it and gave it a name. Not for any particular reason, just for the heck of it.
With some of the nicknames for one of their buildings in Burbank, I could definitely think of some possible reasons. Hopefully, the reasons for the GP are different than my ideas, and they are much less depressing.
working under Disney's direction. As others have pointed out this is an example of where a system like this can be carried by getting a few things really right and faking the other ones well enough.
I think because it was close enough that Disney probably was already working on it before the book came out and the book came out without (presumably) knowing about what Disney was working on.
Then you have two pop culture references to an animatronic Lincoln being released within two years of each other.
That is odd.
I would agree with you if the book had been out longer before Disney immitated it.
Now that I think of it, it was close to the 100th anniversary of the civil war and not long after Lincoln’s death so Lincoln must have been on people’s minds. The civil rights movement would be another reason people would be thinking about Lincoln’s legacy too.
The book had not even been published at the time. Dick wrote it in 1962, it was serialized in 1969 and published in book form in 1972 so it could not have influenced Disney at all.
I would though think that this Star Trek episode must have been informed by Disney's Lincoln
even if there is no explict connection drawn (everybody knew about Disney's Lincoln then, it was talked about a lot) It is mentioned specifically in the 1967 film
where the protagonists are being lectured to by an animatronic figure and once character says "He's like Abe Lincoln" so there is some expectation at that time that people would know about him.
Honestly an idea I've had with the recent surge of interest in LLMs is some type of robotic pet. I feel like LLMs can understand human language and images(hopefully soon) to the extent that a pet should be able to. Some kind of robotic dog that can not only express emotions, but understand what you say to it, seems like it would be pretty successful if done correctly.
The nice thing is that it doesn't even have to understand perfectly, a GPT-4 level of vision/language understanding would be perfectly fine. It doesn't have to hold a conversation, just be able to be happy or sad when it should.
I feel like the big complication at the moment would be getting it to navigate outside of controlled environments. Mainstream tech is getting there (iPhone LIDAR mapping, Meta Quest room scanning) but there's still a lot of dots to connect for a robot far less inherently steady than a Roomba.
What I would bet we'll see at first is some of these little guys in Disney parks, in fenced-off areas or small stages that are kept meticulously clean by staff, where they can interact with visitors without needing to handle all the complications of the real world yet.
You'd be surprised how much a pet can understand.It's more like we can't understand them.They know all our ticks and can sense emotions before even we realize.
What is the total addressable market for a Disney/Star Wars branded e-pet droid? Seeing the kids pull a robot with this much 'personality' on a leash made it seem like an obvious use case. The robot wouldn't have to be that cheap to be less expensive than buying and caring for a dog. It has to be as adorable as this robot though.
Dog's don't become superseded by next year's flashy model, and owners who become bored of them can't put them in the loft to gather dust. I suspect the target market will never be the same as real pets. Maybe you could spin those things as positives.
This is the move right here. And it doesn't have to be a GPT-4 I-know-absolutely-everything model. I think they could train a <1B param model that is capable of being cute and interactive.
Dialogue is the area where the interactivity of video games falls down completely. I mean, dialogue in a video game can be just as good as the dialogue in a movie but it has to be "on rails" because of the lack of linguistic competence of today's computers.
Even with a much better system it has to be "on rails" in that a video game character will get in trouble if you get it to enough an extended conversation which can be somewhat answered by "people have various ways to set boundaries" and that that of course can be part of the characterization.
Y'all are probably sick of me talking about how Tamamo-no-mae in the game Fate/Extella is the pinnacle of characterization in modern games, but being based on a legendary character who is able to charm people by talking intelligently about any subject who is connected to a photonic crystal computer that has recorded all the Earth's history and having the relationship she does with the protagonist any answer she has to why she doesn't realize her promise would be terribly disappointing.
Video games have the separate complication, though, that dialogue needs to holistically work with the rest of the plot. There's no such requirement for a very fancy Furby.
I've wanted an evil Teddy Ruxpin that is really evil and wants to lower your utility function and tries really hard to do it (violate the first law of Robotics) but just isn't strong enough to really hurt you. I figure though the bar for creating a catastrophe is not really that high, all it has to do is start a fire.
I am not sure about that! It seems like small models are emerging that are a bit more specific but can be very small, and thus have much lower latency. For example: https://arxiv.org/abs/2305.07759
This might be worked around with a characteristic verbal affectation that makes their utterances initially long-winded. Once the LLM's lag is passed, the proper content may begin to flow. Thinking fast and slow.
Oh, not even. You off-board that hardware into a nearby cluster. No reason the brain has to be in the chassis when radio is so high-bandwidth.
... a Disney of the past would build four of these and just have them wander around EPCOT's Future World, just to set the tone. This Disney will probably purpose them for their Star Wars experiences. Still great to see.
Disney Imagineering (is it still called that?) always seemed fascinating when you looked at it as part of a company like Disney. Like it doesn't really make sense to exist within Disney if you think about it too hard but it also is critical to what makes Disney... Disney.
I am always fascinated ever time they release a research video (I remember the one talking about snow simulation for Frozen).
It seems like it could be a fascinating place to work and (from the outside) seems unique given their priorities. It isn't to release a consumer product, a new b2b project, its to support some other narrative purpose.
I do find it fairly fascinating that at one point in the video I was watching it and figured it had to be CGI. I wonder if that is just still not used to this on a regular basis.
The focus on robotics research starts to make more sense when you dig into the financials.
Disney’s FY2022 operating income was $12.7B. $7.9B (!) of that comes from theme parks. That’s 65%.
Everything else Disney does (Pixar movies, Disney movies, Star Wars movies, television, streaming, touring musicals, merchandise, partnerships, IP licensing) is the remaining 45%.
I think that is even underselling it some. The theme parks don't exist without the content. With normal ads it's still possible to enjoy the product without being advertised to first.
That's why I don't care one bit about box office stats for any Marvel or Star Wars movie. It could literally sell 0 tickets and still be worth it for the downstream effects on merchandising and promotional tie-ins.
> Like it doesn't really make sense to exist within Disney if you think about it too hard but it also is critical to what makes Disney... Disney.
Disney himself was a huge fan of technology and both him and the CEOs after him recognized that there's a part of the industry they're in (entertainment) that requires secrecy; literal "magic" is mostly "things you can do that other people don't know how to do." So they keep that research in-house because they want first-mover advantage on illusions, effects, and experiences that nobody else can do.
That's the best way to conceptualize Imagineering: it's a magic factory. In that sense, absolutely essential core-business-model stuff for the park-and-show entertainment sector of the company.
(Some of the more recent CEOs didn't grasp this aspect and actually did outsource some of the work done in the past half-decade. I'm going to be real interested to see what the park scene looks like in the Orlando area in the next decade or so as the technologies third-party vendors developed on Disney's behalf diffuse directly into Disney's competition).
hah right? It's not even subtle with how similar it can be. It just needs to be able to jump on my shoulder in a few generations.
Like ok I get it, Disney isn't going to paint up a prototype or reveal too much of their plans with something like this likely in the works years in advance.
This wouldn't be the first time they showed a prototype that was rather clearly for a specific purpose (like the spider man flying robot)
On the other hand Walt Disney was a creative maniac with a large fortune and desire to invent theme parks. Imagineering is the one valve in the heart of Disney.
I agree 100%, based on the desire for theme parks it fits.
But from the outside looking in what is "Disney". I would wager that for the vast majority of people Disney is Mickey Mouse, Frozen, Lion King, etc. basically media. Obviously I would say that people would also say the parks. But I doubt the word "Tech" is anywhere in the vast majority of people's opinion on what Disney is.
It is wild to think that behind what is a media company, is an organization with strong engineering talent. I mean it seems like that the robot engineering specifically rivals most of what else is out there.
That isn't saying there are not more advanced robotics, Boston Dynamics projects are a big example.
But there is also something to be said that this is probably going to be in a park in just a few years, if not sooner. It isn't going to sit in research stage forever.
But it is a really weird timeline that Disney is associated with advanced robotics and other crazy things that Disney Imagineering does.
Disney corp. and their brilliant animators and technicians invented various parts of animated films during an intense period of RnD, both technical and creative. This culminated in Snow White which was the first animated feature film (afaik) and it was a huge hit.
Hollywood was the Silicon Valley of the 20s and 30s. They were using and inventing cutting edge tech!
I love applying this to our technology companies. Sure, we get products like Google Search and Gmail and Facebook and Messenger and Instagram and Tiktok. Everybody calls them tech companies. Like everyone assumes Disney is media. But in reality they are ad companies. Like Disney is a theme park company.
Disney has always been on the cutting edge of technology. They priced the first feature length animated film. Created the "multi-plan camera" to create depth, they partnered with and later acquired Pixar who pioneered computer animation.
Yeah, I think when they do star wars shows/movies they seem to prefer practical droids and aliens when they can do them convincingly. They did do a BD in the past in Mando, but I think it was CG. Maybe they are working towards a practical BD for a future series.
I'm not sure I see what you mean with that? Prompt engineering used to be crucial for getting good results, however with each successive iteration it has become less and less relevant.
Prompt engineering is an extremely simple form of behavior authoring. I'm curious about how you arrived at the conclusion that newer models make prompts and prompting strategies less relevant.
An example of what I mean with prompt engineering becoming less relevant as models get better is quoted at [0]:
> Researchers from Microsoft tested GPT-4 on medical problems and found "that GPT-4, without any specialized prompt crafting, exceeds the passing score on USMLE by over 20 points and outperforms earlier general-purpose models (GPT-3.5) as well as models specifically fine-tuned on medical knowledge (Med-PaLM, a prompt-tuned version of Flan-PaLM 540B).
Or AI will just get to the point that you will describe the Behavior you want in very simple language, maybe referencing certain known characters and it will do all the hard work for you.
I wonder if anyone recalls the little worker robots in the 1972 movie "Silent Running". [1]
I don't know what it was about those fellows but I felt emotionally attached to them. It was not that they had any visible means of conveying emotions. They just went about their tasks quietly and diligently.
For some reason they pulled at my heart strings. It was quite heart wrenching to see them destroyed in the last part of the movie.
> As far as this robot goes, the character doesn’t have an official name, and Disney isn’t ready to comment on where we might see it. But based on how it looks and sounds, we have some guesses.
In the embedded video, there's a part where they are pulling a rug, and I'm so confused about this very small jump he does when he transitions from letting the rug get pulled to standing on the floor [0].
It feels like it knows that it has to do it this way, but it's something you wouldn't expect from any other robot, like Boston Dynamic's Spot or Atlas. They'd know that they have to recompute after the fact that they're partially on the ground where one leg still gets pulled away while the other one is already on fixed ground.
This is a great question that enticed me to watch that moment a fair few number of times. I’ll provide a lame, handwavy explanation and hope to be corrected.
The right foot was on the floor while the left was on the rug, which was in motion and and raised delta the thickness of the rug. In such a mode, the bot is inclined to prefer the stable right leg and lifts the left leg. (I wonder if it can balance on one leg indefinitely? If not, that makes this sequence even more impressive.) Putting the center of gravity over the stable leg increases the probability of the bot staying upright. As opposed to leaning or shifting weight with both legs remaining aground, lifting the unstable leg entirely from the moving surface maximizes the goal of staying upright by minimizing the force of friction. I also think the rug was jerked at just the right moment such that it maximized the visual impression.
This explanation avoids the need for the bot to predict visually the impact of the rug pull, since the rug pull was not instantaneous and the bot’s reaction came after the rug pull began.
This is a very impressive demo. I do think they nailed the goal of expressively and emotion.
It looks like the same corrective behavior as at the very beginning of the video when the robot is pushed sideways. It does a series of jumps to keep itself upright when its right leg moves too close to the center of gravity. Angular momentum of the body probably plays a role too.
Have some coworkers that used to work for Disney. I hear most left since the California to Florida transition. It’s a shame because the problems sound so interesting but I also hear the pay is very mediocre.
It never came into fruition suddenly but I believe it was always the planned agenda prior. Which makes sense to change your job if you expect that it wouldn’t be in the area you want to live in the long run.
Quite a cool little character, but like everything else, is it a bit too late to the table? It will soon be replaced by a prompt I guess? "Generate me a cute Disney Robot video..."
Very nice, I am so relieved that there looks to be a fine technical solution to the eventual extinction of so many animals. It looks like our zoos will have a perfectly great robotic solution!
I'd be very tempted to try and run a gear train and cable system so that the majority of the weight was near center of mass. If you're going small enough, there are servos that pull a few oz and could easily manipulate fishing line in tubes.
Hardware is more or less half a robot dog, but the cuteness is in the software. Closest open source cute animatronics I've seen is Stack-chan based on M5Stack/ESP32, but it's still far off from Disney cute.
This was the comment that made me go back and watch the video.
Interesting.
It of course has a lot of human elements to get at that association. The antennas as ears. Having eye-likes that can close. The tilting of the head is very good looking and I must say it holds itself well.
Boston Dynamics robots and its clones probably has more axes, if that’s what you want. Dynamixel sells servos. Sony does small batches of robots every five years or so. ABB and Kuka makes tons of robotic arms and they’re available new and used. Bellabots(just an off brand Roomba, almost literally) are everywhere.
Oh, and Aldebaran Pepper are easily found being sent to recycling. Because the onboard Wi-Fi has unfortunate tendency of showing early faults, of course. Totally not because it’s not about having human features.
Apparently the software used to convert CG animations into motions for this guy is part of novel development too. That’s what allows him to have these hallmark Disney movements. Are there open source or hackable alternatives to that? To what…
I should have ignored this comment but I just can't.
Disney has been political for a very very long time. I would even say "Woke" since that seems to be the term that you want to use so much.
I would like to point out the lyrics for Pocohuants "Colors of the Wind".
"You think the only people who are people
Are the people who look and think like you".
This came out in 1995. If that movie came out today people like you would be ALL over it calling it Woke.
This doesn't mean that Disney hasn't had its share of problems when it comes to representation and similar issues. Like Mickey donning blackface in an old cartoon (Is that the "Wholesome" content you want Disney to be making again?).
But creating media is inherently political in some form. Otherwise you are not saying anything of value and what you're making probably shouldn't exist. (There are some nuance to this but let's keep this simple). You are generally trying to say or express something and that can be political in its purist form (may not political in how it's viewed today but still political).
As far as Disney going away from "identity politics". Well that isn't happening. Not only are they still fighting the "Don't Say Gay" bill but they are holding official pride nights at their parks now.
We have seen an increase in representation in Disney media and gotten a lot of praise for doing that. It has been slower than many of us would like, but it's still progress.
The world isn't about to start going backwards and Disney is just following that most people don't give a shit anymore.
For the record I am saying all of this as a gay man with multiple Disney tattoos, including a pride one I very proudly wear on my arm that is always visible.
Disney, the company, goes where the money goes. I mean, yes, there are human beings with beliefs in charge also, but the company has a very sensitive nose for where the wind is blowing in terms of American politics. Misreading those winds leaves money on the table.
Disney World Gay Days were an organized event dating back to the '90s, where people would show up on a specific day and wear red to indicate they were there for a specific reason. This was, broadly, supported by the staff (no surprise that there is a massive overlap between a theater-heavy ecosystem and LGBTQ+ tolerance and inclusion) and never formally endorsed by the company.
... but the company did start selling red-based rainbow-patterned official merchandise for certain months of the year. Plausible deniability, but people have money and the company would prefer they spend it.
If anything, the Disney we see in this decade of the 2000s is a Disney that is responding to what it perceives Americans are willing to support, spend money on, and consume. People blame Disney for "being woke," but Disney's Disney; it's always been Disney. Disney looks at American culture, catalyzes it, makes it feel fun, exciting, and engaging, and feeds it back to the public. Disney's telling the stories its audience wants to hear.
... and that is what scares the hell out of the DeSantises of the country and those who support them.
(Sidebar: if you happen to know staff who have worked at the parks and are willing to talk to you off-the-record, the stories are hilarious about guest behavior. Gay Days guests were basically never a problem. Church youth groups, on the other hand? Cast has to poke their noses into every nook and cranny of the parks to make sure they aren't sneaking off to make out and... let's say "violating decency policy").
(Sidebar 2: I tend to be a cynic and my writing tends to end up with a cynical tone. but I want to turn the elephant around here and highlight something: "Disney responds to what Americans are willing to support" means that they're doing what they do now because people like you didn't shut up when told, didn't sit down, didn't go away, didn't become invisible. There are megacorporations who haven't been able to make the Mouse hear them. The people who stubbornly refused to go away for so many decades that the company started selling red shirts and rainbow mouse ears did something to be proud of.)
So just a couple things, I wasn't referring to Disney World Gay Days but "Disneyland Pride at Night" which was an official event that was held this year.
I was at the event and it had the feeling of it being organized as a big "F U" to DeSantis. (and btw it was a fantastic event).
They also have pride elements around both parks (including a friend of mine was there just a month or so ago and they still had some pride things up. Downtown Disney at Disneyland has (or had) a giant pride mickey reef.
Disney now is past plausible deniability on this.
I do think that Disney did push the envelope a bit. I mean it was back in 2014 that they had a Disney Chanel show had a lesbian couple with kids on Good Luck Charlie.
But you are right that the reality is what Disney is doing (and most companies) is following what is socially acceptable. That is good business. Maybe they will push a bit, especially with Hollywood tending to lean more socially liberal so they are generally a few steps head of the rest of the country.
Gay acceptance was pretty big in the 90's. I remember white papers released around 2000 talking about the inevitable nationwide acceptance of LGB. It was trending up fast. My family is from SF/Oakland. I talked with a lot of LGB academics at that time, I watched it happen. The LGB are also a demographic that has a lot of disposable income and that's part of the reason they were so accepted by corporations.
Life is politics, I’m talking about the writing. I hate listening to supposed liberals defend a heartless media monopoly because it panders to their secular religious beliefs. The Disney woke brand is exploitation of a market, it’s not social justice.
Perhaps you meant to say that you’re happy they’re moving back toward broadly appealing topics instead of those resembling a rather small part of the population.
I prefer good writing over wokie identity politics injected into low grade corporate content. Disney is never producing a film of the same quality as Priscilla Queen of the Desert. And I’m sick of people defending Disney mediocrity because it panders to them or pretends to be high-minded.
Is that to say there is only the display of emotion?
I am curious if the two robot monkeys forget all about the history of the ladder.
Will the future not-robot monkeys discover the truth?
I am always amazed that I am not upset at the computer but at my failure to operate it in a way to get the desired outcome. The ability to generate emotion exists within the imageo and not the observed or learned.
> The ability to generate emotion exists within the imageo and not the observed
I was thinking the exact same thing while pondering the parents assertion.
Qualia is just the notion of being something. The classic example is a bat: what is the qualitative experience of being nearly blind, but having wings and a highly accurate echolocation system?
Plants too have qualia. What is it like to be a tree. To feel the nurture of the sun, the change in seasons, and to heal a broken branch.
The qualia of a stone is less exciting, but we can still imagine being a dense pack of minerals. Perhaps rolling down a mountain and losing some weight. Suddenly finding ourselves on the ocean floor, drifting aimlessly, while providing cover and nutrition to all sorts of life. After staying stationary for thousands of years.
That rock has seen things, man!
Down at the atom level things get more excited. What is the qualitative experience of interacting with other atoms? Or a photon? Of being entangled? State transitions?
Anyway. It's clear to me that this robot absolutely has qualia. I can imagine what it's like to be be a circuit board, with an operating system, a bunch of sensors, a neural processing engine, and a hard shell. How it feels.
It makes me appreciate being human more. But also evokes empathy for that adorable lump of silicon, metals, and plastics. Don't let the GP comment bring you down, little one. You are perfect just as you are.
If qualia is required for emotion, then the appearance of emotion can be evidence of qualia. Saying it requires qualia to truly be emotion is putting Descartes in front of the horse.
The impossible parts of problems could be solved by visual sleight of hand or some artistic trick, leaving you open to explore and solve some really interesting application specific problems with technology. The much, much harder part is making any of this research work robust and compelling enough to actually show in the parks.