Do they not? "At scale" is somehow often synonymous with "with less effort and care per instance", but that doesn't have to be the case. Things work worse if you scale only one part of the equation (the "customers"), but do not scale the other side appropriately (the "service provider"). That should be obvious.
> Text campaigns are most likely to be effective when they:
Are sent by a school and even staff member who students know and trust
Add meaningful and novel value above and beyond existing outreach and support efforts like giving students real-time information on the status of their financial aid applications
Provide content that is personalized to the student and where they are in the college application and financial aid process
Having a new org sending out mass messages to all students is not scaling The described. It is something different.
Scaling it would be convincing more school staff and schools to make a similar effort.
The positive effect was not caused by "getting a message", it was caused by "getting personal advice from someone you know". They scaled the wrong thing.
Things also "work" if you are able to scale down a cost (employment of human labor such as teachers and counselors) while maintaining or scaling up a source of revenue (student enrollment.)
I worked for a company that offered this service (among other data-y things.) It employed lots of bright-eyed young people who were extremely happy they had found a tech job where they could do good, helping struggling students succeed at college. It was a dream job. However, the information we heard trickling in from the customers (public universities and community colleges) was pretty demotivational. They ones that were shrinking were panicked about their future existence, and the ones that were growing were basically like, yeah, hiring a proportional number of front-line staff to interact with students like in the past would just be silly. Nobody is going to do that. Run along, idealistic tech people, save these poor kids with analytics and text messages, because we're not paying to do it the old-fashioned way.
This really gets at the heart of why I have very little allegiance and essentially no desire to donate to my alma mater: Why would I care about or donate to a business that really doesn't care about me individually besides $$$ signs? Sure, I patronized them for four years of my life, but I've done the same with In & Out for longer and you don't see me sending them a check for nothing. Much like In & Out, the positive memories of my alma mater were more so about the people I was with than the location or business proposition itself. Ok, I'm ending this metaphor before it becomes too contrived...
Realistically you donate so that the school has a bigger budget and more opportunity to gain prestige making your diploma more valuable. That's really only relevant to a small subset of the population and rapidly shrinking. Four decades ago degrees had meaning but now they are largely participation trophies that come at great expense.
Khan academy is absolutely fantastic. However, it's only a step above giving students free textbooks or other self-learning resources: It mostly work for students that are already self-motivating. This tends to be a minority of students, and most require some degree of a human support network to succeed.
Things also "work" if you are able to scale down a cost (employment of human labor such as teachers and counselors) while maintaining or scaling up a source of revenue (student enrollment.)
This is an unfortunate truth. Schools grow, don't scale human labor to accommodate additional students, overall student performance & dropouts increase, and their answer is to recruit more students in a downward spiral. Never mind the fact that it costs more money to do this than simply hiring 2 or 3 more advisors to do real human outreach. I saw this cycle over the course of 4 or 5 years where I work. After providing hard analysis of this pattern each year, I, working with colleagues on the academic side of things, managed to push for those advising resources and the results have more than paid for it.
The problem is that many of the idealistic tech people are only looking at the problem from the tech point of view. I have resisted efforts over the years to assimilate me into a central IT division because the viewpoint I get through being embedded in an operational division strongly informs my analytical efforts. I see the business cycle, I talk to advisors, I construct & analyze surveys that mix quantitative and qualitative factors-- there is no detail of the student experience that is too small for me to ignore. It hammers home the point that analysis & tech can only help focus the efforts of human labor, not replace it.
Some problems can be improved with a simple contact point.
I have a colleague who works in this space in a research capacity. They were looking at use of different systems to measure behavior and identify behaviors that are correlated with poor performance.
In one scenario, they surveyed a cohort who was leaving and entering the dorm after 1AM, resulting in poor AM class attendance. 40% of the responses were essentially: “I went to meet my [girl/boy]friend and have sex.”
Getting a text message to tell that kid to get his beauty rest isn’t going to work. Just treat people like adults, imo.
Some problems can be improved with a simple contact point
Absolutely. Most schools don't employ enough advisors to have sit-down appointments with every student every semester. However, I did see a school implement a policy where 1st semester students-- a higher risk population-- were required to have a sit down like that before they were allowed to register for the next semester. That single effort resulted in a 15% decrease in 1st-year dropouts.
I went to a big public university where they had two freshman tracks, a "college within the university" where, if you knew to sign up, you'd be in a cohort of students to meet some general education requirements and would get access to faculty motivated to help you. The problem was that the kids who needed the least help tended to discover this program.
The "normal" path was being dumped into the University bureaucracy blind, and you had to learn to sink or swim. It was tough -- the registrar in those days was an experience like DMV. I'm pretty sure the system was devised as a funnel to disqualify people for popular majors like Business, Computer Science, etc.
the kids who needed the least help tended to discover this program
Heck yeah did you hit the nail on the head. I've analyzed the data over a period of years for which students utilize tutoring services: They very much tend to be students that were already doing alright before they started getting tutoring.
Conscientious self-motivated students are precisely the ones most likely to seek out help on their own. As a result, for students that go on academic probation we now require them to attend a certain amount of tutoring hours each semester, and the software we use to manage advising caseloads allows faculty to raise a flag for students performing poorly so that advisors can pro-actively reach out and direct them to tutoring resources.
This has led to a huge increase in tutoring resource utilization, but unfortunately they are now at the limits of their capacity. Our next hurdle is to analyze the academic outcomes of students using those resources, which will hopefully show increased performance & allow us to petition for more support services.
Fortunately our leadership is not completely short-sighted on these issues, but as obvious as "more one-on-one advising & support" would seem, they (not too unreasonably) still demand very concrete evidence to justify additional expenditures. However it would be nice if they viewed customer service of this sort as the profit center it is, rather than a cost center. The problem is that things like the benefits of quality tutoring of freshmen may not yield tangible results for a few years, so things move slowly.
Excellent point. It’s a problem with operational definitions which plagues a lot of research.
But “scale” in this context implies economic efficiency. You can’t give every kid a full time counselor. You can write a script to send sms messages.
One question for the tech community, to what extent can advances in ai and other technology impact this question of scale/efficiency. If the student doesn’t know his counselor is a bot, maybe the nudges become more impactful.
I'm a professor. My university bought a big piece of enterprise software, designed precisely to track and boost student success in various ways.
The software sent me an email asking me to do something, ostensibly from someone in our advising center, signed by my assistant chair instead, but obviously corporate-sounding and autogenerated. It reeked of fakery.
Is this sort of thing supposed to accomplish something?
That tells me you have a team of people who have not done their jobs, or let too many decisions be made by committee.
We have a large piece of software at our institution that communicates automatically with students, staff, and faculty based on various triggers (grades, dates, etc). The hardest part of that process was writing the messages. It took 1 semester to get the system operational, and two years to dial in the messaging.
The worst thing you can do is create a "specific" message for an individual with generic language targeted at groups. Which, subsequently, it seems is what most institutions do. No idea why. It's easier, I suppose?
Yes. What it accomplishes is to abuse the trust you have built with your students through genuine interaction to spread their corporate (ok university. Same shit) propaganda by making it seem like it comes from you. Most people can smell that from a mile away so you should absolutely not do it if you care about the respect of your students.
It depends on how it's implemented. These systems can work, but they require very tight coordination & buy-in from faculty as early as the procurement stage. Schools that just buy & force them on faculty don't achieve much.
The types of messages faculty get like that are usually a check-in where they're asked to give a status report for each student. This can be a quick good/ok/bad type of thing, or it can be something like a brief survey. My personal opinion is that the first option is best: These systems should put as low of a burden on faculty as possible. I really minimalist implementation that is almost as good is to ask faculty to only flag students in trouble.
What this should do, if the college has properly implemented it, is trigger the advisor to reach out to students struggling to determine 1) Why and then 2) What can be done to help: tutoring, assessment of whether the course is really required, determination if the course can be withdrawn from if not required or if the student is beyond the point of recover, or should have taken a pre-requisite or lower-level course in the subject first, etc.
If your college has not communicated this sort of thing to you, they are doing it wrong. I would still encourage you to do what is asked unless it is a very large burden. If that's the case, reach out to the advising unit overseeing these initiatives to understand how they're using it. But if it's not too much of a burden, even if they've done a poor job communicating with faculty, it may trigger a meaningful intervention with the student. Even if it doesn't, it will still allow collection of data that can be analyzed to determine trends, the significance of whether these activities by faculty correlate with student success, etc. And if they do, that will then be used to feed back into the advising process that these things are actionable to the benefit of students.
These systems can work. In my current job, after implementing such a system we saw first-year drop outs decrease by about 6%.
> The software sent me an email asking me to do something, ostensibly from someone in our advising center, signed by my assistant chair instead, but obviously corporate-sounding and autogenerated. It reeked of fakery.
>The software sent me an email asking me to do something, ostensibly from someone in our advising center, signed by my assistant chair instead, but obviously corporate-sounding and autogenerated. It reeked of fakery.
So did you do the thing? Did it improve a student outcome? The 'fakery' is just a string that could be updated. Telling you what needs to happen and for whom is the critical function. But then again getting faculty to play ball often may as well be cold fusion...
Because, and I am not exaggerating in the slightest, this is the worst idea I have ever seen directly related to my career. This concept is an idea that I am fighting all the time.
Teachers and students alike around me sort of blindly "get used to" these notifications flying around you and even begin to expect them even when they don't work.
Essentially, they become the equivalent of spam in your email that you ignore, except they're harder to ignore and people still have this wildly misguided faith that they can think for you.
As a first measure -- I'd love to remove all "notification" type services from higher-ed; after a bit of a learning curve, as in -- yes, you must visit this website or call this person or read the syllabus to get things done.
You develop your own system of remembering things, maybe it's paper, maybe you email yourself, whatever works for you
It does work "at scale" but you need to launch, fail, and iterate just like any product.
Georgia State University is a great example of successfully implementing the goal of "college success" and continuing to innovate. They started by solving the problem of students accepting their offer letter, but never choosing classes during that crucial first summer.
First tool they built used a combination of AI chat box and counsellors to answer common questions, help students fill out forms, and/or get financial aid. At each step of product development they asked "what's next?". For example choosing a degree is "hard" for first generation college students. So they built salary, and career earnings during the UX where you pick your courses.
Another non-SMS solution they provide is automatically depositing money into student's accounts who are low and struggling. This especially helps 4th year students who may have burnt through their financial aid.
This isn't specific to the article per say, but something I've been discussing with my colleagues revolves around gamification and nudge theory in education. There's a lot of pressure on gamification in financial trading after the GameStop saga in January, and with the pre-existing pressures from loot boxes, I'm starting to ask "Why is it okay for education to do it?"
I think nudge theory is a subset of all the elements that make up gamification, but still, the idea that something is okay to do because "we're doing it to help people" seems a bit odd.
If you accept that the use of grades is just a way of gamifying individual learning to establish player scores and a competitive leaderboard, then schools been at it for a long, long time.
When you look closely, a whole ton of things are gamification. Even something as simple as a problem set is very gamified, if you choose to see it from that angle, since real challenges almost never come in the form of neat bundles with a single unambiguous solution. I'm honestly not sure whether I think that's a useful angle or not, but either way it seems inescapable to conclude that there's a category of healthy gamification towards good ends.
If you have signed up for college, then I don't think it is wrong to get an SMS, similar to what a doctor might send. I think the key is "person really wanted this at some point in the recent past", which is not the case for gamefication.
SMS used to be a very effective channel for getting conversions from people. But marketing channels are limited resources, they are most effective early in their discovery when organizations begin using it, but once their novelty is sucked dry the effectiveness tapers off until it’s no more effective than a flashing banner ad.
At scale implies automation, especially guessing at budgets to support these programs.
I think schools know there has to be a human at some point in the interaction chain.
A frustrating experience I had with this is a school I was at (expensive uni) tried to farm out this responsibility to other students.
In a mentorship program designed to do 1:1 student:student pairings and used “SMS intervention” this program asked students to take up to *10* student mentees, with the idea that that the real mentor load is pretty low (quick convos).
To me, this felt like an outrageous ask, as if they were trying to sub in students paying to be there with the tasks paid advisors are responsible for.
This was at grad school, and I think indicated something funding challenges behind the scenes. I imagine it’s similar/worse at high school levels. Maybe not the same exact problem, but it makes sense why this specific program linked has trouble.
> Implementing a texting campaign with hundreds of thousands of students may require organizations to forego advising and have texts provide just one-way content or to have large caseloads that limit how much engagement advisors can have with individual students.
> Another trade-off is that students may have less trust in a state agency or national nonprofit sending the messages.
> Finally, state- and national-level campaigns have typically sent content that is uniform across all students, whereas earlier, local campaigns often personalized information to students’ college plans or FAFSA (the United States’ government financial aid application) filing status.
> Even local text campaigns may face obstacles to efficacy that similar campaigns did not encounter to the same degree a decade ago. [...] As message volume through text increases, we should reasonably expect that any one message won’t get the same attention—and eventually, that people will migrate to other communications channels.
I work in the industry. There are a couple of reasons they don't work, at least as currently used.
1) Over saturation/overused: Especially as High Schools begin using these, students get too many, too frequently, and it just becomes a sort of background noise. Back when students still had email as their prime mode of digital communication, we had the exact same issue. Studies on their effectiveness were based on relatively short time spans where nudges were new & novel: The novelty has worn off.
2) A simple nudge can't help a complex problem. A college may nudge "You haven't registered yet! Don't forget to register for the next semester". That's an extremely frustrating message to receive when you're poor and can't pay next semester's bill, or some other roadblock.
3) It has been used to justify not scaling human resources for students, so any benefit is offset by lesser real human contact with advisors. People complain about administrative bloat at universities, but that doesn't apply to advising: I've seen caseloads rise from ~300/advisor to ~500/advisor. This is especially problematic for overcoming #2 above: Advisors can often help with complex issues that student's believe are insurmountable.
My observations on how to do it better:
Only use nudges for low-stakes issues, and even then choose carefully so students don't become numb to them. High-stakes issues need human outreach. With limited bandwidth by advisors to do this work, it needs triage: Even if it's high stakes, don't waste time reaching out personally to low-risk students. Determining risk requires a two-pronged approach:
1) A statistical model that uses historical data to identify students at risk of poor performance or attrition. A competent data analyst can build a reasonable model without having to go too far into advanced data mining. "Deep Learning" tends to be fairly useless here: There's not enough data. There's also a few good vendors out there that will perform more advanced risk modeling, partitioning a school's population into a dozen or more groups and building models for each of them. These aren't cheap, but I've seen it pay off when correctly paired with one-to-one human outreach initiatives. In a single year I saw it reduce attrition by 1%, not to mention academic performance increases, and that more than paid for the cost. I've also seen schools pay the invoice and expect things to magically change without aligning their resources to the risk analysis output and waste a lot of money.
2) A rule-base system of identifying students at risk. Things like a late payment; a good student getting their first D or F in a class; delaying registration for the next semester; requesting transcripts to be sent to another school; and many other factors.
Using the high risk population identified in #1 and #2, that is where you focus your one-to-one human outreach efforts.
This title cut-down to "why aren't text messages designed to boost college success working at scale?" was extremely confusing until I looked more carefully at the original article and saw it says "text message INTERVENTIONS". To save characters--if we are ignoring the half-rule to accept the upstream author's title--I would suggest changing "text message" to "sms" instead of dropping the key noun "interventions".
I still can't actually parse the title. What is "college success working at scale"?
Edit: Ok, halfway through the article it became clear: The proper reading is
> "Why Aren’t SMS Interventions which are Designed to Boost College Success Working at Scale?"
and not the way I read it initially:
> "Why don't we design SMS interventions so that they boost college success working, at scale?"
Funny enough, this is a case where languages with different capitalization for noun vs verb (working, in this case) would immediately resolve ambiguities. Well, except for Title Case, I guess.
> Text campaigns are most likely to be effective when they:
Having a new org sending out mass messages to all students is not scaling The described. It is something different.Scaling it would be convincing more school staff and schools to make a similar effort.
The positive effect was not caused by "getting a message", it was caused by "getting personal advice from someone you know". They scaled the wrong thing.