Just posted this on Anthony's blog, but thought I would post it here as well:
--
Hi Anthony,
I'm sorry that you feel we were't straight forward, although our goal was to be completely transparent in our communication with you about the status of this feature. Since the time of launch, we thought we had clearly messaged that international SMS has been an unsupported feature, and that the entire product was in beta status. We had published this fact on our SMS product FAQs, as well as in the public GetSatisfaction forums. Our apologies that we weren't clear enough about that fact, which caused you to roll international SMS features into your app.
We do pride ourselves on putting developers and their applications first, and in being open in our communications, product capabilities and limitations. Please feel free to email me at jeff@twilio.com, I'm always open to feedback, questions, comments or concern.
SMS is just a huge pain in the ass. The carriers have a strangle-hold over this and can cut messages off on a whim. We had an experience once with Verizon where they "accidentally" cut-off our short-code to Poll Everywhere on a Friday evening due to some "clerical errors". 72 hours later they finally had somebody "fix" the situation (support doesn't work on weekends there), and in the meantime we were left in a really bad place trying to explain to our customers how Verizon screwed us over.
I've dealt with some total dopes at Celltrust (I need to call these guys out, they're atrociously bad. Please don't do business with them unless you absolutely have to) where our service to Canada was entirely out for a week because they changed how their API worked. They failed to communicate that to us and nobody at their company could tell me how it worked so I had to fire up nc -l, point their gateway at that server, find somebody from Canada who could send a text, and see what came in. After the fact, I told them how horrible the experience was and provided recommendations on how they could improve it. They threatened to cut us off as a client because they felt I was being mean to them instead of acting on the feedback that could have made their service better.
Twilio thinks this is assine and are trying to change how this all works. We've shared with them the pains we've experienced with SMS aggregators and they share the same thoughts with us that its a totally unnecessary amount of pain that developers face to build SMS applications. They're really trying to make this easier and lower the bar.
Anthony, while your experience wasn't that hot, it was far and above what you'd experience elsewhere. At least Jeff acknowledges that Twilio could do better. That is so far above and beyond anything else in the industry. Throw on top of that an amazingly well executed API and the drive to make a difference in this industry and you're in good hands.
Hi Jeff,
I had the same problem. It was not very clear from your pages or FAQs that international SMS was not supported. Now I see it's been made very clear in this page http://www.twilio.com/faq/international, that's great, the previous message was confusing to say the least. FWIW I complained about this on the getsatisfaction forums some two weeks ago http://getsatisfaction.com/twilio/topics/international_sms-1...
Fortunately I don't have a live app that depended on your service, so all it cost me was $50 deposited to start playing around.
I like Twilio and I'm a customer. It seems they did the best they knew how with this situation. But that was a really terrible apology and seems more insulting than assuaging to anyone affected. It's like there's a rule that once you have more than 10 employees all written communication needs to be jargon filled, passive-aggressive business drivel.
Better would have been:
"Anthony -
We're glad to have you as a customer, sorry we screwed up here. We posted notices to the status API and support email - but if you didn't get it, we didn't do our job! I definitely gained some perspective from reading your post.
We'll work even harder on future product updates to get notice to all our customers. If there's anything I can personally do to make the situation right, email me: jeff@twilio.com.
Again, very sorry. Thanks for being our customer and letting us work to earn back your confidence.
Jeff Lawson - Twilio CEO
P.S. Task.fm looks really cool. Best of luck!"
Danielle from Twilio wrote a much more sincere and helpful apology in the thread above...she should probably handle the "responding to blog rants" duties from now on.
Isn't "I'm sorry" the very first thing he says after "Hi Anthony"? And as if that wasn't enough, he goes on to say "Our apologies" later in the paragraph.
He said he was sorry that you feel we did x. So he is apologizing for for the fact that Anthony feels a certain way, not the same as saying "I am sorry.".
I was also planning on using twilio for a project and when looking at their faq for the initial research, it wasn't very clear that it was something completely unsupported that could be disabled at any moments...
This is why programmers make horrible customer support people, it doesn't matter if you have nothing to apologize for. You say your sorry because no matter what the circumstances the person is unhappy with your service and your company - saying your sorry and fixing the problem keeps a customer. Sticking to your guns and saying you have nothing to apologize for just loses you a customer.
A professional organization NEVER turns off features -- final, beta, experimental, it doesn't matter what you call it -- without warning unless external pressures force you to. What you do is determine who is using the feature and send a warning to those customers. Then after waiting a while (a few days to a few months), you can turn it off.
Predictability is critical to B2B services. It is more important than price or performance. It allows customers to make an informed decision. It prevents dissatisfaction and PR disasters.
I also think this should be a lesson for younger businessmen. Customers will even get angry if you cut undocumented features. Or fix bugs they relied upon. Any change that impacts customers needs to be communicated in advance.
IMO, if you truly "put developers and applications first", you would have done this. I humbly suggest some soul searching.
From the Google Cache of Twilio's International FAQ:
> You may send SMS from a Twilio number to an international number and find that it works. However, this is an unsupported feature and service outside the US may change at any time.
I want to chime in here, because this thread is painful for me to read. We are developers, we work with dozens of APIs on a daily basis, and we work really hard to help developers succeed.
I just want to lay out what we're currently doing to achieve our goal of being one of the most accessible API companies in the world, and encourage you to tell us what we can do better. Right now we communicate status via our API, documentation, FAQs, forums, and social media channels like Twitter, Facebook, and IRC chat. We can also be reached by email 24/7 as well as over the phone from 7am to 7pm Pacific Time. Finally, we publish the status of all our services at status.twilio.com
In the case of international SMS, we saw high rates of undeliverable messages starting on August 16th and we have communicated this via our status page, forums, and email support channel. When it became clear to us that we wouldn't be able to restore service, we emailed customers who were impacted (yesterday) to let them know. While the service hadn't been working for awhile, people were checking back with us regularly, and we wanted to keep them in the loop so they could find new providers if needed.
As others have noted here, international SMS is very hard, we're trying to be open and keep everybody in the loop. We're very sorry for any trouble this has caused. You can always reach me at danielle@twilio.com
Well it would have been clearer if it said: However, this is an unsupported feature and service outside the US may change at any time WITH OUT WARNING.
The reason he is upset is that Twilio didn't give him a warning window.
Has anyone noticed an abundance of ranty blog posts (with authors who cannot distinguish between their/there) on HN lately? Here's the issue in brief:
> The email from twilio informed me they had, without warning switched off all international SMS functionality. ...they were able to massively degrade the functionality of our app – an app which not only do people pay for, they rely on.
Which I suppose is interesting enough, but then:
> But when it comes what really matters, being able to offer my customers a reliable service, you need a written agreement.
Which isn't exactly profound wisdom. If you're charging people for a service, you better make sure that you can hold your backend providers accountable for their end of the chain.
Well, since they don't provide an SLA you can't really hold them accountable. Does that mean you shouldn't build services for which you charge money on top of twilio?
Is twilio better suited to internal use rather than for building SaaS apps?
If you give an SLA for your service, you had better have SLA all the way back down the chain. So yeah, it does mean that you shouldn't charge money for services built on Twilio (until they have an SLA) if you intend to provide reliable service yourself.
SMS is the ugly step-child of international communications, a thing that should cost $0, and should be easy has been made in to a nightmare to deploy and a patchwork of carrier related troubles.
I can fully understand their decision to go US only for now, but I really hope that at some point they'll be able to roll this out reliably world-wide, that alone would lay the world literally at their feet.
So many services are now hard to impossible to implement because there is no easy way of doing this, in my own site I would have several applications for it right away but only if I can guarantee international delivery.
I've tried a few over the years, and I've yet to find one that can actually deliver on a service level agreement, would you mind sharing which companies you have good experiences with?
I'll have a look back for the main one I used and had success with. They offered delivery reports, raw data sending etc I never had any trouble with SLA when I was using them for sending out ringtones. Will see if I can find the provider I used. It would have been British.
I work with a mobile marketing company that uses OpenMarket as their SMS hub, and I have been quite pleased with them so far. They provide a consistent API and callback feature for sms that bridges over to the mobile networks.
I invite the author to sample the basket of shit that is SMS delivering services out there before complaining about Twilio.
We have paid for an "SDK" with one provider, they came to our office and installed a hardware device which we also had to buy, and they still can't manage to deliver 10% of our messages. That was in Australia.
After that mess, I personally wrote Twilio a letter asking them if I can help them establish themselves in Aus and the Middle East; which they declined, saying they were getting around to it themselves, which is good.
Bottom line: I would chalk this up as a slight oversight by a bunch of very capable, if very busy people, bless their heart.
They are one decision away from being fucked over by Skype and Google, if those two ever decide to compete with them. So I am rooting for our kids, even if they have to make gut-wrenching decisions like this on the spot.
Remember, just a year and a half ago they were nothing but a RESTful URL that accepted a few XML messages. They have come a long way, imo.
In India, the SMS deliverability is very good. Thanks to fierce competition between "bulk-sms" companies. I personally use a simple, non-nonsense service http://smsplatter.com/ and have been lucky enough to get mission-critical alerts (from servers) delivered every single time.
That only kind of works... Actually, once when we needed to deliver many messages, we tried to get 6 usb gsm dongles. Then we discovered that you can overload 1 gsm transmitter with 6 standard dongles - maybe the equipment was just weird in some way, but at some point messages simply started to be rejected. Another problem is the delay - you need ~4s to get an acknowledgement which seriously limits the throughput. Then there are other problems with mass-sending... there is one 8-bit counter, but I can't remember now for which functionality (either replies, or delivery confirmations), which killed the idea in the end (it was supposed to be mass-notification, so thousands of texts / hour).
Just find someone who provides smpp gateways and save yourself the pain...
Ok right. I wasnt really thinking of bulk mail. Gsm modems work fine though for less frequent usage like sending notifications when the motion alarm is triggered etc..
Twilio isn't really for bulk sending either, is it?
When we launched the Twilio SMS Beta we tried hard to support sending SMS messages to both US and International destinations. When there were problems, we worked with our customers to collect forensic data on hundreds of carriers worldwide and pass it to our carriers partners to debug.
At Twilio we are dedicated to working with top quality carriers and technology. After months of working to fix problems, were not able to deliver the reliable International SMS service our customers have come to expect.
We apologize for any problems this has caused for our customer and we'll work to bring back International SMS service after were able to deliver on the quality we do the rest of Twilio services.
Hi Anthony, Twilio service status available via our public status page http://status.twilio.com/ We communicated degraded international SMS service to customers on August 16.
You bring up a good point that the information might not be readily discoverable. We'll work to make the status page more findable and to extend the API (http://status.twilio.com/documentation/rest) with features such as RSS to let customer subscribe up-to-the-minute status information.
Presumably they have a list of all their customers with contact info. Not saying you should spam them often, but for the message "no more international support"? I think it's worth telling everyone.
As a heavy user of twilio (I work at groupme.com, we're entirely text message based, and built entirely on twilio), I'll say that every time we've broached the topic of international SMS, they've always cautioned us that it's experimental, and so we've explicitly marked our product as such (USA only for now).
That's not to say that we don't want international bad, but we pretty much trust their judgement as to whether or not they say a product is ready or if it ain't. Twilio does good work, and they're amazingly responsive to developer requests/feedback.
So Twilio as an unsupported service which they charge others for it.
They choose to close the service and don't have the courtesy to inform users before that and now say the service was beta/unsupported and they are sorry for it?
That's really lame...
They have the obligation to inform users they're going to close a service, whether it's beta or not.
The answers they gave is not the way to do it, giving an excuse because the service was unsupported and completely ignoring their fault in warning their users shows that they don't give a damn about the users.
There's nothing wrong in coming here and apologize for not giving a warning before closing the service, but they choose not to do it...
No they didn't:
"I'm sorry that you feel we were't straight forward, although our goal was to be completely transparent in our communication with you about the status of this feature. Since the time of launch, we thought we had clearly messaged that international SMS has been an unsupported feature, and that the entire product was in beta status. We had published this fact on our SMS product FAQs, as well as in the public GetSatisfaction forums. Our apologies that we weren't clear enough about that fact, which caused you to roll international SMS features into your app."
This is not an apology for not informing the developer before closing the service...
Just quoting what you quoted: "Our apologies that we weren't clear enough about that fact, which caused you to roll international SMS features into your app."
Reads like an apology to me.
Read on for a bit, they acknowledge that their communications were lacking and have improved on that by making it easier to find this information.
Look, what he's doing his apologizing for not making clear it was an unsupported service.
What he should have done is apologizing for closing the service without telling developers about it.
They can try to deflect this as they want but the fact remains, he should have sent a warning the service would be closed at date X, not closing the service and tell afterwards they're sorry for not making it clear...
The state of SMS is ridiculous. We go through a gateway. Certification was painfully slow, and the cost... OMFG the cost of a text. -It's as if the alphanumerals had to be mined, shipped, refined, and then placed into each message by hand.
I applaud Twilio for what they are trying to do. IMO, they are playing interference for an ugly beast.
This is worrying. We're using Twilio in my app - they've been good to us so far, certainly very friendly and helpful. The service has been spotty at times, but mostly reliable. Most of our customers are based in U.S./Canada, and all paying customers are; we also had a backup SMS delivery built recently. However, providing absolutely no notice (didn't seem like I missed an email from them) of such a significant change in service certainly isn't a good way to go about this.
This retort to Twilio is just as unprofessional as their "poor service."
As others have pointed out, SMS is an unsupported feature and service outside the US can change at any time.
So in conclusion, apart from twilio losing one of there early adopters (task.fm isn’t a big customer, but that doesn’t mean we can be screwed over) they are also damaging there brand. They used to be all about developers.
I think only getting miffed that your OWN service wasn't perfect and then posting to HN about how task.fm was "screwed over" is more damaging to twilio (is it?) than an SMS outage.
The author didn't read Twilio's own SLA and hardened entrepreneurs don't burn bridges publicly everytime something goes wrong.
I know trying to build an awesome web application is hard and its frustrating when supporting services go down, but that is the nature of our beast. We build on top of each other, so we must respect and support each other. Getting mad and cutting off your own hand because you are angry isn't helping build anything.
Google's API has bugs that my start-ups web application runs into all the time. Sometimes they change API behavior, which the visualization screws up the entire UI in the process. This effects thousands of our users and we have to release work-arounds as fast as we can. But I don't blame Google - it was our choice to use their APIs.
Generally, this sort of thing will continue to happen with all web service APIs. As a community of developers, I would ask them why it happened and provide constructive feedback about how important it was. You might just find they'll listen and you'll have more pull.
Great start-ups have a short list of early adopters or engaged users that they trust and continually go back to for product advice. Writing a post like this pulls the rug out from under your own service because you refuse to use it, it will cost you more, and twilio will suffer from lack of valuable constructive feeback.
Really just my POV, but hopefully you'll find why posting inflamatory articles like this only hurts entrepreneurs and our community at large. At most it sets an example of how we treat each other.
It's 11:24 in phoenix. I just finished a 6 hour sprint working on an app that will rely 100% of the Twilio api, so the headline caught my attention. By the time I came back both the CEO and CTO gave an explanation and apologized. I can respect that. I hope Task.fm gets back on track as well.
www.tropo.com and www.teleku.com does both international voice and SMS I believe. There's also www.cloudvox.com, but I'm unsure if they also support international.
Yes, Tropo does international voice and SMS. We've got numbers in 36 countries and speech recognition in 9 langauges.
If you buy a number from us outside the US, we can't do SMS on that yet, but we're working on it. In the meantime, feel free to send texts to and from anywhere from your US number. We won't even charge you any extra for the international sending.
Teleku is now owned by Tropo and as such can do international SMS as well. CloudVox doesn't do SMS worldwide, but I was talking to one of their founders recently and he mentioned they're working on it.
I have founded Nexmo.com, Low Cost SMS API (in private beta), early this year after spending 7 years in the Wholesale SMS industry. Delivering awesome SMS service on a global scale is tricky – but it should not be! We just finalised building our back-end SMS network.
Here are some challenges the industry faces:
———————————————————-
-Many intermediaries to reach end-users:
Between an App and an end-user you can get up to 4 or 5 intermediaries operating their own infrastructure. Adding intermediaries not only increases cost but also exponentially reduces quality – QOS is measured in terms of availability, delivery ratios, speed, features supported, etc…
Ask your provider about their sourcing strategy? Who do they buy from? Do they have a strict policy in terms of suppliers’ selection? Can they share delivery ratios?
-SLA does not really matter:
What if the providers of your provider do not guarantee any SLA?
A provider can potentially guarantee an SLA on its own infrastructure but can’t control what happens afterwards.
-Non transparent fees:
Customers get charged on SMS sent to networks that are not or poorly covered but listed as LIVE in the provider’s reach list – even when you sign a written agreement.
-Tricky to reach all Networks:
Some networks in the world are tricky to reach and sometimes the only reliable way to deliver SMS is to connect directly. Other are installing SMS filters to cut certain type of traffic (e.g. Spam). For instance, if your provider mentions networks such as Comcell Colombia or IAM in Morocco, you’d rather check again because these networks are impossible to reach with full feature support/reliable service without direct connections. Currently there are at least 26 networks that are tricky to reach and 22 that are filtering.
Another rule of thumb is to check whether in Europe their prices are above $0.055, since European Wireless Carriers (also applicable to other regions/countries) require a termination cost for reliable SMS routes. If below that price, it could mean that your provider (or the provider of your provider!) is using unsolicited routes that are not reliable/has limited life time.
———————————————————————————————————
Delivering hassle free (and Low Cost!) global SMS termination is not obvious but we are working on it :)
Tony Jamous
twitter: @nexmo
e-mail: tony.jamous [@] nexmo.com
I was wanting to use Twilio for UK SMS sending but the system couldn't send to my own mobile. I emailed again a few weeks back and it was still "unsupported".
I'm glad it didn't work on my own mobile as I would have definitely integrated it into my site and no doubt run into the same huge problem here!
Presumably, Twilio knows who their top users of the international SMS feature are. Why not just look at the logs and email the top users, telling them the service is going to be discontinued?
This isn't rocket science, people.
Sorry buddy; your post is basically a hit piece about an experimental service which stopped working. It seems they sent out an email and marked it on their status page.
So I'm going to categorize you with the other startup people who have pretty blog and hurt feelings for your own mistakes.
This is an important lesson for any startup. Don't rely on free services, insist that you pay for them and set up an agreement.
But it is also a lesson from the other end. Twilio would probably not have done this lightly. By not offering agreements for there free service, they had the option of choosing to disappointing their users instead of loosing what might be a lot of money. It's never the easiest decision but it in some cases it is the best option.
--
Hi Anthony,
I'm sorry that you feel we were't straight forward, although our goal was to be completely transparent in our communication with you about the status of this feature. Since the time of launch, we thought we had clearly messaged that international SMS has been an unsupported feature, and that the entire product was in beta status. We had published this fact on our SMS product FAQs, as well as in the public GetSatisfaction forums. Our apologies that we weren't clear enough about that fact, which caused you to roll international SMS features into your app.
We do pride ourselves on putting developers and their applications first, and in being open in our communications, product capabilities and limitations. Please feel free to email me at jeff@twilio.com, I'm always open to feedback, questions, comments or concern.
Sincerely,
Jeff Lawson Co-Founder & CEO Twilio.com