Our domain was abruptly blocked by our registrar this morning. Our NOC team and myself tried to get in touch with them and they tell us "Contact our legal". Even I could not get in touch with anyone beyond their phone operator. The domain was restored, but as DNS takes time to restore, we are still facing issues. They later claimed there were abuse complaints about Zoho.com emails (which is our personal email service with millions of free and paid users). We received a total of 3 complaints from them and two of them have been acted upon and one is under investigation.
Once we dig our way out of this, we will find ways make sure no one takes down our domain again this way.
Just FYI, I'm one of the maintainers of a mid-size forum regarding opensource virtualization/containers and thus spam is a daily occurrence.
While the fight against it is rather dire and no end will ever be in sight, I'll nonetheless never stop (tool assisted) fighting.
Anyway, @zoho.com addresses used by spammers started to pop up circa a month ago and increased rapidly in occurrence. As we use stopforumspam to report and track spammer info (and surely are not the single forum seeing those @zoho.com domains) you may got a few flags raised somewhere.
Not sure what caused this sudden (from our POV) attraction of spammers using zoho, you may want to look into some defense against this. While a full solution may not be achievable it's often enough to be faster than other providers, aka the tiger defense ;-)
It sounds like the spammers found a way to automatically create new @zoho.com email accounts, and the single way to stop them might be using a CAPTCHA service from the direct competitor, Google. At least that was the unfortunate case for the privacy focused German email provider Mailbox.org[1]:
> We recently detected activities on our servers where bot nets were used to create hundreds of thousands of e-mail accounts for the sending of spam e-mail. Although we take this as a compliment – somebody out there must be convinced our infrastructure is up for the job – we needed to find a solution to stop this abuse of our service, of course. We subsequently deployed a number of different CAPTCHA systems to help our servers identify bots during registration. However, spammers were able to circumvent all these solutions shortly after they were put in place. [...] We therefore decided to use Google’s CAPTCHA for the time being, because out of the set of solutions we tried thus far, this one seems to work best.
If you’d like to use a strong captcha approach without using a competitor you might want to check out http://funcaptcha.com (I have no affiliation, have heard good things and been presented it on a couple of sites)
The "Book Demo" button and "read white paper" button seem to be broken, which does not inspire much confidence. The first button just takes me to the bottom of the page while the second button does nothing.
Not sure if this is the same, but I once came across a website with a captcha where you had to rotate a dog so it stood upright, but it was lagging so bad that it would skip several frames, making it impossible to time the angle correctly. After several minutes of trying I gave up and went to a different website with an inferior service, but which did not waste my time.
Both buttons work for me on mobile. Can't be sure, but that page looks like a JavaScript heavy "single page app" type situation, so if your JS is turned off that might explain things.
Incidentally, both links just pop up a sign-up form.
I suppose due to the increasing risk of being broken by competing neural networks, recaptcha appears to be moving towards a model based on usage heuristics in v3. This is something that is more easily achievable by a small startup, so I hope to see competition for this type of solution if there isn't some already.
> recaptcha appears to be moving towards a model based on usage heuristics in v3
I always thought that Google has a huge competitive advantage here, because most people browse the web being logged into their Gmail accounts, and, therefore, as with Google Analytics and Google Adsense, Google knows that it's you who is viewing that page. It can then present extremely time-consuming CAPTCHAs to anonymous visitors, most of whom are likely to be bots or the spammers themselves.
...or running a logged off browser with cookied restricted to the browser session. I spend my time solving captchas which I am getting sick of. My immediate reaction now when presented a captcha is to browse away.
That is pretty terrible if the web is being split into "google knows who you are and approves of you visiting this website" vs not being tracked by google and being treated as a second class user.
Using google with a vpn (PIA) was a non-starter. I usually had to solve 3 or 4 puzzles before I could get to results. Privacy is important to me and it is just as important for them to deny me it.
Interesting--I'm trying Nord right now and while Google has been fine, Amazon blocks me regardless of what I do and I ended up having to add some static routes for Craigslist.
They are not treating you as second class citizen, they are saying they haven't trust you to be human yet. Which is the whole point of capcha.
You want Google to not know about you. You want to be a stranger to them. And you are complaining that they don't trust stranger, which you want to be, as much as someone they know?
If it's about using only Google's services, then yes I agree, but the point is if lots of random sites all decide to use Google for captchas.
This has already happened with tor and Cloudflare, but at least that changed for the better recently (see https://www.zdnet.com/article/cloudflare-ends-captcha-challe...). In that case it was just one CDN using captchas to discriminate against a group of users, so that one change by the CDN could fix the issue. If too many random sites are independently blocking or slowing down anyone not logged into Google, then that'll turn the web into Google's web.
I can relate to what the previous poster said. The worst thing is that this happens even for services I pay for. Some of them even do that for logging in.
Yeah I'm with you. I like to browse with everything logged out, and I clear all content on browser close.
I'm so bloody sick of helping Google train their self driving cars. I swear I'm going to have PTSD about stop signs and store fronts for the rest of my life.
I've started deliberately getting them wrong - probably won't make a difference, but makes me feel a little better about it (I drew the line when I started getting CAPTCHAs on services I have paid for).
>> I'm so bloody sick of helping Google train their self driving cars. I swear I'm going to have PTSD about stop signs and store fronts for the rest of my life.
> I've started deliberately getting them wrong - probably won't make a difference, but makes me feel a little better about it
I don't even have to try anymore to get them wrong on a regular basis. Now, I think it's now more like training Google users to make the same recognition errors as its self-driving cars than training the cars to do a better job.
Ditto here. And some of the worst offenders are retailers! You're trying to get someone to spend real money, and you think it's a good idea to make them screw around with 20 picture puzzles in a row before they're able to do that?!
I can only fathom these shops, both management and the webdevs, have no idea how unprofessional their site looks to anyone that isn't using a vanilla ISP connection. And my experience is coming from using a single longstanding VPS address, not even a shared VPN.
A sensible scheme would allow a certain rate of login attempts per any IP before hassling a user, but Google is obviously more interested in getting their training data than making sure you don't lose customers!
As a network engineer for an ISP, I can tell you that StopForumSpam reports generally don't make it on our radar. Cisco Talos IP reputation, SpamHaus, SpamCop and various other DNSBLs do make it on our radar and are proactively monitored by most responsible ISPs.
That being said, the proper way to report abuse to an ISP is to email the official point of contact for abuse associated with their IP netblock. In the case of Zoho, that contact info can be found here: https://bgp.he.net/AS2639#_whois
ARIN rules require that all IP netblock owners provide a valid point of contact for abuse issues. ARIN validates the points of contact annually. I believe that RIPE, APNIC and LACNIC have similar rules.
If an ISP doesn't act on the abuse after it has been reported to their abuse point of contact, then you have a legitimate complaint against them.
Hi
Sorry for the issue caused to you.
Can you provide few email address to abuse at zoho.com, so we would take appropriate action after investigations.
Regards.
Rajasekar
Zoho Abuse Monitoring Desk.
Thank you for your notification, will check on this and block those who spam using our system.
However please put up an email to abuse at zoho.com so it would help us provide clue to our investigations.
Reg
Rajasekar
Zoho Abuse Monitoring Desk.
If you have 40M users I suspect the annual cost from the registrar is very small part of the budget. Get a registrar where you don't have to deal with a phone operator.
I work in this industry and it's a very clear separation between bulk registrars and those that maintain fewer but high value domain names. The latter usually give you a personal contact person to call and work proactively to deal with threats to companies' domain names and trade marks. I don't think I have ever heard of a domain being abruptly suspended by such a registrar.
The cost is usually 5x-10x that of the cheapest registrars so there is naturally a balance to be struck, and as I work in this industry I might be a bit biased. However the damage when waiting on the TTL when registries update NS records sounds very substantial when they first suspend and later restore a domain name in what sound as a very reckless behavior.
Yes, that is good advice. We are reviewing all our processes about domain registries right now. Major lesson learned, and I would encourage other companies to think this through and learn from our experience today.
I learned this the hard way just a few months ago with Namecheap. Those guys dumped all of my personal information to some people (my name, address, phone number, etc.). I have kids in my home and all they offered me was $100 in Namecheap credit, which I didn't accept out of principle. I spoke with a lawyer and the privacy laws in the U.S. seem to make it not even worth going after them. Registrars basically can do what they want and it's hard to hold them accountable.
I almost has a domain frozen with namecheap after one warning.
If I missed the warning email or checked my email after 24 hours they would have completely suspended my domain. I'm talking about a site with MILLIONS of visitors per month and ten thousands of posts per day, not some small blog.
I did some work for a client in 2017 who was starting a cryptocurrency business. This involved buying a domain name for him to transfer to him later.
Well in 2018 there was some internal strife in his business that ended with a lawsuit being started. The opposing party started sending subpoenas to Namecheap asking for all information from 2018 onwards in relation to his account. What ended up happening was they released all of my information about my purchases, domains, personal information(anonymized credit card info, my actual physical address, information about my other unrelated clients domains, etc.)... going back to the start of my account.. several years worth of data prior to 2018. All clearly out of scope of the subpoena they were served.
Not only that, Namecheap never notified me of this.. in violation of their own privacy policy. They're supposed to notify their customers of the release of their information in relation to subpoenas by email or certified mail. Instead I found out much later from my previous client when he was given a copy of all of my information. And presumably his opposing parties in the crypto space were also given all of my information.
Seems kind of messed up to release all of that erroneously, without warning... especially to shady people in the crypto space.. you know, with people getting kidnapped over this stuff.
TL;DR Namecheap will drop your info, even if you paid to protect it as soon as they're given a single demand letter. And they won't stop at just giving up the info that's asked for (with 0 fight and 0 notification to you) there's a chance they'll release ALL of your account information.
Cloudflare Secure Registrar - I know you guys probably in some ways compete with Cloudflare, but maybe give them a call. Or for that matter become your own registrar and get into the corporate registrar business. With this experience under your belt, no doubt you'll crush it!
FWIW, CF's registrar is nice, but also represents an extreme form of lock-in on the part of Cloudflare -- the registrar subscription is specifically tied to your enterprise plan and will be terminated if you are not using other CF products.
Oh, fantastic! I'll let my former colleagues know, assuming no one else has reached out to them (this was a pretty specific piece of feedback we had re registrar, so great to hear that it's changed).
He didn't own the name, he found a way to change the DNS records; while being registered at MM, google.com is still pointed to Google's own DNS servers.
They're a reseller like everyone else. If I'm not mistaken they actually use eNom for customers buying domains on any of their platforms (though not for their own domains).
I get emails for a friend's domain that was originally registered through Google Apps (G Suite) many years ago, and I see emails with "enom" in them going back all those years.
Cloudflare Secure Registrar. Few people know that Cloudflare operates a registrar, but they do. The pricing is $enterprise, as it should be:
"Cloudflare Registrar is the highest level of registrar security. It protects your organization from domain hijacking with high-touch, on and off-line verification of any changes to your Registrar account. Cloudflare is an ICANN accredited registrar providing secure domain registration for high-profile domains."
I don't like to give recommendations since it either mean promoting the company I work at which just feels like mixing professional and private, or promoting competitors which just feel worse. Instead I prefer giving general advice on what to look for when picking a registrar.
Having a personal contact at the registrar for example might sound unnecessary, but it means that a person at the registrar should know the company involved and the impact of the domain or domains before any serious action like suspension are made. In large and bulk like registrar this isn't the case and as such no one likely knew what Zoho.com was or how many users it would effect. It was likely just an other $10 annual fee among millions of other domains, and as such it is very easy to just suspend and forget and later try fix any issues if those are raised. Cheap and quick solution but very costly if the owner values the domain name above that of $10.
The cheaper, and easier way, if you're looking to start selling domains with a lower barrier to entry (but less control over how much you pay/how you sell your domains) is to find a white-label reseller registrar.
The first aspect is that every* TLD has it own registry and system. For the generic ones you got ICANN accreditation process, but there is also a bunch of registrar reseller that act as a middle man between ICANN and other registrars.
Usually most processes involve some form of capital investment and/or technical capability. Country specific TLD can either be easier or much much harder depending on which country.
Thank you for bringing this up. It was due to our greylist setting for *@zohocorp.com domain, we have now excluded the greylist for abuse addresses.
Please resend your complaint to our abuse address.
Regards,
Zoho Account and Abuse Monitoring Desk.
Zoho has 40M users and apparently $350M in revenue. Why are you using a consumer grade domain registrar[0]?
The gold standard for any enterprise is MarkMonitor. You can pick any other enterprise level service which would mean you don't resort to lowering yourself to begging on Twitter to find a contact at a pivotal service provider
This has damaged you beyond DNS propagation, I don't know how anybody in tech is going to take you seriously again without some serious action
Yep, this incident shows deeper problems. As an outsider, I now question their security team, their devops, their entire company and internal policies.
This should be higher up in the comments. DNS is a seldom thought of security / point of failure. Dyn's whole business model is basically: we won't turn you off until we talk to you.
Mirai was an extremely rare event. I understand businesses were impacted, but it's unfair to hold a three-year grudge against any Mirai victims who are otherwise responsible infrastructure operators.
This happened to us (Weebly) years ago when we had godaddy as our registrar. I highly suggest you transfer your domain to someone competent like Safenames or MarkMonitor.
Transfer your domain to a major registrar. Tierra.net looks like some bs cheap registrar and doesn't have any social media updates on their accounts since 2017. I'd recommend Namecheap.
When I worked as an SRE at Stack Overflow we used name.com for all our domains (and R53/GCP/Azure for DNS). Never had any issues, and worth adding to any short list you come up with.
If you do whois lookups against the top 50 websites you'll see a lot of them use a small set of registrar's. But not all of them accept small businesses.
I use name.com for all my personal domains because it's cheap and supports a lot of unusual TLDs. But I would never trust a $100M company to it. Who cares about saving ~ $100/year.
Probably because you want enterprise grade support, a real person that you can call and will help you solve your problems without having to deal with low level support before.
I almost has a domain frozen with namecheap after one warning.
If I missed the warning email or checked my email after 24 hours they would have completely suspended my domain. I'm talking about a site with MILLIONS of visitors per month and ten thousands of posts per day, not some small blog.
Perhaps. But surely they've run into some sort of technical issue from time to time. Isn't posting such to Twitter a reasonable expectation? I mean, if they don't want to proactively communicate with customers, maybe they have a culture where they don't want to hear from customers at all? Hello Google ;)
I don't recommend Namecheap. A few months ago they dumped all of my private information erroneously, including physical address, for a whois guarded domain. They admitted to it too and all they offered me was $100 in Namecheap credit.
Spoke with lawyers and from what i was told in consultations there's basically nothing I can do about it.
TL;DR Namecheap will endanger your family and they give 0 fucks.
Handshake.org is trying to solve this problem for good by decentralizing DNS at the root TLD level. I'd look into this if you want to make sure no one takes down your domain ever again.
Disclosure: we're building a registrar on top of Handshake. We can also help you claim "zoho" on Handshake for free if you're interested.
I'm sure that Zoho has many talented engineers, but to manage abuse on the scale of 40M users you might benefit from engaging with one of the firms that specializes in this area.
We(Gridmarkets) use multiple Zoho services and are a very satisfied customer. Would like to say we understand and stand by you as you sort this issue out.
> Once we dig our way out of this, we will find ways make sure no one takes down our domain again this way.
Would look forward to an official email with regard to what steps were taken to mitigate this going forward.
Thank you for responding to this quickly. I saw this just a little while ago; I use Zoho Writer and Show for presentations and team-based doc editing and I have for the last decade. If Zoho goes down, I'd be very much lost. Thanks for providing a great service for this many years and I hope it keeps going for many more.
This event seems to have been triggered from abuse complaints - and involved the registrar not reaching out to the client in question.
Curiously enough, I had a very similar incident with Namecheap last week: an unsubstantiated email (without subpoena, judge's order, or even validation of who actually sent the email) - was sent to namecheap abuse /alleging/ (correct, no proof) trademark infringement.
Namecheap rolled over and provided all information to the third party - and didn't bother to inform me of the incident. The only way I found out was a menacing legal letter using the address that I have on file at namecheap.
If Namecheap doesn't respect due process (ie, requiring legal documents to turn over customer information) or customer privacy (Hi, we have just had to turn over information) - on a 10+ year customer, I'm not sure that you're in a better position than Terra.
"Upon the receipt of a valid criminal subpoena, unless the circumstances or subpoena warrant otherwise, Namecheap may promptly notify the customer whose information is sought via email or U.S. mail"
Two things seem unclear:
1) The phrase "unless the circumstances or subpoena warrant otherwise"
2) The use of "may" in "may promptly notify the customer". Why is that not "shall" or "must"?
I believe that's for criminal subpoenas. For civil subpoenas they actually change #2 to "will." However in my experience they never notified me.
"Upon the receipt of a valid civil subpoena, Namecheap will promptly notify the customer whose information is sought via email or U.S. mail. If the circumstances do not amount to an emergency, Namecheap will not immediately produce the customer information sought by the subpoena and will provide the customer an opportunity to move to quash the subpoena in court. Namecheap reserves the right to charge an administration fee to the customer by charging the customer’s Namecheap account."
Email sent. I'd love to be mistaken on this. As re-iterated in the email, the email + address used in subsequent C&Ds were to a personal address only used in NC.
Don't take Ted up on his offer. Namecheap released all of my personal information erroneously and all they offered me was $100 in Namecheap credit.
This company literally has 0 morals and doesn't care about making sure people are treated right. Also, good luck getting through their regular support. It's straight from a script with 0 deviations.
I run a forum site with MILLIONS of visitors and about 5,000 TB of traffic per month. Namecheap.com suddenly sent me a link warning that they will suspend my domain completely within 24 hours, if I did not delete two problem images (which were inappropriate/troublesome images but in the context of the forum posts, "a very poor attempt at humor"). I deleted the images and avoided being suspended, but the way they threatened to suspend my domain due to two images was ridiculous. If I missed the warning email or checked my email after 24 hours they would have completely suspended my domain. I'm talking about a site with MILLIONS of visitors per month and ten thousands of posts per day, not some small blog.
They may be suitable for some blog, but I can now say to NEVER use them for any enterprise site.
btw, I just noticed that Zoho.com domain TLS certificate expires next year. Hope you have automatic checks for the timely renewal.. I have been a fan of Zoho and hope you make a comeback.
I highly recommend AWS Route53 domains paired with their DNS service. Pay for the AWS support plan so you can call. I suspect Zoho is a multi-million dollar company at this point, should not be using a discount registar.
If you’re providing email service, you should be actively monitoring public blacklists, not waiting for your registrar or hosting company to notify you. Even if your domain isn’t banned, your users’ emails may be bounced by other servers. That you don’t seem to know any of this means you aren’t employing the right people.
I'm pretty sure you're over inferring stuff from that post. It's not credible that 20+ years old company serving email for millions of users wouldn't know the most basic stuff about running an email server, don't you think?
Zoho customer support is worse than useless. They’ll happily call you back only to tell you that they can’t help because it’s someone else you need to speak to. It’s a real shame because the Zoho product line is vast and with more focus it could be brilliant. But every part of it has little issues that will never get fixed because it feels like there’s nobody within the frontline staff seems to want to really understand.
EDIT that all sounds more negative than I wanted it to. I think Zoho should charge people more and offer the service to go with it all while fixing the niggles that make their products feel flakey.
Ironically zoho for some bizarre reason would not let us update our company credit card. Calls and email to tech support were useless. We ended up having to just move to another service since we had no way to pay.
So zoho is quite happy to terminate their users with no recourse.
I can't help but wonder if the 'free email' thing hurts the company more than it helps.
A couple of years ago my parents wanted to trial individual email for their very small business (just a few employees). Google had recently removed the free tier from Google Apps, and Zoho was the only company we could find that offered a similar service. I set it up and they used it for ~6 months with quite a few issues in that time, whether service outages or (more commonly) emails not being delivered due to Zoho's spam filter or the recipient's spam filter giving false positives. Customer support was near enough to useless.
So when we decided that it was worth upgrading to a paid plan we didn't stay with Zoho, due to the poor experience. If Zoho charged just a dollar or two for accounts it may help reduce all the problems they have with spammers abusing the service and the flow-on effects that has on their customer support.
I am using Zoho Reports to generate some daily PDFs from a DB. If they could just make it so I could copy reports and point the reports at new queries it would be fantastic. As it is, I just want to throw the whole thing out and will eventually make something to replace it.
I tried to become a customer a few years ago, but could not get access to their services despite two weeks of back and forth with support. I simply gave up and assumed it representative of what was to come later.
As a small start up when we ran into a similar problem while using google domains, they gave us a very hard time with bad support...something we could not afford then. I dont have a good alternative but wanted to mention this experience with google domains. Hope that helps.
Yes, all of these possibilities are under investigation. We have just recently secured ICANN approval to be a domain registrar. With our scale, this has become important now.
It is more likely Zoho is getting into/already in the business of selling domains and hosting as part of their portfolio and that's why they are becoming a registrar.
This isn't really a problem for companies the size of Google - while they may well refuse service to competitors or prohibit usage via terms of service, if they do allow a competitor on board there's no way they treat them any differently - there will be huge legal ramifications of they do.
For most of history Google offers essentially no support. Recently Google has started making phone support available for Google Ads (AdWords) putting a contact number on their customer facing website.
Maybe domains registered more recently work differently, but my Google domains use a random user account generated by Google for eNom (the provider they were contracting with at the time). That makes each domain it's own virtual customer (I couldn't just login to eNom or Google see a list of all my domains). I need to log into Google Domains as a separate account for each domain, and then that takes me to GSuite which links to eNom.
Also, last I checked, unlocking a domain for transfer to another registrar required emailing Google/eNom. There's no interface for it. For a while the entire UI to choose to cancel a domain just disappeared as well.
Our domain was abruptly blocked by our registrar this morning. Our NOC team and myself tried to get in touch with them and they tell us "Contact our legal". Even I could not get in touch with anyone beyond their phone operator. The domain was restored, but as DNS takes time to restore, we are still facing issues. They later claimed there were abuse complaints about Zoho.com emails (which is our personal email service with millions of free and paid users). We received a total of 3 complaints from them and two of them have been acted upon and one is under investigation.
Once we dig our way out of this, we will find ways make sure no one takes down our domain again this way.