Wait.. they are saying the app itself is making NTP requests?
> Confirmed - starting up the iOS Snapchat app does a lookup to the domains you listed, and then sends NTP to every unique IP. Around 35-60 different IPs.
Hmm. Is that a fraud prevention thing or something? No way on earth a user app should be getting its own time
Snapchat have a lot of "fraud protection" in the form of time sensitive tokens hashed with secret keys generated by strange .so libraries. This is used to keep third party apps from using their API, obviously a lot of user's devices have incorrect clocks, so when they reduced the secret token time frame lots of users probably started getting API errors, so this is their attempt at a solution.
See my other post here, and the problems in the (third party) iOS NTP library "features" and its use.
There "createAssociations" in that library without any parameters contacts all the IPs behind the big domain list of the NTP servers!
And according to the forum every IP is contacted -- behind one server name there are 3-4 servers in the DNS in this case, I get 31 server! Real "distributed denial of service" attack.
Yes, thank you, the third party library that Snapchat used without even thinking what it does by using some (wrong!) defaults. I thought it was obvious from the links I've given. I've edited my posts to name it as such.
I've never used Snapchat but I believe one of its features is time-expiring photos. If they do the expiration in the client then this may be a way to check if a user is getting around it by setting the system clock backwards.
NTP doesn't cryptographically verify the time. All you do is have your router redirect these NTP requests to your own server, which is set to serve the wrong time.
Ironically, part of an HTTPS handshake involves sharing the server time in a cryptographically-verifiable manner. I am not sure why they don't use that! https://github.com/ioerror/tlsdate
Optional for the implementations. So basically your ssl library is unlikely to do it. And at that point having a "getTime()" API call in your service is simpler than having a custom patched SSL implementation.
The third-party library used by Snapchat didn't have any "maximum of servers" (using 30 at once(!)) and defaulted to many in the ntp.org pool, across all the continents!
If you want to prevent users from altering their time use your server and do a time compare with your server.
NTP can be easily intercepted and altered so it would make a lot more sense to do this via a encrypted certificate pinned communication path increasing my work load drastically to alter the time.
I snapchat going to pay for the DDOS they created?
Stupid decision by SC for sure, but is there a reason there is no rate-limiting on the NTP servers? I'm not up to date on their structure. Maybe it's just not feasible because they don't synchronize clients?
The NTP servers are all run by random members of the community. There's no real coordination between them (outside of being part of the pool), so you can't really rate limit between servers.
The library was only doing a couple request per NTP server, so rate limiting really wouldn't have helped.
Even without thinking about storing some context, replying to NTP request is probably similarly expensive as evaluating whether it should be rate limited.
Also, NTP runs on UDP rather than TCP, and it only requires one packet from each party to exchange time, so it's harder to rate-limit it without making your server unreliable.
The way they achieved their goal was misguided but what you've described skips over large parts of the problem. Why would the server time and phone time be in sync?
They could just use their own canonical time (from their server) instead of hammering NTP.
It doesn't say anything about synchronisation between phone and server.
EDIT: In fact it is easier to implement it this way than using NTP. I've implemented something similar and I found it easier to add an API endpoint that returns time() than to ship an NTP client...
But then you need to manage a server and ensure that the time is running accurately. Why add such a large level of responsibility for little to no gain?
How is that extra responsibility? Do you think Snapchat aren't already running servers?
And you don't care whether the time on the server is running accurately if you're just using it to generate tokens which are checked against the time on the server. It just needs to be consistent, it could be an arbitrary counter and would still work fine as long as it counted up reliably at consistent intervals.
Because instead of depending on a 3rd party, you depend on yourself (and in this case you'd hammer your own server, and notice). It is not far fetched; Windows and OS X work this way. By default, they use their own NTP servers. OS X even has separate servers for different regions (Europe, Asia, and Americas).
Because that's the point of NTP; internally both operating systems use UTC, and in practice most clients can be in sync to within 1 ms or so of the true time, given reasonable connection speed and quality.
For whatever reason, ntppool.org is blocked at my work.
And of course, you don't get the page that states why when the website is served via https. Not that I need to see the page to know it was either blocked for "hacking" or "entertainment", and I'm guessing it's not entertainment.
Edit: This probably explains why our clocks have been off by 45 minutes since Monday. I guess it will be entertaining to see how long it takes for IT to figure this one out.
I got told off for diagnosing issues in the past. The IT director is a megalomaniac and interprets it as a challenge to his power. The only time I offer suggestions now is when one of his employees specifically asks me for help.
Edit: I realize "got told off" didn't really capture what happened. I came in early one day and noticed we were having a dns issue. I manually refreshed my DNS cache and it started to work. I sent him an email to let him know that the DNS cache was expired. He told me I was out of line and complained to HR.
I had to go meet with HR, which was pointless since they think he is on a power trip as well. Anyways he added a line to the IT policy that specifically prohibits "performing a diagnosis on the network or any of IT managed systems."
I got into trouble at university when I was running a CAD session on X (Cadence VLSI design FTW, not). Some asshat had telnetted in and was trying to brute force root because it was a faster machine than the crappy sun4's dotted around. It was spewing all over the framebuffer. So I logged into another box and sent him an email saying pack it in and that I was trying to work and that I'd report him for AUP violation to the Sysops.
He complained (?!!!) and the next thing I was in front of a tutor getting a bollocking for it. No explanation was allowed to be returned or appealed, permanent black mark on my record.
And that's when I learned about university politics, gave them the fat middle finger and got a job and left a massive 11 page long diatribe about the charlatans at the place.
I sent a complaint to my boss and HR about IT recently that was around 30,000 words. It was basically 500 words describing the level of incompetence of the IT director, the remaining 29,000 words were just supporting evidence. I'd say it was a massive waste of time, but the IT director has basically made his expertise unassailable, and the only higher authority he recognizes is Microsoft.
I basically went through all my tickets and emails, took IT directors claims, read the MSDN articles on the topic, and pointed out all of the places where what he said was not only wrong, but grossly wrong. Things like "sometimes databases lose data." That's funny, cause I'm sure the team that built SQL Server 2012 would have something to say about that. Why don't we look at the documentation on ACID principles.
I've come to expect the IT director to be a moron at this point, and I had been trying to roll with the punches. However, my work has been going missing, and I got in trouble for it. He denied losing it, then he blamed the database for his incompetence. I was so angry I was awake for 3 days straight cross referencing everything.
The lost data wouldn't have even been a problem if he had just told me within a day or two. I only find out it's missing when we try finding it weeks or months later. Then I have to waste my time doing a forensic investigation in to the scope of the problem.
The most annoying this is that the last person in this role never had any of these issues because nobody ever audited our data. I have managed to instill the idea that data can be 100% correct, and we should always be checking our data to make sure it's right. So now I get blamed anytime information is missing or incongruous.
I know that otherwise well-reasoned emails seem like a rant once they reach a certain length. I kept the body of the email to 500 words, and included a pdf of supporting documentation. I let the email sit for a week before I sent it, and had another manager read it as well to make sure the tone was alright.
My entire argument was contained in the body of the email. I knew that was all it would take to get my boss on my side.
The PDF was aimed at HR. The IT director has a lot of power under the IT policy, and he has used it in a retaliatory manner in the past. My goal was to stop the IT director from retaliating so I could do my job while I look for a new one.
I write my emails as if they could be leaked to the general public at any time, and I'm certain that the email wouldn't reflect poorly on me, even out of context. Maybe there was a better course of action, but I don't think I did much to hurt myself.
As unfortunate as it is, most people, especially us engineering-types, are not good at seeing what drives peoples' decision making. That means we do things like this that only make us look worse under the belief that they will help (including the belief that if it doesn't help now, perhaps it will later; this is slightly more likely, but still extremely unlikely).
The important people often see those things, assume it's an incoherent screed from a disgruntled piece of crap, and delete it without reading (I know from personal experience; such completely accurate and valid diatribes have gotten me fired on the 3rd day of my 2-week notice and, on a separate occasion, a running joke among the big shots for months after my departure, where one would specifically talk about how the email came in while he was on the toilet, at which point he cackled and promptly deleted it without reading).
To anyone non-technical, that letter is all mumbo jumbo. It seriously might as well be in a foreign language. They are not going to check a few of the cross-references and see that you're obviously right. Even though you might hope they'll do this just one or two times, they won't.
They are not going to ask the people called out in your letter to account for your accusations.
They are going to write you off as an unhinged, angry, and worthless nothing-tier employee/student/whatever, make fun of you for a long time, and then forget all about it.
Humans base their decisions on personal trust/credibility. The way to win against an evil IT director is to obtain far more trust/credibility in the eyes of his bosses than he has, and then to use that credibility to your advantage.
That's a lengthy and difficult process, especially when you start out as a subordinate and the boss has a lot more access and ability to frame your efforts to his advantage.
I personally have never had the patience to undertake such political subterfuge seriously and I find such undertakings both incredibly frustrating and soul-crushingly phony and hypocritical, but I am now convinced it is the only reliable way to get real career success and mobility. Thus I accept that any career success I enjoy will be lucky/accidental.
Employment and career IS a popularity contest. Good software is somewhere between the 10th and 20th most important career concern for a developer. The number one concern for anyone trying to make it in white-collar America is to be as well-liked and popular as possible. Most of the time, love of colleagues and love of bosses are symbiotic and they feed off of each other (as long as you're sycophantic efforts aren't TOO obvious), but to the extent that a situation arises wherein someone's love has to be preferred, prefer to get the love of the bosses.
This is the sure path to career success. Disregard truth, objectivity, and practicality. No one cares about you or what you think, they don't care about what you judge to be practical or wise. They didn't really hire you for your experience or insight even though they want to pretend they did. They hired you because they thought you would make them feel and look good.
Not only bosses, but people in general, care only that your presence and actions are generating pleasant feelings for them. Do this reliably, disregard everything that is not this, put only as much energy into the tasks required by your actual job description as you must to be passable in the unlikely event of a performance audit, and pour the rest of your energy into social development. If you're going to make it as a company man, that's the only reliable path.
I don't disagree with you about the dishonesty of living a suck up life, but try actually being poor for a little while. Your tune will change to "Where would you like me to suck?" and "How hard?" very quickly.
I have the advantage to be able to leave a job and find another relatively easily, and I would definitely agree that I'd rather leave then suck up, but for those who don't have that kind of mobility, sucking up and playing politics can literally make the difference in making car / house payments, sending your kids to college, etc.
Yeah, this is also an element that's important not to discount, and can help describe why it's hard for engineers and other high-IQ roles to accept that social compliance is the #1 factor in all ongoing voluntary associations.
We're really spoiled in tech because the field is so abundant. We offend someone or get offended, and we're off to greener pastures within weeks if not sooner. I've come to believe that hopping around like this, which I've done for most of my career up to this point, is not healthy, but the availability means that technical people don't have to learn to conform as well as everyone else to survive. And while that means we may be able to keep a job, it's hard to move up.
These things that we struggle with are just normal life to a lot of people. They had to swallow these compromises early in life when it was apparent that good feelings were all they would be able to offer.
This contributes to the cycle because those people pay their dues to the establishment, go through the process, and get used to the circlejerk. They then expect everyone else to do the same.
When someone wants to come in and challenge some of their thoughts, opinions, or practices, even minor things, in what the challenger feels is a sporting way or a way to drive an interesting and inoffensive discussion, the "good feelings violation" siren fires off in the non-technical person's head.
This brings in a large flood of negative feelings and resentment, including but not limited to jealousy that you can express your thoughts openly while they've always had to kowtow, a sense that you're entitled for thinking you should be able to do this and "dictate from your expertise" (as was expressed about me once) instead of "climbing the ranks" the hard way and then quietly and subtly implementing your opinion after you've won the social clout like everyone else has to do, and a sense that you may represent a threat to the perceived competency of the challenged person (and those least competent will be the most aggressive protectors of this perception) in the minds of the people whose trust they live off, which is really everyone -- colleagues, subordinates, and superiors -- which means it's very difficult to overtly question or discuss anything done by anyone, even in what you believe is a polite or considerate way.
Exceptions are basically not made to this. The potential of substantial data loss, massive security holes, etc., are irrelevant. If someone grossly incompetent like this is on your team, the smart move is NOT to challenge or disprove, because again, no one evaluates proof on any basis except "which proponent do I trust more?". The smart move is to frame the situation such that his failures are opportunities for you to deduct from his social clout and add into your own, without ever firing an alarm in anyone's mind that you're trying to do this.
It's all an image game. As an individual contributor, you can avoid a lot of this game as long as you're non-threatening, churn out semi-reliable work, and are at least not annoying if not socially pleasant. Once you try to move up the ladder, even just one rung, image and likability goes from 65% of the equation to 99%.
Yeah, I definitely agree with that to a large extent. That's why I stated that any career success I personally enjoy will be based on luck and/or accident instead of political skill. I've tried to bring myself to find some way to reconcile these paths and still feel that I'm not throwing away my integrity, but I've been having a hard time doing that. Trying to find that reconciliation has actually been a large part of my personal struggle over the last year or so.
I think the thing is that there are degrees here. A lot of the things that we are socialized to consider "bad taste" are not actually immoral and unfair (for example, aggressive SEO), and you have to meet on that playing field if you ever intend to be competitive. But you have to identify the limits of what's just getting into the nitty gritty of business and what's crossing the line into being a cheater/liar/phony.
There also seems to be some people for whom social camouflage comes naturally. These people don't feel like frauds or phonies when they cater their preferences and likes to match those of the group around them. It's hard to compete with these people because they have no compunction about being yes-men and they have no malfeasance behind it, since they don't really even realize they're doing -- they only realize that they're making the people around them happy. They truly just have nothing original or important to contribute and don't realize that they're mimicking everyone around them, which, as we've discussed above, is a great skill to have on the path to career success.
I think that developing an active, interesting conversational style and taking care to frame arguments in the most emotionally influential way possible without altering, distorting, or seriously misrepresenting them is perfectly in bounds. Most people do this intuitively to some extent or another, and I believe that intuition can be improved with practice and training, and that that's a great thing for anyone to develop.
The problem is that that's about where I stop. Even if there is nothing immoral about dragging yourself to a baseball game with the bosses, it still feels painful and phony to do that kind of thing. Your unscrupulous and/or unaware competitors, however, will waste no time immediately making themselves as likable as possible by adopting all of the boss's favorite things and habits, refusing to criticize him or anyone whom he holds in high esteem no matter how grossly dangerous their actions are, etc.
It's a tough game. Sometimes I try to believe that there's some cove of people that aren't this way out there, but it always gets shot down as soon as I start believing it again.
I'm starting to think that the only bonds that allow real honest participation are permanent and non-revocable bonds like parent-child and sibling relationships. People still get offended in these and rarely they may even fully withdraw, but most of the time everyone accepts each other and has to get over the perceived slights. I'm not sure there are many voluntary relationships (certainly not relationships where the continued relation is predicated upon a regular payment) where people do that.
I encountered a similar IT manager in high school. I kept telling him that netsend wasn't locked down and that any user could run a .bat.
He told me I was wrong. So, I wrote a .bat with a netsend command and emailed it to all staff. Multiple staff clicked on the attachment.
Once they figured out it was me, they made me start a computer club with the IT manager as the supervisor of the club. First order of business was locking down .bat execution.
> Once they figured out it was me, they made me start a computer club with the IT manager as the supervisor of the club. First order of business was locking down .bat execution.
That seems like a great way to handle this situation. Some ignorant other schools might have kicked you to the curb because you were spreading "malware".
Im endlessly glad that my high school was extremely tolerant of us exploring the system and messing with things as long as we didn't try to cause harm (eg, deleting one file is okay, trashing a whole network drive is not) and reported what we found to the school IT manager. Most of us ended up as techs for the school and district as a student job. Several of those students went on to be whitehats. (Who knows where careers would've gone if they'd been discouraged and come to view the system as the enemy.)
They also were way cooler than they had to be about the several times we took down the network or broke the porn filters, or the time we port-scanned a district tech's machine, or had a whole collection of malware on the network drive, or....
I just worry about how students like me would fare in schools these days.
In secondary school I discovered Windows Registry - in particular, the LegalNoticeCaption and LegalNoticeText keys, which let you set a pre-login information dialog. I left a funny message on a computer in the school lab, saying something to the tune of "What a shame it isn't Linux".
Two hours later, I faced an angry teacher (who was also a math teacher and the lead teacher for our class). She said something about the next person after me freaking out that "the computer has viruses". Got a bad note for behaviour, spot-check of math homework leading to three F grades, and she also tried to take away my notebook with notes about Windows Registry -.-
Currently looking. I finally came to my last straw recently.
A coworker sent me an email because some data I was in charge of adding to the system was missing. I looked, and somehow data in our database had gone missing. I use the data to add information to another database, and that database had the information in it. So somehow he managed to lose information in SQL Server.
A few weeks later, my boss brought me in to his office because about 25k rows had incorrect information. I went to check my notes, and all of my notes from a period of 10 days were missing. I had been creating a changed file log (because files have been lost in the past), and I could see I had created notes during those dates, but they didn't exist anywhere.
The only evidence I had done my job at all during that time were the emails I sent to other people. (Thankfully Microsoft hosts these, so the IT director can't mess it up.) I had sent somebody an email that basically said "I found error [x], but I fixed it."
For a little while I was questioning whether I was insane. I mean, I would never have believed it was possible for data that you've added to an ACID database to just disappear. If you can't trust ACID principles, what can you trust?
I've seen this happen as well. Data "mysteriously" reverted and it turned out to be an unrequested database restore. They reverted to a recent backup but didn't mention it to anybody so a few days of data was missing.
That is definitely what happened. One manager found out about this and called him and ended up verbally tearing in to him for like an hour. She is really task-oriented and gets really frustrated with how often I want to change things just to see what happens. It felt really good to hear her step up to defend me, because she would only do that for someone who she really respects.
What kind of firm are you in (and perhaps more important: country/culture)? I've actively been a friendly pita wherever I worked and always found people to help search for answers, even if the question was not really in the line of work. Have a discussion related to tax and payrolling going on right now. Couldn't make sense of company policy even when reading / quoting the law (ianal). Still the surroundings stay friendly (or I'm in massive bubble). Getting into trouble for having a (truthful) opinion, sounds like a showstopper to me. Just don't email the big shot CEO if you haven't got your story straight, that would be my perferred level of openness.
I really like the term "friendly pita". When people complain about tech support being unhelpful, I've often found the opposite. Be genuinely friendly and relentless and I've found even incompentent people will fix things. Which I also genuinely appreciate.
Sincerity helps, both as a shield and as a way to diffuse the inevitable frustration that comes from troubleshooting tech.
Of course, I've got a strong force of will, too, so I tend to mix poorly with charlatains like the grandfather's post. I really feel for the souls who have to work under that jerk.
I'm definitely a friendly PITA. Most of my coworkers get it, though some definitely think that I'm just creating more work for myself. It's really only the IT director who is so defensive, and unfortunately he has the power to back it up. HR goes out of their way to find answers to questions, and my immediate coworkers are really driven.
If you want to sidestep your IT, you can report the site to your filter provider as miscategorized. I'd argue that ntppool.org is not related to hacking. Most of the filter providers have publicly available forms or email addresses for this stuff. There's not a high burden of proof to worry about either.
If a cached DNS record expired it would not be a problem. If a bad or old record was cached before TTL, clearing your cache works to update it. Maybe ITs ego didn't like your terminology of DNS cache being expired, and telling them theirs is- by definition expired records get looked up again because they no longer exist locally. As former IT it was mildly irritating to get unsolicited advice, But still it sounds like a redicilous over reaction on ITs part. I'm assuming you didn't actually clear a local IT managed server's local DNS cache.
Right, the DNS records weren't expired, they were no longer accurate. I'm not sure what the word for that is. The Windows DNS Cache on the local machines was inaccurate. Basically, I couldn't access any websites I had accessed before, but if I tried accessing a new site it worked fine. If I forced a DNS request for a site I had already visited, that site would start working but all the other sites would still be broken.
I knew what the problem was because I had run in to that issue a few years back with my own computers. After I updated my DNS, all the Windows computers were having issues, but none of the Linux ones were. That's when I learned that Linux doesn't typically cache DNS records on local machines.
It's called the resolver cache, and Linux does it too. Most commonly with nscd (use "nscd -i hosts" to clear it, keep in mind this is not specific to DNS requests). In the specific situation of a web browser it's more complicated and you'll find that the web browser caches these things too. If you run Chrome check out chrome://net-internals to get a glimpse of what's going on there.
Thanks for the tip on net-internals. I think I used a similar feature in Firefox a long time ago, but I've never used this tool in Chrome before.
I know Chrome in Linux definitely doesn't cache DNS requests. I believe Linux only keeps the DNS info around until the socket closes, but I don't know the actual implementation. I looked in to the ncsd man page, and it looks like DNS info isn't cached, only open sockets.
I know Chrome uses the system DNS cache in Windows, and my understanding was that all browsers in Windows used the system DNS cache.
I don't know a lot about how sockets are handled. I thought they were discarded as soon as they were closed, but they could function similarly to a DNS cache. I though a DNS cache stuck around for a lot longer though.
I'm currently looking for a role, but I've worked here for 3 years because I have been able to basically do whatever I want. I'm a combination of analyst, data scientist, and marketer. I'm in charge of our appeals, from strategy all the way to the money coming back in house.
We have over half a million constituents. It's really great if you are interested in testing. I've sent out mailings where I'm testing 10 or 11 different factors. I've tested basically everything: the size of the font, the weight of the letterhead, the format of the coupon, how the letter is mailed, the dimensions of the envelope, the structure of the ask, etc.
Even with the added expense of testing, I've decreased our cost to raise a dollar by over 20 cents.
On top of that, I've automated all sorts of processes. At this point, I think my processes save about a man-month per month.
So, I have a lot to feel good about, but I'm really being limited on the technology side of things at this point. Biggest barrier is that I haven't quite decided what I want my next step to be at this point. I'm afraid I'll end up with a job with a much narrower focus and get bored.
Good call. I thought it was possible to message me as long as the email field was filled in. I hadn't put my email in the description because it's my actual name at gmail. I've added it now.
you've got a great skillset. I think you could find a similar role at another company where you weren't being limited by a shitty IT department and internal politics.
Thanks, that's a real confidence boost. My formal education was in biology with an almost minor in computer science. I am a little self-defeating sometimes because I don't have a stats/marketing or CS degree, and a lot of job postings have a relevant degree as a requirement.
The IT may be terrible, but I'm very thankful for the opportunity I've had to develop these skills. It's really unfortunate how one or two people can ruin an organization. The rest of the people here are incredibly kind and incredibly driven people.
I don't know why an org would be blocking NTP. But ideally, IT would run their own NTP service (if the shop is big enough), block NTP to any other org, and sync their server with upstream. Similar to package management, DNS, etc. - self-host it all.
Blocking UDP 123 upstream is a common tactic to avoid participating in NTP DDoS botnets.
You're best off running your own NTP infrastructure that's isolated from the public Internet NTP servers if you want stable time sources. Put a CDMA NTP server in two locations, let them sync, and go. You can get cheap EndRun servers on eBay for a few hundred bucks.
It's tempting to believe telecom has got their act together, but I've found the public Internet NTP infrastructure to be more stable than CDMA/GSM refclocks. YMMV.
It's why you need multiple sources. The NTP protocol and implementation handles sudden skews from a source. When done properly, CDMA clocks can offer a much more stable source than the public internet, and you can filter out UDP 123 upstream.
Yes, but when your IT is incompetent they do the first half (block NTP to anywhere outside your network), then don't provide their own time source. Then try to blame users when they complain that their time is always wrong.
We do have our own time source. Our NTP source is locked to that server via GPO. NTP requests aren't blocked, because I can still set NTP on things like my printer to other NTP sources and receive updates.
I'm willing to bet they don't realize that they've blocked their server. What I want to know is how the drift got to be this bad in a matter of days. It's like whatever computer they are using as the NTP server doesn't have a real-time clock.
"It's like whatever computer they are using as the NTP server doesn't have a real-time clock."
It sounds like their domain controller holding the PDCE FSMO role is a virtual machine. This advice might be outdated but last time I checked it was better to host it on a physical machine. If they're restoring DBs from backups without notifications and managing their Active Directory environment like this I can only imagine how fragile your infrastructure really is. Good luck, you're going to need it.
>It sounds like their domain controller holding the PDCE FSMO role is a virtual machine.
Called it. I didn't even think about the fact that it was a virtual machine until you mentioned it. Everything makes so much sense now. The randomly disappearing files, random issues with file locking. I just assumed the problem was they had no idea what rules to use for file replication.
> I can only imagine how fragile your infrastructure really is.
Yeah, it's a nightmare. Ignoring the "files disappear and are never found" issue, files get overwritten with earlier versions all the time. I'm not entirely sure what their hosting situation is, but I suspect they are running virtual machines in virtual machines.
On top of that, most applications are only available on VMs through a Citrix connection. This isn't inherently a problem, but they have it set up so everyone is logged on to the same machine. This causes issues with most MS Office applications, because they assume one user per computer is accessing the files. So file locking is completely broken, because Windows looks at the file and says "Hey, that's me, I have that open!"
It's the biggest nightmare with Access applications, (which we have a ton of), because Access assumes that everyone has their own copy of the front-end (client) database, and each independently accesses the back-end (server) Access database. Optimistic locking doesn't work at all with the "everyone uses the same file from the same computer" approach, and pessimistic locking barely works. I've set up the client databases to automatically create a copy for each user, but as you can imagine it's a bit of a nightmare to keep everything updated.
Odds are high that someone set up a high-accuracy server on the local network and expects everyone to get their time from that. ...which is not a bad idea if they had the sense to make it known that they've done so.
AD and a few other systems get very snippy if the time various servers have isn't in sync, although they don't have much of a problem if every server has the wrong time (within reason).
"ios-ntp is often (mostly?) used to make sure someone hasn't fiddled with the system clock. The complications involved in using multiple servers and averaging time offsets is overkill for this purpose. The following skeleton code is all that is needed to check the time."
And that "skeleton" contacts just "time.apple.com"
But the library really has the default possibility of contacting a lot of the ntp.org servers from a big list ("createAssociations" with no parameters!) and it's bad.
As we know, the developers like to just "copy-paste" whatever is where. Or use any defaults. "Hey it works."
I think it's pretty safe to assume that the developers have no idea that things like vendor zones exists. NTP, like DNS or SMTP, is ubiquitous infrastructure, not something the average developer sets aside time to understand.
They didn't have to use ntp.org pool at all. They obviously wanted to check if somebody changed the time on the phone and to still use "the internet time." And for that contacting one Apple's server (time.apple.com) were enough.
It seems that the iOS library author "helpfully" provided the default of contacting 30 servers from the ntp.org pool.
I'm guessing of cause, but it's entirely possible that they used the ntp.org pool, because that's what a quick Google search would tell you to do. Getting a vendor zone require that you know it exists.
> I'm guessing of cause, (...) they used the ntp.org pool, because that's what a quick Google search would tell you to do
You're wrong, see my upper comment in this very thread, I've wrote already: they used the third-party library which has as default in the call without any arguments the huge number of the ntp.org servers. The servers hit and the servers in the library sources match exactly. They used that default, resulting in 30 servers being hit at once from a single app (and the app is probably used by tens of millions of people).
" I wrote this library a long time ago for the iPhone 3GS and earlier. At that time, as best I could tell, the iPhone obtained its time from the phone company and, for me in South-East Michigan, it was not unusual for the iPhone time to be at up to two minutes variance from true time. Since my app was predicting the location of the International Space Station and a two minute error represented about 600 miles, I wrote this library so people wouldn't be staring at the wrong side of the sky!"
"Since then, the world has changed. First, iOS devices use the NTP protocol as a time source -- I stopped using my own library years ago because iPhones were delivering sub-second accurate times natively."
"I've thought for a while that this library had passed its "sell-by" date and was of minimal value; this Snapchat incident (about which I've heard only a little, and that obliquely), makes me want to remove it. I've been involved in network software, on and off, for about forty years and regard myself as a good net citizen."
The defaults up to now were really, really bad. His correction now is to remove the list. But if then the users just use any list, they are doing it wrong again. For effectively all the scenarios in practice, just time.apple.com instead of any other list and the limited number of IP addresses used from there, by default 1, should be enough for the iOS apps, or even better that they use their own servers. The swift library with the similar goals, mentioned elsewhere, has that IP-limiting loop, this library probably not.
This is a really weird comment format. It almost suggests a lack of syntax highlighting.
/*┌──────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Prepare a sort-descriptor to sort associations based on their dispersion, and then create an │
│ empty array for associations to fill .. │
└──────────────────────────────────────────────────────────────────────────────────────────────────┘*/
for use of two different styles of boxes, + some additional typographic experiments.
I actually kind of like it. Not enough to start using it yet, but then again, in Lisp code I make a judicious use[0] of ^L characters and form-feed-mode. Form feed character seems to be a forgotten but pretty neat invention.
FWIW my teenage daughter has been complaining about this latest Snapchat update for iOS the past couple days. It constantly crashes and causes the phone to reboot itself. Looking at Twitter, there's tons and tons of people reporting the same issue, so it seems pretty widespread. Wonder if it's related to this NTP issue.
I've seen a test version of an app cause issues such that the launcher would crash (i.e. "respring"). Probably some intersection of particular api usage and OS bugs.
To the unobservant user it looks like the OS is rebooting, although it's much quicker than a real reboot.
psh, iOS can be rebooted from safari by web pages - just send them enough images with 3d transform + filters and watch the memory allocator kill the whole system trying to compose the thing.
Yes, I believe they can. I haven't had my iPhone (6S) crash more than 5 times that I can recall in the 14 months I have had it. Every time it has rebooted I have been in a third party app. The phone begins acting strange, gestures may stop for a few moments then it goes black and the Apple logo shows up on the screen as it reboots.
Two of them have been in the last month or so. Unfortunately the phone doesn't just reboot. It shuts down completely and refuses to boot back up with a low battery warning. I believe this is actually hardware related but I have had it crash when trying to call an Uber. I think Facebook may have caused a crash too.
Uber was especially painful because hailing taxis in my city (Seattle) is essentially impossible and pay phones are not a thing that exist in 2016. Luckily I was near a bus stop and know how the lines work without my phone.
It was a shocking illustration of how dependent I have become on my smartphone.
Evidently they can. I witnessed it myself, even did a cold start (hold power and home until you see the Apple logo) and it still happened on a completely up to date iPhone 6.
It took 4 days, to zero on the root cause. As is usual in a complex scenario like this there are a few false positives, some suspects abusing the protocol and alas final redemption. Amazing work by a dedicated group of technical folks in coordinating (just via emails, I suppose) and tracing the root cause.
App Store vetting varies wildly and tends to trend towards more close introspection on smaller apps. Well known apps such as Facebook Messenger and Snapchat for example can get an update reviewed and pushed out faster than a standalone developer.
Yeah, it's not like Apple is in charge of the entire operating system running the app and verifying said signatures. Having them sign your app is definitely what gives them too much power.
Well, it is only client-side code though. Yes, it'd require some discipline to have proper separation so there is no leakage but I imagine it can be done.
to be clear, this is the reaction I anticipated when I wrote the gp. I don't think this is a cut and dried topic. I was merely expressing one side of the argument.
I don't think Apple compiling and signing binaries changes much when it comes to the average user and their security and trust model. It is my impression that Apple could easily intercept any keyboard input if they wanted to and frankly I wouldn't put it beyond the people at some other companies like Facebook or LinkedIn but I don't think we have found any good reason to lose faith in Apple's (or Google's) ability and will to do the right thing. Of course, ideally the idea of Trust No One is great but at some point you have to trust someone because who can read all the lines of all the code in the world?
I don't think Apple needs access to the source code to do things it needs to do. I was just saying that they have enough clout that the platform iOS has a solid foundation and can weather the storm that would inevitably rise from such a divisive decision.
This really isn't the sort of thing that would show up on an App Store review. The tests aren't done by engineers, so something's only going to get caught if it's noticeable by the user or gets caught by automated tools. (e.g. use of private APIs.)
Excessive NTP queries aren't going to cause noticeable issues on the device or on Apple's test network, especially with only one or two people testing simultaneously.
Oh, if my app did this I'd get in all sorts of trouble and I imagine never approved by the app store. I suspect Facebook properties just play by a different set of rules, unfortunately for those of us who believe in meritocracy and fairness.
edit: snapchat apparantly isn't a facebook property, but it has very deep pockets and as such my comment still stands.
Snapchat is not a Facebook property. Nor do I think Apple has resources allocated to review a library used by any app to facilitate if queries on the ntppool are kosher.
On my device, at various points, Snapchat and Uber have both been completely nonfunctional for days on end. I didn't think building phone apps could be so difficult for these large companies.
I do some work for the Network Time Foundation and we were not contacted by snapchat as far as I know. Anyone have a contact there, they probably need our help.
People from the NTP Pool community were talking to them (including myself, briefly). Given the available information I'm not sure why you think they need help from NTF ...
> Confirmed - starting up the iOS Snapchat app does a lookup to the domains you listed, and then sends NTP to every unique IP. Around 35-60 different IPs.
Hmm. Is that a fraud prevention thing or something? No way on earth a user app should be getting its own time