The "issues with IPv6" are with education, operation, configuration.
I personally ran WiFi networks with 8000+ wireless clients on a single /64 subnet (my employer's CiscoLive conference), and assisted/consulted in running the networks with more than 25000 clients on a single /64 subnet (Mobile World Congress).
The default kinda suck, and the bugs may happen. but the statement "IPv6 is not ready for production" is wrong.
I'd be happy to volunteer a reasonable amount of time to work with the OP or others having a network of >1000 hosts, to debug the issues like this, time permitting, vendor independent. (glass houses and all that).
There are bazillion knobs in IPv6 and a lot of things can be fixed by just tweaking the defaults (which kinda suck).
Network of <500-700 nodes generally don't need to bother much. It's not optimal a lot of times with the defaults but it will work.
EDIT:
the seeming "charity" of volunteering the time isn't. I want to understand what is broken, so we can bring it up back to IETF and get it fixed + make better educational publicity to prevent folks shooting themselves in the foot. It'll make it to the stacks in another decade, but it will.
IPv6 is powering nontrivial millions of hosts now - so the correct words to use "needs tweaking for my network", not "not ready for production". Let's see what the tweaks are and if we can incorporate them into the protocol, if necessary.
Playing devil's advocate: your experiences of "CiscoLive conference" and "Mobile World Congress" sound like conferences, which are the kind of scenario where most hosts are connected to the network for less than a day each time.
If I understood it correctly, his privacy addresses issue wouldn't show up in that scenario, since it needs several days using the same network connection continuously for each host to accumulate enough privacy addresses.
University campus typically have similar, although a bit less dynamic pattern of usage, than conference type scenarios. We actually observed on different (not just conference, but also campus) that wireless networks in general seem to have the same curve of connection duration distribution, though with different parameters. (If you have a large WiFi net, and would like to collaborate on re-confirming or disproving this hypothesis, please let me know).
The privacy addresses refresh depends on 2 factors:
2) frequency of reconnection. Some platforms (iDevices in particular) decide to make a new the IPv6 address every time they reconnect. Great for privacy (And that's exactly how I ran the abovementioned networks, also take a look at the http://tools.ietf.org/html/draft-desmouceaux-ipv6-mcast-wifi... for the observed leave/join stats), but if the infra does not have enough leeway to accomodate the extra addresses.
In any case even then not all is lost - clear the "A" bit in the RA to turn off the SLAAC, and use DHCPv6. Poor Android clients will be off the IPv6 grid because they don't bother to implement IPv6 for the reasons left unnamed here, but the rest of the gear will work just fine, with a single, predictable address, that an admin controls.
So, again - it all depends on the local situation and there's no unsolvable problems. There are existing and production-proven solutions.
You imply that DHCPv6 allows admin control, but as I understand it, it is not guaranteed that the request (DUID) can be mapped to a known device (MAC) (Type 1 DUID).
Especially with installation media, it is just not feasible to custom script them to force Type1 usage. This makes bringup and configuration mgmt just rely on IPv4 to get an identity.
I'm running into difficulties with this in our campus deployment. - I need static addresses in DNS (DDNS is just dirty, especially with DNSSEC) and also reverse records for kerberos.
If you talk install, then with most of today's PXE ROMS first thing you want to do is to bootstrap into iPXE to get IPv6 support.
That gives you also the support for pulling in your install via HTTP, with passing the UUID via the parameter in the boot URI.
You can then drag this parameter all the way to the final boot of the client.
I've done a PoC for a zero-touch install setup where you boot up a fresh empty machine and after a while of unattended magic it ends up with the Ubuntu installed and configured to a hostname set to its UUID. The address is registered via DDNS but not direct from the client - it's done via call-home server.
IPvFoo independent and no need to tag to MAC address or any address family whatsoever.
I've a 20-page write-up draft with scripts and all, though needs a ton more polishing for the real world. If it's of use to you, drop me an email, I can send - would be interesting to hear your opinion on that.
I spent a few days designing a thing like this for server deployment, but it was merely pen and paper notes, I would love to read through the draft you've written.
Couldn't find your email in profile - drop me an email at ayourtch at gmail / cisco dot com.
It's too raw to just be dumped on the web, and I'd be curious to hear from a few folks doing this as a "day job" about the usefulness if I were to invest more time in it.
And somehow juniper (and juniper support engineers for large routers) is unaware of these things? I'm skeptical. Particularly for ongoing problems where second and third level support get involved.
Don't be so surprised. There can be corporate-political problems to engineers solving the problem. For example, the official marketing line is "we support feature X!", and while the engineer wants to say "that doesn't work right, turn that shit off" he can't really do that.
Or maybe the solution is to not use Juniper for ipv6. I really don't know. It's possible that Juniper's development process and goals prevented solid ipv6 support - another case where even if their engineers were aware, they couldn't do anything to help this guy.
I wouldn't be surprised if this turns out to be a MIT-environment-specific issue.
E.g. Juniper support wasn't aware of it because they assumed no one would try to use stateless configuration in what would generally be a managed environment using dhcpv6.
Quite a few large environments use SLAAC and they work just fine (though I work for cisco so a bias is quite possible ;-).
But there's certainly something specific to MIT's setup that makes it melt down in such a spectacular fashion with v6. "Forwarding loops" (preexisting?) from the blog raise some suspicion.
The local environment frequently has a huge impact - and for the most extreme/"obvious" cases it took me to recreate the setup in the lab and show folks how it works fine with their config and traffic to make them believe that it was not just stupidly broken - because in their environment they could reproduce the bug every time!
Normally when the big dogs for a product get called in it is to find out why their product has an issue, they are the code experts for that. This doesn't mean they are inherently experts in every protocol their product happens to be using.
Shooting themselves or shooting an innocent and necessary technology. I personally see IPv6 adoption going to a full stall if users are told that advertisers found a way to tattoo cookies on their machines. I am worried by the hostile view on privacy addresses in that article and many of the comments.
Seriously, tracking via your v6 address is a minor concern. Browsers already provide more than enough information to an advertiser to do a really good job of tracking a person as he changes his attach point to the Internet.
Let me ask you a question. (Please address this question, and not some hypothetical situation that isn't covered by this question.) If the combination of my User Agent string, screen resolution, and installed browser plugins uniquely identifies my PC, why should I be concerned that I also have a stable, globally unique IP address?
I agree. This is not great. Though I believe that if this start being used, browser vendors will crack down on this (it is shocking that they give away so much information). But I read that this technique still leaves thousands (if not hundred of thousands) of collisions (particularly on more standardised mobile platforms) so I don't expect that to ever be widely used.
But the fact that there is a leak somewhere else in the house isn't an excuse to create another leak.
Also, browsers evolve and react relatively quickly. Not the IP protocol. When you see how hard it is to switch from IPv4 to IPv6, I don't think I will live to see IPv8! So making a disastrous decision from a privacy point of view in IPv6 will have long and lasting consequences. It's not because Chrome v30something is broken that we should leave a flaw in the IP design.
On "still leaves thousands (if not hundred of thousands) of collisions", here's some research:
"In 2010, Eckersley demonstrated that benign characteris-
tics of a browser's environment, like the screen dimensions
and list of installed fonts, could be combined to create a
unique device-speci c ngerprint [7]. Of the half million
users who participated in Eckersley's experiment, 94.2% of
those using Flash or Java had a unique device ngerprint,
and could thus be identi ed and tracked without the need
for stateful client-side technologies, such as browser or Flash
cookies."
Yeah but keep in mind that as soon as you install or update anything on your machine your signature changes. And I would be surprised if there was less than hundreds of thousands collisions on Android or iOS browsers.
In any way this is a flaw in the current versions of the browsers, but by no mean a structural flaw that justifies giving up randomized IPv6.
Who's talking about removing IPv6 "privacy" addresses from the standard?
I am addressing your assertion ("I personally see IPv6 adoption going to a full stall if users are told that advertisers found a way to tattoo cookies on their machines.") with facts that demonstrate that today, even behind an IPv4 double-layered CGN, your web browser makes your machine just as trackable as if you had a single, globally-unique IP address.
(In fact, if you've a mobile device of any sort, your web browser makes you more trackable than IPv6 [as deployed] does, as your address changes as you change attach points, but your device and browser fingerprint does not.)
ay is addressing your misunderstanding of the effectiveness of browser and device fingerprinting.
Neither of us are talking about removing "privacy" addresses.
The question is not about removing privacy addresses from the protocol but whether users will be able to use them. If network administrators, or more worryingly ISP force onto users things like RFC7217 then I think this is a problem. All I am saying is that I am a bit worried of reading lots of comments advocating effectively static IPs.
I wonder if the genie is not already out of the bottle anyway. I think pretty much every recent version of windows that support IPv6 has randomized IPs turned on by default so this will require a lot of efforts to reverse this.
I am not expert enough to form an opinion on the problem described in the article but it looks to me more like a problem with the implementation of multicast than with randomized IPs, and therefore disabling randomized IPs (the solution the author opted for ultimately) is in my opinion not something I would like to see generalized.
And again browser fingerprinting is a short term problem. How many versions of each of the major browsers have we seen in 20 years, versus how many versions of the IP protocol have we seen over the same period?
> The question is not about removing privacy addresses from the protocol...
This is correct. The issue at hand is talking about just how short-sighted and mis-informed this statement is:
"I personally see IPv6 adoption going to a full stall if users are told that advertisers found a way to tattoo cookies on their machines."
The only person who's talking about IPv6 "privacy" addresses is you. I don't understand why you keep bringing them up in a conversation about the effectiveness of browser fingerprinting.
I skimmed RFC 7217. It's not incompatible with "privacy" addresses. The RFC in question even mentions the orthogonality:
"The method specified in this document is orthogonal to the use of temporary addresses [RFC4941], since it is meant to improve the security and privacy properties of the stable addresses that are employed along with the aforementioned temporary addresses."
I don't see what your problem is.
If you're not interested in talking about the effectiveness or lack thereof of browser fingerprinting techniques, then I've no interest in continuing this conversation with you.
whether to allow the randomized addresses or not is a choice available today with no protocol changes whatsoever.
A network admin can turn off the "A" bit on the prefix in the router advertisements and turn on "M", and the compliant hosts will use DHCPv6 to get the address, which one will allocate statically per-user. This is exactly the same as "no privacy addresses".
ay's comment is great. You dodging my question? Not great.
I have to ask, have you even visited panopticlick? Go visit, using the browser that you typically use to browse the web. Slow your roll, and take time to actually understand what the site is telling you.
Also: "I believe that if this start being used, browser vendors will crack down on this".
You... are clearly not aware of what's happening in the Internet advertising space.
Visit https://panopticlick.eff.org/ (I mean it. Do this. Don't just blow hot air.) and ask yourself which of those things that the browser has been happily giving away for ages could easily be removed without breaking much of the useful functionality of the web.
Well, I thought my comments did answer your question.
- the fact that this signature works today (and I am not convinced it is actually so unique and stable to be really useful, again you don't have flash or java on iOS nor can you really customize the browser, so you will have lots of collisions) doesn't mean it will forever, for instance flash, java and silverlight are on their way out of the browser. And the more plugins and fonts you have, the more unique your signature is, the more likely it will change as a result of updating something (typically at least once a month) and therefore the less likely it will be useful to tracking you (your identity changes regularly).
- the fact that there is a flaw already doesn't mean we should introduce another one.
- IP protocol is evolving much more slowly than browsers and therefore a flaw in the IP protocol is a lot more problematic than a flaw in a current version of the browser.
If you're not convinced that browser fingerprinting is a useful tool in an advertiser's tracking toolkit, then you need to read what the EFF, lcamtuf, and other security researchers have to say. Or, you could just go work for a grey or black-hat internet advertiser.
It seems to me that the presence of a "bazillion knobs" is a bug in itself. A stable system can not depend on puny humans for its operation—nor should a design be able to blame puny humans for its failure.
The maxim "every tuneable parameter is a design weakness" applies for top-down design, but not so much for something as foundational as the IP layer.
In the 70s, TCP's inventors imbued it with solid fundamentals, but could not have anticipated how the protocol would need to evolve in the future w.r.t. security and performance. For IPv6, there are unknowns that will only encountered at scale, and implementors are hugely motivated to play fast and loose with the spec. (TCP Fast Open converts!)
Bottom-up systems have inherent wiggle room, and shipping them with sensible defaults is only possible in the short term. That's not to say we should design over-complicated systems, but just that fine tuners are not a weakness.
It's a very noble goal, and a pity HN has only one upvote per reader :-)
But, designing a ship that 10 years post-release can fly and dig under ground on a whim is a pretty high bar - and would be interesting to learn of successes elsewhere, if you know.
Some of the "weird" IPv6 knobs today are actually very useful, though from what I know, I must've been one of the first known instances to abuse some of the non-defaults at a 10K+ users' scale.
So, I'd say - keep "no knobs" as a goal, but still leave the dirty knobs somewhere to turn in case of emergency. You'd be thanked, if by accident your protocol get used 15 years later :-)
> My philosophy for a long time has been "every tunable parameter is a design weakness, and every one that must be tuned is a bug."
It's completely true that good defaults are essential. But no one person is competent to make a decision for everyone everywhere. It should always be possible to change something but ideally it shouldn't be necessary.
The interesting thing about software: excepting DRM, it's always possible to change something. Don't like a "fixed" parameter in your IP stack? Patch your kernel.
I'm honestly surprised people don't take more advantage of "tuning the source" as an solution to providing end-user flexibility (the only project that I can think of that does is nginx.) Maybe it's because we have "dev ops" (devs who end up doing ops) but no "op devs" (operators who know how to code.)
It's still possible even with DRM, it's just more work. You have to get out a debugger and write new code.
But what I meant was that you shouldn't try to make modifications impossible. Yes, it's futile anyway, but it's a bad idea independent of that.
If you've designed the system right then the users shouldn't need to modify it. So if they do it's a signal that you've missed something and then you can improve it. Whereas if you declare yourself the final authority and refuse to facilitate user modifications, or worse actively interfere with them, then any mistakes you make are silently uncorrectable.
100% agreed. This is why I'm liking elementary OS so much. I still need to pop into the command line for some things, but the standard GUI software is less excessively configurable than most. They have an HIG and a small community of developers who actually pay attention to it.
A lot of it is based on the GNOME core applications, so it's similar. I haven't used GNOME enough to say more than that. For the people who want extra options, there's elementary-tweaks, similar to gnome tweak tool.
How many users will want to change the file naming or folder hierarchy convention for their music library if they're managing them through a GUI application? Why is there a big list of "visible columns" checkboxes instead of setting that by right-clicking the columns? Does crossfade really need a whole tab for a checkbox and slider (and why does it need a checkbox when 0 seconds implies no fading)? Do people actually use crossfade anyway? I never have.
In short, there's room to clean a lot of things up. Even if it's minor, doing it across the entire system makes things a lot nicer. Plus there are some additional system-wide standardization features that elementary is doing, which should prevent a lot of inconsistency from software from all reinventing the wheel in different ways:
How many users will want to change the file naming or folder hierarchy convention for their music library if they're managing them through a GUI application?
I do.
Do people actually use crossfade anyway?
Yes.
The past decade has shown a disturbing trend in design. B-level designers think they can ape Apple by optimizing for the middle of the bell curve and alienating the edges. The problem is that the actual distribution of feature usage is multidimensional in a big way, and these designers delete features on one axis that otherwise very average users absolutely depend on.
There's the saying that 80% of users use 20% of features, but no users use the same 20%.
I'm not saying it's ideal for everyone, but it's an Ubuntu derivative, and it will continue to exist alongside Ubuntu. Noise will exist alongside Rhythmbox. I like a lean, well integrated player as the baseline "everyone gets this by default" option. Out of curiosity, what's the naming/organization system that you use, and is there a reason you don't like "Music/Artist/Album/# - Song.mp3"?
And re:crossfade, if people other than me really use it, then my complaint is that they added an entire "playback" tab in the settings for it. If there's only one playback setting, give it a header under General. It's the visual equivalent of a "for i in range(0):" loop; all it does is make your interface harder to read.
Noise is still an early version and I'm sure there are features that will be added, but "Print CD jewel case insert" isn't on the high priority list. That might have been neat when iTunes first added it, but it's just more cruft now. 1% of people might burn CDs in iTunes today, and only 1% of that 1% will even realize they could print an insert to go with it.
Even on OS X where every computer ships with a pretty effective (if not always snappy) media player, there are plenty of alternatives for people who don't like iTunes.
> and is there a reason you don't like "Music/Artist/Album/# - Song.mp3"?
There are tons of reasons I don't like that format. It's extremely limiting and it only applies to a few genres of music. It tries to fit all music into a rigid hierarchy of bands, albums, and tracks, a hierarchy which applied to a reasonable amount of popular music from the '70s to the '00s, and it makes me wonder if the designers of music players ever listen to anything else.
Suppose the song is a collaboration. Which artist directory does it go in?
Suppose the song is a promo track that you downloaded from a band's website or from Bandcamp. What's the album? What's the track number? What happens when the track is eventually released on an album?
Suppose the "song" is the third movement of Bach's Unaccompanied Cello Suite No. 1 (BWV 1007) as performed by Yo-Yo Ma. Who's the artist? What's the album? (If you're going to say "the album that Yo-Yo Ma's recording was released on", you'll have to be more specific.)
Now can you give answers that put things in the same organizational scheme when it's the second chorale of Wachet Auf (BWV 140) as performed by the Bach Cantata Pilgrimage, directed by John Eliot Gardiner?
Suppose the song is Monstrous Turtles, by Zircon, a remix of Koji Kondo's "Super Mario World" music, released as OCRemix #1558. Is "OCRemix" an album under "Zircon"? Is it track number 1558? What about the fact that most OCRemixes are not by Zircon? Does it go in the same place if you download it from Zircon's web site instead of OCRemix?
> There are tons of reasons I don't like that format. It's extremely limiting and it only applies to a few genres of music. It tries to fit all music into a rigid hierarchy of bands, albums, and tracks, a hierarchy which applied to a reasonable amount of popular music from the '70s to the '00s, and it makes me wonder if the designers of music players ever listen to anything else.
And here are Rhythbox's five options for library folder naming.
Your library sounds like a perfect case for unchecking the "Keep Music folder organized" and just managing it yourself, since there's no single pattern that will work for the whole thing. Incidentally, that's the one library organization option that elementary OS has.
And taking another look at Rhythmbox, either there's no option for "Let me organize my own damn files," or they hid it somewhere unrelated. Maybe you can launch it with a command line flag to do that...
Millions of people are using it today, one and a half decades after it was conceived. It's not perfect but nothing is.
If you have examples of tech to compare with - please bring them in!
EDIT: and yeah, a lot of knobs does sound like a bit of a poor design, but on the other hand, today, 1.5 decades later, I am grateful I can tweak some of the defaults!
Again, if you know something knob-less that'd get flawlessly deployed in 15 years after it was designed - shout, would be very useful to try and learn from that success!
The nearest comparison to IPv6 is DNSSEC. IPv6 has done a little better, but that's not saying much. Both suffer from massive over engineering, and lots of false starts and back pedaling over the years. Trying to solve too many problems, and therefore failing to solve the one problem that's of immediate concern.
False starts and backpedaling are also known as "iterating". It's disappointing that many of the planned features never got production-ready. But the protocol is now more likely to be implemented than ever before, which is the primary "feature" to which all others are secondary.
I was looking for the example of the sparkly unicorn protocol from 2000 the people seem to evoke, which does not need any knobs and is available on every device to update some other protocol from 1970s.
(I suppose the hard meta-question to protocol designers is: how do you make sure your children will not hate you for the protocol you have imposed on them today ?)
UTF-8 updating ASCII is probably the best example of this going well. And even then there was a terrible mistake made in the design of Korean encodings which had to be fixed.
Both yours and tedunangst example (SSH) have one important commonality which is not present in IP: they become immediately available to two parties as soon as these two parties upgrade.
With IP, all the intermediate nodes need to upgrade to use the new protocol (tunneling tricks aside).
This difference would probably be a significant factor in the speed of adoption. Tunneling obviously can emulate the 2-party model, but then we're left with the problem of managing the crazy amount of tunnels (and security issues) and sunsetting them.
I have no experience with Korean and Unicode but, I guess, it's jamo composition?
With it, there are two distinct and not-really-compatible (since anyone in real world practice rarely cares about Unicode normalization) ways to specify a single Hangul character. And that should make text processing a bit painful.
Anyone who writes software to deal with unicode really better care about unicode normalization!
You are going to have all sorts of edge case bugs and unexpected behaviors if you don't, definitely not limited to Hangul. There are multiple un-normalized ways to write all sorts of on-screen glyphs, including Latin alphabet ones.
I thought wbl meant something to do with the combination of Korean and UTF-8, whereas jamo composition is no more related to UTF-8 than any other unicode encoding.
They moved the Korean encodings at one point in the early 1990's. Luckily this was before anyone used it, but it was still a big mess. I don't know why they had to move it.
...and thanks to the knobs for being able to workaround the day1 bug uncovered just this year in at least one of the widely used vendor's server implementations while getting the fix.
But yes, with just 2 parties involved into any instance of the protocol enabling easy upgrade/downgrade path, and tight engineering it is indeed a beautiful protocol.
I would say IPv4 had that property. Not exactly knobless but there are reasons it replaced just about every other networking protocol in existence and simplicity and robustness probably had a lot to do with that. Of course hierarchical routing was a huge feature for the Internet but not necessarily compelling enough to drive out every other LAN protocol.
> I would say IPv4 had that property. Not exactly knobless
I can tell you haven't tried creating/setting up your own router with NATing, separate subnets (secure, insecure), inbound and outbound VPNs, firewalling and traffic shaping. Or basically anything non-trivial with IP in general.
The amount of knobs which you can (and must) set correctly for this thing to fly is enough to give an experienced engineer severe paranoia.
Actually I have done most of those things at one time or another.
I'm not saying IPv4 is simple (especially if you want to do complex things with it) but it works well when configured correctly and is simple enough to be deployed in businesses of all sizes as well as home networks.
I didn't get involved with networking much until shortly after most other protocols were rapidly disappearing but the fact that IPv4 replaced IPX, DECNET, AppleTalk, etc in only a few years time suggests that it probably wasn't that much of a nightmare to set up, even in it's early days.
My concern with IPv6, based on this post, is that it may have lost some of the robustness and relative simplicity of IPv4. I mean layer 3 <-> layer 2 address mapping is fundamental to any network and if that's breaking with IPv6 it could be a pretty serious issue.
It's free, very high quality content, and it is written as if you co-design the new IP protocol together with the author - you start from "why" and arrive to "how".
Absolutely that's why all network pros advise against tuning the timings for a v4 network even though in some edge cases it can help unless you really really know what you are doing.
How can you be so certain that the author's issue is not real?
If you have enough information to be sure that the author's situation isn't a genuine problem with IPv6, why don't you also have enough information to say what the administrator's mistake is, and its solution?
I don't mean to be snarky considering your generous offer to help, but your certainty that the problem isn't with IPv6 seems odd, given the specificity of the author's description and analysis.
That's not what I said. There are plenty of very real and very painful ways to shoot oneself in the foot with IPv6, especially in a larger kind of network that I can imagine the folks at MIT have.
But based on my experience with running the real-world IPv6 networks of large scale, there has to be a better solution than just "turning off IPv6" - even if it is a great interim band-aid after being up for 40 hours! (and I emphasize with that!)
The blog entry says: "The most serious issue is that there appear to be bridge loops occasionally being created". IS this something related to IPv6 ? Specific to IPv6 ? What's the root cause ? IPv6 with default settings in a larger net has very nonlinear behavior in the case of unreliable connectivity, and can make the situation worse.
The other issue, multicast group overload, should be solvable by making the switches blind to MLD.
This will make it as "bad" as IPv4, but in todays' network size it will work fine because IPv4 works, mostly. Meantime they would need to kick their vendor's representative to have it fixed.
Also, a nit about the whole issue at hand: the "worst offender" is iOS, which will create a new privacy address everytime you disconnect and reconnect to WiFi. And if you were to "fix" that, imagine the wrath of the "privacy advocates" about this.
Privacy addresses in IPv6 are very valuable for a reason that due to a fast refresh cycle they help to uncover a lot of corner cases that would otherwise pop up with a kind of "shrug, something weird" type of RCA. And fix them before the IPv6 usage hits 50% (cf my comments on the other thread).
(From the privacy standpoint, privacy addresses are a pretty entertainingly weird construct.)
So to summarize this somewhat long-winded post: no question there's a problem. Whether it's in "IPv6 the protocol" or "IPv6 the vendor implementation" or "IPv6 the configuration" is another story. And that'd be something to understand and fix - but needs more info/context/legwork IMO.
I have dealt with MIT network admins who while not infallible are pretty darn competent. If people like this don't know or cannot fix the problem then there is a very real issue with the process by a which ipv6 is taught. It is not reasonable to have a super duper airplane if the pilots you have cannot fly it safely.
The complexity and general human unfriendliness is a major downgrade from ip4 to ip6.
By no means there's any doubt in MIT admins' competency. Just that by reading the encounter, I feel sorry for the cards they've been dealt with, so to say:
1) a s/w hang for which there was no way to advance in a getting a solid RCA. This alone is a very hard one, and I know the ugly feeling both from the customer's side and from the vendor's side when all you have as an action plan is to try to reduce the potential exposure to what you think the problem is. I've been there myself multiple times, both as a customer, and as a vendor support.
2) Bridge loops. You can fix an active loop by disconnecting links, but root-causing such a problem is not a piece of cake, by far.
4) TCAM overload (?) due to privacy addresses rotation. "TCAM space dedicated to IPv6 multicast is only 3,000 entries. That would be just barely enough to support all of CSAIL". The network gear frequently allows to re-carve the TCAM space - maybe they might have taken some space from IPv4 which is by default is provisioned much more generously (for various reasons outside the scope for this thread).
Besides just brute-forcing, there might be other strategies in dealing with the temporary addresses - actually shortening the valid lifetime and bringing it closer to the preferred lifetime should work. This gives me some useful material to update my IPv6 troubleshooting talk with.
So, no, MIT folks are by far not stupid - it's the problem that is complicated - and, after staying up for 40 hours, whatever helps to reduce the pain is good.
But I'm confident there's more learning here to be done from this unfortunate incident.
The title should probably read,"Bugs in JunOS caused network downtime."
This isn't really news. There are bugs in all routing and switching OS's. That's why they hire support people. This isn't me trying to rag on Juniper. I know lots of people who work for JTAC and they're incredibly smart folks. I'm sure they'll get this sorted out and fixed in JunOS and, like any bug, merged into upstream releases.
This isn't me trying to point out that IPv6 is infallible. There might be some design choices that the IETF made with IPv6 that were stupid, but mostly they got it right, and it's too late now to change most of them anyways.
This is just reality with new software. New software has bugs.
You can blame it on MLD, but really MLD is no more complicated than IGMP. You can blame it on NDP, but really NDP isn't much more complicated than ARP.
At a minimum IPv4 required ARP to function. However, in reality it also required atleast IGMPv2. Since without IGMP, or some way to manage multicast, how are you going to get something like VRRP to work. Link-layer multicast is not new to IPv6.
A small nit to add: IPv4 did not have 1+ multicast groups per every host. That's a dramatic difference in terms of capacity on the middle-gear which escapes if one thinks of IPv6 as "IPv4 with bigger addresses".
And you're right about the additional mcast addresses required by IPv6 NDP. It's called the 'solicited-node multicast address' that every host must join.
My historical guess about why multicast was chosen over broadcast for NDP was due to NBMA networks. In the 1990's NBMA networks(frame-relay) were much more common than they are today. And NDP just makes more sense than ARP over NBMA networks. This is just a guess.
Someone else suggested DOCSIS was the reason that multicast was chosen instead of broadcast. I doubt this. I'm not that familiar with DOCSIS, but I think it has a broadcast type link-layer. Also, IPv6 predates DOCSIS.
It could well turn out that multicast was the right choice for NDP once we get over the inevitable roll out problems. NBMA networks could return in 20-30 years. You never know.
The "classic" rationale to choose multicast over broadcast was to try and limit the amount of time to process NDP traffic vs. the time the hosts have to process the ARP broadcast traffic in IPv4. 15 years ago that took a nontrivial amount of host resources, and with the way NDP constructs the solicited node multicast, even if you just flood every packet on the link, you still can filter the packets that are not for you at the NIC HW level.
And since there are 16 million unique solicited node multicast addresses, in principle the scaling is pretty impressive.
On NBMA: you can have such a network today in a public WiFi scenario where you do not hosts directly talking to each other but want them to access the internet. In the wired case the moniker is "Private VLANs".
With IPv6 you can clear the on-link bit, and make NBMA work quite elegantly. But, depending on the exact details, http://tools.ietf.org/html/rfc4903 does list quite a few interesting challenges - quite an informative read.
It's a gem: free, very good quality material, and written as if you are co-designing with the author by solving various problems you see on IPv4 networks, and the protocol evolves into what is the IPv6 today.
Thanks again for the excellent response, and for the history lesson on why multicast was chosen over broadcast for NDP.
I've done tons of work in private vlans over the years. I was part of a months long effort to find and document bugs in private vlans at a vendor, where my primary focus was to find and document bugs only in private vlans.
Residential ISPs love private vlans because it prevents different customers from seeing each other's broadcast traffic. DHCP Snooping to prevent things like ARP spoofing, in combination with private vlans to limit broadcast traffic will mostly lock things down pretty well.
It's funny you should provide a link to a resource from the Deploy360 Programme. I just finished doing a stint with the Deploy360 Programme as a writer working on, among other things, IPv6 resources. My real name is Andrew McConachie.
We've also hit this Intel Ethernet driver bug, even though we don't have IPv6 deployed. Linux will send MLD packets on bridged ports by default, triggering the Intel driver bug on Windows machines.
With only two Windows machines saturating their Gigabit Ethernet connection whenever they went into standby, we managed to crash the university's switches big time (we're a group with our own VLAN within the university's network, so we make use of their network equipment).
Naturally, because the issue only occurs during standby, and usually users don't log off thus preventing Windows from sleeping, we first hit the bug during the Christmas holidays (2013). The culprit hosts were all in use for just a couple of months. In the end, it took a couple of hours to reproduce this bug during working hours!
We fixed it by using different NICs (we didn't want to rely on the Intel driver to be updated after a clean install; Windows Update doesn't have the fixed version), and by disabling MLD snooping on the Linux hosts, since we aren't yet using IPv6 anyways. This prevents the Intel bug from being triggered in our environment.
As someone watching from the sidelines I had no idea there were such major issues with IPv6. It seems like IPv6 has been out there for a long time (about 10 years) in terms of being supported by OS's and networking hardware, if not ISP's. So I would have thought that cutting edge institutions (like MIT) would already have years of experience with it and have worked out most of the kinks by now.
If this is not the case what does it mean for more widespread IPv6 adoption ? If such adoption is significantly delayed or stalled what will the consequences be, both for current Internet growth in the face of IPv4 address depletion and for new technologies like IoT ?
I haven't been on the front lines of new protocol deployment for a long time now, but the pattern then (and it appears unchanged) was that larger deployments brought out 2nd and 3rd order issues with the protocol. The old joke was "How can you tell someone is a pioneer?" answer, "Count the number of arrows in their back." which expressed that folks who adopted new protocols bore much of the burden of their failure and revision. Sounds like CSAIL has made some great progress in this respect.
There are already quite large both enterprise and service provider networks already using IPv6. It's more in-between the "pioneers did not document the thorns they hit, so the others would not" and "the future is not evenly distributed yet, so we don't know about it" territory.
I've volunteered myself to understand which of the two and to do whatever is actionable.
IPv6 has been sold as "just like IPv4, but bigger" but it's not. It's much more than that, leading to the schism between the "adopt or die" and "not on my watch" people.
That depends which part of the stack you're working on.
If you're writing web apps that receive a REMOTE_ADDR field, then the protocols look very similar; you just need to parse and store bigger values, and perhaps account for the fact that users tend to control a prefix instead of a single address.
At the BGP and packet forwarding layers, everything's the same, with bigger numbers.
The linked article relates to IPv6 over Ethernet, which is the area where IPv6 added the most new features and quirks.
Sure. The people who got "128-bit IPv4" aren't the ones complaining. The network admins who deal with all the rest of ipv6 are the ones who don't like the imposition.
> what does it mean for more widespread IPv6 adoption?
If I were a security hacker (whatever shade), I'd be spending all of my time looking at IPv6, from device drivers through kernels and routers up to user-level programs.
IPv6 is simply not hardened in the way IPv4 is. It will be some day, but it's going to have to earn it the hard way, same as IPv4 did.
Indeed, IPV6 is a possible entry point for pen testers, as plenty of admins will leave ipv6 enabled, but not bother to setup ip6tables. Plenty of software is setup to bind on "all interfaces", which includes your link local address. Fortunately, takes some effort to scan the link local address space.. But if they are based on MAC, and you can just ask for the MAC of the host via ARP...
Yup. iptables should really have a "figure out how to mirror this" mode, since 99% of iptables rules can be translated to ip6tables rules by sprinkling 6s around, and almost everyone screws it up initially.
I've got a bit of a headache right now, but IIRC the second remotely exploitable OpenBSD bug was because they had IPv6 enabled by default. There are a lot of grues lurking in there.
The parent wasn't just implying it being enabled, but also likely buggy in the implementation as well. E.g. Maybe an overflow with the next header implementation.
Almost all large organizations decided to delay adopting IPv6 until the last possible moment because this reduces various costs. But this meant that many deployment problems were not discovered until very late.
I used Ubuntu as an example, but it is hardly the worst offender. We have seen
Windows machines with more than 300 IPv6 addresses
Wow! I don't operate a very large network, but I do operate an IPv6 network and I've never seen one of our machines use more than 2 addresses. I feel like they've got some other configuration option or oddity going on that's causing a lot of these problems, but I am guessing they're much smarter than I am, so I don't know what to say.
Could someone elaborate on this? I've never seen this behavior on an IPv6 network, and I'm just running a server or two with radvd and no custom switch configuration.
I had a raspberry pi on my network via ethernet and saw it had accumulated a ton of IPv6 addresses. Ended up just disabling privacy extensions on it. If your machines reset their network connection (ie wireless reconnects) then they'll flush their IPv6 privacy addresses.
A house may have multiple people living there, multiple devices, visitors, etc.
You could track a person connecting through multiple locations by using their MAC address used to assign their IP. i.e. if I am connecting with my laptop from various places, my IP address will be roughly the same, only the network prefix will change.
You are completely right. Privacy IPv6 address are an illusion when used in a household that has effectively an active /64 per subscriber.
(Those who say "but IPv4..." last time my IPv4 address changed, was half a year ago). Anyway this is mostly a political give-in, that happens to also help ruggedize the rest of the stack (the relatively rapid change of privacy addresses uncovers more bugs than we'd otherwise find - so from that standpoint they're to be advocated). But privacy.. huh.
Last IETF there was precisely this discussion that one might want to force a new /64 on themselves, and privacy addresses have nothing to do with that. (DHC WG, FWIW).
"effectively an active /64". Comcast folks did a ton of testing with the CPEs, and very very few could ask for anything than an active /64, that's why this qualifier :-)
There's active (and pretty cool) work in the HomeNet IETF workgroup, with running code and all (http://www.homewrt.org/doku.php) to make the multi-subnet home network a sane reality.
But a different /64 from within the same /56 helps not much, and indeed to correctly reflect the spirit of the discussion in the DHC working group, would be to say "I want to be able to press a button and release my currently used allocation and get some completely new one, be it /48, /56 or whatever".
The proper solution is Stable Privacy addresses, recently standardized in RFC7217 [1]. The interface ID is essentially hash(secret || prefix || mac-address), so it stays the same when you're on the same network, but changes as soon as you go somewhere else.
But that defeats the purpose of a privacy address. The idea is precisely not to use your address as a way to track individual users, like a cookie tattooed on your skin. I fail to see how a hash of the MAC address is better than the MAC address itself.
Because even with traditional privacy addresses, the first 64 bits of your address never change, and you can be tracked that way. If you happen to share those first 64 bits with many other people, then privacy addresses provide value, but that's simply not the case on your home network.
Now keep in mind that a stable privacy interface ID is not a hash of just the MAC address, it's also a hash of the network prefix and a secret that's specific to your system. Including the prefix ensures that the interface ID changes completely when you move to a different network, preventing you from being tracked all over the world. Including a secret ensures that someone can't build a lookup table from MAC address||prefix to stable privacy interface ID. So it's quite superior to using just the MAC address.
Not really, how many networks do you use in a week. Perhaps 3 or 4. And with this, once a link has been established between the IP, it becomes a permanent static link. It is trivial for advertisers through browser cookies to establish this link, and they only need to do it once.
You say that like if I was actively letting myself track.
I block third party cookies, I use AdBlockPlus. But I can't block all cookies and html5 storage (most websites wouldn't work anymore). And it only takes one website to see me coming with the same cookie from different IPs to establish that link (take a pick: google, facebook, twitter, etc). And once that link is established it will be pretty valuable (as it is static) and you can bet that these boys will exploit it.
So to me RFC7217 is pretty much as bad as static addresses.
I'm not sure why websites would even care what IP address you're coming from if they've got a cookie, which provides an even stronger way to track you. They might care what the network prefix is, since that would identify your geographical location, but not even privacy extensions prevent that.
You have ways to block third party cookies so that you cannot be tracked on the internet but if you have a website that is advertising focused (say google) and see you coming from two different IPs (and it doesn't need to be a third party website, it can be just when you search something on google), then it will be able to associate these IPs to your profile. Then every time you visit another website, that website will be able to get from google your ID from your IPs even if you block third party cookies. And this is the end of privacy.
If privacy addresses pose multicast problems, then the multicast protocol or implementation needs to be fixed. But static IP addresses isn't the solution.
I think you're putting too much faith in blocking third party cookies. Through collusion with Google (such as an IFRAME or JSONP), sites can get your Google ID from your Google cookie without needing to consult your IP address or use third party cookies. Web browsers open up such a huge can of privacy worms that I don't think it's very useful to talk about IP addresses until you fix the browser.
Personally, I clear all cookies, cache, and local storage (except from a handful of trusted sites, Google not being one) when I exit my browser (and I exit my browser often), and I occasionally examine my browser profile for traces of super-cookies. (And even this is probably not sufficient because of browser fingerprinting.) I don't think privacy extensions provide an appreciable privacy improvement over stable privacy addresses.
I find the Privacy addresses a pain when it comes to managing traffic on my network. My (mostly Apple) devices will never use their DHCP provided v6 address to connect to a public service and this means that my gateway doesn't know what the hostname is for that connection.
I ended up writing a little script to scan the "ip -6 neigh" and correlate hosts together based on MAC, then write that to a hosts file for DNSMasq to use. Hack, but it works for me.
In fact you know it's a home, but an advertiser has no way to tell. It could be a restaurant, an office, a flatshare, a gym or anything else. If they see two IPs on the same /64, all they can assume is some form of weak correlation.
Correlating "stable privacy addresses" (cf agwa comment) is trivial. How many network do we use in a day? Home, office, mobile provider, perhaps a couple more. Chances are that advertisers will see you using the same cookie across multiple networks and if your IPv6 address is persistent then you can bet that within a week all your stable IPv6 will be mapped to your profile. It only takes one observation to establish the link and then it is a static link.
Ideally you would like your /64 to be renewed regularly (it's not a bad idea to reboot the modem from time to time anyway) and to have randomized IPv6 addresses.
I just checked a couple of freebsd 10 boxes I have. Both have 5 addresses in 'deprecated autoconf temporary' state, and 1 in the 'autoconf temporary' state.
Privacy addresses are enabled via these sysctls:
* net.inet6.ip6.use_tempaddr (default 0)
use temporary (privacy) ip6 addresses
* net.inet6.ip6.prefer_tempaddr (default 0)
prefer them over the default ip6 address for egress
The lifetimes are controlled via sysctls:
* net.inet6.ip6.temppltime (defaults to 86400)
Specifies the “preferred lifetime” for privacy addresses
* net.inet6.ip6.tempvltime (defaults to 604800)
Specifies the “valid lifetime” for privacy addresses
So if private addresses are enabled.. by default a temporary ip6 address is changed every 24 hours, and will linger for 7 days. However, they are not enabled by default on freebsd 10.
They're enabled on quite some of the Linux distros. And if someone did not know the consequences of what they were doing when they decided to play with the reachable/valid lifetimes on the prefix, they could end up with a lot of addresses.
I'd love to know more on the origin of the story to understand how we can fix the defaults in IPv6 so they do not blow people's network - but I have a vague feeling someone played with them anyway.
You may find https://communities.intel.com/thread/48051 and similar useful; I've run across mention of it before but have not personally encountered it (but we don't usually have WoL enabled, so...)
"the entire TCAM space dedicated to IPv6 multicast is only 3,000 entries"
Some mid-level switches normally have TCAM space for hundreds of thousands, or millions of entries, IPv4 or IPv6. Maybe their vendor artificially crippled their line of switches, or maybe the switches were deployed with a configuration error. It is probably the former though. Network vendors like to make you believe some features cost a lot to implement and that you really need their highest-level gear, when in fact even the biggest TCAM in silicon cost a few tens of dollars, at most.
Nobody on this thread really seems to be talking about one of the issues brought up in the IPv6 analysis (though I'm not sure if it caused their outages - any time I read a post mortem with the phrase, "bridge loops" I usually don't look any further - that alone is enough to bring a network down)
If I read the post correctly, one of the roots of their problems seems to be that either (A) Traffic flooding causing excess traffic as a result of multicast packets being flooded over their network, or (B) If they used MLD snooping to reduce the flooding, the switches they have only support 3,000 entries for multicast groups - which are quickly exceeded with the privacy IPv6 addresses that are generated by the hosts (each of which creates it's own multicast entry, and some of their hosts had 10+ addresses)
Other than turning off privacy based IPv6 addresses, and moving to something like RFC 7217, is there a solution? Increasing the number of multicast entries on the switch to something larger, say, around 30,000 entries combined with reducing the length of time in which a privacy address is valid (and therefore requiring a group) from one day, to say, one hour?
What's the reasoning for dropping ARP? It seemed like a simple architecture. The post seems to indicate IPv6 requires a ton more hardware resources. And if Juniper doesn't have a basic feature like MLD snooping after all this time, uh? Shouldn't practically designing a high-volume switch be part of creating such a fundamental protocol? (I know designing 2 elegant implementations of other protocols would have fixed a ton of things in nasty protocols like HTTP - dumbass things like line folding and comments-in-headers.)
Is this a case of idiocy seeping through the IETF because they can? It's pretty easy to write something down on paper if you don't have to implement engineering and product management on the result. Or because you're out of touch with reality, like the source routing feature which was kept in IPv6 despite it only ever being a problem in IPv4? Or is this a case of the protocol being superior and vendors just being very lazy?
ICMPv6 ND and ARP are essentially same protocol. Main difference is that ARP runs directly over some L2 protocol, while ICMPv6 ND uses IPv6 multicast as underlying protocol regardless of underlying L2 transport. Practical motivation to do neighbor discovery using multicast instead of broadcast is to limit overall volume of ARP/ND packets that have to be processed by host's L3 stack (and enable reasonable intelligent switch to just drop them), when this mechanism was designed, excessive ARP traffic was real problem for many kinds of ethernet or ethernet-derived (eg. DOCSIS) access networks.
It seems like they really wanted to push multicast as a "first-class" mode of the protocol, and by weaving it into the core of IPv6 they've forced everyone to have a somewhat exercised implementation of it in their stack.
-> Broadcast is just a special case of multicast, where every host listens to an "all hosts" multicast group;
-> Since this is a new protocol, there is no need to keep broadcast; just mandate that every host join the "all hosts" multicast group (notice that the numerically first two IPv6 multicast groups are "all hosts" and "all routers", implying they were the first ones thought of);
-> Since we will always have multicast, instead of flooding ARP-like requests to everyone, let's create many "buckets" so each ARP-like request will go only to the hosts to which it's relevant, saving traffic and processing power.
IPv6 is old. It's possible that when it was designed, global multicast was just around the corner. Designing the protocol to use multicast would have seemed natural back then.
So pushing something no one uses, and has significantly narrow uses, and still isn't deployed publicly, into the core... How is that anything but self-serving protocol designers? How do they get away with that?
Multicast has much more than "significantly narrow uses". When we were studying IPv6 in my network protocols course 20 years ago(!) I remember thinking how useful that would be if widely deployed.
A good portion of the bandwidth being used on the Internet could be done over multicast if it were widely supported.
For example, popular content on Hulu could be multicast. You'd download the beginning of the video over HTTP but simultaneously tune into the multicast that started the most recently. Once the two streams meet you drop the HTTP connection. If widely deployed this would reduce costs for both Hulu and ISPs, and for places where multicast doesn't work it can fall back to HTTP.
As I mentioned above, this misses the point. All you are getting is that instead of having a unique identifier (your MAC) you have 4 or 5 (the number of networks you roam between within a day). This will be trivial to defeat with a browser cookie. The identifier are still static which means that it only takes one observation to establish a permanent link between these IPs.
A small domain encryption scheme, rather than a hash, would make more sense... that way there'd be no collisions. SDE isn't that much harder to put together in hardware.
The subnet space is 64 bits long. Collisions between pseudo-random values only become likely when you have billions of nodes, so it's not worth the effort to preemptively avoid them.
This would also require all nodes to implement the same algorithm, which means it's not incrementally deployable.
Edit: The actual RFC7217 algorithm contains a DAD_Counter field in the hash input. In the event of a collision, the counter increments, generating a new address.
the random address is changed regularly, typically daily, but the old random addresses are kept around for a fairly long time
I don't understand this part. I have Ubuntu machines in a network which is technically /48 but only one /64 prefix is announced by radvd and they all have only two addresses, one derived from MAC and one private/random changing over time. They certainly never have eight.
Are those previous addresses not visable in ifconfig or ip -6 addr show?
A significant factor is that privacy addresses won't be released if there are active connections using that address. So if the machine is running something with long lived connections; big downloads, push notifications, IM clients, ssh sessions and so on, then it can end up a growing stack of unreleased addresses.
A typical admin who ssh's to a few boxes every so often, but then leaves those connections open in a tmux/screen/terminal-tab session can cause havoc.
I haven't seen my Ubuntu servers use privacy addresses, however Ubuntu desktop does use privacy addresses, and I've seen my Ubuntu desktop grow more IPv6 privacy addresses over time on my local network.
This is now the second major network that I've heard ran into this exact problem. The update to Windows caused the end-user nodes to send out lots of IPv6 packets The access layer switches went to full CPU utilization, and you ended up with packet storms across the network. There really should be an advisory about this.
I have no idea where to even start - this article was written by someone who has no large scale IPv6 deployment experience. There are errors upon, back-to-back, errors in what's assumed and the expected results and assertions with the vendor (Juniper) and the protocol operation (IPv6).
I'm not surprised that it's towards the top of HN but it shows the relative understanding of the HN crowd with regard to complex network related topics.
I'm pretty familiar with "complex network related topics" - and I think this is the most interesting IPv6 related post on HN in 2+ years (there may have been others that got by me).
I"m curious - are you someone with "large scale IPv6 deployment experience?" - in particularly, how would you have approached their issues regarding MLD snooping and TCAM exhaustion?
It may be interesting, but my point was that there is nothing in this article to learn from with the exception of problematic code from a network vendor.
Yes - I am. I deployed a 4 state IPv6 overlay servicing 250k subscribers on one of the very first (fully rolled out) DOCSIS 3 HFC networks in the US. I was responsible for the security and performance of the architecture. This rollout started in 2010. The TCAM exhaustion was self inflicted based on the hints of design throughout the article and the original authors understand of MLD is, just generally, incorrect unfortunately.
This comment would be better if instead of insulting its audience, it taught them something. Many people here would appreciate a chance to improve their "relative understanding".
As it is, although it alludes to questions of substance, it doesn't actually say anything about them, and therefore is just noise.
I think that is a very telling point in and of itself. I've sat on the boundary between the devops / security and networking world my entire career. I know many people who are great at *nix and tooling of backend infrastructure but choose to ignore the plumbing, and complexity contained within, the planning and architecture that can make the pieces above it that much more successful. When I read an article like this there are blatant points that showcase how green the OP is with regard to networking. I happen to have an advantage in this space and the point being is that I don't claim any expertise in compiler design, never will. However - it pays to bring people in to help. While you only see the story from the OPs perspective there's an entirely different side from the vendor most likely (probably around fundamental design if I had to throw a dart).
Sure, I knew I'd get downvoted for the post - but sometimes that's a calculated position to show a point, and my point is - sometimes you need help, this article was very one-sided and rife with flaws. Not something I generally expect to read on HN - but I know there is likely a lot of that out there that I "just miss" because I know that I don't know.
I personally ran WiFi networks with 8000+ wireless clients on a single /64 subnet (my employer's CiscoLive conference), and assisted/consulted in running the networks with more than 25000 clients on a single /64 subnet (Mobile World Congress).
The default kinda suck, and the bugs may happen. but the statement "IPv6 is not ready for production" is wrong.
I'd be happy to volunteer a reasonable amount of time to work with the OP or others having a network of >1000 hosts, to debug the issues like this, time permitting, vendor independent. (glass houses and all that).
There are bazillion knobs in IPv6 and a lot of things can be fixed by just tweaking the defaults (which kinda suck).
Network of <500-700 nodes generally don't need to bother much. It's not optimal a lot of times with the defaults but it will work.
EDIT:
the seeming "charity" of volunteering the time isn't. I want to understand what is broken, so we can bring it up back to IETF and get it fixed + make better educational publicity to prevent folks shooting themselves in the foot. It'll make it to the stacks in another decade, but it will. IPv6 is powering nontrivial millions of hosts now - so the correct words to use "needs tweaking for my network", not "not ready for production". Let's see what the tweaks are and if we can incorporate them into the protocol, if necessary.