Dependencies? I helped upgrade a couple third-party modules, too. That's part of the job. If you choose to use a dependency, you're vouching for it. That means if the current version of the language no longer supports your favorite library, you need to fix the library, or find a new one. If we switched from phillips to torx, and your favorite tool brand didn't make torx drivers yet, you've got to either convince them to start, or switch brands. Professionals don't get to use this as an excuse to badmouth torx, and stick with phillips.

If people spent half as much energy upgrading as complaining, this would have gotten done 5 years ago.

Apple took only 3 years to go from first announcing a new CPU architecture to releasing an x86-only OS. 2 years later, Rosetta stopped working, so there was no way to run old apps on current hardware at all. That's got to be one of the advantages of proprietary systems. It's amazing what you can accomplish when you have no choice. People complained a little but they got it done.

It's tedious yes. Boring even. But most projects get away with 2 weeks of investment. And yes, it pays back. Python 3 is a vastly superior language when it's about introducing less bugs or debugging existing ones. It's just not stuff what had be sold to the public. People want to hear about better perf or fancy features, not better error messages or banned logical errors.

The hardest code base I saw ported was the Twisted project. It's a good counter example: it was long, hard, and required very gifted people (thanks Hawkowl !).

But most projects are not like that. They are full of dumb operational logic that can be ported 80% by 2to3, and a bit of manual fix.

The fact is, people love to complain, so you will hear massively people with that particular example from the movie industry, or this guy who coded this Fortran extension that was in such a tough situation. Well guess what, that's not what most migrations are about.

Most migrations are Django/flask websites, math teachers exercises, physicists scripts, sysadmin tools, etc. Straightforward stuff. I should know, I've been moving from industry to industry for 10 years, all my clients are different, but most of them end up with similar stuff in their repo because Pareto is a thing.

Nonetheless, the screams from the first ones scared the later.

Dynamic typing on the language side, and a less than perfect test suite on the user side are not a good combination for large projects facing a project wide migration.

The existence of the cloud means that you can scale wide instead of scaling tall, which makes developer time and complexity management more valuable than hardware efficiency. Python is optimized for primary applications in exactly that kind of environment, where the performance characteristics of something like C++ don't make up for the clumsiness of the language.

In my very brief experience with python over the last few weeks this is very common. Half of our dependencies were abandoned before python 3 existed, when mercurial shuts off the hg we'll even lose the source to some of them. Python 3 get's the blame but the real problem is that the company has ignored maintenance for decades, it's entirely there own fault. There's no "business value" in maintenance until the whole lot needs to be rewritten.

Another underlying cause is package tools like pip, they make it too easy to take on dependencies with zero thought given to their maintenance windows and management think the can just ignore upgrading/replacing them regularly.

Are you perhaps thinking of when Atlassian removes the Mercurial repositories hosted on Bitbucket?

Mercurial isn't going anywhere or shutting down anything; it's open-source software with some very large users and committed developers.

(The majority of engineers at Facebook all work in one enormous Mercurial repo. Facebook naturally employs a few people to work full-time on Mercurial, and I don't think they're the only ones.)

Atlassian's decision about Bitbucket is regrettable. Especially regrettable is to actually delete repos for so many open-source projects that may not have active maintainers (rather than keep them online but read-only.) To my mind it marks a stain on their reputation that should make anyone think twice about relying on Atlassian for years to come.

But you should be able to avoid losing the source to any of your own dependencies. Between now and May 2020, go through all your dependencies and make sure you make your own clone of all the source repos.

The original programmer is long gone, and I while I am a semi-competent Python scripter and a slightly above average C programmer, I'm not looking forward to having to dig deep into this code, and patch it up to keep running.

Where did you come up with this "2 weeks" estimate? That has not at all been my experience.

Many months.

Step zero is education. Your team has been programming in Python 2. You need to make sure they know the differences in the environments, how to use create cross-compatible code, and how to use six. They also need to setup a second dev enthronement, and become comfortable switching between 2 and 3 regularly.

Step 1/2 is prioritizing. Determine how important is the move to Python 3, and what other features and dev work will you have to sacrifice to make it happen. While having a nice plan in place may provide some level of comfort to management, you can be sure it will thrown out, amended, extended, and/or ignored throughout the project. Upgrading Python dev environments brings no near or mid term value to the company. So expect developers to continue their current work-load while also attending to this tech debt.

Step one is getting the libraries into shape. This is easy if your project only relies on actively developed libraries with good teams behind them, but nearly impossible for abandoned libraries with no new commits in the last few years. Sure in retrospect, it was a bad idea to use these libraries, but at the time, they were incredibly powerful or popular. Generally, for abandoned libraries, it's easier to find a newer Python 3 library than work on fixing someone else's code. But this may mean rewriting large parts of an app, and may alter functionality, so constant communication with product mangers and a flexible approach is necessary.

Second step is working on your own code. One or two modules is no problem. But more than a dozen takes time. Scripts like 2to3 are not helpful, you need to use tools like six and modernize.

Third step is testing. If you have a good team, then you should already have good tests with known coverage. In that case, you should make sure your coverage matches, and take a very hard look at the code that is not covered. If your tests cover less than say 60% of code, you need to invest quite a bit of time of either testing directly, or building many more tests.

Step 4 is partial rollout. There will inevitably be issues you didn't think about or catch, so you need to plan the roll-out carefully and either split traffic or at the very least be able to quickly revert.

Step 5 is to watch carefully for customer complaints. Any complaint may or may not be related to the switch from 2 to 3, and you need someone on your team who stays on top of that, and can communicate the issue to the correct module owner.

Step 6 is deciding when to drop support for Python 2 altogether, as there will have to be a time where you have the two environments running side-by-side while you're testing. After all of this work, you'd think this part would be easy, but in every organization some folks will be wary about dropping support for something that already works.

Obviously, this isn't just a linear progression, you can do some of these in parallel, and will likely have to take a step back a few times. My back-of-the napkin, experience based, non-scientific estimate suggests that a 100k LOC project can be managed by two developers in two-three weeks, but a 1M LOC project (including libraries) is roughly 5-8X that much work.

Given the reach and popularity of Python, of course you will always find plenty of testimonies saying they suffered. Again, it's important to remember you can't make everybody happy, especially if your job is important. If porting is easy for 90% of projects, you did a great work, period. 10% of millions of users having it hard is still going to raise a lot of voices against you by the sheer power of numbers, but you did ok.

There is also dishonesty. Most people reporting this problem or that problem didn't actually encounter it. They are reporting somebody else experience they heard about because they wanted to make a point. Or vaguely tried something and ran away after they saw 10 minutes of fiddling didn't solve it.

In 10 years going from company to company, I met only 2 projects that were hard to port, but much, much more complainers. When I looked at what was really going on, the fact was just that they were afraid to port, and so just repeated all the stuff other people told them would go wrong. Or they tried running a few scripts for 5 minutes on Python 3 and gave up seeing too many error messages.

I know there are honest situations in the lot. But again, the most honest people are not making noise, because they just ported their code and noticed it wasn't the hardship they've been told.

...but the last to actually finish migrating. In fact, it's still ongoing.

> Given the reach and popularity of Python, of course you will always find plenty of testimonies saying they suffered

Everyone suffered. Everyone had to deal with Python2 versus Python3 bullshit. It's not just about migrating some codebase.

Everyone had to use an inferior version of Python, whether it was because library choice was limited (Python3) or because the language was intentionally left without feature backports (Python2). There was a huge amount of plain "brokenness" in the ecosystem and it was a huge waste of everyone's time.

Updating software version is part of the job. Like creating tests, writing documentation, training newbies and dealing with customers. It's not hard just because we don't like to do it. Porting to Python 3 was not hard for most people. They just really, really didn't want to do it.

I get it. I didn't want to add that on my plate too. Like I didn't want to migrate from my Centos 7, I didn't want to move from mysql to postgres and I didn't want to learn the entire setup of Webpack. 3 times. But "suffering" is a big word that has no place for the vast majority of projects.

You are stuck with a tough choice: Do I start out with Python3 and tons of broken packages? Do I limit myself to Python2 and face a costly migration later on? Do I run the extra cost of supporting both? This the choice you had face for the better part of ten years of migration. Perhaps it's not obvious that all Python-based software was worse for it, but that's what happened.

I'm not talking about some web backend service where most of what you do is trivial stuff. You can write and re-write that in almost anything, it doesn't matter.

Conda ships on 3.7, and I'd consider it the definitive "scientific" stack.

This is the best analogy I have yet read. Thanks - you nailed an argument I have had at multiple gigs/clients way to often.

> Professionals don't get to use this as an excuse to badmouth torx, and stick with phillips.

D'accord. Exactly. I had very fruitful discussions with professional builders about tools, tool brands and their respective strengths and weaknesses - but never has a professional decried Bosch Pro vs Makita vs DeWalt. One has ones favorites - and that is fine. But badmouthing - I have never encountered.

I don't think there's a time I can remember when Norm Abram (probably now one of the greatest or at least most prominent carpenters of a generation) ever mentioned the particular tool brands he was using.

And looking back on earlier seasons of This Old House, he was hand-nailing thousands of nails a day (nowadays he and everyone else often uses pneumatic tools for this).

I'm sure he prefers a certain brand, but unlike popular YouTube personalities and more 'celebrity' builders, he isn't a brand ambassador for Craftsman, DeWalt, Milwaukee, Ryobi, etc.

But yeah, when you talk about 'builders' (the more common carpenters you encounter on most job sites), it seems like most of them will die on a sword defending whatever brand(s) they've sunk a fortune into.

I'm a DeWalt guy, mostly because I invested in a 20V Max drill a long time ago and have a ton of batteries for all my electric tools now. But I used to use Makita before switching systems. I wouldn't call myself a master carpenter by any means, but like AvE, I find good and bad in almost all the 'top tier' lines from tool manufacturers.

You want it written in C for a 16bit bargain basement ALU? I can do that. You want two implementations, one in Pascal and the other in PHP? Not a problem. You want my advice? No? That's fine too, the advice is expensive and who knows after all these years if it's even worth anything.

At my first job we used to build some hydraulic models in hard wood that where works of art.

Our wood shop was so good they made a museum quality piece of furniture for our retiring boss and it was better than the best Chippendale furniture when new.

I do like the grips and ergonomics of some brands more than others.

The one piece of concrete advice I got was “Whatever brand you get, go for the 20V, not the 12V”. Last week I saw that same guy at work showing off his 12V driver. So I’m pretty sure nothing matters, and I’ll probably just get the brand with the prettiest color.

That's fine. I'm sure there are some that also don't have a budget for fire insurance, or a security system, or whatever. For anybody who wants to gamble with no fire insurance, or not fixing tech debt, that's a conscious choice they get to make. Now they have to be responsible for the outcome(s) of their choices.

There are many maintenance mode projects where the switch to 3 represents absolutely no upside and is all downside risk. Not everyone is developing web apps in constant flux that are exposed to the Internet. That is a tiny fraction of python’s use cases.

A security system and fire insurance is at most around $2000/month, and it's a cost that is understood and often required by law. Diverting a $100k+/year employees to spend their time on tech debt is quite a bit more expensive.

Companies rightly try to minimize time devoted to tech debt.

If you built software using Python 2 at any point in the past 5 years, you have no right to complain about the added technical debt of updating to Python 3. But not wanting to incur that technical debt in the first place would also have been a good reason not to use Python 2 in the first place; whether that means adopting Python 3 early or simply using a completely different language.

Congratulations. You're fired.

"Hey boss, we're using an outdated programming language for which security updates are no longer provided, because we were told there's no budget for fixing technical debt."

"Congratulations, you're fired."

Same outcome either way... might as well make some effort to do the right thing along the way...

People will continue to use py27 going forward with likely a pay-for-support runtime because that’s cheaper and less risky than changing a million lines of python code.

Like it or not, there are a lot of teams that choose to incur technical debt to avoid what is seen as needless work on something that is working fine at the moment.

But hey, if some 3rd party group effectively forks the language and maintains a Python2 branch after the official EOL date, good on them. But I'd argue that using that is risky in the same way that using a Pale Moon or Waterfox is (arguably) riskier than using mainstream Firefox.

For what it’s worth, Perl 6 can load Perl 5 libraries: https://github.com/niner/Inline-Perl5

Python3 broke compatibility for spurious reasons. The language didn't change that much, it could have kept compatibility with minor concessions.

> Java or COBOL maybe have done a better job of not breaking backwards compatibility...

Literally every single other major language has done a better job at it. That's a big part of why they are major languages. Also, most of these are statically typed, so any minor breaking change is far less risky to pull off.

> ...but Python 3 isn't exactly Perl 6 here.

Well, maybe if Perl6 didn't happen Perl wouldn't be irrelevant today.

C++'s upgrade woes make Python 3 look downright trivial in comparison.

If you're talking about ABI breaking, that's a different story. Python breaks the ABI in every major release as well. That generally requires a recompile, not a major code migration.

Security through obscurity, baby...

Every non-tech business that maintains technical infrastructure finds itself doing that kind of tech-debt work regularly (if, often, chronically behind the vendors recommended schedule) for every platform they use Py2 to Py3 may (or may not, really) have been a bit more painful than a (multistep, because those types of shops never do it timely enough that a single-version-bump makes sense) .NET framework, Visual Studio, SQL Server, and Windows Server upgrade (and, yeah, those often happen together as multistep upgrades in slow enterprise shops), but it's not a fundamentally different kind of problem (and replacing dependencies that didn't make the same upgrade, often because they were abandoned years before the firm made it's upgrades, is often a part of that.)

The thing about debt is that ignoring it doesn't make it go away magically. If business relies on tech for crucial functions, it's only a matter of time before debt comes due.

If you don’t push your clients/employer to have room in their budgets for these types of things, you are the one being naive.

1. Python core dev pretended Python 3 was good and ready by, like, 3.1. It wasn't.

2. While your problem may have been painless (I'm glad), that doesn't mean that everyone who complained was just complaining.

(I use Python 3 as the default now, but as someone intimately involved with an async IO library at the time it came out, I maintain that a) the transition was botched b) core dev did not listen to any of the problems people pointed out for _years_.)

To give you an idea of what botched means: several people, names withheld to protect the guilty, had to wade in 6 months worth of shit to get a 2.x release that made TLS vaguely OK, in, like, 2014?

IIRC the messaging from Guido and Python core dev circa PyCon 2008 and 2009 was that it was good enough for people to start thinking about how they might migrate. This resulted in experiments with different approaches to migrations, maintenance of versions for Python 2 and 3, etc, but at that time there wasn't a clear end date and no rush to migrate. It seems like a lot of people misinterpreted what was actually said. Similar misinterpretations seem to happen within the wider internet community around the GIL and perf considerations. That cliff is finally here ten years later after a lot of experience with migrations.

3.5 fixed % formatting for bytes, further making 2/3 transitions easier (or harder, depending on your use/abuse of strings vs bytes).

3.6 added the first 'exciting' new feature: f"formatted string {literals}" if you don't have an asynchronous type project that can make good use of async/await.

(Or in language-lawyer terms: CPython 3.6 introduced this behavior and documented it as an implementation detail, and for Python 3.7+ it's guaranteed a feature of the language proper.)

    % python3.7 -c "import sys; print(sys.argv[1])" "$(echo -e '\xff')"           
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' in position 0: surrogates not allowed

The key surprising thing that's going on here is this clever hack (clever, but a hack):

> In Python, file names, command line arguments, and environment variables are represented using the string type. On some systems, decoding these strings to and from bytes is necessary before passing them to the operating system. Python uses the file system encoding to perform this conversion ... > > On some systems, conversion using the file system encoding may fail. In this case, Python uses the surrogateescape encoding error handler, which means that undecodable bytes are replaced by a Unicode character U+DCxx on decoding, and these are again translated to the original byte on encoding.

https://docs.python.org/3/library/os.html#file-names-command...

This is meant as a way of fudging the fact that (a) for UI purposes, you want to treat filenames as text strings; (b) your Linux filenames are probably all encoded as UTF-8 (or your locale encoding); (c) but they might not be -- they could be arbitrary bytes, except only NUL; (d) and if they are, you really want to not munge the name when you go back and try to operate on the file.

The fudge is that filenames get decoded as (by default) UTF-8... but if invalid, the offending bytes get stuffed into the UTF-16 surrogate space. Then filesystem APIs encode as UTF-8, except they look for that surrogate hack and turn those to the original bytes, so it all round-trips.

It goes pretty wrong if you try to hand such a hacked-up string to something that just expects to encode normal real Unicode with UTF-8, though. That's what's happening in your example.

The magic words are `os.fsdecode` -- that's how you get back bytes round-trip clean from that hack.

I would expect any stable release of software to be "good and ready." Can you explain what was wrong with Python 3.0 and 3.1?

> core dev did not listen to any of the problems people pointed out for _years_

What problems were those?

[1]: https://bugs.python.org/issue19977

[2]: https://www.python.org/dev/peps/pep-0540/

[3]: https://docs.python.org/release/3.5.0/whatsnew/3.5.html

I appreciate the general point that that's far more recent than Python 3s initial release but it's also a large exaggeration to say "until this most current release".

Also to understand why it took so long is because Python 2 kept improving after Python 3s release. Python 2.7 was a great release and many further features from Python 3 got backported in it's 2.7.x point release.

The post you're responding to starts verbatim with `Python 2 to 3 (at least by 3.3 or so)` and your reply is about Python 3.1, released 3 years earlier. The amount of changes was quite big.

Just let it go...

Good for you. Some of us had code bases of considerable size and complexity though.

The fact is that for working software on the python platform, this upgrade represented work that had to be done that for a legacy app that was still chugging... little benefit. If you already coded around the python 2 limitations for Unicode eg, then python 3 was not a help at that point for something already in service. It was just more cost for no benefit.

> If people spent half as much energy upgrading as complaining, this would have gotten done 5 years ago.

These aren’t exchangeable. Why do people on the internet think bitching (either constructive or not) is some valuable currency? Often times the people complaining had no means or position to of the work. 4 promotions later I sure as hell wasn’t going to participate in that 2 to 3 mess on a project produced years ago, but I could still opine in the situation. I also had little incentive to fund it.

Your analogies are bizarre. No idea what you’re trying to convey with philips v torx but I’m going to wager it’s explanatory power in this case is shit anyway. I can still demolish it, having actually worked in manufacturing there were times we told a supplier (ie Python) to fuck off and piss up a rope because what they were proposing was not compatible with our existing tooling and it would be too costly to convert for little benefit to us.

I respect apples prowess in the consumer space, but there’s a reason you don’t see their products regularly put into industrial roles where your timeline is more than 5 years. Apple products are disposable, many applications in industry are expected to last. Python is a general purpose programming language (or at least billed itself as such). Your comparison is poor.

Isn’t that comparable to java? Oracle gives you paid security updates on version 8. You can keep using the previous version without security updates or migrate to OpenJDK 8, which have their own security group. More options, seems better to me.

In the vast minority of cases.

Not in most electrical engineering (except power plants), nor in computer engineering. 60 years ago was 1959. What software/computer project from that time is still running?

Even moving outside of the electrical domain, how many physical products outside of civil engineering is expected to last that long. I certainly can't expect support for my car for longer than 25 years.

I've yet to see a CPA adequately depreciate software assets, anywhere. Maybe we should help them out by building better depreciation schedules. There seems to be a lot of CPAs that don't believe software depreciates over time, and maybe that's the largest disconnect in labeling it "tech debt", because accountants hear that term and think they can ignore it on a balance sheet they don't care about, but "tech depreciation" might actually scare them straight (until they find the tax advantages in calling it that).

In a previous job that was mostly in an engineering/manufacturing department, but with a lot of Perl/Python automation scripts, we had an internal conference. One of the keynote talks was when not to automate using SW. It went into the cost of maintaining SW over the long term - including the fact that authors leave, and people who understand their scripts require higher salaries. Most people who write these scripts are not hired for SW roles, so their replacements likely cannot debug/extend.

But I get your point, those are special cases for preservation. For mainline things, especially with today's processes for continual integration, dependency checking and advance build tooling, dependency rot is something that should be accounted for in all software project plannings. If your dependencies are a few months out of date and you don't have the time to update them and re-run tests (people write tests right?) things are just going to hurt more and more later.

> PSF

These are two very different kinds of institutions!

People who need COBOL support from IBM are paying a lot of money. Giant piles of money can get you many kinds of help that people won't volunteer to do for free... among them, maintaining ancient software in amber.

If you need Python 2 support and you are willing (and able) to pay the kind of money that IBM's customers pay for COBOL support, you'll be OK. For a start, Red Hat (aka also IBM!) shipped Python 2 in RHEL 8, which means they'll be supporting it until 2029 at the earliest.

Er, why not? It's not like there's some kill switch in Python 2 that will make it stop working after January 1st, 2020. If it works now, then it'll still work, you're just not guaranteed fixes anymore. At least, not for free. As stated in the article, paid support options exist from several vendors.

Name one.

If you haven't put in the priority to update your Py2 to Py3 apps by now, I really think your shop has the wrong priorities.

It's not just about Py2/3. Dependency rot is one of the worst form of technical debt. It often shows broken CI, broken security scanning, lots of generally broken processes that will just keep hurting a team further and further down the line.

CG/Visual Effects industry is still firmly using Python 2. Only in 2020 are they taking the first step to transition to Py3 [1]. Users and studios are held back because the Python runtime is used inside major applications; Nuke, Houdini, Maya as well as libraries and APIs. None of them have released a version that runs Python 3 yet.

The reasons for delaying it (mentioned in a footnote on that page) makes sense to me. Previous years were focusing on coordinating updates to GCC, Boost, C++14, Qt, and waiting on Python bindings for Qt.

Also, I've worked at a couple studios many people have probably heard of and none of them have unit tests covering much of their code. The focus is on tools that facilitate in-house artists where responsiveness to needs are valued over architecture and completeness. Requirements change for each project and previous requirements are often sacrificed (until a new project needs them in a few years).

I'm itching to move to Python3, but even for standalone tools I've felt it better to choose a completely different language (or Python2) instead trying to mix Python2 and 3 because having them co-exist creates more headaches in managing the environments, dependencies, and coding styles.

[1] https://vfxplatform.com

I am not sure how you could possibly know this. You have no idea what else they were working on instead.

I think in this case at best the analogy grossly oversimplifies the issue. If it really were just a “screwdrivers” problem as explained then the 2 to 3 migration would have been mostly trouble free and would have happened. Clearly it did not go that way so that analogy can not possibly be appropriate.

I think many people underestimate the challenge that the 2 to 3 migration presents for large enterprises. The core issue is that even though the migration for any given module is normally really easy, the total effort required to migrate is still essentially O(n) in module count/file count, because even with current tooling you still need to have an engineer look at every module to do the change safely. Even if it only takes ~5 minutes per module to make the changes and validate that it works correctly, this becomes a giant undertaking when you have tens of thousands of files to migrate.

The fact that it takes a long time also creates other problems. Your business isn't going to hit "pause" on other development, so there will be changes constantly introduced into modules you've already "swept". It's going to be hard to make sure 100% of your engineers and code reviewers are knowledgeable about the specific requirements to make sure the code works in both 2 and 3, so you would really like some automated safeguards to make sure they don't introduce anything that won't work in 3. Pylint helps with this, but won't catch everything. Unit tests are obviously essential, but:

1. Even a well-tested project won't have tests that cover 100% of code paths and behavior.

2. You're stuck running the tests on both python2 and python3 for the duration of the migration, which doubles the resource (compute, memory, etc.) cost of your Python CI and regression testing infrastructure for the duration of the migration.

I see far too many commenters attributing the delayed migration to laziness or complacency. In reality, most big companies have passionate Python advocates who really want to be on Python 3, but the scale of the problem and the lack of tooling to tackle it with a sub-O(n) amount of effort make the overall project risky and expensive for the business.

Compiler errors sure would be handy here :^)

Also, your screwdriver analogy makes no sense. The choice here is not about "your favorite tool brand". Mixing and matching tool brands is straightforward. The more appropriate analogy with screwdriver heads would be to imagine that you now need to change out all your hand tools, manufacturing lines, assembly robots, and equipment from multiple vendors from phillips to torx all at once. Sure, you can do some prep work, but you've got to coordinate a cutover at some point, in sync with tools that you have no control over or have to recreate from scratch. All for zero benefit in the end, other than the hope that they don't make the same mistake again with Python 4.

Also I think you're too harsh on the analogy, most of the time there's a way to write python2 and 3 compatible code, so which means the transition can go smoothly.

I’m not sure what more they could have done. Python 2 was released almost two decades ago, with an original planned sunsetting that got pushed back to 2020. That’s plenty of time and respect for their users.

>All for zero benefit in the end, other than the hope that they don't make the same mistake again with Python 4.

You’re free to stick with python 2 for as long as you’d like. You’re free to fork the project and continue develop it. To say there is no benefit to new and evolving languages is odd and short-sighted.

Not broken 2? None of the breaking changes in 3 were necessary.

That seems like a very specific example...

_Everything_ I relied upon, every single dependency in my projects, gradually migrated over the course of a year and nothing broke.

Furthermore, I now have asyncio, aiohttp and friends to play with, and it's been pretty good.

> If people spent half as much energy upgrading as complaining, this would have gotten done 5 years ago.

Yeah, right. Look at the creator of Python himself. Took about 3 years to migrate just his employer, and that's a top-tier software shop [1].

I've heard many people talk like you over the years, always being dismissive of the cost to others in python 2 vs 3. The complaints against the migration were not unfounded.

1: https://blogs.dropbox.com/tech/2018/09/how-we-rolled-out-one...

I wager that in many cases, fixing something that isn't broken never reaches the top of the TODO.

I suppose you don't use Java.

Isn't what code review and CI/CD is for?

As a manager one should be extra careful to be _seen_ eating your vegetables, flossing your teeth and getting your code reviewed.

(What you do in private is a different matter. In some companies office politics can require some skullduggery.)

Contrast this with the Postgres website, which tells me I'm browsing old docs (because Google still offers old links), see e.g. https://www.postgresql.org/docs/9.2/tutorial-window.html

Are there plans to add a banner to Python 2's docs? I think it would be quite instructive.

As a casual python user, it is quite common to stumble in some issue, after somme googling land in a doc page, try my solution, and find it does not work. Then googling after the failed solution, noticing that the previous doc page was describing a Python 2 feature that do not exists (or works different) in Python 3.

I just wish they'd named it something different. If there's neither forward nor backward compatibility, I'd argue they're not the same language and should not have the same name.

https://perldoc.perl.org/perlpolicy.html#BACKWARD-COMPATIBIL...

I'd support that if every Python 3 page had a link saying "if you're still a sorry SOB stuck on Python 2, here's the link to the related library." It's just as bad for Python 2 developers, it's roughly a 50/50 shot of whether you get docs for 2 or 3.

https://bugs.python.org/

Or email the mailing list for more help?

https://mail.python.org/mailman/listinfo/python-list

- https://github.com/python/cpython/commit/46ed90dd014010703c7... - https://github.com/python/cpython/pull/13638 - https://github.com/python/cpython/blob/master/Doc/README.rst...

And because of changes in PostgreSQL versioning, people don't release that the oldest support version (9.3) is rather old.

PostgreSQL 9.6 was released in 2016, 10 in 2017, and 11 in 2018.

But PostgreSQL 9.3 was released way back in 2013...not quite as old as Python 2.7 but close.

Also they are color coded.

In 2019, I feel pretty confident about using Python 3, having used it exclusively for about 18 months now.

For my personal use case at least, this timeline worked out well for me. Hopefully it works out for most everyone. I can't imagine they made this decision without at least some data backing it up.

Python is an open source project I've used and contributed nothing to, so I don't have the right to be a back seat driver. Were it a commercial project and I was a customer, I would be quite upset.

Were I a commercial customer, and had been told to switch to the new version ten years ago, then it's on me if I still have started the migration yet.

https://github.com/getsentry/sentry/issues/8806#issuecomment...

https://github.com/getsentry/sentry/issues/1152#issuecomment...

I mean, even supervisord is migrated at this point, and it was a long-time laggard.

Actually, I can't wait to remove it from my hard drive on Jan 1, since it's one of the most poorly designed apps I've ever used. It's really not surprising at all that they're not able to migrate it, assuming the quality of the app is indicative of the quality of the code. I wouldn't be surprised if the same was true of a lot of applications that are refusing to migrate. (I'll probably have to find an alternative to ebook-convert, the command line tool that's the only part of Calibre I use, or maybe I'll rewrite it in Python 3 myself.)

The gui, Calibre proper, is a total nightmare. Functional, but my god is it esoteric and ugly. I've heard people actually use that gui as their primary ebook reading software and I just can't fathom how they find that tolerable.

https://manual.calibre-ebook.com/generated/en/ebook-convert....

https://imgur.com/a/nlAAlZn

> No, it doesn't [need to convert to Python 3]. I am perfectly capable of maintaining python 2 myself. Far less work than migrating the entire calibre codebase.

That's a very strange quote. I want to have this much self-confidence.

You can write code that's compatible with both Python 2 and 3, but I don't think you can expect to be using Python 2 exclusively and write code that works well in Python 3 unless you're constantly running it with Python 3. Especially, if you don't have much firsthand Python 3 experience. This is much easier if your code is isolated from those dependencies and you write tests that run in both Python 2 and 3, but that's quite a bit more of a commitment than you seem to be implying.

I get your point though about not being able to run full integration tests due to dependencies. Still, if you were making a good effort to keep the code Python 3 compliant, it would make upgrading much easier.

Also, my other suggestion- contributing to Python 3 compatibility for modules you depend on! That will give you Python 3 experience :)

    import subprocess
    import re
    import yaml

    output = subprocess.check_output(['ls', '-l'], universal_newlines=True)
    matching_lines = []

    for line in output.splitlines():
        if re.search(r'total', line):
            matching_lines.append(line)

    with open('matching_lines.yaml', 'w') as f:
        f.write(yaml.dump(matching_lines))

Python 2.7: https://repl.it/repls/OrchidDirtyNotification

The much worse case of this that I dealt with a few years ago was reading in and modifying XML files, and then saving the modified XML contents as strings in a yaml file (a keyed cache). The input XML files (which I didn't control) were not consistent about having the encoding marked, and that really made things a mess for ElementTree.

A taste: https://github.com/ros-infrastructure/rosdistro/search?q=utf...

I know that's not the version of Python you were targeting, and I know you didn't write this program to be cross-platform, but I wrote a similar program and I was repeatedly surprised by those sorts of problems. I don't entirely agree with mikepurvis, but I do feel his pain.

This is simply untrue— there are a dozen small stumbling blocks that mean that writing bilingual code has a real cost. And it's not as easy as just having the right CI setup, especially if it's ten years ago and most of your dependencies haven't migrated.

Q: Ten years ago how would you have explained to your manager that you needed your team to start writing 2-3 compatible code, which obviously takes considerably more time and effort than sticking with 2 compatible code?

At this point, all the code I write is 2/3 compatible, and the things that annoy me are all py3 only features. Backwards compatibility stuff is pretty easy. Most of the easy problems are easy, and six has solved all the hard ones for like 7 years.

And I've been writing some of my code 2/3 compatible since 3.4.

While car analogies are notoriously bad, it's much more like "I'm going to buy a plug-in hybrid and use gasoline until charging stations are widely available". You can write code that works in Python 2 that takes zero effort to run in Python 3. I (and many others) have been doing this for years. Just look at the number of packages on PyPI that run, unmodified, on 2 and 3.

[Edit: gas -> gasoline]

- 2008-2012 Python 3 becoming usable (byte formatting, six library)

- 2012-2016 Libraries (Django, etc) becoming compatible

- 2016-today Applications (Trac, Ansible, Chrome[1], etc) becoming compatible.

[1] https://bugs.chromium.org/p/chromium/issues/detail?id=942720

If the deadline wasn't 2020 but 2024, then you wouldn't get more time, simply we'd be at "not yet ready to migrate" state right now, as major libraries would not have switched yet.

I had to manage a mature codebase upgrading Ruby versions, and it was painful. And that is fairly mild by comparison; not the big jump from Python 2 to 3.

A lot of devs like to not worry about new things.

Library, as is most open source libraries out there, sure.

For a big company-internal Python codebase that also has to wait for all the major libraries to migrate first, that time frame can quickly shrink to 1-2 years, which is very little.

Beggars can't be choosers; if your company wants or needs major libraries to have migrated 4-5 years before the deadline, then your company has to participate in making that migration happen. Or accept that it's going to be done later than you'd like, because the needs and motivation of these library maintainers are quite different from the motivation of Python core maintainers.

“The free libraries we chose to depend upon haven’t migrated.”

Do you have programmers?

“Of course.”

Put them to work.

“We have other priorities.”

Well then. Your choice, your outcome.

Sure, they will have to cope. One of the ways people cope is by accepting security risks and continuing to run with old versions. Another is to decide that Python isn't for them, and move resources toward better platforms with commercial support for older code.

The fact is, that until a few years ago there were too many libraries that were not compatible for most organizations to make the switch. So, effectively the window for those organizations has only been a few years.

I think he's blaming organizations, not engineers.

I also think organizational dysfunction is a problem that isn't a technical problem that the Python project has the capability or duty to solve.

No one is doing that.

OTOH, it's not PSFs job to protect developers from bad management.

If there wind up being major security issues, there will be little choice but for someone to take up the mantle and fix them. Because companies aren't going to switch, and when you've got a botnet ravaging the internet they can't flip a switch and do a big-bang rewrite, someone will have to push out a patch for them.

This isn't the end of Python 2, it's just the Python team washing their hands of it. Ask PHP or Cobol users how their attempts to kill legacy codebases worked out for them.

This whole endeavor was complete folly from the Python foundation to begin with. A big-bang rewrite because they didn't like the syntax, and they didn't even fix the GIL while they were at it.

At no point do you have to wait on anyone else, you choose to.

The Python 2->3 window is an example of where being too nice to too many users harms the project. Once Python 2 is dead, people will choose between being the sole maintainer of a Python 3 fork of some Python 2 library versus a being a sole maintainer of their entire Python 2 stack from the language layer on up.

Still better than my last programming job where the vendor of a vital piece of HW had provided a C++ API (also vital), but refused to provide dlls that would work in anything newer than VS2010. At least with open source you can do something about it.

Programmers ~= Resources ~= Money

=> If you want your dependencies to be upgraded in a reasonable time frame, try to contribute to financing the maintainers (or support them in a manner _they_ find suitable).

Personally, I would guess that many organizations will be running in Python 2 for at least parts of their codebase come 2020.

Is it though? Especially since as late as 2014, everyone was under the assumption that in 2015 security support would end. We were already trying to migrate long before that. The five year extension seems pretty reasonable to me.

Actually, depending on what action you pin dates to, it was twice as long.

One other thing - you can’t run non-64 bit apps anymore. You can still run Python 2.

I feel like this the python community has been the epitome of class regarding communication during a transition.

There are an increasing number of applications which require .Net 4.7, we found one early this year.

Promising "10 years of support" for a product is great, but if/when other vendors decide to undercut this, you can still be SoL :(

Where they only release security updates and no breaking changes?

At the company I work at, we've got a large amount of Fortran code from the '70s that works fine with the latest and greatest Fortran compilers (barring deprecation warnings which help guide refactoring efforts). Same with C/C++ from the '80s.

There is also the case of Oracle, which defines support as "you can read our knowledge base and we will answer a phone before telling you it can't be done", meaning their support is literally for life (of the company). And they still manage to screw it up by deleting old KM articles...

Everything else in IT goes towards shorter and shorter support windows, because the field is always in flux and change (or rather churn) accelerates every year.