Having worked at a JPMo competitor for several years, we did certainly value IT. At most such shops IT spend is around $3 Billion a year, so they're not messing around. Many even created specific dev centers around the world (i.e. London, Moscow, India, RTP, Dallas) specifically to do just dev.
But, the real kicker is at few such firms will you find the tech "talent." They might join mistakenly, but usually they'll get out within a year or two to better pastures at startups or SV. What often remains are the folks that are mainly task masters at CRUD forms, but not anyone that thinks about the merits of various architectures, reads up on optimal coding practices, or that learns in their free time. Nope, those are the high paid "contractors" that are hired when folks can't figure out why a six foot long query is running slowly.
Having worked in the front office at a JPMo competitor for several years, most people around me certainly hate IT.
This is because our IT is diligent and wants to do things correctly and follow standards and practice. But there is no effort to educate the money makers to this effect, they just want results yesterday. So they bitch and wine about IT being a blackhole where requests go to die. And I agree with them in many cases as someone who's sort of midway between these people.
> but not anyone that thinks about the merits of various architectures, reads up on optimal coding practices, or that learns in their free time
That's a bit demeaning IMO. As with any large corporation, there will be people who stay for years and years just coasting and letting their skills stagnate. However, there are also a lot of people who work hard, learn rapidly, and move around a lot internally instead of changing companies.
One might be persuaded to believe that IT is actually something that provides a monetarily significant benefit to an organization... but we know that can't be true. Just because every iota of every business flows through computers and even minor software projects can multiply the productivity of employees by an order of magnitude, that doesn't mean management should be expected to value IT in such a way that they actually are willing to pay significant amounts for it!
I worked for a while for Morgan Stanley and have 20 friends that work in finance.
They pay a lot for IT. Good hacker can make 300k, but the main problem that big banks have is complexity. Years of M&A, 20 systems that do the same. Simply paying more is not a solution.
The problem is the average IBMer will not know how to secure your zOS server. If you ask them how to configure, or how to manage files they'll tell you their aren't any. This is simply amazing double speak because they are, and are not correct. When you first log into this systems you see the old zOS stuff, which looks and feels like MTS system360-67M/70 (its a long story). All your files are data sets, which aren't files, see MTS didn't have files, it had data sets.
But it isn't. Its just a mainframe layer running on AIX. Its called something dumb like encom, unicom, in MTS . That brings you into proper unix mode. Which you may think is running inside MTS, but it isn't. MTS is running inside AIX. Because keeping MTS up to date with modern hardware would be impossible. So we just trick it into thinking its processor is running at the Plank frequency. Not 200 PowerPC chips or what ever IBM shoves in these things today.
Then your in a standard unix environment. If you attack these things from the outside (and get in), you'll see and feel what looks like a BSD/SysV (ish) system. Because that's what it is. Its a unix system.
But nobody configures this crap because the mainframe dudes can just say, "Oh no there are no files on a mainframe, just data sets, they're different." But if I steal a data set in unix mode, its a file. So no, your wrong.
Nobody really goes into Unix mode half the time. They just care about keeping the system360 stuff working. Nobody cares about security because "Its a mainframe, they don't have security issues." Well they do, just like every other unix box on the planet. But unlike every other unix box we pretend they don't need security.
I don' quiet buy your assertion that z/os is an emulation layer over AIX. When I used to while away the days poking around on a z/os machine, it was not that cut and dry. I couldn't just access a dataset from the shell, and the mainframe permissions model was pretty effective at locking down what I could get too. Do you have any references to back that ie an IBM red book?
I was thinking the same thing - I knew that z-series mainframes could run Unix, but I didn't think they always did - but it looks like valarauca is correct (assuming this Wikipedia article is reasonably accurate: http://en.wikipedia.org/wiki/UNIX_System_Services). Wow, that's a whole new attack vector I'm going to have to look into now. Exciting! Thanks valarauca!
Defcon21 there was an hour long talk. (I'm not 100% sure it was 21, could have been 20).
I look at MTS on a nearly a weekly basis. I want to say z/OS's mainframe OS is based on MTS. I don't know this for sure.
This is how I arrived at this conclusion:
(This is all ancient ~6-8 years Pre-Unix System 1)
See the big draw of of the System360 was Michigan University shoe horned in memory visualization (which allowed for concurrent time shared processing). This was released as the System360-67M (M for Michigan), these things exploded off the shelves. IBM received roughly 100 orders for these systems, not requests to lease time, but cash in hand gimme that.
They rolled out the System360-70 which was just the System360-67M, but under a different name. These are going to ship with TSS/360 (Time Sharing System 360). But ya know it never got done in time, and UofM had already finished MTS. Which did explode.
Also OS/360 was batched based. You couldn't have multi-user, multi-process. Which famously modern System360 emulation can do. So who knows.
I honestly don't know. But I'm 90% sure z/OS is just AIX with MTS in some kind of BSD-esque jail. I think MTS is open-ish sourced. It had a big community up until it was mothballed universally in the mid to late 90's.
:.:.:
I certainly can't afford a z/OS system to sit around and pentest myself :P
:.:.:
Also as I stated nobody configures the security of the underlying unix on IBM mainframes. So they maybe fairly easy to own.
I think he is saying that a lot of these companies have a mainframe emulator running z/OS on unix rather than a true mainframe. I'm not sure if that's true or not, but I believe that's why he is saying you can attack from the unix side.
z/OS itself is most certainly not an emulation layer over AIX. It's a complex operating system with 50 years of history behind it.
Could be personal preference, but that actually sounds more fun than a lot of things in tech. Software archaeology on ancient and possibly broken systems? Probably has lots of annoying parts, but also sounds pretty interesting. Is it really less annoying to do "modern" work, like learning the Nth new Javascript framework, dealing with piles of Chef scripts, extending Enterprise C++ codebases, spending weeks testing browser compatibility, etc.?
My experience is that it's a lot less fun in practice than it is in theory. Uncommented spaghetti COBOL code is about as hard to reverse-engineer as an x86 disassembler dump done without debug symbols, IMO.
I used to know someone making a pretty good living doing COBOL for banks. Some people really do enjoy that sort of thing. Also a bonus of less (or even positive) age discrimination, since he was an older guy.
Failing to migrate after a M&A is failure to invest in IT. It's a stupid, short sited, and expensive failure, but good luck finding banks willing to pay for it.
Banks will only become willing to pay after experiencing expensive, massive disasters like what JPMorgan is experiencing. Maybe.
Migrating and/or killing large legacy enterprise applications is monstrously expensive, so the decision makers often decide against it. Relatively speaking, there haven't been enough very large companies that have been burned by that sort of decision, so executives at these companies (and the government) are less likely to see how badly it can go.
Then there's the very real challenge of justifying a big IT spend on retiring legacy applications to the shareholders. And the high chances of large enterprise IT projects going wildly over budget.
As much as we like to point the finger at the c-suite guys who keep doubling down on technical debt until there's a disaster, it's not an easy to make the case for paying to fix the problem. At least not until there are plenty of negative case studies out there like this one.
>>> that doesn't mean management should be expected to value IT in such a way that they actually are willing to pay significant amounts for it!
I was at one large financial company similar to JPMo and every time the development team came up with a good idea, it was met with a laundry list of requirements we had to meet in order to show positive ROI over a very short time period. Not sure if they did to discourage us from trying to help processes and development, or they really believed redesigning some parts of their network architecture could be done in a month or two.
Either way, it was quite frustrating to say the least, and after a few gallant attempts to get something going, most of the younger developers just opted out and left. I saw the writing on the wall and was in the second wave of people who finally gave up and moved on.
JP pays programmers very well. In fact, it's very good for the counties in which those programmers are employed. Well, that is, if they were in a money center. Costs centers, like security and infrastructure, those places they minimize the costs! Part of minimizing costs is all about not paying for things that aren't expressly needed or killing programs that might make sense but cost too much.
I worked there for half a year before quitting and going back to a start up. Nice people but only the people at the top seem to have a clue as to what to fix and they, themselves, can only focus on so many things at once.
So tangentially related but what does it take to increase wages in the US? We've been hearing the rhetoric for a while now that over the past 10/20/30 years productivity improvements simply go to the top and real wages decline with the hand-waving recourse that it's just the way capitalism works. But how do you actually prevent this?
Unions are obviously one solution but can be wrought with other issues. Besides that, it seems there really is no solution to wage stagnation outside of major failures or security breaches which hit the business' pockets directly.
I don't just mean in IT but it does seem to be an area where salary growth doesn't match up _at all_ with the amount of value it adds to a business, especially in the case of IT within a non-IT company.
Wage stagnation doesn't tell the whole picture. Workers are compensated through other means as well (benefits). Also, purchasing power for many goods has increased.[1]
Note, of course, that benefits have also stagnated, if not been cut. Defined benefit pension plans disappear, part time workers (with no benefits) replace full time workers, and health insurance increasingly is paid for by the worker, with increasing copayments.
By that standard, though, finance companies respect their IT staff more than Google respects their engineers, even though the cultural perception is the opposite.
In my view, salaries for programmers should be going down, not up.
For instance, I think house construction is overpriced because I think "what's so hard about putting up walls around some ground?". The answer is everything is hard about it. But that it is hard is incidental. By that I mean, it was harder before, and now it is easier, and it should get even easier (there are 3D printers printing houses now) and therefore cheaper.
A non-programmer thinks the same way about software as I do about houses: "what's so hard about telling the computer to do grab this data and transform it?". Well, all of it is hard, but again, it's incidental. It would have been more expensive in the past because it was harder, and it will be easier in the future (Docker, NixOS, mathematically proven programs that do not require maintenance, etc).
It's nice being a programmer today but we won't ride this forever. As we automate ourselves out of the market, it will be a net benefit to everyone (as no one will have to program as hard or as much as today).
That just means we will be tackling harder problems. The problems we were solving in the 70s were much simpler than the ones today. Today, we're handling things that we would have never considered possible before.
On the other hand, something that hasn't changed in 40 years is that end users are not able to precisely explain what they want. That is why we will always have jobs. There will always be some sort of exceptional rule.
That's a good money (providing it's not a 24h job), but what about treatment? Sales people make more than average devs at Google, but who gets more respect?
Which is why I took a lower paying DevOps job at a startup. I make less, but the people I work with are incredible, not just in their talent, but as a whole. I'd rather not swim with sharks.
Reminds me a bit of the problems RBS had after large cutbacks and outsourcing
had removed a lot of the in house expertise from their batch processing team.
If the security of the JPM website depends on 5 executive level employees leaving within a short time, then there certainly is something to worry about.
"The departures meant that executives with intimate knowledge of JPMorgan’s systems..."
Working most of my life in financial services, executives dont have intimate knowledge unless the systems in place happen to be the same ones they helped build 10-15 years ago (which is true in some cases).
I still consider myself hands on, although not cutting code on a daily basis, and I dont have intimate knowledge of those systems. They will still phone me at 2am despite that.
I'm not surprised at all. The article focuses on leadership brain drain, but I think it's more basic then that. Really complicated systems are getting built and deployed every day. After a while, the folks who built the system don't really know how it works anymore, because they are busy building the next thing.
After a little longer, the underlying infrastructure (OS and native libraries), platform (java, node, ruby, etc...) and persistence (Oracle, MySQL, MongoDB, you get the idea...) starts to become vulnerable to zero-day exploits and you get one of these big breaches. Maybe I'm just being cynical ;)
It's sort of like the old rusty ship that finally sinks. It happens all the time...
The departures meant that executives with intimate knowledge of JPMorgan’s systems... were gone"
Yeah, um, worked there. At least in my line of business and those others I was exposed to, losing the executives may have been beneficial in terms of knowledge because it's possible that the newcomers would try to learn the stuff that the departees never bothered to know in the first place.
Having also worked there, I can confirm. How little most managers and executives know and understand about the systems they own can be quite shocking. Their sole focus seems to be playing office politics to get more funding assigned to their projects.
As someone who used to work in tech at JPM, the exodus doesn't really surprise me. Though I don't think a few execs leaving would precipitate this kind of event.
But, the real kicker is at few such firms will you find the tech "talent." They might join mistakenly, but usually they'll get out within a year or two to better pastures at startups or SV. What often remains are the folks that are mainly task masters at CRUD forms, but not anyone that thinks about the merits of various architectures, reads up on optimal coding practices, or that learns in their free time. Nope, those are the high paid "contractors" that are hired when folks can't figure out why a six foot long query is running slowly.