Having worked on projects related to integration of other systems with the banking systems I can confirm a massive lack of skills, experience, and imagination amongst developers and managers working for banks. They pay hundreds of millions of dollars and get very poor return on their investment in IT because they themselves have no people working for them who have any idea how to build a secure system and the consultancies providing bodies have no idea how to build what the client wants. The turf what between various consultancies and managers make things even worse. The results are mediocre and laughable.
This is by design. For some ideological reason it looks like the biggest fear of bank execs is to depend on talented developers.
The prospect of a developer or team that completely crushes deadlines and deliver beyond expectations is horrifying to banks.
They find far more palatable to pay tens of millions every year to IBM, Microsoft, Oracle and even far less known and ancient entities than, horror of horrors!!! Have to pay FAANG salaries! This is absolutely a no-no for bank bureaucracies
I've heard this many times, also directly from a guy I knew years ago, so I guess there has to be some truth to it.
What I don't understand is how that comes to be. Are just people who are not skilled enough applying to work at a bank? Do they have some weird selection criteria that tends to discard all the good people?
You'd think that they'd hire some good people by accident and those people would be more than capable of running the whole operation, but apparently they don't.
Decisions are all top down.
Projects are run top down. The developer delivers the spec the consultant has approval on. Each idea gies through rounds of approvals.
Have a great idea as a developer? Save it for your next role.
Spot on! Also many projects end after achieving the main goal - a powerpoint slide deck, claiming something was done. No one has ability to ask what was done and why it was done. Also the same project is reworked multiple times, during dev stage, over multiple years as the tech becomes old or other dependencies change.
When you do release everyone claims it is never what they wanted. As a developer your job hangs on the fact that you followed the spec. It makes you happy when they cancel a project at the very end. You get to do the fun stuff (coding) and don't have to deal with the negatives of a release.
> Have a great idea as a developer? Save it for your next role.
And if you don't save it you'll get a bollocking for showing initiative. Because, apparently, our fetishistic obsession with WIP-limiting means we can't walk and chew gum at the same time.
I've never worked for a bank but this all sounds very similar to a (mercifully brief) contract I did for a multinational in a different sector many years ago.
In my experience of these things the environment at many banks greatly reduced the effectiveness of technical software staff. For example when I went to do some workshops for a bank a while back on data science, half the room didn't have their access approved to get the data and a third of the room didn't have access to the big data platform and many couldn't install any analysis software on their work laptops and it had been over 8 weeks since their project started. People who are really passionate about technology will find this sort of situation quite annoying.
The level of technical debt is also huge. So there is just tons of bureaucracy and red tape that you couldn't even make a big difference if you wanted to. The difference of working at a startup vs. a big stodgy company is just so immense. Not saying startups have it all right, but even if you are a 10x dev in the startup world, you would be amazed how quickly that motivation could dissipate working at a place that moves at a snails pace.
They do hire good people sometimes but then those good people leave. One of my colleague was complaining about getting sign off from 12 different people for releasing a minor feature to Prod.
Commenter is talking about retail banks that outsource everything. They get bottom of the barrel outsourcing firms that sends them any hot body that's available (you don't need to be a developer to enter an outsourcing firm).
Add the usual outsourcing issues, managing communications across 3 countries and 3 opposing project managers and little happens.
Banking does have good developers but they mostly aggregate in investment banks and hedge funds, that are inhouse and have more means. Retail banking is unfortunately not a great industry financially or technically.
The same reason that trust fund kids end up being dumb and lazy. When you have a corporate cash machine printing money, you end up getting lazy and stupid.
it seems to me that a 1.3MB file is a perfectly reasonable size to include in a webpage, and much better than a regex for this purpose - a file containing numbers is maintainable without having to also maintain a test suite to ensure that your regex is still valid every time the company is assigned a new phone number.
maybe a server-side lookup would have been a little bit nicer than shipping the whole list to the client side, but a static client-side app can be stored in the CDN layer, doesn't require provisioning any server resources, and sidesteps any security concerns about servers processing user input.
it would have been nice if they'd cleaned up the user input before querying the dataset, but overall this looks like a clever solution to release a feature with the smallest risk and maintenance profile possible.
Several years ago, along with some cofounders, i created a Hearthstone website / database (hsreplay.net). If you look at the card database it's incredibly fast once loaded because all the filtering and displaying etc happens clientside. It downloads a 25-something megabyte JSON (which compresses extremely well), cf. hearthstonejson.com.
Downloading a big JSON file locally is underrated. I implemented the exact same strategy on https://overframe.gg, a Warframe database. It allows a fantastic user experience ... the tradeoff being the slow loading time (which is invisible to most people on good connections).
Overframe was a SPA though and had to lose the approach on most pages because of SEO. The usage were too different. Before that every single page on the site was clientside and available immediately after initial load.
For a tool though, especially one that relies on user input (and thus will load in the background before your user starts interacting with it), it's a great approach as long as your data is somewhat bounded.
Agreed. I’ve written a lot of B2B tools and locally storing a copy of lookup data is perfect when appropriate due to the generally smaller dataset sizes.
Back in the day I’d even do super fast geo proximity locally by including a json of the locations of first three digits of zip codes and truncating the zip code of the user to make the comparison.
Couldn't have done it without Semlar, the current maintainer (fun fact, i never played Warframe, this was a job for a client/friend). Glad it's of use to you!
Agreed, this 1.3 MB JSON data file is perfectly reasonable: it compresses to only 191 kB in the HTTP response with standard gzip compression that all browsers have been supporting in the last 20 years (I just checked with Chrome's dev tools)
I'm struggling to see how a regex would work, either. It's a database lookup, a regex that matches only 74k numbers out of every possible valid phone number in the UK would be absurd.
I think the intuition is that a lot of these phone number come in ranges, so it would not be a regex but multiples? But yeah seems easier to just encode the ranges, than some complex system to create the most minimal regex.
edit: checked the comments of the original post and they apparently used perl to produce a 11kb regex.
I assumed they meant that a regex here could be used to just extract out the numbers and remove anything extraneous. They mention it fails with trailing characters, or typical punctuation used in numbers (eg. dashes).
A little regex to remove out anything non-digit would leave you with just the numbers to compare to the numbers in the list.
Wouldn't it be better to perform a server side lookup to avoid leaking the full list of currently used Barclays numbers in plaintext to fraudsters?
The fact that there isn't any user input sanitization makes me inclined to believe that this implementation decision may not have been well-thought-out (which I'm guessing was the author's point exactly)
when you're offering a service designed to expose a list of numbers, there's no way to design it in a way that doesn't expose that list of numbers. given enough time, a motivated person will extract all the data.
most likely, this page started out as a list of all 74k phone numbers, and at some point somebody built a lookup tool on top of it. there's no reason to assume that keeping the full list secret was ever a design goal.
I guess you don't pay per MB for data? Or live in an area with poor connectivity?
Making life easy for a web developer, rather than saving time and money for users, is unethical behaviour. The developers are putting their (and their company's) comfort over that of the people using their system.
Perhaps that's OK on a day-to-day basis. But this is a system designed to help people in a stressful situation trying to ascertain if a caller is genuine. That's not a situation where you want to needlessly pipe a couple of hundred KB over a dodgy 3G connection.
It’s only 191 kB compressed, which seems like a reasonable thing to send over even a dodgy 3G connection. Not that I think it’s a good idea, but in terms of bank IT I’ve seen far worse.
You have no real idea of the various numbers involved or the ins and outs of the decision, so to accuse people you don’t know of “unethical behaviour” is ridiculous. Get over yourself.
I agree that this approach is actually pretty reasonable for checking if a specific string is in a relatively small list of possible values that can't easily be expressed as a simple set of prefixes or whatever. It's must simpler and easier to maintain.
The bigger problem with this approach is that it exposes a canonical list of phone numbers that are guaranteed to not be flagged as fake by the tool itself. That isn't likely to be a _massive_ concern here specifically, since it will always be easy to find a valid customer service number without that access. But I'd expect a team looking at dealing with fraud or scams to be pretty conservative about information disclosure generally – it would be a good argument to implement this as a backend service.
We’ve no idea how difficult the work environment at Barclays is, I don’t personally feel you should judge other people’s work unless you know exactly what went on there. Clearly there would be changes I would make (in a code review hopefully) but I’m not sure digging people out on hacker news is going to help.
That is a fine tradeoff until you realize that a single, manual verification of these 1.3MB telephone numbers costs more than developing and maintaining the unit tests...
If you sort the numbers, then represent them as the amount you need to add to the prior number[1], it produces a very small file once compressed...about 4k[2].
So, even with the code to turn it back into a list, handle the leading zeros on many numbers, etc, it would be under 10k total, much smaller than the crazy regex.
Not general purpose, and a bit silly, but you can get the list pretty damn small.
[2] The < 4k gz file that you can reproduce most of the list from: https://github.com/tyingq/barclays/blob/main/out.c.gz?raw=tr... Gunzip, then read each line, then add the prior read line to it, prepend a leading zero and print. "Prior read line" is 0 for the first line. There's a small amount of the list you would have to hardcode or handle another way...the numbers without leading zeros, total of 63 numbers.
> If you sort the numbers, then represent them as the amount you need to add to the prior number[1], it produces a very small file once compressed...about 4k[2].
That seems like a nightmare to do searches on. I'd probably compress it with a prefix tree or similar. In terms of off the shelf solutions, sorting the list, saving as a newline delimited text file, and then compressing with LZMA2 gets you down to 22KB.
It's easy to do searches on 74,000 numbers. Unpack them into an array and do binary search. No need for prefix trees when binary search will do just fine. Much simpler, too. The unpacking process is trivial if the list is packed using the method described above, where diffs are stored, not whole numbers.
The point is that they didn't think to do this or anything else to try to keep the bandwidth down, while still managing to deliver a lazy sloppy product that doesn't work very well for its sole purpose.
I don't even know if this qualifies as a lazy effort, as that implies some effort was put in. This was an effort-free implementation. The user, as usual, pays the price.
Google uses a similar scheme of compression in their Safe Browsing v4 protocol: https://developers.google.com/safe-browsing/v4/compression It calculates deltas (adjacent difference) between a sorted set of numbers and then use Golomb encoding on that.
From a UX point of view, what would be the downsides of just rendering a plain text file of 1 number per line in alphanumeric order?
Does this really need any more UI than the browser’s chrome?
Not every web user knows “ctrl+f” or “find in page” on their phone, but those that don’t, do still know how to scroll 74,000 lines of ordered numbers (which is surprisingly easy and fast - i just tried scrolling 74k lines in excel on mobile).
The one use case i’m thinking might be confounded is a screen reader for the visually impaired or blind but even then, i’m thinking this use case is probably covered?
The typical user, faced with a huge sorted list of phone numbers and asked to just look through it, will not know how to find a number of interest and will quite possibly be quite flustered or irritated.
It's not technologically hard to implement a search, so I don't see a reason to only show a big list.
Having all the numbers visible in the page would certainly help it show up in web-search results for those phone numbers, though.
Overassessment of general literacy (the reading kind, not computer or maths literacy) is exceedingly common in technical fields. Averag computer skills are abysmal.
About 5% of computer users have "advanced" literacy, defined as "Some navigation across pages and applications is required to solve the problem. The use of tools (e.g. a sort function) is required to make progress towards the solution. The task may involve multiple steps and operators. The goal of the problem may have to be defined by the respondent, and the criteria to be met may or may not be explicit"
Scheduling a meeting room, or determining "what percentage of the emails sent by John Smith last month were about sustainability" are examples of level-3 tasks.
A quarter of the adult population cannot use computers at all, 14% are at "below level-1" skills, and 30% can only perform very basic level-1 tasks, for a total of 70% of the population which has only very basic skills ... or less.
It's easy to over-estimate the general literacy and numeracy of the population, especially if you yourself are college-educated and work in and/or with information technology.
The United States performs one of the most comprehensive assessments of adult literacy. *The key lesson for me is just how limited it is.*
I've discussed this as "The Tyranny of the Minimum Viable User", which both notes that much of the population has very basic skills, and that this also hampers the very small minority who do.
I made an unrequested but useful internal web app recently and added a 'Search' link. Clicking the link popped dialog that said 'Press Control-F to search'
Rendering files megabytes in size is quite bad in a browser. I'm not sure of the reason, but it fires up slowly, and I suspect memory usage is also quite bad.
The browser isn't optimized for text files though, but for documents and applications. I've noticed some flickering while scrolling longer pages before, and from that I expect that there's some caching of the rendered content of the page.
The text file also gets transformed into a html document with DOM (inspect one in Firefox to see), which likely has its own caching tricks for performance.
> Because no one knows how to build a sensible web service any more, the page loads a 1.3MB JSON file containing every number that Barclays has.
While other comments have already mentioned that it's actually "only" ~190kB which get transferred thanks to gzip compression, there is still a lot of potential for a smaller transfer size by simply using brotli instead of gzip for compression. With brotli the size of this file is <60KB, so only less than a third of the file size of the gzipped data.
Using brotli would be an easy win and improve performance of loading all other assets as well, without the need to change anything in the actual implementation of the website, however Barclays Bank is probably still running a webserver stack which doesn't support brotli yet.
Bank IT departments are filled with stats, Econ and financial math majors who couldn’t get front office jobs after school. They’re great for creating software that prices derivatives, but awful for reliability, security and all things engineering. Meanwhile, the decision makers just use Excel for all the important things. Things blow up all the time, it’s just that it rarely gets public. For whatever reason $200mm in wasted IT doesn’t generate the headlines like a $200mm trading mistake.
Elitist prejudice without any empirical basis. On the contrary, at least on my corner of the world, banks are the ones who completely abhor hiring someone who doesn’t have a 5 year degree in computer science. You could have a thousand stars project in GitHub, come from FAANG and most bank HRs would trow your resume at the trash bin if they don’t see CS.
This website and the one they're referencing shouldn't even exist. Barklays' website should just tell the customer to always call them back. Spoofing the phone number isn't just a possibility, it's normalcy.
I usually find the phone number to call either on the debit/credit card itself, the latest invoice, the contract/agreement I have with them, or their website. There is usually one customer support hotline that can forward you to where you need to go.
It gives a false sense of security. The vast majority of users do not understand phone numbers can be spoofed, and will use this website to check if a call they received is from a "legitimate" number, instructions be damned. It wouldn't surprise me if scammers even incorporate it into their routines.
The bank could easily record whether or not they are in contact with the customer over the phone when they initiate outreach. Then standard practice would be “visit Barclays and enter your phone number to check the call is legitimate and get the name of the agent “ or more realistically even a push notification in app. Asking people to call back is a canonical example of bad UX in the name of security that weakens overall security (like stupid password requirements, rotation requirements, etc).
It can take ages to get through back to an agent and it may not even be the same one or even in the same department.
187k of very compressible strings, producing a small array in memory, in exchange for avoiding all the costs of a web service seems like an excellent trade off.
187k seemed quite expensive, the file actually drops to 49k with Brotli and removing whitespace.
Sorry if this put a backend team out of a job, but I definitely approve of this design. The other number normalization defects could just as easily be fixed client-side
A few years back, I wanted to refactor something in the quite large codebase I was working on at the time.
I stumbled on a class that was very short and simple. If you know enterprise code, that should immediately make alarm bells ring.
And yes, indeed, all this code did was provide a (world-visible) API endpoint which returned the result of "SELECT id FROM <table with millions of rows>" as a JSON array of numbers.
IIRC it was used for something theoretically reasonable like "figure out if this search index contains all rows from that table".
But I'm sure there were less hacky ways to achieve the same result. Maybe even put permissions or firewalls on it.
Along similar lines, I once found a very simple class. It was an endpoint which executed a stored procedure. The name of the stored procedure was passed in as a query string parameter. There was no validation being done. It also ran system stored procedures; if you're not familiar with that, think admin and housekeeping type functions on the database.
We couldn't immediately disable or firewall it as it had embedded its roots deep, and a lot of things calling it didn't know they were calling it.
I tested the 44k optimised Perl regexp but it seemed to let through some other stuff - i.e. replacing the leading 0 with 8 allowed through a bunch of numbers that aren't on the original list.
e.g. changing 02071160299 to 82071160299 still appears to pass the regexp.
Probably the cardinal sin of regex: forgetting the anchors.
Also something really weird: when I view zimpenfish's comment in the post it shows as 54 minutes old which makes sense, when I click it to view direct or reply it shows as 2 days old which is really wonky. Any idea what is going on there? https://i.imgur.com/DvBDcIL.png
Might be something that got resubmitted from the second chance pool and then automatical rewrite only did half the work it should or something is cached or something similar?
AFAIK dang said that's intentional. The view on the thread gets faked to look consistent, but they don't want to modify the actual metadata of the comment you see in the direct view, and something is always going to be "wrong" however you do it.
> But how can you be sure the number displayed on your screen in genuine? You can't. The telecom system is hopelessly insecure and shouldn't be trusted for anything more complicated than dialling the speaking clock.
There is a (set of) protocol for signing telco connections that was developed:
Barclays have a decent mobile app, and you can call them directly there (even a video call).
As far as UK banks go, they're one of the more secure ones - giving you out a pinsentry 2FA device for accessing your account, setting up new payee's etc.
So far, everyone is focusing on the JSON vs regex thing, but that's not the worst of the criticisms leveled here. Not being able to use the +44 prefix, or format the number with dashes are stunningly obvious usability problems that honestly should not have made it out of QA.
You'll see the same thing on different sites with credit card numbers. See, every credit card issuer uses a scheme called an "issuer identification number" to tell you what type of card it is. And, there's a list of them [0]. So, for instance, I can tell that any valid credit card number that starts with 4 must be a Visa. So, why ask me what type of card it is, when you can figure it out yourself?
Those rare sites where you actually want to put your Social Security number in have these sorts of issues, too. Some of them require dashes, some of them don't. Some of them will only allow you to enter 9 characters, but won't tell you not to use dashes. It's infuriating, really. And, any site where you enter an account number can potentially have these types of issues.
It's not like being just a little more flexible with input formatting is a major feature with huge implementation costs and security risks, either. We're talking about failing to take a few minutes' thought in order to be a little more kind to the user, and that's just inexcusable, IMO.
> It's not like being just a little more flexible with input formatting is a major feature with huge implementation costs and security risks, either.
Just in case somebody considers implementing phone number parsing and normalizing: Just don't and use Googles libphonenumber, which offers solid, ready to use implementations in various languages (there also exist ports for other languages too): https://github.com/google/libphonenumber/
In case anyone asks you to implement any input formatting, see if you can get them to change their minds. IMO the most bug-prone and useless feature of websites. Let the user put in whatever they want and just input.replace(/[^0-9]+/, ''). You, your users and your product owners will be happier.
This doesn't work for phone numbers if you want to do something programmatically with them. In Germany it's for example common to include the international prefix _and_ a trunk prefix like this in written phone numbers:
+49 (0)30 23125 123
Your naive approach to just replace non-number characters would lead to an invalid number in such a case. Using libphonenumber you'd get a correctly normalized phone number while the user doesn't get bothered for inputting a technically invalid number: https://libphonenumber.appspot.com/phonenumberparser?number=...
Use of this data, or perhaps to be fairer on the author of the blog, the use of the data in the manner implied by Barclays is not to be encouraged.
People should NOT be trusting calls simply on the basis of the number presented and should ALWAYS validate the caller by another means.
Spoofing of numbers is a reality on communications networks worldwide.
This applies even in places in the US where they have STIR/SHAKEN tokens. STIR/SHAKEN tokens are mostly about the reactive tracing the origin of the call, not necessarily about proactive blocking spoofing.
The UK, like 99.999% of other telecoms networks in the world doesn't even have the equivalent to STIR/SHAKEN and is reliant on carriers implementing (or, more often not) various limited and mostly informal possibilities.
You would have thought a company such as Barclays would be more knowledgeable than to go around saying that you can trust a CLI if it matches one on a list we publish.
This is the relevant complaint. Not that scammers need much help collecting a list of legit numbers to spoof, but now they can tell their victim to go and check the origin number on this service and falsely reassuring them.
I worked in a place that did this on an internal page, but with all the budget codes, financial limits and which manager they belonged to. I don't know if it posed a fraud risk but if there was none then they could have just stuck an Excel file on SharePoint.
At one point I was exposed to an identity management web UI originally developed at Barclays (and still littered with references to their internal systems).
It was based on some completely Byzantine web framework and was incredibly slow. I checked the web requests it was making once while trying to add a user to a group and found that the UI was loading every group from our directory server (~22000 groups) into some JS object that then loaded them all into an html list element. Then it had a separate request to the web server for field validation for some reason.
It also didn’t have an API, much to the chagrin of every attempt to automate any infrastructure system.
> Because no one knows how to build a sensible web service any more, the page loads a 1.3MB JSON file containing every number that Barclays has.
Banks never did. This is par for course. Just be lucky they didn’t send it in FIX, because you know a bald guy clinging to their 2010 Blackberry “because of the keyboard and emails” really wanted to send they payload over the FIX protocol
Reading the article, and given the fact that the list isn't even sorted, I wouldn't be too surprised if it turns out that the list itself was hand curated. More precisely, the hands of maybe 5 or so interns. That sort of thing would even be surprisingly close to a summer job I once had in an accounting department.
The file is transferred compressed with http gzip so only 190kb is transferred.
I can see the point of being efficient but this doesn't seem that bad all things considered. At least the list is also human readable unlike a regex or custom compression.
While the 74k number looks insane, what I think they have is some centrally allocated PBX block or two, and then a handful of numbers for each branch, of which they have a few thousand; so it looks a little less crazy.
Reminds me of that time I got suspended from Twitter because I insinuated that shorting a certain exchange traded note from Barclays had numerous ways to pay out, including Barclays going bust.
You misunderstand the role of modern banks: they're not there to protect your money, they exist to create money through credit. Everything else (payment routing and account management) is just an accessory, a necessary evil. They want your money in "safe" accounts only because the law forces them to keep around a certain amount of cash in case all that credit turns bad.
It seems that you have misunderstood how banks create money through credit. Even if there was absolutely no requirement to keep a certain amount of cash, banks still couldn't create infinite money out of thin air, because that's not how it works. I wrote more about this here: https://www.attejuvonen.fi/money-out-of-thin-air/
It seems that you have misunderestood what my comment says, possibly in order to shoehorn a link to your blog. Please point out where I say that banks can "create infinite money out of thin air". I said they do the minimum that is necessary for them to be allowed to create money, which is what they exist for in the fractional-reserve system. Nobody else can emit the level of credit that banks can without getting shut down by the government.
