I should mention that 99.9% of domains will fall into standard form ( handle.domain or ip.ip.ip.ip )
As such, You are definitely more likely to let a user enter a bad URL they did not intend because it validates then to let a uncommon domain actually be used.
As such- a much simpler regex would likely 'make more people happy' than being 100% correct to tech spec.
None of those is a URI, so a URI validator most certainly should not accept them. Just because browsers tend to understand them as a matter of a historical accident does not mean those are valid URIs, just as tag soup that browsers also tend to understand isn't valid HTML either.
Exactly. The goal was to come up with a good regular expression to validate URLs as user input. There’s no way I’d want to allow alternate IP address notations.
> Why use regex? It's much simpler to write a URL validator by hand...
I actually have a use-case. I am firming up a feature right now that detects when a user types a url into a text field and replaces it on the fly with a footnote-style reference number (much like your comment above). This is done to (1) minimize input string length, (2) draw the benefits of a consistent interface, and (3) avoid screwing around with the fragility of url shortening nonsense.
I may regret this, but here's a link to my dev environment for this feature (please be gentle), to see it in action:
...just start typing in the big text box and add in some urls.
It uses a fairly ugly-looking regular expression[0].
If you take out the unicode mumbojumbo, it's not really THAT tricky of a pattern. It does fail on IP addresses and it may be a little over-aggressive on matching, but, I wanted it to catch things like "abc.com" and "//xyz.com".
Edit: Formatting and clarity. Removed the explicit regular expression because it predictably got garbled.
[0] http://regexr.com/38vsq not exactly the same one I'm currently using, but it's pretty close. See source for most up-to-date version.
I agree, the correct one (+500 chars) looks like a maintenance nightmare to me. I tried to build a real e-mail address validator that would accept also the more exotic forms and there was no way in hell I would have done that in regex.
I also met very few people that actually understood regex. It's a whole new skill and if I use it in my application I don't know if the next guy can pick it up.
It's why you should always write regex (at least, anything beyond a couple of characters) using the `x / PCRE_EXTENDED` flag. It makes them ignore whitespace and anything following a #, so you can split your regex up onto multiple lines and comment it to explain what it's doing.
If you looked at the page before commenting you’d know that PHP’s built-in URL parser is one of the implementations that is being tested. You’d also see that one of the requirements was to not match scheme-relative URLs (e.g. `//foo.bar`) like the ones your commit fixes.
The regular expressions presented do not conform to the URL Standard or to any RFC, but rather to the list of requirements on that page.
I applaud your work in improving PHP’s built-in URL parser.
The @stephenhay is just about perfect despite being the shortest. The subtleties of hyphen placement aren't very important, and this is a dumb place to filter out private IP addresses when a domain could always resolve to one. Checking if an IP is valid should be a later step.
The goal was to come up with a good regular expression to validate URLs as user input, and not to match any URL that browsers can handle (as per the URL Standard).
a.b--c.de is supposed to fail because `--` can only occur in Punycoded domain name labels, and those can only start with `xn--` (not `b--`).
Many of these regexes should not be used on user input, at least not if your regex library backtracks (uses NFAs), because of the risk of ReDoS: http://en.wikipedia.org/wiki/ReDoS
Trying to shoehorn NFAs into parsing stuff that isn't a regular expression is generally a bad idea. (See: Langsec.)
The first one I'm not sure, it looks like a valid FQDN (equivalent to www.foo.bar, where the trailing dot is implicit). The second one I guess it has to do with punycode and internaltional URI encoding?
> The trailing dot makes it an invalid URL, which defines hostname as * [ domainlabel "." ] toplabel.
I disagree. But, this is a tricky one. The relevant specs are:
Spec | | Validity | Definition
URL (RFC1738) | obsolete | invalid | hostname = *[ domainlabel "." ] toplabel
HTTP/1.0 (RFC1945) | current | invalid | host = <A legal Internet host domain name
| | | or IP address (in dotted-decimal form),
| | | as defined by Section 2.1 of RFC 1123>
HTTP/1.1 (RFC2068) | obsolete | invalid | ; same as RFC1945
HTTP/1.1 (RFC2616) | obsolete | valid | hostname = *( domainlabel "." ) toplabel [ "." ]
URI (RFC3986) | current | valid | host = IP-literal / IPv4address / reg-name
| | | reg-name = *( unreserved / pct-encoded / sub-delims )
HTTP/1.1 (RFC7230) | current | valid | uri-host = <host, see [RFC3986], Section 3.2.2>
The only way that URL is invalid is if we are in a strict HTTP/1.0 context.
As a note about RFC1738 being obsolete: these days a URL is just a URI (1) whose scheme specifies it as a URL scheme, and (2) is valid according to the scheme specification.
As the given URL is a valid URI, and is valid according to the current http URL scheme specification (RFC7230), that URL is valid.
The trailing dot makes it an invalid URL, which defines hostname as ∗[ domainlabel "." ] toplabel.
RFC2396§3.2.2 defines hostname as:
hostname = *( domainlabel "." ) toplabel [ "." ]
RFC3986 which obsoletes RFC2396, seems to make fewer claims about the authority section, and specifically says in Appendix D.2 that the toplabel rule has been removed.
At best this lets you conclude that a URL could be valid. Is that really useful? Is the goal here to catch typos? Because you'd still miss an awful lot of typos.
If you really want your URL shortener to reject bad URLs, then you need to actually test fetching each URL (and even then...)
As an aside, I'd instantly fail any library that validates against a list of known TLDs. That was a bad idea when people were doing it a decade ago. It's completely impractical now.
My exact use case was the following: the user clicks a bookmarklet that passes the current URL in the browser as a query string parameter to a URL shortener script. The validation is then performed before the URL is shortened.
In that scenario, and with the given requirements, I can’t think of a case where the validation fails. There’s no need to worry about protocol-relative URLs, etc.
(Keep in mind that this page is 4 years old — I very well may have missed something.)
> If you really want your URL shortener to reject bad URLs, then you need to actually test fetching each URL (and even then...)
I disagree. http://example.com/ might experience downtime at some point in time, but that doesn’t mean it’s suddenly an invalid URL.
> As an aside, I'd instantly fail any library that validates against a list of known TLDs. That was a bad idea when people were doing it a decade ago. It's completely impractical now.
I still don't quite follow the purpose of the validation. Is it against malicious use? In normal use, I would think that pretty much any URL that's good enough for the browser sending it would be good enough for the link shortener.
Another important dimension when evaluating these regexes is performance. The Gruber v2 regex has exponential (?) behavior on certain pathological inputs (at least in the python re module).
Use a standard URI parser to break this problem into smaller parts. Let a modern URI library worry about arcane details like spaces, fragments, userinfo, IPv6 hosts, etc.
uri = URI.parse(target).normalize
uri.absolute? or raise 'URI not absolute'
%w[ http https ftp ].include?(uri.scheme) or raise 'Unsupported URI scheme'
# Etc
That was not an option in this case, as the goal is to validate URLs entered as user input and blacklist certain URL constructs even though they’re technically valid.
John Gruber (of daringfireball.com) came up with a regex for extracting URLs from text (Twitter-like) years ago, and has improved it since. The current version is found at https://gist.github.com/gruber/249502.
I haven't tested it myself, but it's worth looking at.
Isn't it the @gruber v2 column from the page? It looks to have no false negative, but many false positives. The only one which does perfect on the tested set is Diego Perini's https://gist.github.com/dperini/729294
Interestingly it seems http://✪df.ws isn't actually valid, even though it exists. ✪ isn't a letter[1], so it isn't allowed in international domain names. I was looking at the latest RFC from 2010 [2] so maybe it was allowed before that. The owner talks about all the compatibility trouble he had after he registered it [3]. The registrar that he used for it, Dynadot, won't let me register any name with that character, nor will Namecheap.
There is no perfect URL validation regex, because there are so many things you can do with URLs, and so many contexts to use them with. So, it might be perfect for the OP, but completely inappropriate for you.
That said, there is a regex in RFC3986, but that's for parsing a URI, not validating it.
However, some of the test cases in the original post (the list of URLs there aren't available separately any more :( ) are IRIs, not URIs, so they fail; they need to be converted to URIs first.
In the sense of the WHATWG's specs, what he's looking for are URLs, so this could be useful:
http://url.spec.whatwg.org
However, I don't know of a regex that implements that, and there isn't any ABNF to convert from there.
This always comes up in these discussions and is a terrible counter example. RFC 822 is the format for email messages (ie headers) and not a form you'll ever find email addresses in "in the wild" (eg on web forms).
What's wrong with IP-address URLs? If they are invalid because it says so in some RFC, this is still not the ultimate regex. If you redirect a browser to http://192.168.1.1 it works perfectly fine.
And why must the root period behind the domain be omitted from URLs? Not only does it work in a browser (and people end sentences with periods), the domain should actually end in a period all the time but it's usually omitted for ease of use. Only some DNS applications still require domains to end with root dots.
This is the context of a URL shortener, and one of the goals seems to prohibit bad IPs, not all IPs. Bad seems to be defined here as ones in the private IP spaces, and those in multicast space, etc.
According to the RFC, since domains are allowed to be entirely numeric, there is overlap between valid domains and valid IP addresses. The RFC says that if something could be a valid IP address, it is to be thought of as an IP address.
WTF? When will people finally learn to read the spec and implement things based on the spec and test things based on the spec instead of just making up themselves what a URL is or what HTML is or what an email address is or what a MIME body is or ...
There are supposed URIs in that list that aren't actually URIs, there are supposed non-URIs in that list that are actually URIs, and most of the candidate regexes obviously must have come from some creative minds and not from people who should be writing software. If you just make shit up instead of referring to what the spec says, you urgently should find yourself a new profession, this kind of crap has been hurting us long enough.
(Also, I do not just mean the numeric RFC1918 IPv4 URIs, which obviously are valid URIs but have been rejected intentionally nonetheless - even though that's idiotic as well, of course, given that (a) nothing prevents anyone from putting those addresses in the DNS and (b) those are actually perfectly fine URIs that people use, and I don't see why people should not want to shorten some class of the URIs that they use.)
By the way, the grammar in the RFC is machine readable, and it's regular. So you can just write a script that transforms that grammar into a regex that is guaranteed to reflect exactly what the spec says.
This entire rant begins with the premise that the spec matches the real world implementation. Given that one of your examples is "what an email address is", I submit that expecting reality and the spec to match is a beautiful dream, from which a developer should awaken before trying to implement such a scheme.
Except there is no "the real world implementation". There only are lots of implementations that are incompatible with the spec as well as amongst each other. Inventing yet another variant of your own that also isn't going to be compatible with anything is not going to help anyone. Deviating from the formal spec because everyone practically agrees how to do things, albeit differently than in the formal spec, is something quite different from making shit up, and actually tends to be even harder than building things to spec, as there tends to be no easy reference to look things up in, but instead you might have to look into the guts of existing implementations and talk to people who have built them to figure out what to do - and you would normally start with an implementation according to spec anyhow, and only add special cases for non-normative conventions lateron.
Also, what exactly is the problem with email addresses? There is a very unambiguous grammar of those in the RFC, and there are lots of implementations of exactly what the spec specifies. Just because some web kiddies have made up some shit about email addresses and use that for validation, doesn't mean that postfix, qmail, or exim are written by morons.
> Deviating from the formal spec because everyone practically agrees how to do things, albeit differently than in the formal spec, is something quite different from making shit up, and actually tends to be even harder than building things to spec, as there tends to be no easy reference to look things up in, but instead you might have to look into the guts of existing implementations and talk to people who have built them to figure out what to do - and you would normally start with an implementation according to spec anyhow, and only add special cases for non-normative conventions lateron.
The goal was to come up with a good regular expression to validate URLs in user input, and not to match any URL that browsers can handle (as per the URL Standard). I am fully aware that this is not the same as what any spec says.
> By the way, the grammar in the RFC is machine readable, and it's regular.
The RFC does not reflect reality either (which, ironically, is what you seem to be complaining about). If you’re looking for a spec-compliant solution, the spec to follow is http://url.spec.whatwg.org/.
> If you just make shit up instead of referring to what the spec says, you urgently should find yourself a new profession, this kind of crap has been hurting us long enough.
I am aware of, and am a contributor to, the URL Standard: http://url.spec.whatwg.org/ That doesn’t mean there aren’t any situations in which I need/want to blacklist some technically valid URL constructs.
> The goal was to come up with a good regular expression to validate URLs in user input, and not to match any URL that browsers can handle (as per the URL Standard).
WTF? What is "validation" supposed to be good for if it doesn't actually validate what it claims to? Exactly this mentality of making up your own stuff instead of implementing standards is what causes all these interoperability nightmares! If you claim to accept URLs, then accept URLs, all URLs, and reject non-URLs, all non-URLs. There is no reason to do anything else, other than lazyness maybe, and even then you are lying if you claim that you are validating URLs - you are not. If you say you accept a URL, and I paste a URL, your software is broken if it then rejects that URL as invalid.
This does not apply to intentionally selecting only a subset of URLs that are applicable in a given context, of course - if the URL is to be retrieved by an HTTP client, it's perfectly fine to reject non-HTTP URLs, of course, but any kind of "nobody is going to use that anyhow" is not a good reason. In particular, that kind of rejection most certainly is something that should not happen in the parser as that is likely to give inconsistent results as the parser usually works at the wrong level of abstraction.
> The RFC does not reflect reality either (which, ironically, is what you seem to be complaining about).
A spec for a formal language that doesn't contain a grammar? The world is getting crazier every day ...
> That doesn’t mean there aren’t any situations in which I need/want to blacklist some technically valid URL constructs.
Yeah, but blocking IPv4 literals of certain address ranges seems like a stupid idea nonetheless. Good software should accept any input that is meaningful to it and that is not a security problem. And as I said above, such rejection most certainly should not happen in the parser.
> Doesn’t matter – if there’s a discrepancy between what a document says and what implementors do, that document is but a work of fiction.
Yes and no. When there is a de-facto standard that just doesn't happen to match the published standard, yeah, sure. Otherwise, bug compatibility is a terrible idea and should be avoided as much as possible, many security problems have resulted from that.
> This is not a parser.
Well, even worse then. Manually integrating semantics from higher layers into parsing machinery (which it is, never mind the fact that you don't capture any of the syntactic elements within that parsing automaton) is both extremely error prone and gives you terrible maintainability.
edit:
For the fun of it, I just had a look at the "winning entry" (diegoperini). Unsurprisingly, it's broken. It was trivial to find cases that it will reject that you most certainly don't intend to reject. For exactly the reasons pointed out above.
No, it doesn't. This regular expression matches strings that are not URIs, and that should be quite obvious if you compare the grammar in that same RFC to the regex.
