From the report at https://abertschi.ch/default_public/ethz/graalphp/download.p...: "Experimental results indicate that our runtime reaches competitive results with performance gains of up to 859% compared to PHP 7. These preliminary results suggest that a Truffle-hosted PHP implementation might be significantly faster than existing language implementations."
So it will work for really old(-style) php then. I still see new software coming out (and being sold) written in that style: no frameworks, classes or exceptions. Not relevant for this implementation, but many (popular) commercial wp plugins are still written like that; installed one for a client a few days ago and it was a dense mix of code and html without classes, exceptions or namespaces. And that is a popular, new, commercial one.
That's pretty common within the WordPress community, yeah, and it doesn't have to be ugly if the coders knew what they were doing. A lot of it is a mess, though.
Also commercial standalone scripts as I see; I think many are 15+ years old originally and are just updated with better fronts and some new features but all the logic is just the same hacking.
The JVM version implements a JIT compiler; the C version implements an interpreter. So it's the same thing that makes PyPy faster than CPython, Java 1.3 faster than 1.2 and the first release of Chrome's V8 faster than earlier JavaScript implementations.
PHP 8 also introduces a JIT compiler, which is presumably less mature but more complete: https://wiki.php.net/rfc/jit
A majority of the dynamic compilation research goes into JVMs. (The remainder being apportioned mostly to the javascript vms and the clr, with mike pall waving his luajit hat in the distance.)
An overwhelming majority of GC research also goes into JVMs, thanks to Jikes RVM.
Apart from the interpreter part, the GraalVM project also has a focus on creating native images, basically compiling directly to the assembly of the architecture. That's what the native results are from.
Ever seen a java Spring project start in 0.015s, or a java rest service living in 16-32mb of ram? With graal these things are now possible.
They are using CPU intensive synthetic benchmarks to compare tight loops, fannkuch-redux, spectral-norm, and similar done in PHP code and aren't directly implemented in C.
The kind of things they are benchmarking aren't what people typically use PHP for.
The JVM compiles the PHP code to highly optimised machine code at runtime. The C implementation doesn't compile to machine code at all (or I think they only just started doing it and it's very limited.)
Mostly because PHP is implemented very poorly. Its used to be really slow (its a little bit better now with newer versions) and it always amazed me, because of its basically a thin wrapper on C.
I guess PHPs core types are very inefficient and the team behind PHP lacks time to really do a rewrite. A prime example is the PHP "array". Its not a list, its not an array, but more of a weird object thing. PHP is full of these weird things that are probably hard to optimize (in the PHP runtime)
I'll point out that in the latest Techempower Benchmarks[1] PHP makes up 20% of the fastest 25 frameworks. It also makes two appearances before the first Java framework.
PHP might not be suited to every application, but it's certainly not slow.
Thats a total BS claim. I took a quick look at the top 25, and the ones that made the list (top 25) that are PHP based are:
- 11: php-ngx-pgsql
- 12: workerman-pgsql
- 22: workerman
- 23: php-ngx-mysql
- 25: Swoole
Here you see a trend, workerman and swoole are BOTH nodejs clones, they have a event loop and are non-blocking. This means you CANNOT use 95% of core PHP because its blocking by nature. These are all a non-starter for 99.9% of PHP based apps. Also swoole is a PHP C-extension, not a "installable" framework like say Symfony is, taht said these are NOT frameworks at all, not sure why they are on the list?
Heres the REAL rankings of the PHP frameworks that are used in the wild:
- 299: codeigniter
- 323: fatfree
- 369: cakephp
- 370: symfony
- 377: laravel
PHP occupies the lower bottom of the ranking, and will probably always be there becuase og how PHP is built. The core model of execution is always going to be slower than a "running" program. This is evident with the nodejs clones.
I have to point out that what you say applies only on the "Fortunes" benchmark. On the others benchmarks, Java always comes before PHP. I'll also add that most people don't use the top performing frameworks (most people use Laravel and Symfony for PHP, which are near the bottom), but that applies to everything here.
I wasn't clear, sorry. What I meant to say is that most people use Laravel and Symfony, which are nowhere near the top performing PHP frameworks on the benchmark.
For DBMS, the authors of the APOLLO system found the following [1]:
"Code coverage is a frequently used feedback mechanism in fuzzing engines [22, 27]. We found that this metric is not particularly useful for fuzzing DBMSs, since the core components of DBMS (e.g., query optimizer) already have high coverage (e.g., >95%) after running tens of queries."
Yes, you are right, the results are not comparable between DBMS that we tested. For MySQL, the large number of existing bugs prevented us from going further, and also testing more advanced functionality (like FTS). For SQLite (and DuckDB), bugs were typically fixed within hours, which is why we could comprehensively test them. For SQLite, we also tested three extensions (besides FTS, also R*Tree and DBSTAT).
Thanks for your interest! I'd love to test H2, but we are currently lacking the resources to do so. We've received a number of similar requests in the last couple of days and weeks. For each DBMS that we test, we need to add some logic to randomly generate databases and queries, since the SQL dialects typically differ. I've opened an issue in SQLancer's issue tracker for now: https://github.com/sqlancer/sqlancer/issues/23
The name SQLancer suggests that it is made for SQL databases. Do you think it would be easy to adapt it for documents databases like CouchDB or MongoDB?
I believe that the approaches that we designed could be implemented/extended to document-oriented DBMS. Since I don't have any experience with those DBMS, I have no intuition on whether the approaches would uncover any bugs.
Implementation-wise, supporting CouchDB and MongoDB in SQLancer would be feasible. SQLancer shares little infrastructure between the testing implementations for the individual DBMS, and thus imposes only few restrictions on them. We would definitely welcome implementations for any NoSQL DBMS in SQLancer!
Thanks! Here is one interesting example of a bug that was reported on the SQLite mailing list: https://www.mail-archive.com/sqlite-users@mailinglists.sqlit.... The SQL query that triggers the bug is rather complex. Interestingly, it was generated automatically: "I might not spell it like that myself, but a code generator would do it (and much worse!). This example was simplified from a query generated by a Django ORM queryset using `.exclude(nullable_joined_table__column=1)`, for instance."
When the user reported the bug, it had actually already been fixed on the latest SQLite3 trunk based on one of our bug reports. The query in our test case used a “NOT NULL NOT NULL” predicate, which is unnecessary complex as well: https://www.sqlite.org/src/tktview?name=c31034044b. We believe that this is interesting evidence that even “obscure” bugs matter in practice, for example, when the queries are automatically generated, as demonstrated above.
We will also consider testing DBMS with a DeWitt Clause in the future. For now, we have indirect evidence that our approaches could find bugs in Oracle. Specifically, we found a case in MySQL, which we consider a bug, but which was closed by the bug verifiers, partly because Oracle computes the same - we believe incorrect - result. Our bug report can be found here: https://bugs.mysql.com/bug.php?id=99182.
We believe that it is a bug, because:
* the invariants that we are testing using TLP, which is the approach that we used to find the bug, should hold for all DBMS;
* dropping the UNIQUE constraint would yield the result we would expect, and we believe that an index should never affect a query's result;
* an earlier version of MySQL computed the result we would expect;
* and DBMS that strive to support the MySQL SQL dialect, computed the result we would expect (e.g., TiDB).
Thank you! SQLite was indeed a challenging target. We could find and report many bugs in it only because all of our bugs were addressed very quickly (typically within a day). For some of the other DBMS that we tested, bugs were fixed slowly, and for some we also saw many unresolved bugs in their bug tracker, which is why we did/could not comprehensively test these DBMS.
Before we proposed our testing approaches, the state of the art of finding logic bugs was writing unit tests, which does not scale well, since it requires much manual effort. We believe that we provide the first practical approaches to automatically finding logic bugs. Companies have already started adopting them. For example, PingCAP implemented all three approaches that we proposed in https://github.com/chaos-mesh/go-sqlancer to test their DBMS TiDB.
A number of tools are available to test other aspects, like performance or for crash bugs. SQLsmith, for example, is one effective tool that allows finding crash bugs (see https://github.com/anse1/sqlsmith).
Many DBMS development teams use SQLsmith and other random query generators, which can effectively find crash bugs. In contrast, we designed and developed SQLancer to automatically find logic bugs, which are bugs that cause the DBMS to compute an incorrect result set (e.g., by omitting a record from the result set).
How do you design the tests? Do you think of complex cases and hope any dbms may fail the test? Because this is would result in many tests having no problem and sometimes finding a bug, which cant really be an effective way of doing it.
We automatically generate databases and queries. The main challenge is to verify that the query yields the expected result. To this end, we have designed and implemented three different approaches:
* Pivoted Query Synthesis (PQS): The main idea is to generate a query that is guaranteed to fetch a randomly-selected row, to which we refer as the pivot row. If the pivot row is not contained in the result set, we have detected a bug.
* Non-Optimizing Reference Engine Construction (NoREC): The idea is to transform a query so that it cannot be optimized effectively by the DBMS. By comparing the result set of the "optimized" query with the result set of the "unoptimized" query, we can detect bugs that affect the DBMS' query optimizer.
* Ternary Logic Partitioning (TLP): The main idea of this approach is that we can partition a given query into multiple "partitioning" queries, each of which computes a partition of the result, and to then combine these partitions to derive the same result as the original query. If the two results differ, we have detected a bug in the DBMS.
According to the PQS paper, they pick an existing row in the database and generate random predicates until they find one that’s true for that row. They then use that as a WHERE clause and verify the original row is contained in the result set. A similar procedure is used to find relevant JOIN clauses.
You could use a fuzzer and fuzz several DBMS products at the same time. If their db's states or query results start to diverge, you probably found a bug in one of the product.
