I am excited to reveal my latest project WPDirectory, a web tool which allows lightning fast regex searching of the entire WordPress Plugin & Theme Directories.
When I first saw the WordPress Plugin Directory Slurper tool was used to download every available Plugin in the Directory so the files could be searched locally I was really excited. It was a great tool but there was a lot of room for improvement- it could take a 24+ hours to download every plugin .zip one by one and some of the forks required complex local PHP installs.
I had been doing a lot of server work in Go and thought its concurrency and single cross-platform binary with no dependencies would make a much better Slurper, which spawned WPDS (WordPress Directory Slurper)…
I really enjoyed making WPDS and it is certainly usable but almost immediately I was left with the feeling that it was still not an ‘acceptable’ solution. The single cross-platform binaries with no dependencies were easy to use, the concurrent downloads made it very fast, yet it could still take 1-2 (or more) hours to download all the plugins and searching could still take 30-60mins (or higher) depending on PC hardware and the tool used.
It felt like the process as it stood was prohibitively difficult, so time consuming that people would avoid doing it or where they needed the results the wait was disruptive and delayed decision making.
So I put WPDS on the back burner while I researched alternative options. I had an intimate look at the ripgrep tool, which provided the fastest searching I could find. Needing to make searching faster is a common requirement and it even has a common solution- indexing.
I was introduced to trigram indexing by Etsy’s Hound project (it is also used by PostgreSQL). It provided real-time searching of code repositories using trigram indexing through Google’s codesearch library.
You can begin a more in-depth look with this Russ Cox article. The short version is that by splitting text into trigrams (batches of three characters) it is possible to work out if a file could contain a regex search match. Thereby avoiding any files which cannot contain a match and significantly speeding up searches.
The important part is that small scale tests showed that searches could possibly be measured in seconds instead of minutes, it was clear this had real potential.
This was the point at which I realised that searches could potentially be made so much faster than it would become possible to run it as a web app, along with a host of other benefits that would bring- like easy sharing and collaboration.
So began my journey with WPDirectory, with the first prototype going live on June the 27th. I had a whole host of wonderful people help me with testing through Twitter, which identified more bugs/problems that I could have imagined but ultimately validated the prototype. There was clearly a demand for the service:
Oh, nice, this is currently the best way of finding plugins that come with built-in @wpcli commands@schlessera
Very cool tool Helpful for finding plugins which use deprecated php (7.2) functions like each() or create_function()@mfgmicha
Fast forward a month and I have worked through all the major bugs and WPDirectory is nearing production status. I am focused on frontend UX now and will shortly begin advertising the service publicly.
While I am committed to further improving and maintaining the tool, the wider community will always be welcome. You can view the progress and get involved at github.com/wpdirectory/wpdir. The server is written in Go and the frontend in JS (React), but you do not need to code to get involved. Ideas, suggestions, bug reports and feedback are all valuable contributions.
I hope that WPDirectory will help speed up decision-making across core WordPress teams and encourage community use, from security research to development, that was previous prohibited by the difficulty.