Replacing an Asset Pipeline
At Onfido, a lot of our major applications are built with Rails. For our assets, therefore, we have been using the Rails asset pipeline (Sprockets).
Unfortunately, Sprockets also has some hidden drawbacks. As our projects grew in size, we began to really feel the following issues:speed ― Sprockets is very slow. datedness ― Sprockets has not incorporated CommonJS, leaving lots of packages in npm as out of reach. Equivalent gems (for example jQuery) tend to be very dated.
There are Rails solutions for supporting CommonJS code, but nothing that truly addresses the speed issue. We began to consider dropping Sprockets, but what else is there? Tools like Gulp are great, but very project-agnostic. While agnosticism works for the wider community, creating a gulpfile.js for our Rails-oriented applications to manually recreate Sprockets in every one of them would be duplicative and unnecessary. So what else is there?Enter Pinion
We decided to build our own tool to simulate a lot of the core functionality of Sprockets. It leverages Gulp to define all of its tasks and keeps it quick and easy to extend.
How much faster are we talking? I'll go further into this, but we're currently seeing our pipeline roughly doubling in speed.
It’s entirely open-source, and you can check it out on GitHub . Take a look through the README to get an idea of the tasks that it offers.The Tablecloth Trick
Unfortunately, swapping pipelines mid-project isn’t as simple as changing one or two configuration flags. There are several issues that you might hit when trying to swap over to a more modern pipeline
As mentioned previously, Rails does not support CommonJS. Instead to load in other files, it offers a //= require method. This method inlines the relevant file in place of the comment, which is subtly (but significantly) different to how a CommonJS require() call works.
As mentioned in the Pinion README, we included some helpers to mimic the default behaviour of asset_path etc.
But on top of this, we needed a way for developers focusing on Rails work to not even have to notice that they had swapped asset pipeline. For us, this involved adding in some dusty scripts to automatically boot up a $(npm bin)/pinion instance whenever a container wakes up.
This will obviously change depending on your workspace and requirements.Gems
Lots of gems provide assets and some extra functionality for assets. These can’t just be pulled out, and instead need to be replaced with similar code performing the same functionality. This has been the hardest step for us.Goodbye Sprockets?
Getting rid of these gems has been such a fiddly task that, in fact, we haven’t entirely finished yet. So, how are we using Pinion with these gems? Well, it’s a little messy.
First, it’s useful to look at a diagram of what an asset pipeline is from a high level.
In this case, we take our static assets and run them through the pipeline. The assets will either be raw source, or references to external sources that it wants the pipeline to bring in (through some form of package manager).
But there’s no reason to only have one pipe. At Onfido, we currently have two isolated pipes that make up our asset pipeline.
As well as Pinion, we are using Sprockets to build a subset of our assets folder. This subset contains gems that run through Sprockets, providing certain assets to the application. We strictly enforce that new source code written should be run through the Pinion pipeline, rather than Sprockets.
As we replace these gems with equivalent code, we slowly bring ourselves to where we want to be:
So how much faster actually is Pinion? It's hard for us to give an exact number, being split between Pinion and Sprockets at the moment ― but we can compare our double-pipeline speed, to our previous Sprockets speed.
The old Sprockets pipeline yielded us this speed...time RAILS_ENV=production bundle exec rake assets:precompile # ... real 2m54.564s user 2m21.140s sys 0m17.420s
The new Pinion+Sprockets pipeline currently yields us this speed...(time NODE_ENV=production $(npm bin)/pinion build rev &) && (time RAILS_ENV=production bundle exec rake assets:precompile &) # ... # Pinion finishes real 0m52.285s user 0m38.620s sys 0m5.120s # ... # Sprockets finishes real 1m30.985s user 0m43.770s sys 0m18.100s
So already we're seeing our pipeline running at 192% of its previous speed. And we hope to see that number continue to rise as we deprecate Sprockets fully!Conclusions
We’re still in a transition period at the moment, but this is certainly something worth thinking about quite closely before making the switch.
Unless you’re starting a green project, swapping over from Sprockets to Pinion can be quite a serious undertaking ― whether or not it’s worth the end result will depend on your workspace requirements.
But please check out Pinion if you’re interested, and contribute if you like it!