Magento Varnish caching problem with `esi:include` tags sometimes being outputted in html on url's that contain `/customer`

magento-varnish-esi-include-problem.md

This discusses an issue in Magento when Varnish is active, that on certain pages (mostly containing /customer in the url) you'll see the top navigation missing and an esi:include tag being included in the html. This is related to Varnish not understanding certain compression algorithms, like Brotli.

Note, we're only talking about GET requests here, not about POST requests.

1. Magento has a way to say if a page is cacheable or not. A page is cacheable by default, unless it mentions cacheable="false" in one of its blocks, in its layout xml file.

2. Based on if a page is cacheable or not, Magento will send different caching HTTP headers back

3. Just for information, following pages are considered cacheable (unless the old skool captcha module is active, which makes most of these uncacheable):

   • customer/account/confirmation
   • customer/account/create
   • customer/account/forgotpassword
   • customer/account/login
   • customer/account/logoutSuccess

4. And following pages are considered uncacheable:

   • customer/account/createpassword
   • customer/account/edit
   • customer/account
   • customer/account/index
   • customer/address/form
   • customer/address/index

5. Magento has a feature that if Varnish is enabled and a page is considered cacheable, it will output an esi:include tag for the top navigation block, as it's more efficient to re-use that block instead of the backend having to build and return it for each page.

6. Magento's default VCL file for Varnish contains this section in the vcl_recv block:

       # Bypass customer, shopping cart, checkout
       if (req.url ~ "/customer" || req.url ~ "/checkout") {
           return (pass);
       }

   This basically says, if /customer or /checkout is part of the url, don't fetch the content from the cache, just send the request to the backend each time, ignore what Magento's cache headers say and don't store its results in cache.

7. In Varnish, when executing return (pass) in the vcl_recv section, it will skip changing encoding headers (like brotli) when passing the request to the backend:

   Unless returning from vcl_recv with pipe or pass, Varnish modifies req.http.Accept-Encoding: if the client supports gzip req.http.Accept-Encoding is set to "gzip", otherwise the header is removed.

   And it will thus potentially receive compressed data back it doesn't understand. That's fine as long as this data doesn't contain an esi:include tag.
   So in specific cases, if for example the browser supports brotli compression and Varnish does not, when executing a return (pass), Varnish will pass the request to the backend and the backend can then return a brotli compressed result. Varnish can't understand that response, and it will return the response to the client as-is. So, if such a blob of compressed data contains an esi:include tag, Varnish won't be able to see that and will thus return it as-is to the client without resolving the ESI tag first.

8. Note that Varnish has support for brotli compression, but only in its Enterprise solution, not in the free Community one.

9. In following cases a return (pass) is triggered from the VCL in the vcl_recv section and thus in these potential cases Varnish can receive potentially compressed data it can't read:

   • When a request is not a GET or HEAD request => this is fine, such requests are not cacheable and should never contain an esi:include tag from Magento
   • When calling the pub/health_check.php file, this is fine, it's not going to contain an esi:include tag
   • When the url starts with /media or /static, this is fine, those are static assets (js, css, images, ...)
   • When the url contains /graphql and is authenticated, this is fine, those won't contain an esi:include tag
   • When the url contains /customer or /checkout. This is the problematic section, as some of the urls containing /customer will be cacheable and will be able to contain an esi:include tag

10. I believe that forcing from the VCL file that all pages containing /customer in the url to be uncacheable from Varnish is wrong, this should not be hardcoded and Varnish should follow the instructions the HTTP cache headers tell it, if a page is cacheable or not. Removing this will avoid the return (pass) and will not run into problems with brotli compression.
    So I would suggest we change:

        # Bypass customer, shopping cart, checkout
        if (req.url ~ "/customer" || req.url ~ "/checkout") {
            return (pass);
        }

    into:

        # Bypass shopping cart, checkout
        if (req.url ~ "/checkout") {
            return (pass);
        }

    This /customer exception was added here, but it doesn't contain much explanation why it was added, so I believe this shouldn't have been accepted.

11. An alternative solution is to disable brotli compression in the backend webserver, but I think the prefered solution is to do it through the VCL file itself.