Menu Search
Jump to the content X X
Smashing Conf Barcelona

You know, we use ad-blockers as well. We gotta keep those servers running though. Did you know that we publish useful books and run friendly conferences — crafted for pros like yourself? E.g. our upcoming SmashingConf Barcelona, dedicated to smart front-end techniques and design patterns.

Automating Art Direction With The Responsive Image Breakpoints Generator

Four years ago, Jason Grigsby asked a surprisingly difficult question: How do you pick responsive image breakpoints?1. A year later, he had an answer2: Ideally, we’d set responsive image performance budgets to achieve “sensible jumps in file size.” Cloudinary built a tool3 that implements this idea, and the response from the community was universal: “Great! Now, what else can it do?” Today, we have an answer: art direction!

Further Reading on SmashingMag: Link

Since its release earlier this year, the Responsive Image Breakpoints Generator has been turning high-resolution originals into responsive <img>s with sensible srcsets at the push of a button. Today, we’re launching version 2, which allows you to pair layout breakpoints with aspect ratios, and generate art-directed <picture> markup, with smart-cropped image resources to match. Check it out8, and read on.

Responsive Image Breakpoints: Asked And Answered Link

Why did we build this tool in the first place?

Responsive images send different people different resources, each tailored to their particular context; a responsive image is an image that adapts. That adaptation can happen along a number of different axes. Most of the time, most developers only need adaptive resolution — we want to send high-resolution images to large viewports and/or high-density displays, and lower-resolution images to everybody else. Jason’s question about responsive image breakpoints concerns this sort of adaptation.

When we’re crafting images that adapt to various resolutions, we need to generate a range of different-sized resources. We need to pick a maximum resolution, a minimum resolution and (here’s the tricky bit) some sizes in between. The maximum and minimum can be figured out based on the page’s layout and some reasonable assumptions about devices. But when developers began implementing responsive images, it wasn’t at all clear how to size the in-betweens. Some people picked a fixed-step size between image widths:

Rectangles showing the relative dimensions of a group of srcset resources that use a fixed-step-size strategy, with the following widths: 300, 400, 500, 600, 700, 800, 900, 1000, 1100, and 1200
Rectangles showing the relative dimensions of a group of srcset resources that use a fixed-step-size strategy.

Others picked a fixed number of steps and used it for every range:

Rectangles showing the relative dimensions of three groups of srcset resources that use a fixed-number-of-steps strategy. The first group has these widths: 300, 600, 900, 1200. The second: 300, 467, 633, 800. Third: 300, 867, 1433, 2000
Rectangles showing the relative dimensions of three groups of srcset resources that use a fixed-number-of-steps strategy.

Some people picked common display widths:

Rectangles showing the relative dimensions of a group of srcset resources scaled to common display widths: 300, 512, 600, 768, 800, 1024, 1200
Rectangles showing the relative dimensions of a group of srcset resources scaled to common display widths.

At the time, because I was lazy and didn’t like managing many resources, I favored doubling:

Rectangles showing the relative dimensions of a group of srcset resources scaled using a doubling strategy, with these widths: 300, 600, 1200
Rectangles showing the relative dimensions of a group of srcset resources scaled using a doubling strategy.

All of these strategies are essentially arbitrary. Jason thought there had to be a better way. And eventually he realized that we shouldn’t be thinking about these steps in terms of pixels at all. We should be aiming for “sensible jumps in file size”; these steps should be defined in terms of bytes9.

For example, let’s say we have the following two JPEGs:

300 pixels wide (37 KB)
300 pixels wide (37 KB)
1200 pixels wide (333 KB)
1200 pixels wide (333 KB)

The biggest reason we don’t want to send the 1200-pixel-wide resource to someone who only needs the small one isn’t the extra pixels; it’s the extra 296 KB of useless data. But different images compress differently; while a complex photograph like this might increase precipitously in byte size with every increase in pixel size, a simple logo might not add much weight at all. For instance, this 1000-pixel-wide PNG10 is only 8 KB larger than the 200-pixel-wide version11.

Sadly, there haven’t been any readily useable tools to generate images at target byte sizes. And, ideally, you’d want something that could generate whole ranges of responsive image resources for you — not just one at a time. Cloudinary has built that tool12!

A screenshot of the Responsive Image Breakpoints Generator13
A screenshot of the Responsive Image Breakpoints Generator (View large version14)

And it has released it as a free open-source4815 web app.

But the people wanted more.

The Next Frontier? Automatic Art Direction! Link

So, we had built a solution to the breakpoints problem and, in the process, built a tool that made generating resolution-adaptable images easy. Upload a high-resolution original, and get back a fully responsive <img> with sensible breakpoints and the resources to back it up.

That basic workflow — upload an image, get back a responsive image — is appealing. We’d been focusing on the breakpoints problem, but when we released our solution, people were quick to ask, “What else can it do?”

Remember when I said that resolution-based adaptation is what most developers need, most of the time? Sometimes, it’s not enough. Sometimes, we want to adapt our images along an orthogonal axis: art direction16.

Any time we alter our images visually to fit a different context, we’re “art directing.” A resolution-adaptable image will look identical everywhere — it only resizes. An art-directed image changes in visually noticeable ways. Most of the time, that means cropping, either to fit a new layout or to keep the most important bits of the image visible when it’s viewed at small physical sizes.

On small screens, we want to zoom in on the image's subject17
On small screens, we want to zoom in on the image’s subject.

People asked us for automatic art direction.

… Which is a hard problem! It requires knowing what the “most important” parts of an image are. Bits and bytes are easy enough to program around; computer vision and fuzzy notions of “importance” are something else entirely.

For instance, given this image…

A white cat, off-center18
(Image source: Cloudinary19) (View large version20)

… a dumb algorithm might simply crop in on the center:

The subject of the image has been cropped out of the frame21
(Image source: Cloudinary22)

What you need is an algorithm that can somehow “see” the cat and intelligently crop in on it.

It took us a few months but we built this, too, and packaged it as a feature available to all Cloudinary users23.

Here’s how it works: When you specify that you want to crop your image with “automatic gravity” (g_auto), the image is run through a series of tests, including edge-detection, face-detection and visual uniqueness. These different criteria are then all used to generate a heat map of the “most important” parts of the image.

The master rolled-up heat map24
The master rolled-up heat map (View large version25)

A frame with the new proportions is then rolled over the image, possible crops are scored, and a winner is chosen. Here’s a visualization of the rolling frame algorithm (using a different source image26):

The rolling frame, visualized. Bluer squares mean higher scores; the green square is the current pick.

The result? Our cat, front and center:

27
(Image source: Cloudinary28) (View large version29)

Neat!

It was immediately obvious that we could and should use g_auto‘s smarts to add automatic art direction to the Generator. After a few upgrades to the markup logic and some (surprisingly tricky) UX decisions, we did it: Version 2 of the tool30 — now with art direction — is live.

Let’s Take A Tour Link

How do you use the Responsive Image Breakpoints Generator?

The workflow has been largely carried over from the first version: Upload an image (or pick one of the presets), and set your maximum and minimum resolutions, a step size (in bytes!), and a maximum number of resources (alternatively, you can simply use our pretty-good-most-of-the-time defaults). Click “Generate,” et voila! You’ll get a visual representation of the resulting image’s responsive breakpoints, some sample markup, and a big honkin’ “download images” button.

Screenshot of the tool's inputs31
Screenshot of the tool’s inputs (View large version32)
Screenshot of the tool's outputs33
Screenshot of the tool’s outputs (View large version34)

The new version has a new set of inputs, though, which enable art direction. They’re turned off by default. Let’s turn a couple of them on and regenerate, shall we?

Screenshot of the art-direction inputs, with 'Desktop' and 'Smartphone' selected35
Screenshot of the art-direction inputs, with “Desktop” and “Smartphone” selected (View large version36)

The first output section is unchanged: It contains our “desktop” (i.e. full) image, responsively breakpointed to perfection. But below it is a new section, which shows off our new, smartly cropped image:

Screenshot of the '1:1 Aspect Ratio' section37
Screenshot of the “1:1 Aspect Ratio” section (View large version38)

And below that, we now have all of the markup we need for an art-directed <picture> element that switches between the two crops at a layout breakpoint.

Screenshot of the picture markup section39
Screenshot of the <picture> markup section (View large version40)

Finally, there’s a live <picture> example that shows you what all of that markup actually does.

41
Screenshot of the “Live Picture Element in Action” section (View large version42)

Let’s circle back and look at the art direction inputs in a little more detail.

Screenshot of the art direction inputs, annotated to point out what each thing does43
Screenshot of the art direction inputs, annotated to point out what each thing does (View large version44)

Each big box maps to a device type, and each device type has been assigned a layout breakpoint. The text under the device type’s name shows the specific media query that, when true, will trigger this crop.

Below that, we can specify the aspect ratio that we want to crop to on this device type.

Below that, we specify how wide the image will appear relative to the width of the viewport on this type of device. Will it take up the whole viewport (100%) or less than that? The tool uses this percentage to generate simple sizes markup45 — which specifies how large the image is in the layout. If you’re using this code in production, you’ll probably want to go back into the example markup and tailor these sizes values to match your particular layout more precisely. But depending on your layout, inputting rough estimates here might be good enough.

And there you have it: simple, push-button art direction.

Automation Link

What if you want to work with more than one image at a time? If you’re building entire websites with hundreds or thousands (or hundreds of thousands!) of images — especially if you’re working with user-generated content — you’ll want more than push-button ease; you’ll need full automation. For that, there’s Cloudinary’s API, which you can use to call the smart-cropping and responsive image breakpoints functions that power the Generator, directly. With the API, you can create customized, optimized and fully automated responsive image workflows for projects of any shape or size.

For instance, here’s Ruby code that will upload an image to Cloudinary, smart-crop it to a 16:9 aspect ratio, and generate a set of downscaled resources with sensible responsive image breakpoints:

Cloudinary::Uploader.upload("sample.jpg",
    responsive_breakpoints: { 
        create_derived: true,
        bytes_step: 20000,
        min_width: 200,
        max_width: 1000,
        transformation: {
            crop: :fill,
            aspect_ratio: "16:9",
            gravity: :auto
        }
    }
)

If you work only on the front end, all of this functionality is available via URL parameters, too! Here’s a URL powered by Client Hints46 and smart-cropping47 that does the same thing on download that the Ruby code above does on uploadand it delivers different, dynamically optimized resources to different devices, responsively:

https://demo-res.cloudinary.com/sample.jpg/c_fill,ar_16:9,g_auto,q_auto/w_auto:breakpoints/sample.jpg

A tremendous amount of smarts is packed into that little URL!

Final Thoughts Link

But back to the Generator. Now, it can do more than “just” pick your image breakpoints — it can pick your art-directed crops, too. And it will generate all of the tedious resources and markup for you; upload one high-resolution original, and get back all of the markup and downscaled resources you need to include a scalable and art-directed image on your web page.

Have I mentioned that the Responsive Image Breakpoints Generator is free? And open-source4815? Give it a whirl49, and please send us feedback. Who knows, maybe we’ll be back again soon with version 3!

(vf, il, al)

Footnotes Link

  1. 1 https://cloudfour.com/thinks/how-do-you-pick-responsive-images-breakpoints/
  2. 2 https://cloudfour.com/thinks/sensible-jumps-in-responsive-image-file-sizes/
  3. 3 https://www.smashingmagazine.com/2016/01/responsive-image-breakpoints-generation/
  4. 4 https://www.smashingmagazine.com/2016/01/responsive-image-breakpoints-generation/
  5. 5 https://www.smashingmagazine.com/2014/05/responsive-images-done-right-guide-picture-srcset/
  6. 6 https://www.smashingmagazine.com/2016/09/responsive-images-in-wordpress-with-art-direction/
  7. 7 https://www.smashingmagazine.com/2015/12/responsive-images-in-wordpress-core/
  8. 8 http://www.responsivebreakpoints.com/?utm_source=Smashing_Mag&utm_medium=Byline&utm_campaign=Art_direction_responsive_breakpoints
  9. 9 https://cloudfour.com/thinks/sensible-jumps-in-responsive-image-file-sizes/
  10. 10 https://www.smashingmagazine.com/wp-content/uploads/2016/09/cloudinary-logo-big.png
  11. 11 https://www.smashingmagazine.com/wp-content/uploads/2016/09/cloudinary-logo-small.png
  12. 12 http://cloudinary.com/blog/introducing_intelligent_responsive_image_breakpoints_solutions?utm_source=Smashing_Mag&utm_medium=Byline&utm_campaign=Art_direction_responsive_breakpoints
  13. 13 https://www.smashingmagazine.com/wp-content/uploads/2016/09/generator-screenshot-large-opt.png
  14. 14 https://www.smashingmagazine.com/wp-content/uploads/2016/09/generator-screenshot-large-opt.png
  15. 15 https://github.com/cloudinary/responsive_breakpoints_generator
  16. 16 http://usecases.responsiveimages.org/#art-direction
  17. 17 https://www.smashingmagazine.com/wp-content/uploads/2016/09/art-direction-example-image-large-opt.jpg
  18. 18 https://www.smashingmagazine.com/wp-content/uploads/2016/09/smartdumb-uncropped-large-opt.jpg
  19. 19 http://res.cloudinary.com/demo/image/upload/white_cat.jpg
  20. 20 https://www.smashingmagazine.com/wp-content/uploads/2016/09/smartdumb-uncropped-large-opt.jpg
  21. 21 https://www.smashingmagazine.com/wp-content/uploads/2016/09/dumbcrop-example-image-large-opt.jpg
  22. 22 http://res.cloudinary.com/demo/image/upload/c_fill,ar_4:6/white_cat.jpg
  23. 23 http://cloudinary.com/blog/smart_automatic_image_cropping_maybe_you_can_always_get_what_you_want?utm_source=Smashing_Mag&utm_medium=Byline&utm_campaign=Art_direction_responsive_breakpoints
  24. 24 https://www.smashingmagazine.com/wp-content/uploads/2016/09/white_cat_g_auto_tests-large-opt.png
  25. 25 https://www.smashingmagazine.com/wp-content/uploads/2016/09/white_cat_g_auto_tests-large-opt.png
  26. 26 http://res.cloudinary.com/demo/image/upload/mountain_scene_panoramic.jpg
  27. 27 https://www.smashingmagazine.com/wp-content/uploads/2016/09/smartcrop-example-image-large-opt.jpg
  28. 28 http://res.cloudinary.com/demo/image/upload/c_fill,ar_4:6,g_auto/white_cat.jpg
  29. 29 https://www.smashingmagazine.com/wp-content/uploads/2016/09/smartcrop-example-image-large-opt.jpg
  30. 30 http://www.responsivebreakpoints.com/?utm_source=Smashing_Mag&utm_medium=Byline&utm_campaign=Art_direction_responsive_breakpoints
  31. 31 https://www.smashingmagazine.com/wp-content/uploads/2016/09/generator-screenshot-inputs-large-opt.png
  32. 32 https://www.smashingmagazine.com/wp-content/uploads/2016/09/generator-screenshot-inputs-large-opt.png
  33. 33 https://www.smashingmagazine.com/wp-content/uploads/2016/09/generator-screenshot-outputs-large-opt.png
  34. 34 https://www.smashingmagazine.com/wp-content/uploads/2016/09/generator-screenshot-outputs-500-opt.png
  35. 35 https://www.smashingmagazine.com/wp-content/uploads/2016/09/generator-screenshot-art-direction-inputs-selected-large-opt.png
  36. 36 https://www.smashingmagazine.com/wp-content/uploads/2016/09/generator-screenshot-art-direction-inputs-selected-large-opt.png
  37. 37 https://www.smashingmagazine.com/wp-content/uploads/2016/09/generator-screenshot-square-large-opt.png
  38. 38 https://www.smashingmagazine.com/wp-content/uploads/2016/09/generator-screenshot-square-large-opt.png
  39. 39 https://www.smashingmagazine.com/wp-content/uploads/2016/09/generator-screenshot-picture-markup-large-opt.png
  40. 40 https://www.smashingmagazine.com/wp-content/uploads/2016/09/generator-screenshot-picture-markup-large-opt.png
  41. 41 https://www.smashingmagazine.com/wp-content/uploads/2016/09/generator-screenshot-picture-live-large-opt-1.png
  42. 42 https://www.smashingmagazine.com/wp-content/uploads/2016/09/generator-screenshot-picture-live-large-opt-1.png
  43. 43 https://www.smashingmagazine.com/wp-content/uploads/2016/09/generator-screenshot-art-direction-inputs-annotated-large-opt.png
  44. 44 https://www.smashingmagazine.com/wp-content/uploads/2016/09/generator-screenshot-art-direction-inputs-annotated-large-opt.png
  45. 45 https://cloudfour.com/thinks/responsive-images-101-part-5-sizes/
  46. 46 http://cloudinary.com/documentation/image_transformations?utm_source=Smashing_Mag&utm_medium=Byline&utm_campaign=Art_direction_responsive_breakpoints#automatic_image_width
  47. 47 http://cloudinary.com/documentation/image_transformations?utm_source=Smashing_Mag&utm_medium=Byline&utm_campaign=Art_direction_responsive_breakpoints#automatic_cropping
  48. 48 https://github.com/cloudinary/responsive_breakpoints_generator
  49. 49 http://www.responsivebreakpoints.com/?utm_source=Smashing_Mag&utm_medium=Byline&utm_campaign=Art_direction_responsive_breakpoints

↑ Back to top Tweet itShare on Facebook

Eric is a Developer Evangelist at Cloudinary and lives on beautiful Orcas Island. He loves visual communication, learning through teaching and building, and the web.

  1. 1

    I’ve been using this tool since it first came out and it is amazing and I love that their now taking on the challenge of automating art direction, I believe however that computers will NEVER be able to figure out how I want MY image to look like. That is the whole beauty or art direction.

    2
  2. 2

    This is in essence what the browsers should do.

    Amazing, useful tool. +1

    Thank you, thank you, thank you!

    0
  3. 3

    Maitreya Vyas

    October 3, 2016 12:32 am

    Nice to see progress on it. Thanks! :)

    0
  4. 4

    I prefer imgix for stuff like this they have a great API and use the great fastly cdn

    0
    • 5

      Andrea Verlicchi

      October 6, 2016 5:10 am

      Cloudinary also have a great API and serves images through a CDN (Akamai)

      0
  5. 6

    This is awesome! Started playing with this a couple of days ago on our website in work (http://www.transcendit.co.uk/it-support-north-east). Haven’t tried the API yet but that’s the next step.
    Super excited, it’s really VERY good!
    Thanks for the great post

    0
  6. 7

    George Hapsis

    October 8, 2016 5:37 am

    Wow! Just discovered this! Its so practical and easy!
    Thank you!

    0

↑ Back to top