We have a crop tool in the editor, that “suggests” a default crop based on the selected aspect ratio. The suggested crop starts at the top-left corner and expands with the selected ratio until it reaches one edge of the image.

The suggested crop for a 1:1 ratio

The problem with this dumb behaviour is that it doesn’t help our editors to select the main part of the image. They always have to adjust the suggested crop because the interesting area is rarely close to the top-left edge of the image.

We had a company hackahtlon recently and I set out to improve this situation by introducing a library that can detect the interesting parts of the image. I only had 6 hours to work on this.

First I realised that our 3rd party cropping library was bumped a major version, so I updated that, and refactored our component to work with the new version and to accept a middle man that preprocesses the image and suggests a crop window.

Advertisement

That took me a while :)

Doing the “smart crop”

Due to higher saturation and more skin-like colour Mona Lisa’s chest area is far more interesting than her head.

Advertisement

I found an open-source lib that can detect the interesting parts of the image by processing the image and weighing parts of the image with high saturation, lots of edges or skin-like colours before other areas.

To avoid focusing on chest areas to much we could pre process the image with OpenCV and tell the smart crop lib to weigh some pre-selected areas more than others, but that would be an overkill for this project.

Advertisement


Integrating the lib

I had to make the two 3rd party libs and our components working nicely together.

The cropping component works with percentages. So a set like this {x: 10, y: 10, height: 70, aspect: 16 /9} tells the view to show the crop rectangle 10% of the image width from the left and 10% of the image height from the top, make it 70% of the height and the width should be 1.7 times the height.

Advertisement

I had to convert this to pixel coordinates to work with the smart crop lib, then back to percentages to show the user. This is how ours and the two 3rd party lib integrates together.

  1. Find largest area we can fit in the image based on the selected aspect ratio.
  2. Convert that area to pixel coordinates.
  3. Pass the image to the smartcrop lib along with the width and height of our largest window.
  4. Get back the new coordinates of that window
  5. Convert to percentages, pass it to the cropper component.
The new crop window is correctly assuming that the elephant is the most interesting part of the picture.