A computer program that recognises sketches pioneered by scientists from the School of Electronic Engineering and Computer Science (EECS) could help consumers shop more efficiently.
The sketches of a pair of shoes or piece of furniture for example, are drawn directly by hand on a touchscreen and recognised using a sophisticated image retrieval system, where the top 10 retrieval accuracy is already close to 100 per cent on some object categories so that it always displays the desired product on the first page.
“What’s great about our system is that the user doesn’t have to be an artist for the sketch to be accurate, yet is able to retrieve images in a more precise manner that text.” commented co-developer Dr Yi-Zhe Song, Director of the SketchX Research Lab, from QMUL’s School of Electronic Engineering and Computer Science.
He continued: “With the proliferation of touchscreens, sketching had become a much easier to do and in some ways is actually preferable to text-based or photo searches.”
Fine-grained Sketch-based image retrieval (SBIR) overcomes problems with using words to describe visual objects in words, especially when dealing with precise details, and with using photos, which can restrict the search far too narrowly.
Co-developer Dr Timothy Hospedales and Director of the Applied Machine Learning Lab also from QMUL’s School of Electronic Engineering and Computer Science, said: “For the first time users are presented with a system that can perform fine-grained retrieval of images - this leap forward unlocks the commercial adaptation of image search for companies.
“Current image search capabilities provided by online retailers only perform category-level searches so for customers it is still a tedious “browsing” exercise to find your ideal item. Now you can simply sketch your mental image of your ideal product and go straight to it.
The computer program system is designed to emulate the human brain’s processing through arrays of simulated neurons. It was trained to match sketches to photos based on about 30,000 sketch-photo comparisons, learning how to interpret salient details of photos and how people try to depict them in hand-drawing.
The research team has been working with Queen Mary Innovation for the past year, and have already identified commercial interest from multiple major retailers as well as potential investors.
The research was accepted at International Conference on Computer Vision and Pattern Recognition (CVPR) will be presented at the end of June.