EECS researchers have built the first computer program that can recognise hand-drawn sketches better than humans.
Known as Sketch-a-Net, the program is capable of correctly identifying the subject of sketches 74.9 per cent of the time compared to humans that only managed a success rate of 73.1 per cent.
As sketching becomes more relevant with the increase in the use of touchscreens, the development could provide a foundation for new ways to interact with computers.
Touchscreens could understand what you are drawing enabling you to retrieve a specific image by drawing it with your fingers, which is more natural than keyword searches for finding items such as furniture or fashion accessories. The improvement could also aid police forensics when an artist’s impression of a criminal needs to be matched to a mugshot or CCTV database.
The research, which was accepted at the British Machine Vision Conference, also showed that the program performed better at determining finer details in sketches. For example, it was able to successfully distinguish the specific bird variants ‘seagull’, ‘flying-bird’, ‘standing-bird’ and ‘pigeon’ with 42.5 per cent accuracy compared to humans that only achieved 24.8 per cent.
Sketches are very intuitive to humans and have been used as a communication tool for thousands of years but recognising free-hand sketches is challenging because they are abstract, varied and consist of black and white lines rather than coloured pixels like a photo. Solving sketch recognition will lead to a greater scientific understanding of visual perception.
Sketch-a-Net is a ‘deep neural network’ – a type of computer program designed to emulate the processing of the human brain. It is particularly successful because it accommodates the unique characteristics of sketches, particularly the order the strokes were drawn. This was information that was previously ignored but is especially important for understanding drawings on touchscreens.
Timothy Hospedales, co-author of the study and Lecturer in computer science, said: “It’s exciting that our computer program can solve the task even better than humans can. Sketches are an interesting area to study because they have been used since pre-historic times for communication and now, with the increase in use of touchscreens, they are becoming a much more common communication tool again. This could really have a huge impact for areas such as police forensics, touchscreen use and image retrieval, and ultimately will help us get to the bottom of visual understanding.”
More information: The paper ‘Sketch-a-Net that Beats Humans’ by Q. Yu, Y. Yang, Y. Song, T. Xiang and T. Hospedales will be presented at the 26th British Machine Vision Conference on Tuesday 8 September 2015.