Recognizing Coins with Seeing AI

The constant struggle of a blind coin collector is how to identify a coin.

The easy way is, ask somebody what it is. But it is not all that simple. It is not other people’s hobby after all, and I can’t expect to have somebody around to help me all the time. Over the years using experience and technology I have perfected my methods, but of course, it will be impossible to identify all coins without vision. I just found a new method which will greatly help me identify coins much more independently.

If you are new to the blog, here are some ways I can figure out my new coins. This is how I identify coins in general. Another way is by using elimination techniques, though it only works in certain cases. Then just recently I found another app which connects me with volunteers, which appears to be the most reliable.

I’ve been using an app recently, called Seeing AI from Microsoft, which helps me recognize objects and text just by taking a picture of it. The app is designed to help blind people better understand their environment. I found a way to use this app to recognize some coins. But first some explanation of the background.

When I hold a coin, if I would know what is written on it, I would be able to tell what the coin is in most of the cases. Of course, this assumes that I could read different alphabets and scripts. Character recognition has been around for decades, I can take a picture of some text or scan it with a scanner, and a character recognition software can read it to me. These days it can even handle handwriting. But in case of coins it is not all that simple. The biggest issue is that character recognition softwares are looking for straight text, while on a coin text is very often read around the edge of the coin. There are some coins where text is straight, or relatively straight, but character recognition systems are also designed to read from paper and letters on a coin would have to be processed differently. Then the next problem is that a coin can be older, worn, or use a completely different font from what you would expect in a book.

Over the years I have tried to use character recognition for coins, and talked with a few people about how to solve this problem, but there wasn’t a real solution. The closest one is Lookzee which is an app that is specifically created to identify coins. The problem here is that it doesn’t have enough coins in the database. Once it will be able to handle most coins this will be the ideal solution. In the meantime, I could not find anything. When I ran character recognition on a coin, it did not give me any useful results.

This week, I tried Seeing AI again, I took a picture of a coin, and ran it through the character recognition. To my surprise, the app gave me enough information to make it useful. Again, if you are new to the blog, this is how I take a picture of a coin. The app receives continuous improvements so what didn’t work a few months ago may work now, it is worth checking back to come up with new solutions.

Seeing AI has a function where I can take a picture I took on my phone and have it identified. The app tells me what is on the picture, and if there is any text it will recognize it as well. The app has a feature built in where I don’t even have to take a picture with the camera first, it will just take a picture inside the app and will recognize it. For some reason this feature didn’t give me such good results.

Just like any text recognition system, Seeing AI will look for text to process, it will even try to recognize text which is upside down or somewhat tilted. So when I scan a coin, it probably gets confused because characters are in all kinds of directions, so it will identify the most number of readable characters and read those to me. Imagine it this way: hold a coin, and read all the characters from it which are not tilted more than 90 degrees. Probably the actual recognition is restricted to less than this. Depending on how I hold a coin, this may or may not give enough recognizable text to make it useful. But if I take a picture of a coin, then a second one after turning it 90 degrees, etc., after four pictures I should have most of the recognizable text positioned in a way where Seeing AI can recognize it. Now I just need to do the same thing on the other side, and put the pieces together. Now the problem will be that there will be some overlap, some missed characters, some inaccurately recognized ones, etc. So I need to extract enough usable text from the eight pictures to make sure I get a sense of what it is.

Of course the concept of text recognition is much more complex than how I explained above, but this will be sufficient for our purposes.

Let’s see how it works in practice. I took a random coin, I didn’t know which was the obverse or reverse, and I took 8 pictures of it. I’ll show you step by step how I came to the conclusion of what the coin is, without sighted assistance, only using Seeing AI.

Below are the eight pictures and a copy of the text under each, the way Seeing AI recognized them.

probably a black sign with white text

DIEZ PESOS brrrrrrrrrr 2010

The first line is the actual picture description, which I will omit under the following pictures. Usually when I take a picture of a larger surface with a coin on it, Seeing AI will be able to tell that it is a coin, this is somewhat deceiving because almost the entire picture is the coin itself. Seeing AI is looking for things to recognize in a persons environment, I am sure it would be capable of giving more info about a coin, but let’s face it, this is not what blind people want to do in general.

The second line is more interesting, it will right away tell me that it is diez Pesos, at the end of the line I get a date of 2010, and some brrrrrr in between. This is probably coming from part of the coin design which is recognized as letter “r”s. This is better than usual, now I could even use a method of elimination to find out what the coin is. Actually, if I put “diez pesos 2010” into the Numista search, it will only find the Mexican 10 Peso coin, KM# 616. This is too good to be true, if the coin said “1 cent” and no date for example, it would be less obvious. So let’s keep analyzing the rest of the pictures.

2310 rrrrrrri
2 PESOS

If I read this picture first, it would likely mislead me. It got Pesos right, but there is a number 2 before it from somewhere else, if the first one didn’t spell out diez, I could now think that this is a 2 Peso coin. But diez is more difficult to be misrecognized than 2. So, we will put this aside, this text did not add to the information we already had.

6053d 2310

This piece of text is completely useless, probably the orientation of the coin was such that it could not make much sense of it.

PESOS

This was the fourth picture of the first side, we got back to a recognizable string of Pesos, but we already know it.

S UNIDOS MEXICANO soaVisa

Here we get “Mexicano”. Well, game over. We know all we need, the coin is from Mexico, on the previous pictures we established that it was 10 Pesos, now I can run a Numista search with “diez Pesos Mexico” and I can get more information about the coin. But let’s continue with the picture analysis, because sometimes we don’t get it right from 5 pictures.

DOS UNIDOS ME EXICANOS

From this picture again, we know that the coin is from Mexico.

ESTADOS UNIDOS SONVOTE

This is also good information, but not narrow enough, because many coins have the text “Estados Unidos”, for example Brazil, Mexico, etc., though many of these are larger commemoratives so it is easier to narrow down the possibilities.

MEXICANOSSLS IDOSM DOS UNI ESTAD

Here the text narrows it down to Mexico again. When it comes to text recognition, sometimes we have to guess, based on what it reads, we can figure out what the original text could be. For example, often times the number 1 and the letter “l” show but based on the context, we can tell if it is “1998” or “l998”.

If we were lucky enough to check the second and the fifth pictures first, it would give us enough information. But if we don’t know how to hold the coin for best recognition, chances are slim we would get it right from two pictures. I found that I can get relatively good results from taking only two pictures of each side, rotating them 180 degrees, but it assumes that at least one of the pictures has a good text orientation. I found that I can get more reliable results from four pictures per side. I also found that I get a much better recognition rate if I take monochrome pictures.

Let’s look at another example. I will not include pictures for this one, since the previous examples showed how it works. I took eight pictures of a coin, and the only text I was able to get from these was “brasil”, “centavos”, “2017”. When I put this into the Numista search, it came up with four coins, 5, 10, 25 and 50 Centavos from Brazil. The coin I scanned had a smooth edge, with some small ridges on it. The five Centavos has a smooth edge, the 10 and 25 Centavos have reeded edge, and the 50 Centavos edge description says smooth and inscripted. Based on elimination, this must be the 50 Centavos. When gently touching the coin, one can also feel the number 50 on it, though it is not the easiest to identify. Ok, I was cheating a bit, because I took this coin from my leftover change pile I brought home from Brazil. But for all practical purposes, I would have come to the same conclusion if I didn’t have a clue what this coin was.

This method does not work in all cases, I found it most reliable with Latin alphabet text. However, I had some luck with the Cyrillic alphabet, but in this case I need to know what the Cyrillic letters look like. Do you remember “CCCP”? It is really “SSSR”, so if I know that C stands for S, P stands for R, etc. when recognized based on the Latin alphabet, I can get some idea of what it could be.

Now why is this recognition method so impressive? After all, it takes eight pictures and about 2-5 minutes per coin to have a chance of being able to identify it, assuming that the coin is readable and it uses a Latin alphabet. If you can see, in case of most coins it would only take seconds. This way, I am able to identify many of my new coins without asking for help. I can further improve the recognition rate if first I touch the coin and try to position the text under the camera without much tilt. Also, if I visit a country, Brazil for example, and I know that I only have Brazilian coins, it is easy to identify the dates and values. I will only have to ask for help or use other methods for the ones which Seeing AI could not recognize.

It may not seem much, but in reality, it will give me new opportunities to identify coins and grow my collection much faster. This will become a true breakthrough in collecting without vision.