I find it fascinating how we are able to tell just by looking that ‘ gebracht’ is most likely a German word and ‘ reconstituer’ looks like a French word. We are able to spot certain patterns in words that give us clues as to which language the word belongs. An example is an ‘eux’ at the end of a word is an indication that the word is French, whereas words ending in an ‘o’ tend to be of Spanish decent (e.g. renacuajo, basurero etc). The aim of this project is to teach a computer to do the exact same thing; to be able to recognise which language a given word comes from.
Due to the complexity of this task, using a deep neural network will most likely be the most accurate technique. This project was undertaken in python, and I used the Keras package to handle the neural network side of things. Note that Keras uses the TensorFlow backend, so we are able to generate some nice TensorBoard visualisations.
The plan of this project is to create a huge labelled word library from scratch, with words from five different languages, each labelled with their language. The way that this will be done is by scraping a load of Wikipedia articles. We will then convert these words into binary vectors so that they can be inputted into the neural network. The network will be trained on 85% of the words and be validated on the remaining 15%. We will then (hopefully!) have a trained network which is able to accurately predict which of the five languages a given word is from.
Lets dive in. The first thing I did was to pick which five languages I want to use for this project. I chose English, German, Czech, Swedish and French. I thought that these languages had enough distinct patterns in them that the network would be able to recognise. The next thing I did was to find around 15 or so Wikipedia articles for each language, making sure to vary the topics to ensure a diverse range of words were picked up. Below shows the first file ‘config.py’, essentially the configuration settings for this project.
We see it only contains two variables: ‘max_letters’ and ‘language_tags’. Max letters is the maximum length of word that the scraper will pick up, and hence the maximum length of word that can be inputted into the neural network. We then see a dictionary ‘language_tags’. The dictionary’s keys are the Wikipedia tags of the five languages that I am using, and the values are lists of the names of Wikipedia articles in that language. It is important that all of this can easily be changed. If I wanted the network to use ten languages, I would simply have to add another entry into the dictionary, or if I wanted the network to be able to predict 15 letter long words, I would just update max_letters.
We now move onto the next file, ‘functions’. This project contains quite a few complicated functions to not only scrape for the words and add them to a large word library, but also to turn these words into vectors etc. Hence I have created a separate file which houses these functions to keep the code nice and organised. The first function is called generate_dictionary and it is shown below.
The aim of this function is to create a long list of words that were found in all of the provided Wikipedia articles for a given language. Its parameters, tag and max_word_length are fairly self explanatory. They are the language tag and the maximum number of letters that words are allowed to be in this list. Note for the Wikipedia scraping I will be using the wikipedia python package. We set the language of the Wikipedia articles to the desired language in line 9, and then begin iterating through the articles of this language. Line 11 gathers the raw HTML, and line 12 gets the raw text content of the article as one long string. Since we want the text to be all in ASCII, we use the unidecode function to convert the text into ASCII. You may then notice peculiar function called process. This is the next function in this file shown below.
So process takes two parameters, page_content (the string of the Wikipedia article content), and the maximum word length. We begin by using regular expressions to extract only the words, removing any numbers, links, punctuation etc from the article. Then, it is important to remove any capital letters, hence we use the lower function on this text. Then, the split function is used to create a list from the remaining words. So now we have a list of every word in the article. However we must still remember that we cannot include words more than max_letters in length. We filter these words out in lines 6–8, and return this fully processed list. Then, generate_dictionary (the function above this one) returns this list, and we are all done.
The next important function in ‘functions.py’ is convert_dic_to_vector. This function takes a list of words (like the one generated using the above function), and returns a list of vectors representing each word. The way I am representing a letter in this vector is as follows:
a=10000000000000000000000000
b=01000000000000000000000000
c=00100000000000000000000000
…
z=00000000000000000000000001
We then string these together to form a word. Since all the vectors must be the same length, we must fill the unused letters (max letters-used letters) with zeros. For example the word ‘hello’ would be represented as.
00000001000000000000000000
00001000000000000000000000
00000000000100000000000000
00000000000100000000000000
00000000000000100000000000
00000000000000000000000000
00000000000000000000000000
00000000000000000000000000
00000000000000000000000000
00000000000000000000000000
00000000000000000000000000
00000000000000000000000000
The first 5 letters are populated with h, e, l, l, and o, with the remaining 7 letters populated with zeros. Also, it would be one continuous string of numbers without line breaks (the line breaks were for visual clarity). This means each word vector will be a 26*12 = 312 digit long string of ones and zeros. So now we must implement a function which converts a list of words into a list of vectors in the form shown above.
The function takes a list of words and a maximum word length. Then for each word in this list, we create an empty string called ‘vec’ (line 4). Then, for each letter in the word (line 6), we convert this letter into a number (a being 1, b being 2 etc) (line 8). We then create the vector from this in line 9 by adding zeros up until this number, then a one, then fill the remaining spaces with zeros. Remember, this process is repeated for each letter in the word. This vector is then added to the ‘vec’ string. Once this is done, we need to fill the unused letters out of the 12 with zeros. This is done simply enough in lines 11–13. We then append this ‘vec’ string to a new list, and repeat for all the other words. The function then returns this list of vectors.
The forth and final function in ‘fucntions.py’ is called create_output_vector. This function creates the output vector for a given language. (shock, I know!) Below I will list what I mean by ‘output vector’.
English = 10000
Czech = 01000
German = 00100
Swedish = 00010
French = 00001
These will be used to label the data. i.e. the vector for the word ‘hello’ shown above will be labelled with the output vector corresponding to English (10000), whereas the word ‘ typhuvud’ will be labelled with the Swedish output vector (00010), since it is a Swedish word. Below shows the function.
The parameter tag_index is the index of the language in the language_tags dictionary (from the config file), and the number_of_languages parameter (obviously) takes the number of languages being used. This one line function then simply creates an output vector as I have described above, and returns it.