Intelligent Shopping List

The Eddi Nez Shopping List is a project I did in 2019 as a submission for the EDEKA Smart Shopping-list Competition. It was one of the most exhausting projects, I have ever done, because I made it entirely with Javascript, IndexDB and HTML instead of PHP and MySQL, which would have been VERY easy, since the project involves a huge amount of data and computing speed. I used JS, IndexDB and HTML because I wanted the project to also work Offline, that way it could even be turned to an Hybrid Mobile App, which I ever made up my mind to make it an App. I also did not find out about the competition, until I recieved the ad on my Instagram few days to the end of the competition about one week, so I had to mix up working, studying and writing a completely JS based code at the same time. WTF?! The short version of this post is here

The project comprises of different parts of programming, one of this is my favourite – the autocorrect and autosuggest algorithm.

I had recently – at the period of coding – read about the Autocorrect Algorithm from Peter Norvig,which he wrote in 2006. Peter Norvig is an American computer scientist and the current director of research at Google Inc., and used to be its director of search quality. I would really recommend, that you visit his Website, if you are a computer freak – lol, I meant geek. So basically Peter Norvig writes in Python, which means his algorithm and code was in Python (-in this post, Python is a programming language not a snake). So the first problem I had was the implementation of his code in Javascript. Fortunately, Peter Norvig was a not a terrible programmer and explained every step he took down to each detail (In fact, I loved it so much I wrote the PHP implementation, Autocorrect-PHP ).

A quick summary of the algorithm:

1. Take a very large text of the language, you want to create the autocorrect for.
For example, you could take a “.txt” file, that contains a whole book of a popular Novel. Peter used a concatenation of publicly-available english books from the Gutenberg project. Since, I was building an auto-correct for german foods, I had to outsource – unfortunately, I couldn’t find a publicly available list of german ingredients and food, that also provides how frequently these recipes are used, which means I had to crawl websites, I crawled about 300 pages from delicious.de overnight – I had to do this about 3 -4 times. Thanks to a web crawler from Dexi.io, I used.

2. Seperate each words and create an array of words and permutation (how often the word occurs in the text).

3. Get the word from the user

4. Is it a correct word? Check how many changes do you have to make to get a correct word – do you have to had letters, remove letters or swipe two letters’ positions? e.g
a. teh is an incorrect word, if you try to make the from teh, that is a swipe of ‘h’ and ‘e’.
b. ric -> rich/rice is an addiction of letter ‘h’ and ‘e’ respectively
c. Computerd -> computer is a subtraction of letter ‘d’.

5. Get all possible correct words and show th e word with the highest frequency as first result. For example in every english text (considering a very large text of approximately 3000 words), the word “the” should appear more often than the word “they“, so if a user types “tehy“, your algorithm, should favour the suggestion of the word “the” before “they“.

Like I said, Peter explained it better. check out the explanation here.

Well, after getting random collations of words, ingredients, qualifiers and quantifiers from the spider, I had to seperate food names from ingredient names and also both of these from qualifiers like gram, kilogram. Essentially, the crawler delivered phrases like ” 100g rice , 1 kg of rice, 3 kilogrammes of beans, 3 cups of vegetable oil, 2 small tablespoons of salt, 2l Milch, 2 coconuts”. Did you notice the new problem? Exactly, there was no reasonable way for a program to easily differentiate qualifiers from the ingredients, sometimes there are spaces between qualifiers(2 small tablespoon), sometimes there are spaces between the ingredient (vegetable oil), sometimes the qualifiers are abbreviations (compare 1kg and 3 kilogrammes), sometimes words and qualifiers are not separated with “of”, sometimes the qualifiers are grammes,sometimes kilogrammes, so basically I had to write a ton of Regular Expressions to check a very large file. Well, I found out Sublime Text can not search through .txt files larger than 5 mb using Reg Ex and I had to create a php file that searches with reg ex . Thank you, Rasmus Lerdorf, for making PHP!!

At the end of the day after always being awake until 3 AM, I had been able to create seperate arrays of the ingredients, qualifiers, assigned qualifiers and their abbreviations – in german. I also learnt some cool words such as “Bund”- apparently bund is a german word, that fits almost all ingredients, it could mean one piece or an appropriate quantity of ingredient – doesn’t matter, weiter.

The next step was choosing the right qualifier for each ingredient, for example you can say 1 litre of water but you can’t say 1 litre of rice, even though you can say 1 kg of water (ice). You can also say 1 piece of Mango but not 1 piece of beans -very inappropriate. Also each ingredient may also have more than one qualifier, 1 can of cocacola, 1 litre of cocacola, 1 bottle of cocacola =>a many-to-many relationship – doesn’t matter, weiter.

Although, there are still a lot to do after that, I would end the journey of the auto-suggestion there.

The less difficult problems were storing the data and the user input locally. At first, I used the JS localstorage, but there was a big problem, I had to implement a search function. You can not efficiently search the localstorage, so I had to rebuild the whole storage logic again using indexDB with DexieJs, I used Dexie in one of my other top secret projects (RainbowCoil), I won’t talk about this here – I could get arrested, lmao, seriously.

I also had to create a custom search algorithm that considers the german umlauts, because german introduced 5 more letters that are not found in the english language!, this means the search should understand Wässer should also show Wasser as a search result, Äpfel and Apfel too. Mark Twain wrote a joke about the german language.

In conclusion, I wrote a whole ton of codes in the night, till I finally submitted the code. There are still a lot of things, I did apart from the storage and word suggestion, such as, an image for each product using a placeholder service, sorting of lists, addition of similar item – for example, 1kg and 500 grammes should be 1.5kg, but those two were the most exhausting, since they exposed my programming to a new part, I have never explored before.

You can test the code here at https://recipe.piccmaq.com.ng/ . By the way, Eddi Nez means EDeka INtelligentes Zettel – I had no better name. It is however only compatible with german words and it is in german.

Here is a Google translated copy of the email, I recieved after my submission was reviewed by the team.

Hi Michael,

our partner company EDEKA DIGITAL and we really appreciated your submission for the Code Competition – “INEZ The intelligent shopping list.”
You’ve done you a lot of thoughts in your solution. The selection of winners and evaluation of the solutions is not easy EDEKA DIGITAL.
Unfortunately it was not enough this time for a place on the podium 🙁

Your rating:
Setup: 5
Basic functions: 7
Code Quality: 6
Documentation: 6
Features: 6
GUI: 5

The evaluation and your feedback has been created by the Department of EDEKA DIGITAL:
A really nice solution! Here are some points that we noticed in the evaluation and peculiarities represent your solution: – Web-based application JavaScript, HTML, CSS
– Database using your own crawler and manually completed data sets
– Instructions on how to launch the application was available
– data is stored locally in a IndexedDB
Functionality:
– Product and units are suggested after the user inputs a product
– Quantities are detected
– Add to shopping list
– Same types of products are recognized and added(case-sensitive)
-Quantity/Amounts of added articles can be changed later
Code:
– The code looks fine
– By using a framework or the MVC pattern, the application could be better structured and would be more scalable

Tip: Look at how we evaluated your solution: https://www.it-talents.de/blog/partnerunternehmen/so-wertet-edeka-digital-deinen-code-aus

We will shortly send a certificate for your participation.

Hopefully you had fun during the Code Competition, and it helped to improve your skills.
I very much hope to have you at the next Competitions back again, we shall, in any case, for exciting tasks. For example, the current Competition:
https://www.it-talents.de/foerderung/code-competition/zf-code-competition-11-2019

The results of the competition and the winners (and their solutions), will soon be introduced on our blog.

If you have any questions or feedback, you can contact me.

Greetings from Bielefeld
Your IT-Talents.de Team

Below is the original Readme.md, file I submitted along. I will probably release the codes on my github soon.

## INEZ - Smart Shopping List
To keep this program as small as possible and to improve efficiency,
this program *only* corrects food products because it was made for Edeka,
where mainly food is sold. This program should also work offline,
since it was built on Javascript. This program was written in just 2 days,
the data used was collected and analyzed in 3 days.
I've only been able to spend 4 hours a day with it, because I work and study at the same time, but I can count on the program to run really well.
This README file is written as Markdown .md, for optimal readability, please use a markdown reader. Because this program was written to Javascript, its speed and efficiency can be influenced by the computer used for testing. With PHP and a good server, parsing should take much less than 1 second. With Javascript it can take up to exactly one second or less - this influence depends on the computer. Due to the fact that I did not have that much time, a few norms were neglected, for example user input verification, but they should not be a major problem because the data does not end up on any server.

**Before you start reading**
Qualifier and quantity description are words that describe a quantity better, for example Kg, liter, glass, bottle

**Features*
1. Auto-correction
2. Autosuggest
3. Long term Saving
4. Replace with synonyms

**To run the program,**
1. Upload the whole files to a server,
2. start index.html

## Creation and data models
Unfortunately, there is no public API for food.
To create this program and get the necessary data,
1. I created a crawler
2. Crawled about 300 pages on delicious.de
3. all ingredients according to recipe
4. extracted with regular expressions qualifications.
5. And write a lot of products with my hands

## Algorithms
The intelligent shopping list works with many algorithms.
Visit the links to learn more.

**The algorithm i wrote*
1. The algorithm for recognizing qualifiers
2. The algorithm for parsing products from sentences

**Public*
1. The auto-correction algorithm of Norvig
https://norvig.com/spell-correct.html,
2. Autocorrection algorithm of Willsen
https://github.com/WillSen/spellchecker-autocorrect
3. Dexi Computer Bots and Spiders
Digital Commerce Intelligence – Dexi.io

4. the Dexie database storage plugin (Dexie.org)
https://dexie.org
5. Jquery - for frontend editing
https://jquery.org
6. Logo from PlaceIt
https://placeit.net/logo-maker

## working step
The program works mainly in 4 steps
1. It checks the Sentence written bz the user
a) Is the shopping article written in the normal form?
50 kg of rice
b) is the article in a tailored form without a space
50 liters of water
c) Absolute form without quantity description
50 bananas

2. checks the qualifier (the quantity description)
a) Is the qualifier a valid and the right qualifier?
eg Kg must not be used for liquids
b) Is it written correctly?
for example, Kiolters instead of kiloliters

3. Checks the product and suggests the right qualifications
if incorrect e.g 1 kg of beans

4. Check synonyms
5. Display result
6. Wait for a confirmation from the user with a click
7. Store in the index database

## files
1. Autocorrect.Javascript
This file contains functions for the auto-correction algorithms
2. core.Javascript
This file contains Javascript functions for the frontend
3. css.css
Css styling file
4. dbcontroller.Javascript
Functions for saving and reading a purchase list
5. index.html
Frontend HTML
6. ingredient.Javascript
contains a collection of kitchen products
7. readme.md
Declaration on the progress of the program
8. realdatalist.Javascript
List of some products and the right qualifier
9. synonym.Javascript
List of synonyms for specific products
10. libs /
Contains all public libraries like jquery, dexie and bootstrap.