Back around 1997 I was doing a project for Nuon GZM, an electricity provider. We would take over the utilities accounting of National Railroads (NS). We started from scratch, four months late, no software. And we had two months until we had to supply the first batch, half a year of proper checked and processed documents, with EDI to NS. So I said ‘I can do that’. I am a RAD boy, and used MsAccess to develop apps.
The main routine was a matching of banking transactions and manual input (1800 documents with variable ‘keywords’) versus 1800 costcenters (the railway stations, with 6 keys). That took 4 hours on a Pentium with standard ‘dumb’ SQL methods in MsAccess. That was too long. So I had to speed it up. You can take a faster machine, that might save 10-15%, or use a mainframe (that is not my techstack) or use SQL optimisations (that would save 5%). That would not work.
So I decided to dig into basic AI techniques. I had heard very positive things about it, the programmers were very enthousiast about it. And I got a book on games algorithms. Games were far more advanced in crude AI techniques than accounting software. It contained the algorithms of Doom. Doom was fun back then. I learned about BSP binary space partitioning, search space minimisation, and other small techniques.
In a 60 day project you don’t have a month to dig into it. You have two days to read and learn and two days to program, test, debug and implement it.
I learned about ‘stored success’, and decided to work that out.
Your first ‘training run’ is dumb sql, you start with banktransaction 1, key 1, and check for a match on costcenter 1, key 1, then key 2, and so on, that is roughly 60 million matching operations. If you find a match, you pop the costcenter key on top. Then you end up wwith a table with costcenters and their ‘successful’ match-key.
Next time you pick the most ‘successful’ costcenter key, the one with the most matching, and only search for matches with the corresponding banktransaction key. You start matching, banktransaction 1 key A versus costcenter X key A. in 800 matches you have 780 successes (some costcenters change supplier or any other change).
Then you take the next most-found-matching-key and same thing, in 500 matches you score 490 hits. Simple as hell, but you reduce the matching operations to less than 750.000 per run, a 98% reduction. With the programming overhead included, I achieved a 92% reduction, from 4 hours to 20 minutes.
After that, you always pop the “winner” and it will always keep optimising itself.
I don’t know if it is actual AI but it worked like charm.
In the big picture, the programming communities do something similar with open source libraries. You can reinvent the wheel a billion times or list the wheel online with the proper keywords, and anyone searching for a ‘succces’, a working solution, can reuse it. That saves an incredible amount of time of money.
There are a few large language communities, not so much coding language, but english, russian, indian, chinese, arabic and these are also economic blocks. We have to compete. Not just with each other but with the other economic zones. And the language zone that is best at using’ ‘stored success’, that is gonna be the winner.
Most dutch work in the english zone, our english is faulty but we know enough to read and comprehend english. And some of us also write english.
I always got flack on the job back then because sometimes I could spend 3 hours googling. And my managers pissed, “you cost 100 per hour and do nothing”, aw I am so misunderstood…
A good library contains anything from 200 to 2000 hours of work. Research, coding, testing, debugging, beta release, user testing, user feedback, updates and then you get a ‘battle tested’ top quality codebase that is both free and worth 20.000 to 200.000 dollars of work. If I can harvest 200.000 dollars worth of developer work wit 3 hours of searching (300 dollars) I am actually doing great. And the less lines I myself code (because I always figure I suck at coding) the better, the less maintenance you have. Because the libraries are all maintained.
RAD Rapid Application Development was also rather new back then. Most coders still had the usual ‘2000 lines a day’ regime and just kept typing and basically reinventing the wheel all day. It was a total waste of resources.
It is the wrong stimulus, if you pay people to type 2000 lines of code, that is what they are going to do. If you pay them for the end result, they are going to optimize towards that, and start using libraries, ‘stored success’.