EuroBillTracker

This topic will be used for writing down the criteria and calculation methods for the forthcoming hit interestingness score system. We have been collecting ideas in a separate brainstorming topic, now it's time to move forward with the ideas. This topic will be the official specification of the system, so please try to stay strictly on topic and keep the discussion related to the current specification phase (more on this later). If you want to comment the system as a whole (or whether the system would be useful at all), please use the brainstorming topic. I specifically reserve the right to remove any messages from this topic which are not related to the current specification phase. This is for the benefit of those who are interested in contributing to the specification, off-topic chatter would be distracting. Consider writing a separate message for each topic if needed. Thanks for understanding.

History and the current situation
The hit moderation was initially conceived as a method to hide the hits between friends exchanging their notes with each other. That same moderation system was also used for moderating all but one hit from a group of hits between the same persons. The term "interesting hit" was born.

The rules for moderation were a bit vague back then, and quite frankly, the "rules for moderation" situation hasn't improved since the early days. There have been ongoing attempts to come up with exact rules, but that hasn't succeeded. As the rules haven't been exact, some users have felt that they have been treated unfairly. The moderators try to work as impartially as possible, but clashes between hit moderators and users have been common all this time.

The same hit moderation is also applied to hits in which the users haven't had the notes in their hands. A classic example of this are the serial numbers shown in media.

This is a bit of a problem, because now the same hit moderation system is used both for moderating uninteresting real hits and also to combat abuse. This may lead to people erroneously thinking that their uninteresting hits were moderated because they did something wrong, and explaining this difference to users is sometimes very very hard. It hasn't helped that there's no real description of the hit moderation system on EBT's website. The support guys and gals regularly receive queries about moderated uninteresting hits and it's been a bit problematic to answer those messages.

Another problem is that a number of people have the "those users must know each other, moderate all their hits!!!1!" attitude towards some hits. It really shouldn't be forbidden to know other trackers in real life. If a shop owner is a tracker and a regular customer of that shop is also a tracker, there will be hits. The purpose of EBT is to track the movement of the banknotes. As long as the note entries are correct, such persons shouldn't feel guilty about getting hits with each other. Sure, such hits might not be particularly interesting if they happen every time the customer visits the shop, but that's not really their problem. It's EBT's problem.

Another aspect is that I'd very much like EBT to get new users. Expanding on the previous shop example, let's consider a bigger supermarket with a few dozen employees or so. One employee sees someone examining the notes and using the computer. She then goes to ask "Hey, what are you doing?" and gets an answer "There's this website where I can enter the serial numbers of Euro banknotes and if someone has entered them before, I get a notification", with possibly a reaction "Oh, that's interesting, I should try that as well".

Now there could be two reactions from the first tracker: "Oh, I'm not sure if it's a good idea. We'd get lots of hits with each other and the people on the forums would declare jihad on us" or "Sure, I'll help you get registered. We'll likely get lots of uninteresting hits with each other, but that's no problem.". Which one would you prefer? This doesn't mean that "here's the bundle I just entered to EBT, it's your turn now" would be a good idea, but people shouldn't be afraid of getting hits with their friends. If the friendly hits still become a problem (despite the hit scoring system), there could be some other countermeasures, such as allowing only the first entry from a community member. The hit scoring system should alleviate most of these problems, so this isn't particularly critical at this point.

Solutions
A solution to this problem is to separate the two -- we'd have one system for dealing with abuse and another for dealing with the uninteresting hits. I'll talk about the abuse first.

There would still be moderation, but it'd be reserved only for dealing with specified instances of abuse. Some examples would be "incorrect location", "invented serial number", "serial number shown in public", "user has not had this note in his/her hands" etc. Specifically, there would NOT be moderation reasons like "these users must know each other", "once again a boring Helmbrechts-Helmbrechts hit", "less than a 24 hour hit", "similar to the other recent hit" etc., as those will be handled with the interestingness score. I'm also planning to add an option for the moderators to specify the moderation reason. This moderation reason would then be shown to the user on the hit report.

The hit moderation is currently essentially handled jointly by the Database and Support working group members, although there are a few moderators that don't belong to either of these workgroups. I'm hoping that once this system goes live, the Database WG (as it's the most closely related working group) would define the criteria for hit moderation and assign the appropriate moderators as they see fit. Defining the criteria for abuse should be much easier, as they don't need to worry about hit interestingness any more.

At the moment, moderated hits will become automatically unmoderated if some third person enters the note again. In contrast to this, the new abuse-focused moderation decisions would be permanent. If the error is corrected (like, one entry from a hit is removed or the location is corrected to the correct place), the moderation can be subsequently undone at the request of a user. There would also be no more automatic moderation in the current sense.


The hit interestingness score would take care of the other half of the problem by classifying the hits based on their calculated interestingness. The score system isn't a particularly new idea. In fact, it was initially mentioned over a decade ago (hint: search for "moderat*" from the forum and sort by ascending date if you want the historical perspective, and if you want to spend a day reading old messages -- there are plenty).

What we need is a numerical score for each hit, from a range of, say, 0 to 100. The purpose of this topic is to define the methods for how the score is calculated.

Once we have the score, we can do all kinds of nifty things with it. My current thinking is that the current hit list would get a new dropdown menu (or some other user interface choice) where we could select to see hits that have at least the chosen interestingness score. This could be arranged in the form of a "star" rating, where ***** hits would be the most interesting, *** would be approximately the same level as now and single-star hits would let people browse ALL the non-moderated (non-abuse (see above)) hits, including any bundle hits and friendly hits and such. The five-star hits would be rare, perhaps we'd get them only a few per week or so. The four-star and two-star classifications would be somewhere between the five, three and one star classifications, respectively. The thresholds for each category can be adjusted appropriately to get the desired number of hits for each category.

There has been some commentary that it's not going to be possible to take ALL the parameters into account and it's not possible to come up with an algorithm that'd be perfect. That's alright, we don't need a perfect algorithm. We just need something that is suitable for categorizing the hits into the 1-5 star categories. In particular, there won't be a list of hits scored by interestingness, with the most interesting hit at the top. There are various criteria for determining how interesting a hit is, and different people have different opinions. The goal is to have a system where all the five-star hits are considered to be interesting for everyone, the three-star hits for most users, and the one-star hits for those who want to see each and every hit there is on EBT.

Most importantly, the hit interestingness score would guarantee that every user's hits would be treated exactly the same. It's possible (or even likely) that we'll tweak the system after it has been put into use based on received feedback, but all of those changes would be applied to all the other users as well.

We could also have a RSS feed or a Twitter feed or an IRC bot announcing whenever there's a new hit that reaches at least the specified interestingness score. But first we need to come up with the score.

One point in my plan is to unmoderate all the currently moderated uninteresting hits. The hit scoring system will take care of the uninteresting hits and assign a somewhat lower score for them, so that's not the problem. The problem is that there are a number of hits that have been moderated because of abuse, and there's currently no way to sort out the moderation reasons automatically. For this there will be an "opt-in" procedure, in which the user makes a formal declaration that he/she doesn't have any hits that should be moderated, after which the hits are unmoderated and a score is calculated for all the user's hits.

There are also some hits which I'd classify as "accidental". Most "meeting hits" fall into this category. Even though some of these hits are quite interesting on paper, the users involved know that it isn't particularly interesting that the other person brought the note to the restaurant where a number of trackers were having dinner. For this we may need some level of user control of their own hits. In principle, something like "if both users agree, the hit won't be shown on the hit listing" might do. There may also be other reasons for increasing or decreasing the score slightly based on the user's own knowledge of the hit.

Another change would be that we'd get rid of the "interesting hit" term altogether, there would only be "hits". For some users this change will mean a massive amount of new hits and their hit ratio will drop significantly. Some people will no doubt complain that this is unfair, but I can live with that.

The way forward
There are multiple steps that will need to be made before this new hit scoring system gets implemented on EBT. We'll start with the basics and proceed systematically to reach the final specification.

First phase: Determining all the qualities of an interesting (or uninteresting) hit [completed]
Second phase: Assigning a score for each of the metrics discovered in the previous phase (each with a typical range of 0-100) <=== WE ARE HERE
Third phase: A formula that integrates all the individual metrics into one
Fourth phase: If any of the metrics include any weightings or tunable parameters, tune them in this phase
Fifth phase: User interface and implementation detail decisions
Sixth phase: Implementation, testing, deployment (by the development team)
…: Adjust the system based on received feedback as needed

More on these in the next message.

Summary of 1st phase findings

Hit cardinality

• Triple hits are more interesting than regular hits, because they're less common
• Quadruple hits are even more interesting, because they're even less common than the triples
• .. and so on

Distance, travel speed, travel direction

• Notes that have travelled longer are generally more interesting than those that have travelled only a few tens of kilometres
• But sometimes it's interesting that the note hasn't travelled at all. Perhaps we should also take such cases into account, perhaps in the form of "hits with an unusual combination of days * km are interesting".
• Notes that have made an intercontinental trip in just a few days are interesting
• Hits with an equal number of km and days (apart from 0 km/days hits)
• Round number of kilometres (1000, 2000, 3000; 100, 200, 300 to a lesser extent)
• Hits that have travelled straight north/east/south/west are interesting

Days

• Notes that have been in circulation for years are more interesting
• Round number of days (1000, 2000, 3000 days; exactly one/two/three/.. years)

Frequency, uniqueness

• How many hits the users have in common? If it's their first hit, it's more interesting than if it was their 500th
• How many hits there are between the cities?
• How many hits there are between the regions (any region type)?
• How many hits there are between the countries?
• How many hits there are in this particular city/region/country (ie. considering all stops along the hit note's path individually)
• How many notes the user has entered in the respective city/region/country?
• Distribution of denominations (at user and country level)
• For the above, we can split them into time slots: how many of them occurred
  1. in the last 24 hours
  2. in the last 7 days
  3. in the last 30 days
  4. in the last year
  5. since 1.1.2002

International issues

• International hits are more interesting than national hits
• For some countries, a national hit would be statistically more interesting than an international hit
• Hits to/from non-eurozone countries are interesting

Note characteristics

• The first hit with an H note was interesting at the time when the H notes were put into circulation (and the same for the other new serial number letters)
• Europa notes? Non-Europa notes? Perhaps in the distant future non-Europa hits might be interesting. Note that this system won't go live before the Europa notes are put into circulation. Perhaps we could skip this criteria at least for now, we can add this later on in a few years.
• Unusual denominations -- hits with a 200e note are more interesting than a hit with a fiver
• Hits with unusual printer code / serial number (/denomination) combinations

Comments

• Comments can't unfortunately be used for calculating the score automatically
• The users involved in the hit should be able to shift the score up or down in some specific circumstances, such as "note found from ground" (up) or "I personally brought the note to Finland from the Antarctic, and someone found the note from Finland later on" (down)
• Having a comment (any comment) is better than having no comment at all

reserved for a summary of findings of the 2nd phase

reserved for a summary of findings of the 3rd phase

reserved for a summary of findings of the 4th phase

reserved for a summary of findings of the 5th phase

reserved for a summary of findings of the 6th phase

First phase: Determining all the qualities of an interesting (or uninteresting) hit

The objective of this phase is to list ALL the qualities of a hit that make it interesting. We'll assign the scores in the latter phases, so do not mention ANY kind of numbers or calculation methods yet. If you must, "more interesting" and "less interesting" are allowed to be used.

So, what makes a hit interesting? I'd say that it's unusual. That's a blazingly obvious statement, but it actually helps with the next definitions. There can be some overlap with the qualities, for example it's likely that a hit that has travelled thousands of kilometres is also an international hit, but that's okay. We can still include both in the calculations.

Hit cardinality
Hits are interesting
Triple hits are more interesting, because they're less common
Quadruple hits are even more interesting, because they're even less common than the triples
.. and so on

Distance
Notes that have travelled longer are generally more interesting than those that have travelled only a few tens of kilometres.
But sometimes it's interesting that the note hasn't travelled at all -- this note took six years to cross a railway bridge.

Days
Notes that have been in circulation for years are more interesting

Frequency
How many hits the users have in common? If it's their first hit, it's more interesting than if it was their 500th.
How many hits there are between the cities?
How many hits there are between the regions (any region type)?
How many hits there are between the countries?
For the above, we can split them into time slots: how many of them occurred today, this week, this month, this year, altogether.
The "bundle hits" would also be taken care by these rules.
We should also consider taking into account the total number of hits of each user. Hits from new users are more interesting.

International issues
International hits are more interesting than national hits
For some countries, a national hit would be statistically more interesting than an international hit
Hits to/from non-eurozone countries are interesting

Note characteristics
The first hit with an H note was interesting at the time when the E notes were put into circulation (and the same for the other new serial number letters)
Europa notes?
Unusual denominations -- hits with a 200e note are more interesting than a hit with a fiver

Comments
Comments can't unfortunately be used for calculating the score automatically

Is there something else that you look for in an interesting hit?

[note: please read and understand the first paragraph of the first message before commenting]

avij wrote:Distance
Notes that have travelled longer are generally more interesting than those that have travelled only a few tens of kilometres.
But sometimes it's interesting that the note hasn't travelled at all -- this note took six years to cross a railway bridge.

Days
Notes that have been in circulation for years are more interesting

A metric that would help with these is the speed of the hit. So your railway hit would be interesting because it traveled at a very slow speed. Hits with a distance of 0 km present a problem with this, though. On the other end of the scale, I'd consider a hit that spanned 5000 km in one week more interesting that another one with 5000 km and one year.

avij wrote:Frequency
For the above, we can split them into time slots: how many of them occurred today, this week, this month, this year, altogether.

We shouldn't use natural weeks/months/years, or the results would be periodically skewed (specially on monday, january 1st). I think it's better to use a sliding window that comprises the last 7/30/365 days.

Also, you say "for the above" but I think it applies for the following criteria too.

Note characteristics
The first hit with an H note was interesting at the time when the E notes were put into circulation (and the same for the other new serial number letters)
Europa notes?
Unusual denominations -- hits with a 200e note are more interesting than a hit with a fiver

Maybe we could consider the first hits with Europa notes as interesting ones, but those hits will be eventually our daily bread and butter so they would only be interesting for historic reasons—we could make their score decrease quickly as they become more usual. OTOH, hits with any :note-200:s will be interesting for a long time, until the monetary inflation causes all of us to carry several of those in our pockets every day, so their score in this regard would decrease very slowly, or not at all.

Or as I said above, it could make sense to take into account their frequency, ie hits with Europa notes will be interesting as long as they don't become commonplace, and the same with . It's just that by december 2013 we'll see several Europa hits every day while :note-200:s ones will still be as rare as today.

This could get complex quick, though. I don't know how usual is to get a hit with a D/L in Finland today, but you could pray for hours to hit with one of those in southern Europe.

avij wrote: We should also consider taking into account the total number of hits of each user. Hits from new users are more interesting.

For me that's a crucial point. I don't care much for hits with users who entered an enormous amount of notes. Hits with 'small' users are much more interesting to me.

avij wrote:Distance
Notes that have travelled longer are generally more interesting than those that have travelled only a few tens of kilometres.
But sometimes it's interesting that the note hasn't travelled at all -- this note took six years to cross a railway bridge.

Is it possible or feasible for this criterion (and perhaps also for the time/days) to compare it with an "average" hit. As an example, say that an average hit travels 50 km in 100 days, then hits that deviate from this become more interesting?

R/J wrote:Is it possible or feasible for this criterion (and perhaps also for the time/days) to compare it with an "average" hit. As an example, say that an average hit travels 50 km in 100 days, then hits that deviate from this become more interesting?

We can figure out the methods how to calculate the numbers in the next phase, but perhaps at this stage it might suffice to say that "hits with an unusual combination of days * km are interesting".

I see this very promising. The list of criteria which may influence the interestingness of a hit will be of much help when creating the formula and in the upcomming steps of the process.

I can hardly figure out more criteria to determine the interestingness of a hit, other than the already mentionned. However, there's still something I think we will never reach to implement: curiosities. (Magic numbers!)

For instance (but not limited to) hits with a special number number of days and/or Kilometres; eg: a hit with exactly 2013 km happening in 2013 or a hit with the same number of days than Km or a hit with exactly 10000Km or even a hit with the same number of days than the postal code of one of the involved locations or... I don't know! Or shall these be "easter eggs" otherwise?

If we ignore that along with the interestingness which may be related to a banknote's serial (very low serial numbers, palindromic notes, etc.) I am unable to find any other characteristics modifiying the socalled interestingness.

Cheer up! Nice work!!

Mkay wrote:

avij wrote: We should also consider taking into account the total number of hits of each user. Hits from new users are more interesting.

For me that's a crucial point. I don't care much for hits with users who entered an enormous amount of notes. Hits with 'small' users are much more interesting to me.

I think that hits should not be 'punished' for the sole fact that a power user is involved, but that 'frequent hit partners' must be punished. Each next hit between 2 users gets a heavier punishment than the previous one. Of course, power users have a bigger change of getting big punishments.

If a new route is involved the new route may or may not give some bonus points.

avij wrote:How many hits the users have in common? If it's their first hit, it's more interesting than if it was their 500th.
How many hits there are between the cities?
How many hits there are between the regions (any region type)?
How many hits there are between the countries?
For the above, we can split them into time slots: how many of them occurred today, this week, this month, this year, altogether.

Oh, I did forget one possible metric. How many hits there are in each city, region and country, ie. ignoring the other endpoint(s) of the hit completely.

The final score will be some sort of a combination of all these metrics. This may mean that the system may need to consider 20+ different metrics for calculating the final score, but that's alright. The goal is that no single metric can dominate the hit interestingness score. I've also envisioned that each metric could be given the appropriate weight, thus enabling us to include even slightly unusual metrics like "hit's kilometres matches the days" with an appropriately low weight. We'll discuss the appropriate weights in the fourth phase (ie. not yet).