Lets say i have this series of values, representing speeds: 50,60,60,120,61,65
The 120 is probably a read error, so i need to detect it as wrong and discard it. I know that because is 60, then 120, then 61.
But what if i have 50,60,60,120
And i discard 120 as out of range, but the next value is 121, and later 125. Then the 120 is probably ok. So i discard 120, but then i realize it is ok and i accept it again.
>>1
A catch-and-release type filter using monads in Haskell.
Name:
Anonymous2013-06-25 13:53
Just make a rule so that you can't discard a value from the last n ones, where n is greater than one and lower than the number of last values your interpolation method uses to produce a bound on new values.
Name:
Anonymous2013-06-25 14:21
>>3
I can't do that. I need to discard bad values ASAP. This data passes through a complex-event-processing machine, that needs to triggers alarms as soon as a special situation happens. And in my business a false alarm is much worse that a (bit) late alarm.
Name:
Anonymous2013-06-25 14:26
>>4
I really don't understand your problem. What's wrong with queueing the last n values and doing filtering only on the middle ones, sending only the values which overflow the queue to the event-processing machine?
Sure, it adds latency, but you said yourself it's the best way.
I'm aware that is has to do with statistics. My idea (that i had before posting this thread) was:
* Wait until 10 speed events have been received.
* Calculate the standard deviation for the population, and the median speed for that population.
* On a new event arrival, check if the std deviation of the event was bigger that the std deviation for the previous population.
If so, put the event on a "discard" stack.
* If the next event has the same treatement than the previous (that is, go to the discard stack), realize that the previously discarded event was ok, and take it from the discard stack and put in back in my population.
The events are then sent to the complex event process engine. It can process "late" events too.
I know this approach can be naive for now, but i think is a decent start. But i might be wrong, so i posted here to take advantage from /prog/'s wisdom.
Name:
Anonymous2013-06-26 13:58
What is the possible cause of these incorrect "speed"s? What the fuck are you really doing nigger
Name:
Anonymous2013-06-26 14:30
>>17
It's data coming from GPS devices. When there's issues with satellite connections, sometime we got reports of speeds of ( for example) 190km/h when the previous reading was like 60km/h, but that happens just once. That is, we don't have two incorrect readings in a row.
Name:
Anonymous2013-06-26 14:46
Oh oh right now I remember, you're the guy who wanted to use jewglemaps or whatever the fuck for your little thing. I hope you still aren't going to rely on jewgle. So now that I know what you're actually talking about, how do you know that they really aren't going that fast or rather having such a burst in speed that they're going about as fast and the misread in speed is from the way you're determining it? And what about little shortcuts and things that cut the distance between the waypoints/checkpoints/whatever you're basing it on? Things like google maps calculate the distance as if you're doing very rigid sharp turns and not curving over curners going ~50, you know?
Name:
Anonymous2013-06-26 15:03
>>19
We do not calculate speeds. The message that we receive with the data has the speed. It is the device that miscalculate the speeds. And we are not using google maps, just capturing raw data from the devices.