• 10 dec 2017: forum version update. In case of issues use this topic.
  • 30 nov 2017: pilight moved servers. In case of issues use this topic.
Hello There, Guest! Login Register


Filter settings in config.json
#31
I would love to read the more extensive description Smile
 
Reply
#32
IMHO we do need to agree on some requirements.
I will do so in this post and comments are appreciated.
In order to limit the requirements discussion I define some assumptions first

ASSUMPTIONS:
A) (Assumption A): Main problem is the sensitivity of RF receivers during pauses.
A1) (PROOF A1): You can easily monitor this using "pilight-raw -L" on unfiltered signals. Use a remote control and fire RF pulses in a rapid sequence and you will virtually see no noise.
A2) (PROOF A2): We basically have no issues during reception of pulsetrains.

B) Purpose of the LPF is not to implement more sophisticated signal processing capabilities like Spectral Analysis and FFT (Fast Fourier Transform) algorithms.

C) I assume that the static noise (crackle) during phases of inactivity triggers a fast sequence of rising/falling edges on the RF receiver output.


REQUIREMENTS:
a) Requirement a: Purpose of the LPF is to reduce the number of data forwarded to the main CPU - without loosing any data or accuracy on pulse durations during phases of activity - during phases of RF inactivity (e.q. no RF signal present),
a1) Consequence 1 from requirement a: You can not apply Low-, High pass filters on OOK signals.
a2) Consequence 2 from requirement a: We do not attempt to improve data reception during reception of pulsetrains.

b) Create a known state during reception of an RF signal either HIGH or LOW,
b1) Will depend on RF receiver hardware, we have to make this configurable.
b2) Is required for requirement d), e), f) and g).

c) The LPF delays the forwarding of a signal for a threshold time.

d) A Falling Edge (Rising Edge) defines the end (start) of an RF signal.

e) A Rising Edge (Falling Edge) defines the start (end) of an RF signal.

f) If time between start and stop of a RF signal is below the threshold time, the state change is suppressed (e.q. LPF output remains silent and state change is lost).
f1) We have removed a spike pulse.

g) If time between stop and start of a RF signal is below the threshold time, and requirement f is not applicable (e.q. we are in the middle of the reception of a RF signal) the state change is forwarded (e.q. the state change is not lost).

h) We do need to be extremly fast on detection of input state changes, as the static crackle is the root of the problem.
 
Reply
#33
All true, but i want to hear what yablacky implemented before we can think of better approaches.

What i would like for us to built first is a unit test for our filters. What means we have to built a small program that can generate noise and valid pulses of various protocols. This allows us to test the whole thing in similar setups.

One way to implement this is to just capture a few seconds of pilight-raw output including a valid pulse train without filter for all various protocols. Then put all those pulses in a big file or int array and let a C program generate an OOK signal based on it.
 
Reply
#34
I have added some information about firmware version 6 to the README.md file.
And also some implementation details. Please ask for more info if required.

An interesting point is (g) because the question is:
Can a spike happen that consist of a phase where the receiver does not receive a RF signal?

Btw, the version 6 LPF does what is written in (f) except that it makes no difference between H and L spikes. That's why (g) is very interesting to me.
 
Reply
#35
I see that you have described what the different functions do, but not what the different algorithms are that you implemented. How do you detect spikes and what do you do with it, how is it additional different / to what was already done.

Can you also give a bit more insight in these:
Quote:This filter detects high frequency spikes and removes them completely. This means the leading and trailing edge of the spike are removed.

And this
Quote:All the so far existing filter methods listed above ignore only the trailing edge of a spike and treat the leading edge as true signal, which it is not. Astonishingly that this works et all and it does not so bad.
 
Reply
#36
(01-11-2017, 08:57 PM)yablacky Wrote: ... the version 6 LPF does what is written in (f) except that it makes no difference between H and L spikes. That's why (g) is very interesting to me.
g) If time between stop and start of a RF signal is below the threshold time, and requirement f is not applicable (e.q. we are in the middle of the reception of a RF signal) the state change is forwarded (e.q. the state change is not lost).

We do need to differentiate between RF ON and RF OFF states.
Simply use two Pi's in standalone mode (or use an Atmel to control a RF transmitter) and run the following test:
There is a fundamental difference between:
a) pilight-send -p raw -c "10000000 1000" and
b) pilight-send -p raw -c "1000 10000000"

With a) there is silence on the receiver side, with b) it is not.
--> If we receive valid RF data, there is little disturbance and we simply forward whatever data we get with the delta time of the threshold delay time (f),
--> If we do not receive valid RF data, there is a lot of disturbance and we simply forward data only if the length of an RF signal (ON) is above the threshold value.

Thus we do need to know if we receive an RF signal or not.
Once we do know this, we simply delay the reception of the RF signal by the threshold value.
If we loose the RF signal before the threshold value time is exceeded, we drop the pulse alltogether, as we consider this to be noise.
 
Reply
#37
I guess we are talking about this two situations:
Code:
a) RF on         _
                | |
                | |
   RF off ______| |_______
               < 150 μs

b) RF on  ______   _______
                | |
                | |
   RF off       |_|
               < 150 μs
If detecting a) we have no doubt that it is noise. But why can we so sure?
Because there is no protocol that sends ON pulses of such short durations.

If detecting b) what could that mean?
We know that there is no protocol that sends OFF pulses of such short duration.
Can it happen et all?
Does it make sense to treat it as valid signal?

My humble answers:
I think it can happen if two senders are sending pulses at the same time and its a random situation that a RF off phase of both senders overlap for 150 μs or less. Ignoring that spike or not makes no difference because the surrounding pulses are garbage anyway.
If we detect it under the assumption that only one sender is sending, than it can not be a valid part of the pulse train either, because all known protocols send longer RF off pulses. I tend to say it makes more sense to ignore that spike than as treating it as valid pulse. Because couldn't it happen that the RF signal is so low that the receiver detects an off for that short time while the sender was in fact still sending RF on?
To me it looks that in each scenario it makes no sense to treat b) as a valid signal that has to be passed rather than being ignored like a).
 
Reply
#38
(01-11-2017, 10:38 PM)curlymo Wrote: Can you also give a bit more insight in these:
Quote:This filter detects high frequency spikes and removes them completely. This means the leading and trailing edge of the spike are removed.
This is done by not processing a detected input pin change immediately (like other filters do) but to wait a little time (150 us) before processing it. If a pin change is detected while already waiting for processing of a previous pin change, then the previous and the current pin change are ignored. Ignoring is done by doing as if both pin changes were processed (albeit they are not really).
There are some comments in the code that describe where this waiting is set up, detected and cleared.

(01-11-2017, 10:38 PM)curlymo Wrote: And this
Quote:All the so far existing filter methods listed above ignore only the trailing edge of a spike and treat the leading edge as true signal, which it is not. Astonishingly that this works et all and it does not so bad.
Even without version 6 LPF a short noisy pulse is detected by a filter. But to that filter, the leading edge of a spike (and they don't know at this point, that it will be a spike) is the end of the current pulse. The pulse is then processed immediately, namely before the next pin change is detected. If that next pin change comes within 150us the filter knows that it got the 2nd edge of a spike and that the already processed pulse did in fact not yet end. But this knowledge comes too late, processing the pulse was started too early. All the filter can do is to ignore the 2nd edge of that spike.
 
Reply
#39
One thing i was thinking of for an algorithm is the following. In OOK we know that meaning in a signal is put in varying pulses. Long and short pulses.

Currently we filter pulses based on their length. Pulses too short and too long are considered junk. But in that case we totally ignore the patterns pulses follow in abstract sense.

So, we can say that after one long pulse there should be between one and three short pulses. And after a short pulse we expect to see at least a long pulse after max. three short pulses.

This means we can make the filter 'smarter' by saying (just some arbitrary thresholds):
1. If pulse > 500 then open filter for 4 pulses
2. If pulse < 500 then wait until we received high pulse within 4 pulses.

This requires us to buffer at least 4 pulses and if this little check fails, we just don't pass anything through. If the pulses are valid we have just 4 pulses latency.

It also means that a long pulse (such as a footer) disables the filter for a short time which can an improvement for some protocols, the actual filter takes place on just the short pulse patterns.
 
Reply
#40
I agree with your analysis that a) is triggered by white noise and b) is mainly triggered by other protocols and not white noise.

Once we do know that a LOW logic level means: NO RF signal present, we can program the hardware such that a rising edge is detected. Discard or not discard (a rising edge, e.q. a spike pulse) that is the question.

After detection of a rising edge and due to b) we do not need to detect each falling edge immediately, it is sufficient to detect it with an accuracy of 10 to 20µS after the state change. Due to b) there are also certain conditions that will allow us to ignore a rising edge as well.

Key issue is that we do not need to look at each individual pulse or its duration, we do need to look at the elapsing time after a rising edge condition occured
We monitor the signal state in a timed loop and record the state in a ringbuffer.
We delay the forwarding of pulses by the max duration of a pulse to be supressed (ignored, removed, ...).
If we want to suppress spikes of 150µS and use a timed loop of 20µS duration, we do need less than ten bit of RAM storage plus two pointers (Read, Write) to implement the logic required to remove spikes of 8*20µS from the input signal, or to ignore them.

(01-12-2017, 07:10 PM)curlymo Wrote: One thing i was thinking of for an algorithm is the following. In OOK we know that meaning in a signal is put in varying pulses. Long and short pulses.

Currently we filter pulses based on their length. Pulses too short and too long are considered junk. But in that case we totally ignore the patterns pulses follow in abstract sense.
Only pulses too short should be considered. Long pulses can simply be silence, e.q. no RF signal present, or a long active RF signal is present
(01-12-2017, 07:10 PM)curlymo Wrote: So, we can say that after one long pulse there should be between one and three short pulses. And after a short pulse we expect to see at least a long pulse after max. three short pulses.
No, Manchester code has a series of long pulses (Binary presentation is 101010...) or a series of two short pulses (Binary presentation is either 000000... or 11111111...), or a combination of LONG and SHORT pulses, with two SHORT pulses having the length of one LONG pulse.
LONG LONG LONG LONG LONG means 01010 or 10101,
LONG SHORT SHORT LONG SHORT SHORT SHORT SHORT means 00111 or 11000,

(01-12-2017, 07:10 PM)curlymo Wrote: This means we can make the filter 'smarter' by saying (just some arbitrary thresholds):
1. If pulse > 500 then open filter for 4 pulses
2. If pulse < 500 then wait until we received high pulse within 4 pulses.

This requires us to buffer at least 4 pulses and if this little check fails, we just don't pass anything through. If the pulses are valid we have just 4 pulses latency.
As said before, bi phase manchester code is coding a bit condition with a single long pulse, e.q. change from logical 0 to 1 and vice versa; and with two short pulses, that a bit has not changed.
(01-12-2017, 07:10 PM)curlymo Wrote: It also means that a long pulse (such as a footer) disables the filter for a short time which can an improvement for some protocols, the actual filter takes place on just the short pulse patterns.
No this behaviour should not be implemented as it removes the first couple of valid pulses, one of the protocols currently affected is quigg_gt7000, as that protocol as an 80000µS footer.
In particular will this assumption cause problems with bi phase codes as the SYNC bit required to find the start of a bit stream is at the beginning of a pulsetrain immediately before the first pulse presenting a valid data bit.
 
Reply
  


Possibly Related Threads...
Thread Author Replies Views Last Post
  Bandpass filter problem mayonezo 15 7,101 10-26-2020, 09:05 PM
Last Post: Michelebup
Rainbow Lowpass filter - how DOES it work? DE8MSH 16 8,528 04-17-2020, 10:13 PM
Last Post: curlymo
Question Passive Component Band-pass filter ripper121 8 4,050 12-16-2017, 01:38 AM
Last Post: wo_rasp
  Filter kit in 315 receiver? ceandre 1 2,053 04-14-2017, 10:10 PM
Last Post: wo_rasp
  Low Pass Filter and Receiver in Series? re-post Gustavo Woltmann 1 2,255 03-08-2017, 05:25 PM
Last Post: pilino1234
  Low Pass Filter and Receiver in Series? Hauke 3 4,057 08-30-2016, 08:25 AM
Last Post: wo_rasp
  New filter version curlymo 0 2,228 04-02-2016, 08:05 AM
Last Post: curlymo
  Band-pass filter parts curlymo 60 79,452 01-08-2016, 12:23 AM
Last Post: gneandr
  low pass filter francois.leprieur 3 3,379 12-20-2015, 10:18 AM
Last Post: curlymo
  Low-pass Filter wiring hazzard 4 4,111 11-15-2015, 08:35 AM
Last Post: woutput

Forum Jump:


Browsing: 1 Guest(s)