• 10 dec 2017: forum version update. In case of issues use this topic.
  • 30 nov 2017: pilight moved servers. In case of issues use this topic.
Hello There, Guest! Login Register


Soens Innovation (TFA similar protocol)
#1
I bought a new indoor/outdoor weather station with 3-sensors (http://www.amazon.de/gp/product/B00G1GNY...ge_o01_s00). It looks exactly the same (but black) like the weather station shown on https://wiki.pilight.org/doku.php/tfa. That's why I thought it uses the TFA protocol, but it doesn't.

I spent the whole day to identify the logic behind the raw code and I got it. I just need some help on how to implement the protocol for pilight, or even better if someone can implement it for me Smile

It should be very easy to implement it, because it's nearly identical to the TFA protocol, there are just some differences on the positions and it uses °C instead of °F.

I don't understand what all the values (except RAW) are for, but here is what I found:
- The rawlen is always 76
- The pulselen is mostly 251 and sometimes 252 or 253 (didn't noticed other values)
- The pulse is mostly 7 or 19 and sometimes 8, 18, or 20

Now I split the raw code into two columns (same like for TFA). The values for the first column are always round about 500 and are useless?!
I transfered the second column to a binary code with the following rule:
- values between 1750 and 2016 (min/max I've ever seen in this range) equals 0
- values between 3750 and 3780 (min/max I've ever seen in this range) equals 1
- The last raw value is always round about 8500 (and marks the end?!)

After that I have a binary code of length 37. I recorded several raw codes for different temperatures and humidities. And here is what I found:

The first 5 digits are always 10011, no matter what I do, I think they belong to the ID. The digits from 6 to 14 change after I reinsert the batteries, so these digits must belong to the ID (like in TFA).
The digits 15-16 are for the channel (00=CH1, 01=CH2, 10=CH3).
The digits 17-28 are for the temperature, but it doesn't follow the same rule as the TFA protocol. If all digits are 0, the temperature is 0°C, otherwise, if all digits are 1, the temperature is -0.1°C. If the decimal value increases by 1 the temperature increases by 0.1°C. So for temperature >=0°C you can calcularte the decimal value of these digits and divide it by 10, this results always in the correct temperature in °C. But if the temperature is <0°C you have to invert the binary values, calculate the decimal value, add +1 and divide it by 10. This always results in the correct temperatures if they are negative. Because the first two digits (17-18) of the binary code are only 1 if the temperatures are below 0°C, they can be used to identify this case. The remaining digits from 19 to 28 can then be used to calculate the temperature.
The digits from 29-36 are the humidity in binary code, so it's only required to calculate the decimal value, then you get the humidity in %.
The last digit 37 is probably for the battery. Using completely new batteries, the value is always 0 for all recordings.

Summary:
- 37 binary digits in the form of 10011AAAAAAAAABBCCCCCCCCCCCCDDDDDDDDF
- ID: 1-14
- Channel: bin2dec(15-16)+1
- Temp sign: (17-18) 00=positiv, 11=negative
- If Temp positiv: bin2dec(19-28)/10 [in °C]
- If Temp negativ: -1*(bin2dec(inv(19-28)) + 1)/10 [in °C] or (bin2dec(19-28) - 1024) / 10 [in °C]
- Humidity: bin2dec(29-36) %
- Battery: 37 (0=ok, 1=bad) (not sure if this is really the true behaviour)

These rules result always in the correct values (>40 trials).

It would be really nice if someone is able to implement this or can help me to do so.

Thanks in advance and sorry if my english isn't perfect Smile
 
Reply
#2
Based on your description, i have uploaded an updated tfa protocol driver.

Follow the steps in the pilight manual to compile that version.

You do need to clone the following github repository:
git clone https://github.com/wo-rasp/pilight.git --branch tfa_s

Everything else is as described in the pilight manual.

You need to configure your device as tfa device in config.json.

As i have no physical hardware i have not been able to fully test it.
Please report here if it is working (of not, use pilight-daemon -D) to check the debug messages.
 
Reply
#3
Thanks for your fast reply and your work! I really appreciate it!

I compiled your version, but something goes wrong while calling the tfa parseRaw() function. Here is the debug output:

Code:
[Mar 20 23:22:13:696371] pilight-daemon: DEBUG: possible tfa protocol
[Mar 20 23:22:13:697031] pilight-daemon: DEBUG: recevied pulse length of 258
[Mar 20 23:22:13:697515] pilight-daemon: DEBUG: caught minimum # of repeats 1 of tfa
[Mar 20 23:22:13:697981] pilight-daemon: DEBUG: called tfa parseRaw()
[Mar 20 23:22:14:766731] pilight-daemon: cpu usage way too high 99.999563%
[Mar 20 23:22:14:767337] pilight-daemon: WARNING: checking again in 10 seconds
[Mar 20 23:22:25:768743] pilight-daemon: INFO: ----- Thread Profiling -----
[Mar 20 23:22:25:769308] pilight-daemon: INFO: - thread socket: 0.000000%
[Mar 20 23:22:25:769745] pilight-daemon: INFO: - thread ssdp: 0.002751%
[Mar 20 23:22:25:770162] pilight-daemon: INFO: - thread sender: 0.000000%
[Mar 20 23:22:25:770542] pilight-daemon: INFO: - thread broadcaster: 0.000000%
[Mar 20 23:22:25:770785] pilight-daemon: INFO: - thread 433gpio: 0.000000%
[Mar 20 23:22:25:770831] pilight-daemon: INFO: - thread receive parser: 96.581360%
[Mar 20 23:22:25:770911] pilight-daemon: INFO: - thread events client: 0.005316%
[Mar 20 23:22:25:770961] pilight-daemon: INFO: - thread events loop: 0.000000%
[Mar 20 23:22:25:770991] pilight-daemon: INFO: ----- Thread Profiling -----
[Mar 20 23:22:25:771031] pilight-daemon: cpu usage still way too high 96.597159%, exiting

seems to stuck in a loop. I tested it serveral times, but always crashs with the same debug output.

Is it helpful if I post some raw-codes?



Edit:
Too much noise around here. It's not the weather station which crashes pilight, it must be another transmitter. Because I get a nearly correct output if I press the TX button:
Code:
[Mar 20 23:47:32:313574] pilight-daemon: DEBUG: socket write succeeded: {"message":{"id":21,"temperature":2.39,"humidity":47.00,"battery":1,"channel":3},"origin":"receiver","protocol":"tfa","uuid":"0000-b8-27-eb-a3b30b","repeats":3}

It shouldn't be 2.39°C, but 23.9°C. The rest is absolutely correct. Nevertheless, there must be a bug in the parseRaw() method.



Edit2:
Did some more trials. Most of the recognized "possible tfa protocols" are parsed correctly. But after a few seconds (up to a minute), pilight receives a "possible tfa protocol" at which the CPU goes up and stays there. Is there a way to output the raw message? If I use pilight-debug I see the raw codes, but it seems they aren't parsed in this mode, at least pilight-debug keeps running. And it isn't possible to run pilight-daemon and pilight-debug in parallel.
 
Reply
#4
Okay, the temperature issue is fixed.
The following additional checks for protocol validation are now implemented:
- only exact rawlen of 76, 86 and 88 will qualify as a valid tfa protocol

For validated protocols:
- rawlen 86 and 88: The CRC must be valid
- rawlen 76: the first 5 bit received must be 10011 or decimal 19

Can you please do a test for negative temperature values by putting one of the sensors in a freezer ?

For the high CPU usage, i do not think that it is a parseraw() issue (i am using the same tfa module code here for my tfa DOSTMAN 32.3200 devices).
There are reports that it may be a WEBGUI related issue and that CPU usage gets back to normal once multiple times a button is pressed on the GUI - Can you confirm this ?

And just to be on the save side: can you get the complete raw pulsetrain using pilight-raw -L ?
"pilight-raw" will report all RF pulses, i want to check if there are more pulses present on the RF layer than currently known.
"pilight-raw -L" does the same but will insert a linefeed and summarize the number of pulses received and the duration whenever it encounters a pulse with a duration > 5100. This will help you to identify your pulsetrains. In particular i want to know if there are additonal pulses between the 8500 footer pulse and the beginning of your 10011 header pulses.

Edit 2:
Your description that sometimes the CPU usage is skyrocketing is interessting, in particular after potentially having received a tfa protocol (it actually may not be a tfa protocol, but something else). I am definitely interessted in getting that issue analysed in more detail.

Yes, i am able to create a special pilight version, outputting the raw codes received for further handling (those are not all raw codes received) - use branch tfs_s_debuf, compile and start pilight-daemon -D. The raw codes are enclosed and clearly marked.

In the thread profile you have cut it off too early, if you still have it please report which thread is eating up CPU time (i assume its the GUI).
 
Reply
#5
Ok...Because you use the first digits for validation, I did a ton of battery reinserts just to be 100% safe and I found something new:
- Pos 1-4 remains the same for all trials (=1001) and can be used for validation
- Pos 5 belongs definitely to the ID, but changes not that often
- Pos 6-12 still belongs to the ID (I checked that each position change at least once)
- Pos 13 is 0 for all trials (maybe this one is for the battery?! I have to check that)
- Pos 14 took a while until I got it. It's 0 if the transmitter is sending automatically (once a minute) and it's 1 if I press the TX button (manual sending).
- Pos 15-end is the same as described in OP

The ID makes more sense now, because it has a length of 8 bit. Also the constant value (1001) has a length of 4 bit. Sounds more plausible than 10 and 5, respectively.

I checked out your latest version and did the required changes to the tfa.c file myself (didn't implemented the TX-button recognition). Now it works like a charm! Even the CPU stays at its normal level.

The heavy CPU usage couldn't be related to the webgui, because I'm not using the webgui (it's disabled). I'm using FHEM with the pilight module. And I didn't cut the thread pofile too early, just scroll down a bit Smile
Code:
[Mar 20 23:22:25:770831] pilight-daemon: INFO: - thread receive parser: 96.581360%

Therfore it's definitely the parser, but since you implemented the checks, the proplem is gone.

But if you are intrested which raw code causes the high CPU usage, I can make some records for you. I will post them later today.

Again, thanks for all your work!
 
Reply
#6
First, I did some trials for negative temperatures, works perfect!

I commented out the checks (exact rawlength check and SOENS header check) and put out the batteries of my weather station. Here are 10 logs where the CPU usage explodes (including the raw codes):
Code:
[Mar 21 18:39:43:985356] pilight-daemon: DEBUG: **** Received RAW CODE ****
55 46 322 65 351 147 337 47 332 51 300 106 531 123 252 66 279 106 456 96 534 69 256 116 212 70 151 127 47 45 334 49 44 499 145 300 41 255 101 236 52 835 120 244 80 681 93 480 86 436 2889 79 48 229 668 133 91 242 162 168 120 199 101 52 228 47 259 193 46 101 365 81 212 425 46 74 198 233 47 77 194 93 345 63 405 125 261 48 405 102 1875 67 1494 41 3066 97 3555 1036 243 117 310 144 229 132 353 87 694 96 801 96 82 43 502 273 346 80 1085 48 256 47 323 74 958 41 8103
[Mar 21 18:39:43:985592] pilight-daemon: DEBUG: ***************************
[Mar 21 18:39:43:985638] pilight-daemon: DEBUG: possible tfa protocol
[Mar 21 18:39:43:985668] pilight-daemon: DEBUG: recevied pulse length of 238
[Mar 21 18:39:43:985693] pilight-daemon: DEBUG: caught minimum # of repeats 7 of tfa
[Mar 21 18:39:43:985720] pilight-daemon: DEBUG: called tfa parseRaw()
[Mar 21 18:39:44:887610] pilight-daemon: WARNING: cpu usage too high 86.258062%
[Mar 21 18:39:44:887807] pilight-daemon: WARNING: checking again in 10 seconds
[Mar 21 18:39:55:888985] pilight-daemon: INFO: ----- Thread Profiling -----
[Mar 21 18:39:55:889119] pilight-daemon: INFO: - thread node: 0.005445%
[Mar 21 18:39:55:889441] pilight-daemon: INFO: - thread sender: 0.000000%
[Mar 21 18:39:55:889504] pilight-daemon: INFO: - thread broadcaster: 0.000000%
[Mar 21 18:39:55:889545] pilight-daemon: INFO: - thread 433gpio: 0.000000%
[Mar 21 18:39:55:889584] pilight-daemon: INFO: - thread receive parser: 96.212218%
[Mar 21 18:39:55:889645] pilight-daemon: INFO: ----- Thread Profiling -----
[Mar 21 18:39:55:889676] pilight-daemon: cpu usage still way too high 96.225472%, exiting


######################################################


[Mar 21 18:42:31:115796] pilight-daemon: DEBUG: **** Received RAW CODE ****
75 1055 121 59 355 145 223 67 198 631 72 197 138 67 235 67 187 88 126 67 63 65 181 140 62 64 155 101 69 597 67 165 497 1584 71 55 173 1515 69 101 70 64 561 222 75 162 88 68 130 194 67 124 66 74 553 560 582 1222 569 3714 542 3699 545 1847 523 3743 536 1833 527 3746 516 1858 521 1862 521 3767 507 1875 517 1858 520 3740 511 1886 499 1871 509 1868 511 1869 508 1906 485 3753 502 3756 511 1882 488 3755 503 1874 511 1899 483 3753 498 3779 505 1879 486 1895 490 3761 496 1905 477 3774 504 1881 495 3789 478 3768 498 1870 495 8569
[Mar 21 18:42:31:116151] pilight-daemon: DEBUG: ***************************
[Mar 21 18:42:31:116225] pilight-daemon: DEBUG: possible tfa protocol
[Mar 21 18:42:31:116268] pilight-daemon: DEBUG: recevied pulse length of 252
[Mar 21 18:42:31:116306] pilight-daemon: DEBUG: caught minimum # of repeats 1 of tfa
[Mar 21 18:42:31:116345] pilight-daemon: DEBUG: called tfa parseRaw()
[Mar 21 18:42:32:130587] pilight-daemon: cpu usage way too high 100.029422%
[Mar 21 18:42:32:131007] pilight-daemon: WARNING: checking again in 10 seconds
[Mar 21 18:42:43:132041] pilight-daemon: INFO: ----- Thread Profiling -----
[Mar 21 18:42:43:132232] pilight-daemon: INFO: - thread node: 0.004777%
[Mar 21 18:42:43:132703] pilight-daemon: INFO: - thread sender: 0.000000%
[Mar 21 18:42:43:133020] pilight-daemon: INFO: - thread broadcaster: 0.000000%
[Mar 21 18:42:43:133264] pilight-daemon: INFO: - thread 433gpio: 0.000000%
[Mar 21 18:42:43:133480] pilight-daemon: INFO: - thread receive parser: 90.052565%
[Mar 21 18:42:43:133826] pilight-daemon: INFO: ----- Thread Profiling -----
[Mar 21 18:42:43:134099] pilight-daemon: cpu usage still way too high 90.083063%, exiting


######################################################


[Mar 21 18:44:00:798611] pilight-daemon: DEBUG: **** Received RAW CODE ****
68 298 49 130 54 138 60 137 44 141 56 217 45 352 309 44 26 43 157 85 97 61 19 41 20 98 253 43 144 562 52 119 70 106 67 117 74 20 51 21 50 145 534 105 215 21 173 49 126 53 68 90 42 132 1088 215 19 235 70 173 1839 124 868 179 163 202 1194 148 127 972 147 20 125 47 111 48 259 472 170 86 138 48 151 70 253 117 257 51 169 21 56 86 110 111 44 118 48 91 229 142 208 196 55 618 263 49 269 185 131 808 93 48 220 58 141 64 315 45 341 26 193 1695 73 94 826 318 263 511 88 54 69 72 93 46 125 70 2141 1309 665 128 7642
[Mar 21 18:44:00:798884] pilight-daemon: DEBUG: ***************************
[Mar 21 18:44:00:798931] pilight-daemon: DEBUG: possible tfa protocol
[Mar 21 18:44:00:798961] pilight-daemon: DEBUG: recevied pulse length of 224
[Mar 21 18:44:00:798987] pilight-daemon: DEBUG: caught minimum # of repeats 1 of tfa
[Mar 21 18:44:00:799013] pilight-daemon: DEBUG: called tfa parseRaw()
[Mar 21 18:44:01:971713] pilight-daemon: cpu usage way too high 92.018860%
[Mar 21 18:44:01:972652] pilight-daemon: WARNING: checking again in 10 seconds
[Mar 21 18:44:12:974143] pilight-daemon: INFO: ----- Thread Profiling -----
[Mar 21 18:44:12:974522] pilight-daemon: INFO: - thread node: 0.005213%
[Mar 21 18:44:12:974815] pilight-daemon: INFO: - thread sender: 0.000000%
[Mar 21 18:44:12:975002] pilight-daemon: INFO: - thread broadcaster: 0.000000%
[Mar 21 18:44:12:975537] pilight-daemon: INFO: - thread 433gpio: 0.000000%
[Mar 21 18:44:12:976137] pilight-daemon: INFO: - thread receive parser: 90.913008%
[Mar 21 18:44:12:976705] pilight-daemon: INFO: ----- Thread Profiling -----
[Mar 21 18:44:12:977202] pilight-daemon: cpu usage still way too high 90.929935%, exiting


######################################################


[Mar 21 18:47:20:118611] pilight-daemon: DEBUG: **** Received RAW CODE ****
154 86 205 75 224 59 220 122 194 77 65 132 68 85 69 162 59 256 108 152 69 191 112 82 58 1680 70 488 68 200 68 223 67 287 68 156 92 57 183 145 65 100 67 143 111 68 109 318 72 137 222 104 68 67 100 67 855 114 68 86 676 1027 169 89 67 1107 109 232 88 259 152 122 74 64 63 170 127 201 68 265 70 64 63 63 214 93 109 73 92 119 205 156 303 207 139 150 59 191 81 56 229 58 65 481 68 71 336 171 74 73 55 245 315 112 246 67 64 374 81 56 167 254 184 81 56 213 71 279 67 187 144 118 68 65 153 96 96 2308 70 190 7606
[Mar 21 18:47:20:120137] pilight-daemon: DEBUG: ***************************
[Mar 21 18:47:20:120642] pilight-daemon: DEBUG: possible tfa protocol
[Mar 21 18:47:20:121138] pilight-daemon: DEBUG: recevied pulse length of 223
[Mar 21 18:47:20:121631] pilight-daemon: DEBUG: caught minimum # of repeats 1 of tfa
[Mar 21 18:47:20:122132] pilight-daemon: DEBUG: called tfa parseRaw()
[Mar 21 18:47:20:761460] pilight-daemon: WARNING: cpu usage too high 72.990715%
[Mar 21 18:47:20:761896] pilight-daemon: WARNING: checking again in 10 seconds
[Mar 21 18:47:31:762794] pilight-daemon: INFO: ----- Thread Profiling -----
[Mar 21 18:47:31:763173] pilight-daemon: INFO: - thread node: 0.004452%
[Mar 21 18:47:31:763402] pilight-daemon: INFO: - thread sender: 0.000000%
[Mar 21 18:47:31:763586] pilight-daemon: INFO: - thread broadcaster: 0.000000%
[Mar 21 18:47:31:763760] pilight-daemon: INFO: - thread 433gpio: 0.000000%
[Mar 21 18:47:31:763939] pilight-daemon: INFO: - thread receive parser: 99.982089%
[Mar 21 18:47:31:764136] pilight-daemon: INFO: ----- Thread Profiling -----
[Mar 21 18:47:31:764321] pilight-daemon: cpu usage still way too high 99.998622%, exiting


######################################################


[Mar 21 18:47:57:836320] pilight-daemon: DEBUG: **** Received RAW CODE ****
73 58 63 80 101 70 266 59 292 139 507 71 189 105 67 279 67 70 411 63 194 61 485 128 284 68 596 71 73 57 187 70 410 63 497 457 70 184 280 68 746 301 182 368 67 247 69 159 73 322 63 773 72 70 59 633 61 881 100 232 251 63 429 67 141 3726 73 74 62 68 98 68 428 69 170 2584 298 125 158 361 70 374 70 205 74 152 461 69 268 62 531 101 236 67 491 101 124 71 377 61 380 239 71 234 61 306 73 165 68 131 70 183 160 111 66 276 68 291 71 252 71 1068 68 1258 251 345 70 165 113 66 185 106 257 69 262 249 87 7540
[Mar 21 18:47:57:836684] pilight-daemon: DEBUG: ***************************
[Mar 21 18:47:57:836744] pilight-daemon: DEBUG: possible tfa protocol
[Mar 21 18:47:57:836785] pilight-daemon: DEBUG: recevied pulse length of 221
[Mar 21 18:47:57:836822] pilight-daemon: DEBUG: caught minimum # of repeats 10 of tfa
[Mar 21 18:47:57:836860] pilight-daemon: DEBUG: called tfa parseRaw()
[Mar 21 18:47:59:307770] pilight-daemon: WARNING: cpu usage too high 88.867607%
[Mar 21 18:47:59:308199] pilight-daemon: WARNING: checking again in 10 seconds
[Mar 21 18:48:10:309159] pilight-daemon: INFO: ----- Thread Profiling -----
[Mar 21 18:48:10:309548] pilight-daemon: INFO: - thread node: 0.004232%
[Mar 21 18:48:10:309786] pilight-daemon: INFO: - thread sender: 0.000000%
[Mar 21 18:48:10:309970] pilight-daemon: INFO: - thread broadcaster: 0.000000%
[Mar 21 18:48:10:310146] pilight-daemon: INFO: - thread 433gpio: 0.000000%
[Mar 21 18:48:10:310708] pilight-daemon: INFO: - thread receive parser: 86.259073%
[Mar 21 18:48:10:311336] pilight-daemon: INFO: ----- Thread Profiling -----
[Mar 21 18:48:10:311878] pilight-daemon: cpu usage still too high 86.274968%, stopping
[Mar 21 18:48:10:312551] pilight-daemon: DEBUG: socket write succeeded: HEART


######################################################


[Mar 21 18:50:56:233965] pilight-daemon: DEBUG: **** Received RAW CODE ****
30 1736 39 1430 962 176 376 88 342 982 447 121 216 327 62 404 691 126 301 204 127 334 174 77 70 252 668 61 182 72 347 562 80 353 65 908 72 213 80 199 116 246 305 122 987 66 377 211 262 28 590 155 168 517 270 181 387 62 280 65 155 29 140 163 185 452 378 66 89 129 83 143 76 242 81 29 68 183 126 62 29 154 64 461 1990 466 1982 376 2641 697 119 671 434 328 176 785 121 188 93 30 168 102 438 84 71 3794 563 1175 861 400 2471 101 680 165 139 172 68 171 133 380 4142 407 3268 87 29 241 28 455 440 128 263 3701 388 2039 380 1462 7685
[Mar 21 18:50:56:235494] pilight-daemon: DEBUG: ***************************
[Mar 21 18:50:56:236116] pilight-daemon: DEBUG: possible tfa protocol
[Mar 21 18:50:56:236690] pilight-daemon: DEBUG: recevied pulse length of 226
[Mar 21 18:50:56:237285] pilight-daemon: DEBUG: caught minimum # of repeats 6 of tfa
[Mar 21 18:50:56:237859] pilight-daemon: DEBUG: called tfa parseRaw()
[Mar 21 18:50:57:607334] pilight-daemon: cpu usage way too high 100.084160%
[Mar 21 18:50:57:607461] pilight-daemon: WARNING: checking again in 10 seconds
[Mar 21 18:51:08:608525] pilight-daemon: INFO: ----- Thread Profiling -----
[Mar 21 18:51:08:608643] pilight-daemon: INFO: - thread socket: 0.000000%
[Mar 21 18:51:08:608704] pilight-daemon: INFO: - thread ssdp: 0.003233%
[Mar 21 18:51:08:608775] pilight-daemon: INFO: - thread sender: 0.000000%
[Mar 21 18:51:08:608820] pilight-daemon: INFO: - thread broadcaster: 0.000000%
[Mar 21 18:51:08:608863] pilight-daemon: INFO: - thread 433gpio: 0.000000%
[Mar 21 18:51:08:608918] pilight-daemon: INFO: - thread receive parser: 98.249902%
[Mar 21 18:51:08:608990] pilight-daemon: INFO: - thread events client: 0.003735%
[Mar 21 18:51:08:609043] pilight-daemon: INFO: - thread events loop: 0.000000%
[Mar 21 18:51:08:609096] pilight-daemon: INFO: ----- Thread Profiling -----
[Mar 21 18:51:08:609130] pilight-daemon: cpu usage still way too high 98.266409%, exiting


######################################################


[Mar 21 18:55:07:815937] pilight-daemon: DEBUG: **** Received RAW CODE ****
65 181 1992 57 167 118 173 355 65 191 117 96 177 56 184 56 148 65 59 59 117 208 54 74 247 64 407 64 309 297 64 130 186 83 71 139 211 62 51 102 65 125 350 65 459 431 373 63 61 58 167 64 61 429 155 63 373 931 133 232 2828 122 477 55 629 86 901 2279 65 196 309 149 56 166 183 64 65 58 204 63 168 108 226 85 129 219 376 201 1135 170 204 238 68 131 64 59 305 65 243 64 288 182 103 65 104 380 55 275 200 249 186 63 186 70 162 63 200 65 193 70 128 64 177 70 101 113 63 135 77 54 1224 55 953 136 63 83 63 7673
[Mar 21 18:55:07:816331] pilight-daemon: DEBUG: ***************************
[Mar 21 18:55:07:816412] pilight-daemon: DEBUG: possible tfa protocol
[Mar 21 18:55:07:816455] pilight-daemon: DEBUG: recevied pulse length of 225
[Mar 21 18:55:07:816494] pilight-daemon: DEBUG: caught minimum # of repeats 1 of tfa
[Mar 21 18:55:07:816532] pilight-daemon: DEBUG: called tfa parseRaw()
[Mar 21 18:55:08:768986] pilight-daemon: cpu usage way too high 111.300824%
[Mar 21 18:55:08:769563] pilight-daemon: WARNING: checking again in 10 seconds
[Mar 21 18:55:19:770869] pilight-daemon: INFO: ----- Thread Profiling -----
[Mar 21 18:55:19:771472] pilight-daemon: INFO: - thread socket: 0.000000%
[Mar 21 18:55:19:771873] pilight-daemon: INFO: - thread ssdp: 0.002754%
[Mar 21 18:55:19:772246] pilight-daemon: INFO: - thread sender: 0.000000%
[Mar 21 18:55:19:772604] pilight-daemon: INFO: - thread broadcaster: 0.000000%
[Mar 21 18:55:19:772961] pilight-daemon: INFO: - thread 433gpio: 2.550051%
[Mar 21 18:55:19:773261] pilight-daemon: INFO: - thread receive parser: 99.969786%
[Mar 21 18:55:19:773562] pilight-daemon: INFO: - thread events client: 0.003266%
[Mar 21 18:55:19:773848] pilight-daemon: INFO: - thread events loop: 0.000000%
[Mar 21 18:55:19:774132] pilight-daemon: INFO: ----- Thread Profiling -----
[Mar 21 18:55:19:774403] pilight-daemon: cpu usage still way too high 102.543324%, exiting


######################################################


[Mar 21 18:55:58:820400] pilight-daemon: DEBUG: **** Received RAW CODE ****
358 528 217 31 166 232 54 382 66 181 28 259 58 147 97 224 56 134 183 77 168 182 369 168 53 186 57 174 413 180 69 246 212 141 435 37 678 58 301 169 29 618 1111 386 199 586 223 242 28 404 79 302 62 154 29 392 62 584 28 818 365 28 115 131 301 38 139 62 295 61 235 64 436 149 37 133 3589 101 85 61 435 141 73 231 192 974 1042 87 854 66 232 392 375 61 553 2588 29 492 384 1729 2932 1135 30 276 28 391 62 215 28 141 129 255 144 550 143 106 128 74 149 121 212 64 189 103 28 202 151 144 63 130 28 106 72 189 151 62 274 81 208 142 97 28 168 65 1679 267 7660
[Mar 21 18:55:58:820865] pilight-daemon: DEBUG: ***************************
[Mar 21 18:55:58:820970] pilight-daemon: DEBUG: possible tfa protocol
[Mar 21 18:55:58:821063] pilight-daemon: DEBUG: recevied pulse length of 225
[Mar 21 18:55:58:821157] pilight-daemon: DEBUG: caught minimum # of repeats 1 of tfa
[Mar 21 18:55:58:821252] pilight-daemon: DEBUG: called tfa parseRaw()
[Mar 21 18:55:58:898317] pilight-daemon: DEBUG: cpu: 12.893643%, ram: 0.296607%
[Mar 21 18:55:59:899644] pilight-daemon: cpu usage way too high 100.267840%
[Mar 21 18:55:59:899817] pilight-daemon: WARNING: checking again in 10 seconds
[Mar 21 18:56:10:900816] pilight-daemon: INFO: ----- Thread Profiling -----
[Mar 21 18:56:10:900940] pilight-daemon: INFO: - thread socket: 0.000000%
[Mar 21 18:56:10:900989] pilight-daemon: INFO: - thread ssdp: 0.002635%
[Mar 21 18:56:10:901049] pilight-daemon: INFO: - thread sender: 0.000000%
[Mar 21 18:56:10:901085] pilight-daemon: INFO: - thread broadcaster: 0.000000%
[Mar 21 18:56:10:901121] pilight-daemon: INFO: - thread 433gpio: 0.000000%
[Mar 21 18:56:10:901158] pilight-daemon: INFO: - thread receive parser: 99.989235%
[Mar 21 18:56:10:901217] pilight-daemon: INFO: - thread events client: 0.004059%
[Mar 21 18:56:10:901261] pilight-daemon: INFO: - thread events loop: 0.000000%
[Mar 21 18:56:10:901290] pilight-daemon: INFO: ----- Thread Profiling -----
[Mar 21 18:56:10:901330] pilight-daemon: cpu usage still way too high 100.006175%, exiting


######################################################


[Mar 21 18:57:18:105774] pilight-daemon: DEBUG: **** Received RAW CODE ****
1231 265 690 124 478 157 151 1084 113 200 537 147 78 85 382 81 188 38 247 37 41 179 43 122 307 58 241 45 142 341 264 36 313 47 618 92 500 144 141 38 85 382 49 91 616 44 122 50 41 179 168 50 141 68 81 263 39 581 190 612 114 100 280 749 98 89 242 70 216 43 91 83 178 49 49 147 45 113 247 203 423 126 43 119 46 130 46 131 57 239 113 108 127 44 96 78 150 214 43 180 38 42 40 108 72 48 36 40 2011 117 56 45 56 142 550 44 42 131 263 244 68 96 45 161 317 77 351 49 36 170 97 45 468 43 65 55 113 96 155 100 37 71 372 50 35 73 264 439 261 986 8640
[Mar 21 18:57:18:106811] pilight-daemon: DEBUG: ***************************
[Mar 21 18:57:18:107160] pilight-daemon: DEBUG: possible tfa protocol
[Mar 21 18:57:18:107484] pilight-daemon: DEBUG: recevied pulse length of 254
[Mar 21 18:57:18:107806] pilight-daemon: DEBUG: caught minimum # of repeats 1 of tfa
[Mar 21 18:57:18:108076] pilight-daemon: DEBUG: called tfa parseRaw()
[Mar 21 18:57:19:209831] pilight-daemon: cpu usage way too high 100.088520%
[Mar 21 18:57:19:210458] pilight-daemon: WARNING: checking again in 10 seconds
[Mar 21 18:57:30:211765] pilight-daemon: INFO: ----- Thread Profiling -----
[Mar 21 18:57:30:212392] pilight-daemon: INFO: - thread socket: 0.000000%
[Mar 21 18:57:30:212784] pilight-daemon: INFO: - thread ssdp: 0.003486%
[Mar 21 18:57:30:213173] pilight-daemon: INFO: - thread sender: 0.000000%
[Mar 21 18:57:30:213529] pilight-daemon: INFO: - thread broadcaster: 0.000000%
[Mar 21 18:57:30:213878] pilight-daemon: INFO: - thread 433gpio: 0.000000%
[Mar 21 18:57:30:214165] pilight-daemon: INFO: - thread receive parser: 99.997688%
[Mar 21 18:57:30:214470] pilight-daemon: INFO: - thread events client: 0.003470%
[Mar 21 18:57:30:214754] pilight-daemon: INFO: - thread events loop: 0.000000%
[Mar 21 18:57:30:215020] pilight-daemon: INFO: ----- Thread Profiling -----
[Mar 21 18:57:30:215286] pilight-daemon: cpu usage still way too high 100.014563%, exiting


######################################################


[Mar 21 19:01:57:443128] pilight-daemon: DEBUG: **** Received RAW CODE ****
108 109 195 191 518 64 136 29 435 38 146 100 162 132 64 263 29 149 114 95 122 62 213 196 86 64 29 118 401 593 1327 63 542 63 72 138 87 286 241 169 271 217 168 595 951 838 531 1841 532 3739 550 1829 30 194 40 465 1151 542 1839 552 1816 569 1814 528 1901 475 1880 507 1865 507 3759 539 1854 506 1871 513 3741 505 1889 497 1874 506 1878 503 1866 506 1945 466 3767 498 3752 497 1889 492 3767 496 1877 499 3776 482 3753 504 1898 504 1887 485 1881 494 3764 493 1887 510 3775 493 1881 502 3761 481 1899 495 1866 498 8574
[Mar 21 19:01:57:443476] pilight-daemon: DEBUG: ***************************
[Mar 21 19:01:57:443632] pilight-daemon: DEBUG: possible tfa protocol
[Mar 21 19:01:57:443685] pilight-daemon: DEBUG: recevied pulse length of 252
[Mar 21 19:01:57:443725] pilight-daemon: DEBUG: caught minimum # of repeats 1 of tfa
[Mar 21 19:01:57:443764] pilight-daemon: DEBUG: called tfa parseRaw()
[Mar 21 19:01:58:522896] pilight-daemon: cpu usage way too high 100.083198%
[Mar 21 19:01:58:523521] pilight-daemon: WARNING: checking again in 10 seconds
[Mar 21 19:02:09:524832] pilight-daemon: INFO: ----- Thread Profiling -----
[Mar 21 19:02:09:525469] pilight-daemon: INFO: - thread socket: 0.000000%
[Mar 21 19:02:09:525864] pilight-daemon: INFO: - thread ssdp: 0.003109%
[Mar 21 19:02:09:526252] pilight-daemon: INFO: - thread sender: 0.000000%
[Mar 21 19:02:09:526606] pilight-daemon: INFO: - thread broadcaster: 0.000000%
[Mar 21 19:02:09:526958] pilight-daemon: INFO: - thread 433gpio: 0.000000%
[Mar 21 19:02:09:527244] pilight-daemon: INFO: - thread receive parser: 99.999912%
[Mar 21 19:02:09:527548] pilight-daemon: INFO: - thread events client: 0.003937%
[Mar 21 19:02:09:527837] pilight-daemon: INFO: - thread events loop: 0.000000%
[Mar 21 19:02:09:528100] pilight-daemon: INFO: ----- Thread Profiling -----
[Mar 21 19:02:09:528367] pilight-daemon: cpu usage still way too high 100.016651%, exiting

I receive a lot of stuff, maybe these "raw codes" occur because I don't have a hardware low-pass filter?! I don't know...
 
Reply
#7
@curlymo,
The mystery of CPU usage too high is solved for this driver.
The issue is that commit 9fac9a3 has an invalid check of protocol parameters within validate(). As a consequence pulsetrains exceeding 88 pulses are validated and thus forwarded for further processing by the tfa protocol module.
The decoder loop was relying on the boundary checks made by validate() and thus did not check the value of the loop's terminating parameter tfa->rawlen.
That value shall never exceed RAW_LENGTH, otherwise the index for binary[] will be out of bounds.
This happened and as a consequence memory is overwriten and in this case the thread receive_parser started eating up CPU time.
 
Reply
#8
Nice. Does this also solve the rewrite CPU issues?
 
Reply
#9
I do not know. As discussed in this thread xanker was immediately outlining that something is wrong with the modified tfa driver 9fac9a3, but that the issue was fixed with bad015a, so those were the steps taken to trace the problem of high cpu usage down:
Code:
1. Creation of branch tfa_s_debuf (the f in debuf was a spelling error)
1.1. Based on commit 9fac9a3 a modified daemon.c module printing out the raw data in receive_parse_code was created and is part of that branch.
1.2. The raw data obtained was used to simulate raw data reception on another machine with "pilight-send -p raw"
1.3. I was able reproduce the unwanted event:
1.3.1. High CPU usage was reported after the pilight-send command
1.3.2. 10 seconds later pilight terminated itself.

Lessons learned:
Due to the fact that parsecode is called indirectly, attention should be paid to the implementation details of validate(), however in no case should we rely on the status returned from validate() and I recommend to add boundary checks before such code in question, including an error statement so that this violation does not go unnoticed:
Code:
....
if(tfa->rawlen>RAW_LENGTH) {
    logprintf(LOG_ERR, "tfa: parseCode - invalid parameter passed %d",tfa->rawlen);
    return;
}
for(x=xLoop;x<tfa->rawlen-2;x+=2) {
....

HowTo get your GIT repository updated
This is the original structure before I started to change tfa.c again
Code:
-> git checkout tfa_s
-> git log --oneline --all --graph --decorate -4
* 17561ee (origin/tfa_s_debuf, tfa_s_debuf) Testversion - not for standard use
* 337a457 (HEAD, origin/tfa_s, tfa_s) Bugfixing tfa.c soens protocol version
* bad015a Remove bug in temperature calculation and enhance checking of protocol validation
* 9fac9a3 Support for SOENS Wheatherstation added to tfa protocol https://forum.pilight.org/Thread-Soens-Innovation-TFA-similar-protocol?pid=17605#pid17605

I committed my tfa.c changes on branch tfa_s with commit f2c09e3.

After the update of tfa.c i integrated this new tfa.c code into the debuf branch as well by rebasing the debuf branch on tfa_s:
Code:
-> git checkout tfa_s_debuf
-> git rebase tfa_s
-> git log --all --oneline --graph --decorate -6
* ec57010 (HEAD, tfa_s_debuf) Testversion - not for standard use
* f2c09e3 (tfa_s) Added boundary check in module tfa.c to function parseCode
| * 17561ee (origin/tfa_s_debuf) Testversion - not for standard use
|/
* 337a457 (origin/tfa_s) Bugfixing tfa.c soens protocol version
* bad015a Remove bug in temperature calculation and enhance checking of protocol validation
* 9fac9a3 Support for SOENS Wheatherstation added to tfa protocol https://forum.pilight.org/Thread-Soens-Innovation-TFA-similar-protocol?pid=17605#pid17605

As you can see the old debugf branch with commit 17561ee is still on github, and this will trigger an error message when you try to push the new local HEAD of debuf to github. In the current case I am sure that no one is relying on that old branch, therefore it is safe to delete that published branch by forcing the push to go through.
Code:
-> git push -f origin tfa_s_debuf
-> git log --all --oneline --graph --decorate -3
* ec57010 (HEAD, origin/tfa_s_debuf, tfa_s_debuf) Testversion - not for standard use
* f2c09e3 (tfa_s) Added boundary check in module tfa.c to function parseCode
* 337a457 (origin/tfa_s) Bugfixing tfa.c soens protocol version

And as a last step the changes for tfa.c in branch tfa_s have to be published:
Code:
-> git checkout tfa_s
-> git push origin tfa_s
-> git log -all --oneline --graph --decorate -5
* ec57010 (origin/tfa_s_debuf, tfa_s_debuf) Testversion - not for standard use
* f2c09e3 (HEAD, origin/tfa_s, tfa_s) Added boundary check in module tfa.c to function parseCode
* 337a457 Bugfixing tfa.c soens protocol version
* bad015a Remove bug in temperature calculation and enhance checking of protocol validation
* 9fac9a3 Support for SOENS Wheatherstation added to tfa protocol https://forum.pilight.org/Thread-Soens-Innovation-TFA-similar-protocol?pid=17605#pid17605
As you can see the old debugf branch with commit 17561ee is deleted and we got a neat clean and linear hostory, it is now easy for everyone to use this repository to go through all steps discussed in this thread.
Please notice the first commit message on github pointing to this thread, indicating where more information on the subject is available.

Before the pull request for branch tfa_s is issued for integration of the new tfa-c module into the mainstream pilight development branch, an interactive rebase is used to squash the commits bad015a, 337a457 and f2c09e3 into 9fac9a3, as the debugging and code improvement history is of no interest later on.

You invoke that procedure with the following command:
Code:
git rebase -i HEAD~4

Your favourite editor is opened:
Code:
pick 9fac9a3 Support for SOENS Wheatherstation added to tfa protocol https://forum.pilight.org/Thread-Soens-Innovation-TFA-s$
pick bad015a Remove bug in temperature calculation and enhance checking of protocol validation
pick 337a457 Bugfixing tfa.c soens protocol version
pick f2c09e3 Added boundary check in module tfa.c to function parseCode

# Rebase b2d758c..f2c09e3 onto b2d758c
And you need to specify which files shall be squashed:
Code:
pick 9fac9a3 Support for SOENS Wheatherstation added to tfa protocol https://forum.pilight.org/Thread-Soens-Innovation-TFA-s$
squash bad015a Remove bug in temperature calculation and enhance checking of protocol validation
squash 337a457 Bugfixing tfa.c soens protocol version
squash f2c09e3 Added boundary check in module tfa.c to function parseCode

# Rebase b2d758c..f2c09e3 onto b2d758c
Well and all debug statements are automatically left behind in branch tfa_s_debuf :-) and what's even better after the interactive rebase it also contains the history of our development steps.

Thus the final log history is reduced to one single commit with the original reason why we did all that development in the tfa_s branch and ready for the official pull request:
Code:
-> git log --oneline --all --graph --decorate -10

* 92e508e (HEAD, tfa_s) Support for SOENS Wheatherstation added to tfa protocol https://forum.pilight.org/Thread-Soens-Innovation-TFA-similar-protocol?pid=17605#pid17605
| * ec57010 (origin/tfa_s_debuf, tfa_s_debuf) Testversion - not for standard use
| * f2c09e3 (origin/tfa_s) Added boundary check in module tfa.c to function parseCode
| * 337a457 Bugfixing tfa.c soens protocol version
| * bad015a Remove bug in temperature calculation and enhance checking of protocol validation
| * 9fac9a3 Support for SOENS Wheatherstation added to tfa protocol https://forum.pilight.org/Thread-Soens-Innovation-TFA-similar-protocol?pid=17605#pid17605
|/
| *   118a327 (origin/somfy_release) Merge remote-tracking branch 'upstream/development' into somfy_release
| |\
| |/
|/|
* |   b2d758c (origin/development, origin/HEAD, development) Merge pull request #251 from cxxcoder/add_arctech_switch_old_telegram_check

Once the pull request is accepted as part of the offical development branch we fetch it back from the upstream repository into the local development branch.
Code:
-> git checkout development
-> git fetch upstream development
-> git log --all --oneline --graph --decorate -2
*   1639c2e (HEAD, upstream/development, development) Merge pull request #258 from wo-rasp/tfa_s
|\
| * 92e508e (origin/tfa_s, tfa_s) Support for SOENS Wheatherstation added to tfa protocol https://forum.pilight.org/Thread-Soens-Innovation-TFA-similar-protocol?pid=17605#pid17605

And the development cycle is closed by saving it to the origin repository
Code:
-> git push origin development
-> git log --all --oneline --graph --decorate -2
*   1639c2e (HEAD, upstream/development, origin/development, origin/HEAD, development) Merge pull request #258 from wo-rasp/tfa_s
|\
| * 92e508e (origin/tfa_s, tfa_s) Support for SOENS Wheatherstation added to tfa protocol https://forum.pilight.org/Thread-Soens-Innovation-TFA-similar-protocol?pid=17605#pid17605

Add the changes to the rewrite branch

In order to get the modifications added to the rewrite branch as well, we do some cherry-picking.
Before you start, ensure that the local rewrite branch is synched with the upstream repository.
Code:
-> git checkout rewrite
-> git fetch upstream rewrite
-> git merge origin/rewrite

Now we can add the changes to the rewrite branch. As all changes are protocol related not interfering with the rewrite modifications, we should not experience major conflicts.
Code:
-> git cherry-pick origin/tfa_s

The only conflict is the overlap in revision numbers (1.2 vs 2.0 on the rewrite branch).
I do propose that the minor revision number matches on both branches, thus i changed the value from 2.0 into 2.2, to match it up with revison 1.2 on the development branch.
It should be noted that the cherry-pick command expects a single commit sha, however due to the fact that a branch is always pointing to a single individual commit sha, we can use the branch name as well.
Code:
-> git add libs/pilight/protocols/433.92/tfa.c
-> git cherry-pick --continue
-> git push origin rewrite

Done. We can issue the pull request on github.
 
Reply
#10
I'm happy everything is running fine now!

I orderd another one of these weather stations and will report back if there are changes to the first 4 bits. Should arrive this week.

Furthermore, I try to confirm the battery bit. But for this I need some (nearly) empty batteries Wink

Thanks a lot for everything!
 
Reply
  


Possibly Related Threads...
Thread Author Replies Views Last Post
  missing protocol of "date_and_time" in Wiki/Protocols fleisch 3 131 08-03-2019, 06:10 PM
Last Post: curlymo
Brick Protocol for FT0073 drobskind 5 2,905 07-25-2019, 10:44 PM
Last Post: fireball
  Oregon Protocol V2.1: THGR122NX jpoilux 208 65,360 07-04-2019, 02:14 PM
Last Post: Tueftler1983
  Brennenstuhl RCR CE1 1011 with QUIGG GT9000 Protocol scootermacro 1 177 06-27-2019, 06:20 PM
Last Post: scootermacro
  mumbi rcs-20gs protocol scootermacro 0 160 06-21-2019, 12:10 PM
Last Post: scootermacro
  How to introduce a new protocol folder? Phunkafizer 2 356 01-15-2019, 07:25 PM
Last Post: Phunkafizer
  rfcontroljs protocol implementation pisperate 0 204 01-13-2019, 11:07 AM
Last Post: pisperate
  MQTT protocol for pilight Chekov 11 2,847 11-03-2018, 04:21 PM
Last Post: curlymo
  getting started guide for implementing new protocol kaililleby 1 788 11-28-2017, 10:24 PM
Last Post: pilino1234
  Logilink EC0003 Protocol ehz 4 1,057 09-24-2017, 07:15 PM
Last Post: ehz

Forum Jump:


Browsing: 1 Guest(s)