• 10 dec 2017: forum version update. In case of issues use this topic.
  • 30 nov 2017: pilight moved servers. In case of issues use this topic.
Hello There, Guest! Login Register

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Ideas for replacement of datetime protocol
The datetime protocol has to be dropped in it's current existance.

With the new codebase, a timerpool was introduced. A short explanation of it's logic.

Old situation:
- 4 functions = 4 threads
- Interval of functions: 1) 3s 2) 1s 3) 10s 4) 5s
- Send message when timeout arrived.

This logic doesn't scale because each new timed function required a new thread.

New situation:
- 4 functions = 4 tasks in threadpool queue
- All intervals are added to a priority queue like this:
2) 1s
1) 3s
4) 5s
3) 10s
- Execute the first function (through the threadpool) that needs execution, then rearm the timer. The execution logic will be like this (when all functions repeat endlessly and rearm at each execution):
2, 2, [2, 1], 2, [2, 4], [2, 3], 2, 2, [2, 3], [2, 4, 3]

This logic scales fine because all timed functions are handled by a single timer and each timeout will be handled by the threadpool.

So in this light, the current datetime device has to be dropped. That implementation still doesn't use the timerpool logic. As in, don't evaluate execution each second, but trigger the timerpool to wait for the first timeout to arrive.

A few ideas how to achieve this:
- Replace the datetime protocol with a cron like protocol. The new cron protocol will fire based on original cron behaivor:
"zeit": {
    "protocol": [ "cron" ],
    "id": [{
        "longitude": 8.53031,
        "latitude": 50.15403
    "year": "*",
    "month": "*",
    "day": "*",
    "weekday": "*",
    "hour": "9, 23",
    "minute": 0,
    "second": 0
This should trigger a cron event each day of each month of each year at both 9:00 and 23:00.

- Another idea is to add a cron feature to each rule:
    "rule1": {
        "rule": "...",
        "active": 0,
        "cron": [ "0 9,23 * * *" ]
This means we drop the datetime / cron protocol altogether, but attach an timed evaluation to each rule. The rule will then only be evaluation on the specific timeout moments.

- Other ideas???
I would prefer the second way, since datetime is mostly used for rules, therefore from the logical point of view it's better to attach a cron to a rule. The first way would rather need more cron devices, isn't it?
Multiple values should be considered as well, e.g. executing a rule during December and first half of January. With one value two rules would be needed. I guess there is no way to handle that in one string?
Just my 5 cents ^^

sent from Z1c
I much prefer the second proposal. This way you don't need a "cron" protocol device for every rule that is time based.
This would also remove the network/pilight-receive spam of the current datetime protocol, which is also repeated once by every single node.
I agree to @diman87. And as the cron property of the rule is an array you would be able to configure more than one cron schedule. I just don't know what would happen if the schedules would overlap or if you configured multiple schedules with the same period. Maybe there should be a validation for that.

But the cron job should consider the current timezone as well. For timezone I think it would be much easier if you could either rely on the system time as usually the host is using ntp or if you could set the timezone in the settings area by setting a property "timezone" with the abbreviation value e.g. shown here: https://en.wikipedia.org/wiki/List_of_ti...reviations.
Timezone is calculated based lon/lat.
but the idea was to remove the device for datetime completely, right? So where to configer lon/lat? And for me calculation of timezone is not necessary if you set the code.
Maybe indeed a global config.
Yes, that would be more sensible than having to define the same long/lat coordinates several times.
(03-05-2016, 11:46 PM)curlymo Wrote: The execution logic will be like this (when all functions repeat endlessly and rearm at each execution):
2, 2, [2, 1], 2, [2, 4], [2, 3], 2, 2, [2, 3], [2, 4, 3]

This logic scales fine because all timed functions are handled by a single timer and each timeout will be handled by the threadpool.

- Other ideas???
I have not yet fully grasped the complete topic, may I ask some questions?
As a prerequisite, I think that I fully understand how pilight V7 is receiving/sending data via the GPIO interface. From the event / rule subsystem so far I have only used those rules that are triggered by status changes.

Form the manual I know that there is a possibility to handle events/rules based on a specific time or on repetitive timeintervals, let me call this a TimedEventRequest (TER). Each TER is assigned to an individual thread, running independently. and I think that I am right that for each timed rule/event/action, in V7 you created an individual thread, each thread defined by what I would call its TimeReQuirements (TRQ) (4 functions = 4 threads).

In order to avoid the context switch between thread executions you want to use tasks instead of threads, by implementing a taskpool for TRQ's, basically reusing the V7 concept without the costs of paying for the CPU time of the context switch. Cost for memory should be more or less the same.

In brief both methods implement a preemptive scheduler for TER's, e.q. those threads could not just only be triggered by time, but by other events as well - correct ?

The new concept you propose is implementing a scheduler (CRON oriented) scheduling the execution of of TRQ's on a timely base.

I fully agree with your concept of adressing the issue of resource optimization.

With the arrival of the Pi3 original CPU power of the Pi architecture has increased by a factor of 10.

I opt for getting the taskpool concept finalized and to focus on major re-writes time permitting only. There is room for CRON based concepts as well, in particular for tasks supposed to run once a day only.

We have still the subject of high unexpected CPU usage

Memory is a certain bottleneck on a Pi architecture, but AFAIK memory overhead of threads for eventing is not the major issue, nor is the RAM of a Pi2 or 3, even not on a PI-B. Memorywise my concern is more with the SD card (i had the first burned out one lately), and introduction of using overlays would be highly desirable. This would allow the introduction of SQL databases by writing to RAM based overlays on 1GB PIs with long term SD card storage (2 or 3 times a day or on shutdown).
Additionally if you completely replace the datetime protocol there has to be an alternative for timed triggers like for sunrise and sunset. I'm thinking of something like a function CURRENT_TIME(<format>) that you could use for example
IF weather.sunrise == CURRENT_TIME(hh.mm) THEN ...

then the inconvenient time conversion could be removed which is currently required.

And then maybe if the time triggers and monitors are reworked you could consider the time evaluation based on the interval used in the rule. I forgot once to set the validation for the second and then pilight tried to send mails every second ;-)
So what I mean is that you could scan the rule for CURRENT_TIME and if there no second is mentioned you could verify only each minute...

Possibly Related Threads...
Thread Author Replies Views Last Post
  program protocol missing in rewrite Niek 1 441 10-07-2017, 08:55 PM
Last Post: curlymo

Forum Jump:

Browsing: 1 Guest(s)