bobaos.pub - KNX TP / UART, Raspberry Pi and Redis

There is no limit to perfection. It would seem that everything worked well, minor bugs were fixed and so on.

Now I will tell, firstly, about the problems that I faced during all the time that has passed since the previous article, and, secondly, about the decisions that contributed to the current status of the project.

→ Article about the previous version

Legend

bobaos - npm module for interacting with BAOS 83x using UART. Returns raw data. Used in all other modules listed below.

bdsd.sock is a script for working with KNX objects. Stores the list of datapoints, when sending / receiving converts values. From DPT1 to true / false, from DPT9 to float. Also listens to Unix Socket for receiving requests from other processes.

bobaos.pub is a new version that uses redis for interprocess communication.
KNX объект/датапоинт is a BAOS 83x module communication object configured in ETS, to which (or not) the group address (a) corresponds. In current versions of iron, the maximum amount is 1000.

Task

The main task is the same as the previous version. A connection to a serial port can only open one. Scripts working with KNX, I want to run a lot. In addition to this, I wanted to implement interprocess communication. Those. so that not only one bdsd.sock process listened to a socket, but each running script could both send and receive requests.

Idea

An idea was born in my head to make my message broker on node.js on top of Unix sockets, to which clients would connect, subscribe to topics and receive / send messages according to the code written in them. I knew that there were ready-made solutions, about which only lazy people hadn’t heard recently, I studied, but the idea of ​​making my decision was intrusive.

And in the end the service started.

Wrote a logger who sends messages to the topic. Subscribers receive and are free to do anything, or rather what is written. Conveniently - logs from several sources can be viewed in one console output.

I wrote, posted to npm package bobaos.pub, which, unlike bdsd.sock, no longer creates a socket file, but connects to a broker. At first glance, everything works as it should.

Problem

Then I launched a script that periodically sends requests to the KNX bus for the night. Waking up in the morning, by the blinking of the LEDs signaling the sending / transfer of data, I realized that something was wrong. Messages did not reach on time. Found out that the samopisny message broker took up almost all of the RAM available from 512MB (from BeagleBoard Black). Further work with nodejs confirmed that memory is the weak point of js scripts.

Decision

As a result, it was decided to switch from samopisny Unix sockets to Redis (by the way, he also knows how to work with them). Perhaps it was worth figuring out the memory, finding the leaks, but I wanted to run faster.

bobaos means communicating over UART with wrapping messages in FT1.2, our communication is synchronous. Those. <..сообщение-подтверждение-ответ..> . The bobaos module, which is responsible for communication, stores all requests in the array, pulls out from there one by one, sends it to the UART, and when an incoming response is resolved, the promise responsible for this request is allowed.

You can go the following way: the service listens to the PUB / SUB channel redis, accepts requests, sends to KNX. In this case, the load on the request queue falls on the js bobaos module. To implement, you need to write a simple module, subscribed to the channel and converts messages using the JSON.parse() method. Further, this module can be used in other scripts.

Another option, which eventually stopped: use the existing task manager on top of redis . There are several choices made on the bee-queue .

→ Under the hood

It describes how the bee-queue works. If you implement this library for other programming languages, you can thus make client libraries for bobaos .

In the second version, all requests are stored in redis lists, are pulled out in turn and sent to the serial port.

Next, a rewrite of the previous version follows, but I already keep all the data on datapoint in the redis database. The only inconvenience that I feel is that all requests are asynchronous, respectively, getting an array of values ​​is a little harder than just turning to an array.

Small optimizations have been made

If previously there were separate getValue/getValues/readValue/readValues/setValue/setValues , now getValue/readValue/setValues accept both a single value and an array.

The getValue([id1, id2, ...]) method getValue([id1, id2, ...]) in the previous version sent a request to the serial port for each datpoint. But it is possible to send a request for several values. Restrictions - the size of the response should be equal to BufferSize , the maximum - 250 bytes; also it is impossible to go beyond the number of objects, for current versions of BAOS 83x modules - 1000.

The lengths of the values ​​are known, the title too. Further, a rather simple algorithm with while loops and awaits =)

1. Sort the array, delete duplicate elements, if any. get the idUniq array.
2. We start the cycle i < idUniq.length , in which we do the following:
   a) Take start: idUniq[i] , for it we count the maximum number of values ​​we can get. For example, if all objects are of type DPT1 / DPT5 (1 byte), then we can get values ​​in the amount of 48. There is one note: if, for example, we have set up objects in ETS #[1, 2, 3, 10, 20] , then, when the GetDatapointValue.Req(1, 30) is requested GetDatapointValue.Req(1, 30) zero one-byte values ​​will be returned in response, even for non-existent data points [4, 5, 6, ..., 30] .
   b) Counting occurs in the new cycle j < i + max (where max is 50, or, if close to 1000, then the maximum is 1000 - id + 1 , for 990 it will be 11, for 999 - 2), if in the process of counting we meet array elements from the original query, then i assigned the variable i by its index.
   c) If in the cycle j calculated length goes beyond the maximum length of the buffer, then we form the element of the "map" of requests {start: start, number: number} , throw it into a separate array, increase the variable i (or assign the index found in the idUniq array), interrupt cycle j , both cycles are restarted.

Thus, we form several requests for bobaos . For example, if you send a getValue([1, 2, 3, 40, 48, 49, 50, 100, 998, 999, 1000]) request getValue([1, 2, 3, 40, 48, 49, 50, 100, 998, 999, 1000]) , then the requests may be for the particular case of the following:

 {start: 1, number: 48}, // 1, 2, 3, 40, 48 {start: 49, number: 48}, // 49, 50 {start: 100, number: 48}, // 100 {start: 998, number: 3} // 998, 999, 1000 

It could be done differently:

 {start: 1, number: 48}, // 1, 2, 3, 40, 48 {start: 49, number: 2}, // 49, 50 {start: 100, number: 1}, // 100 {start: 998, number: 3} // 998, 999, 1000 

Requests would be the same, given less. But I stopped at the first option, since the values ​​obtained are saved to the redis database, respectively, they can be obtained using the getStoredValue method, which I try to use more often than getValue , which sends data on the serial port.

A separate queue is created for the ping/get sdk state/reset service methods. Thus, if there is something wrong with the serial port communication (the frame counter is lost, etc.) and the execution stopped on some task, you can send a reset request to another queue, and accordingly restart sdk .

Client part - bobaos.sub

The bobaos.sub module can be used to control KNX data points in user scripts.

The following example covers all module functions:

 const BobaosSub = require("bobaos.sub"); // в параметры можно передать опционально: // redis: объект либо url // request_channel: "bobaos_req" по умолчанию, // service_channel: "bobaos_service" по умолчанию, // broadcast_channel: "bobaos_bcast" по умолчанию let my = BobaosSub(); my.on("connect", _ => { console.log("connected to ipc, still not subscribed to channels"); }); my.on("ready", async _ => { try { console.log("hello, friend"); console.log("ping:", await my.ping()); console.log("get sdk state:", await my.getSdkState()); console.log("get value:", await my.getValue([1, 107, 106])); console.log("get stored value:", await my.getValue([1, 107, 106])); console.log("get server item:", await my.getServerItem([1, 2, 3])); console.log("set value:", await my.setValue({id: 103, value: 0})); console.log("read value:", await my.readValue([1, 103, 104, 105])); console.log("get programming mode:", await my.getProgrammingMode()); console.log("set programming mode:", await my.setProgrammingMode(true)); console.log("get parameter byte", await my.getParameterByte([1, 2, 3, 4])); console.log("reset", await my.reset()); } catch(e) { console.log("err", e.message); } }); my.on("datapoint value", payload => { // Имейте в виду, что payload может быть как объектом, так и массивом // так что необходима проверка Array.isArray(payload) console.log("broadcasted datapoint value: ", payload); }); my.on("server item", payload => { // Имейте в виду, что payload может быть как объектом, так и массивом // так что необходима проверка Array.isArray(payload) console.log("broadcasted server item: ", payload); }); my.on("sdk state", payload => { console.log("broadcasted sdk state: ", payload); }); 

bobaos.tool

The command line interface has been rewritten. About how I implemented it the following article:

→ We write CLI on NodeJS

The teams have become shorter, clearer and more functional.

 bobaos> progmode ? BAOS module in programming mode: false bobaos> progmode 1 BAOS module in programming mode: true bobaos> progmode false BAOS module in programming mode: false bobaos> description 1 2 3 #1: length = 2, dpt = dpt9, prio: low, flags: [C-WTU] #2: length = 1, dpt = dpt1, prio: low, flags: [C-WT-] #3: length = 1, dpt = dpt1, prio: low, flags: [C-WT-] bobaos> set 2: 0 20:27:06:239, id: 2, value: false, raw: [AA==] bobaos> set [2: 0, 3: false] 20:28:48:586, id: 2, value: false, raw: [AA==] 20:28:48:592, id: 3, value: false, raw: [AA==] 

Afterword

It turned out working stable system. Redis as a backend works stably well. During the development, a lot of cones were stuffed. But the learning process is such that sometimes it is inevitable. From this experience, I note that nodejs processes consume quite a lot of RAM (20MB at the start) and there may be leaks. For home automation, this is critical - because the script should work all the time, and if it grows over time more, then by a certain moment it can take up all the space. Therefore, you need to carefully write scripts, understand how the garbage collector works and keep everything under control.

Updated documentation can be found here .

In the next article I will talk about how using redis and bee-queue made a service for software accessories.

UPD: bobaoskit - accessories, dnssd and websocket

I would appreciate any feedback.