This patch includes a completely reworked client structure.
There is now only 1 client object, that loads the appropriate
drivers. The drivers are fake, aprsis and tcpkiss.
The TCPKISS client was written from scratch to avoid using asyncio.
Asyncion is nothing but a pain in the ass.
This patch adds the new PacketFilter class as a generic mechanism
for doing packet filtering during the packet processing phase of
recieving packets.
The packet phases are:
1. reception and stats collection
2. packet processing.
Each phase has a single thread for handling that phase.
Phase 1:
The ARPSDRXThread connects to the APRS client, and gets packets
from the client. Then it puts the packet through the Collector
for stats and tracking. Then the packet is put into the packet_queue.
Phase 2:
Packets are pulled from the packet_queue. Then packets are run
through the PacketFilter mechanism, then processed depending
on the command being run.
By default there is 1 loaded packet filter, which is the
DupePacketFilter which removes "duplicate" packets that aprsd has
already seen and processed within the configured time frame.
This PacketFilter mechanism allows an external extension or plugin
to add/remove packet filters at will depending on the function
of the extension or plugin. For example, this allows an extension
to get a packet and push the packet into an MQTT queue.
We had a bottleneck of pulling down packets as fast as possible,
which was caused by the time.sleep(1) call in the main RX
thread that used to be needed.
Lots of APRS messages don't have message ids in the packet. This
prevents packet.msgNo from having a value. so when we look the packet
up in the packet_list, the packet key has a None for a value for the
msgNo portion of the key. This results in every packet after the first
from that callsign that doesn't have a msg id, will be marked as a dupe
and will never be processed.
This patch checks to see if msgNo is None. If it is, then just
mark the packet as never being seen and pass it on the chain to
be processed. This also means that there should never be an Ack
sent for those packets that don't container a msgNo value.
This patch updates the aprsd client base class to report login succes
and any error string associated with a login failure. Also exposes
the login status so anyone using the client to check for login failure.
Update webchat, listen, server commands to check for initial login
failures and exit if the login failed. No reason to continue on
if the login fails.
This patch adds a new config option 'enable_sending_ack_packets', which
is by default set to True. This allows the admin to disable sending Ack
Packets for MessagePackets entirely.
this patch refactors the client, drivers and client factory
to use the same Protocol mechanism used by the stats collector
to construct the proper client to be used according to
the configuration
this patch adds the new PacketCollector class.
It's a single point for collecting information about
packets sent and recieved from the APRS client.
Basically instead of having the packetlist call the seen list
when we get a packet, we simply call the PacketCollector.rx(),
which in turn calls each registered PacketMonitor class.
This allows us to decouple the packet stats like classses inside
of APRSD. More importantly, it allows extensions to append their
own PacketMonitor class to the chain without modifying ARPSD.
This patch adds support for processing incoming packets that have
the 'new' acks embedded in messages called replyacks as described here:
http://www.aprs.org/aprs11/replyacks.txt
This allows the admin to set the number of packets to store
in the PacketList object for tracking. For apps like IRC,
we need to store lots more packets to detect dupes.
This patch removes the need for the RPC Server from aprsd.
APRSD Now saves it's stats to a pickled file on disk in the
aprsd.conf configured save_location. The web admin UI
will depickle that file to fetch the stats. The aprsd server
will periodically pickle and save the stats to disk.
The Logmonitor will not do a url post to the web admin ui
to send it the latest log entries.
Updated the healthcheck app to use the pickled stats file
and the fetch-stats command to make a url request to the running
admin ui to fetch the stats of the remote aprsd server.
This does some cleanup with the stats collector and
usage of the stats. The patch adds a new optional
param to the collector's collect() method to tell
the object to provide serializable stats. This is
used for the webchat app that sends stats to the
browser.
This patch implements a new stats collector paradigm
which uses the typing Protocol. Any object that wants to
supply stats to the collector has to implement the
aprsd.stats.collector.StatsProducer protocol, which at the
current time is implementing a stats() method on the object.
Then register the stats singleton producer with the collector by
calling collector.Collector().register_producer()
This only works if the stats producer object is a singleton.
This adds a layer between the client object and the
actual client instance, so we can reset the actual
client object instance upon failure of connection.
refactored all logging of packets.
Packet class now doesn't do logging.
the format of the packet log now lives on a single line with
colors.
Created a new packet property called human_info, which
creates a string for the payload of each packet type
in a human readable format.
TODO: need to create a config option to allow showing the
older style of multiline logs for packets.
This patch adds the new packet_dump_timeout config option, defaulting to
60 seconds. If the same packet matching the from, to, msgNo is RX'd
within that timeout the packet is considered a dupe and will be
dropped. Ack packets are not subject to dupe checking.
This patch rewrites the packet_list internally to be a dictionary
instead of a list for very fast lookups. This was needed to test for
duplicate packets already in the list.
This patch drops packets that have the same data and are < 60 seconds
in age from the last time we got the packet. On RF based clients
we can get dupes!!
Sometimes over KISS clients (RF), we can get duplicate packets
due to having many digipeters in range of the TNC that aprsd is
connected to. We will now filter out any dupe packets that aprsd
is still in the process of doing it's 3 acks.
This patch adjusts the backoff mechanism for aprs client
reconnect to a max backoff sleep of 5 seconds. This prevents
an exponential backoff when connection retrying.
This patch adds basic ratelimiting to sending out AckPackets
and non AckPackets. This provides a basic way to prevent
aprsd from sending out packets as fast as possible, which isn't
great for a bandwidth limited network.
This patch also adds some keepalive checks to all threads in the
threadslist as well as the network client objects (apris, kiss)
This patch is the initial conversion of the custom config
and config file yaml format to oslo_config's configuration mechanism.
The resulting config format is now an ini type file.
The default location is ~/.config/aprsd/aprsd.conf
This is a backwards incompatible change. You will have to rebuild
the config file and edit it.
Also any aprsd plugins can now define config options in code and
add an setup.cfg entry_point definition
oslo_config.opts =
foo.conf = foo.conf:list_opts
This patch decouples sending a message from the internals of
the Packet classes. This allows the rest of the code to use
Packet objects as type hints in methods to enforce Packets
in the plugins.
The send method was moved to a single place in the threads.tx.send()
This patch updates both the webchat and listen commands
to be able to use the new queue based packet RX processing.
APRSD used to start a thread for every packet received, now
packets are pushed into a queue for processing by other threads
already running.
