How many pollers


#1

We have 1500+ subs and probably close to 1700 devices monitored in sonar. We are finding the latency and jitter data to be consistently wrong across all subscribers.

Example: Jitter of 30-60ms and packet loss of 10% and random times (random but consistently across many different subscribers located in a variety of different locations on the network at that same exact time.)

The unfortunate part is that this makes our pollers data useless for tracking latency or jitter problems. seems that this leads us to believe that our poller is simply trying to do too much, or our config is lacking in some way:

ICMP_FORKS=10
SNMP_FORKS=500
SNMP_TIMEOUT=1
SNMP_RETRIES=2
ICMP_TIMEOUT=2
DEVICE_MAPPING_FREQUENCY=10

Basic Machine Spec:
Dual Xeon 3.2Ghz
16G Ram
Intel Chipset 1Gbps (Ethernet) link into the network at head end.

Should we be adding more pollers? Any insight or suggestions would be appreciated.


#2

We also have serious problems with poller optimization, would love to see more discussion of this.


#3

500 forks is probably too high, but it’s hard to tell without some experimentation. The quick test would be to deploy a 2nd poller and split the load and just see what happens. I would cut the SNMP forks down to 250 as well.


#4

We ran into similar issues, using a centralized poller for our customers and infrastructure. High latency, timing out, delays with the poller talking to Sonar etc etc.

The pollers we use are Gigabyte GB-BXBT-2807 barebones with 8GB Ram and a Solid stated HD approx. $180.

We now have 5 poller’s and are currently monitoring 9500 devices with little to no issues a work in progress.

Example:- our techs now see the radios have no jitter but the managed routers show some packet loss or jitter meaning the issue is identified and can be repaired.

Adding a 6th poller for the new Sites just installed.

Below is configuration of one of our poller’s we are testing:-
ICMP_FORKS=10
SNMP_FORKS=250
SNMP_TIMEOUT=2.5 (this is set higher for UBNT Air Fibers polling issues)
SNMP_RETRIES=1
ICMP_TIMEOUT=2
DEVICE_MAPPING_FREQUENCY=5

Some of the devices we monitor include:- Cambium, ePMP, Siklus, Mimosa, Some UBNT, Mikrotik’s, Ligowaves, Ceragons, Web Power Switch to name a few.

Our Poller’s are located across the network at strategic locations. The pollers can also be run of AC or DC 12 volts.

We are in Texas and not had any issues with Temperature as of yet (this will be the second summer), I will let you all know how this summer fairs out for the equipment.

Hope this is of some help, time to get some sleep.


#5

@leigh_Shuck Quick question for you. I decided to upgrade our Poller and bought a Gigabyte GB-BXBT-2807 with a 250GB SSD drive. I have been unable to load Ubuntu 16 onto it. I keep getting an error stating the install drive is read only (ERRNO 30 - Read only file system). Not that I am asking you to trouble shoot my install but did you make any changes to the BIOS or do anything special to get Ubuntu to install? I suppose I am dealing with a hardware issue but that is never my first thought on brand new hardware…


#6

@Chad_Wachs, I have only seen that behavior once before on a ubuntu install and it was caused by an existing partition on the harddrive.
I would try:

  • Throwing that harddrive in another machine and deleting all the partitioning off of it.
  • Try a different harddrive (if this works it probably has something to do with the bootloader written on that original drive)
  • Try changing the SATA Cable (assuming it has one)?
  • If it does the same thing on a different harddrive, try recreating your ubuntu 16 install drive (download a new iso and create it using different software ie. rufus or etcher - weird I know but I’ve seen some odd issues)
  • if none of these work try ubuntu 14 or 18 LTS. Maybe the v16 just doesn’t like your hardware.

Good Luck.


#7

Looks like it was a bad SSD drive. Brand new from Amazon and as soon as I replaced it, no problems.


#8

Sorry for the delay.

Quick question for you. I decided to upgrade our Poller and bought a Gigabyte GB-BXBT-2807 with a 250GB SSD drive. I have been unable to load Ubuntu 16 onto it. I keep getting an error stating the install drive is read only (ERRNO 30 - Read only file system).

Not come across that issue before and have included a link of the ssd that we have used.

I would check to see if there is a firmware fix for the SSD or it could have been a kernel change

**Not that I am asking you to trouble shoot my install but did you make any changes to the BIOS or do anything special to get Ubuntu to install? I suppose I am dealing with a hardware issue but that is never my first thought on brand new hardware…

We install Ubuntu 18.04 lts using an external usb DVD Drive, the only bios setting change is to set the boot power to on

I have included a installation process one of my engineers / programmer Lee Barton Made.

Ubuntu 18.04 LTS Installation Process & Poller Setup

Installation

Use appropriate media and start installation process

If there was a previous version of Ubuntu installed on the device, the installation for Ubuntu 18.04 will fail at the “Profile setup” stage with an error: “No such device or address: ‘/dev/sda2’”.

The partition of the previous Ubuntu installation must be deleted.

Once the install fails, select the option “Exit to shell”

Then enter the following commands:

	$ sudo fdisk -l
	$ sudo parted
	$ p
	$ rm <number of partition shown in the list>

	Continue the rm command for all partitions listed and then quit:

	$ q

	Verify all partitions are gone:

	$ sudo fdisk -l

Reboot and install again

For “Network Connections” stage:

Select interface

Select “Use a static IPv4 configuration”

Enter the following info:
Subnet: entertheip/subnet
Address: ENTER YOUR IP ADDRESS
Gateway: ENTER YOUR GATEWAY
Name Servers: ENTER YOUR DNS HERE
Search Domains: null

For “Profile setup”:
Enter the following info:
Your name: poll
Your server’s name: poll
Pick a username: YOURUSERNAME
Choose a password: YOURPASSWORD

Reboot, login, and apt-get update / apt-get upgrade

Install required software:

$ sudo apt-get install php-cli php-zip php-snmp php-sqlite3 php-bcmath php-mbstring php-dom git fping snmp redis-server monit ntp snmp-mibs-downloader

Sonar Poller Setup
Change SSH port and test:

Run command

$ sudo nano /etc/ssh/sshd_config

Remove the # from “Port 22” and change it to “Port 2222 (enter your port here 2222 is example)”

$ sudo service sshd restart

Login to putty and test that SSH works.

Setup Sonar Poller software:

$ git clone https://github.com/SonarSoftware/poller.git
$ cd poller
$ git describe --abbrev=0
$ git checkout tags/{TAG}
$ sudo php install.php
$ sudo -u sonarpoller cp /opt/poller/.env.example /opt/poller/.env
$ sudo -u sonarpoller nano -w /opt/poller/.env

SONAR_URI=gtek.sonar.software
API_KEY={generated key in Sonar}
SNMP_TIMEOUT=2.5 (change)
SNMP_RETRIES=1 (change)

$ sudo -u sonarpoller php /opt/poller/bin/getWork.php

Make sure the device is headless

Setup firewall rules:
$ sudo ufw enable
$ sudo ufw allow from to any port 2222 (this is the port you changed earlier)
$ sudo ufw allow from yourmanagementipaddress to any port 2222
$ sudo ufw allow from anothermanagementip to any port 2222
$ sudo ufw allow from yoursonarinstanceip to any port 2222
$ sudo ufw status

Re-configure static IP for location:
$ sudo nano /etc/netplan/50-cloud-init.yaml

Possible Problems
If there was a previous version of Ubuntu installed on the device, the installation for Ubuntu 18.04 will fail at the “Profile setup” stage with an error: “No such device or address: ‘/dev/sda2’”. See Installation part 2.

For the GIGABYTE BRIX, if Ubuntu freezes on “Starting reboot…” Or “Stopped LVM2 metadata daemon” upon reboot or right after Ubuntu installation:
Boot into the BIOS
Go to “Chipset”
Under the option “OS Selection” select “Windows 8.X”
Usually it is a lower Windows version selected e.g. “Windows 7”
Save & Exit


#9

Glad to see it was just the HD issue