Mikrotik queue not showing any download speeds


#1

Followed the Sonar cast on setting up address lists and inline devices. Set up 3 address lists for our customers and the respective queues and mangle rules. The queue seems to work (?) but we see zero download data in the Mikrotik.

Should this concern me? Does not seem like normal behavior.


#2

Did a bit more research into this and more testing. If we have a customer who has a public IP address, the counters for marking “down” packets and connections tick up as I would expect. Members where we assign a private IP to their router (which essentially puts them in a double NAT since their home router also assigns privates), those members never show any traffic in the mangle rule on the “down” rules.

However, running a speed test behind the router works perfectly for users with either public or private IPs… The upload and download speeds are queued correctly regardless of their IP, it just that the “down” packet marks don’t ever register with the private IP users - and I’m not totally sure I understand why…


#4

This is not correct, the counters should be going up. You have done something incorrectly, but with the limited information it is impossible to tell what was done wrong. The queues look fine, the problem must be with the mangle rules.

Edit: Actually, I think I see the problem. You are marking the connection and then marking the packets based on the connection mark.

The connection mark applies to both directions. You are probably matching the connection mark alone to mark the packet, so packets going to and coming from the customer are both getting marked “up” and nothing is getting marked “down” - the result is that your customers are getting limited, but instead of getting limited on upload or download, they are limited based on the sum total of their upload plus their download (i.e. a customer in a 25Mbps queue could use 20Mbps down and 5Mbps up, or 15Mbps down and 10Mbps up, or 20Mbps up and 5Mbps down, whatever adds up to 25Mbps or less). I doubt this is what you intended.

The reason you may see a few packets being marked “down” for public IP customers is you are treating download and upload traffic not in terms of the direction of each packet (to or from the customer) but in terms of the direction of the first packet of the connection. Port scanners on the Internet used by hackers will send packets to random IPs that may belong to your customers, those create a connection with mark “down”. This however represents a minority of your customers traffic. Most of your customers download is being marked “up” instead of “down” due to the way you have things configured, any connection that is at first initiated by the customer is being marked “up” and therefore all packets both sent and received as part of that connection are being marked “up”.


#5

Just so you understand the exact sequence of what is happening:

  1. Your customer on the 25 service sends a packet to a web server requesting a web page
  2. The MikroTik applies the connection mark 25 up (mangle rule)
  3. Since the packet has connection mark 25 up, the MikroTik applies the packet mark 25 up (mangle rule)
  4. This traffic hits the upload queue and is counted as customer upload
  5. The web server sends a response with the webpage
  6. The MikroTik knows this is part of the same connection and the packets already therefore have the connection mark 25 up
  7. Since the packet already has a connection mark even before it hits the mangle table, the MikroTik does not apply a new connection mark via the mangle table
  8. Since the packet has connection mark 25 up, the MikroTik applies the packet mark 25 up (mangle rule)
  9. This packet hits the upload queue and is counted as customer upload

9 is obviously not what you want because the response to the customer containing the webpage contents should be counted as download instead of upload.

Then, what is happening when hackers are sending packets to customer devices?

  1. A hacker sends a packet to your customer on the 25 service with, say, a ping
  2. The MikroTik applies the connection mark 25 down (mangle rule)
  3. Since the packet has connection mark 25 down, the MikroTik applies the packet mark 25 down (mangle rule)
  4. This traffic hits the download queue and is counted as customer download
  5. The customer sends a ping reply to the hacker
  6. The MikroTik knows this is part of the same connection and the packets already therefore have the connection mark 25 down
  7. Since the packet already has a connection mark even before it hits the mangle table, the MikroTik does not apply a new connection mark via the mangle table
  8. Since the packet has connection mark 25 down, the MikroTik applies the packet mark 25 down (mangle rule)
  9. This packet (the ICMP reply from the customer to the outside hacker) hits the 25 down queue and is counted as customer download

Again, 9 is not what you want.

If you really want to use connection marks, don’t name them “25 up” or “25 down” because the connection mark is bidirectional, it should be just called “25”. You still need two connection mark rules, otherwise connections initiated from the outside world will not count towards bandwidth usage, but both rules would mark the connection “25”.

When you are marking the packet “25 up” or “25 down” you will need to match both the connection mark 25 and the direction the packet is going in, possibly based on the interface it arrives on, or the IP range it is coming from, etc., then apply the packet mark “25 up” or “25 down”. Then things should work correctly.


#6

Your explanation makes perfect sense to me. What is odd, however, is the up / down rules seem to work perfectly for my customers that have public IP’s assigned to their CPEs. Those with private IPs and are NATed behind us the up / down is not working correctly.

Since the rules are working on the CPE IP (public or private), I don’t understand totally why the Mikrotik would treat these rules differently for public or private IPs.

That being said, I did follow the Sonar instructions for setting this up but I see your description and the problem it creates. Worked for a speed test since that does upload then download but a bi-directional test would be 50% bandwidth due to this rule.

I am not totally following your description on how to do the packet mark (following the connection mark). Connection mark for both up and down does a “25” connection mark. Identical. I then need to take that connection mark and do a packet mark for the queue rules to work properly. That is where I should designate both a connection mark AND an interface match for the packet to get marked? Example: connection mark=25 in-interface=WAN then action=mark packet. Is that what you are thinking? The reverse is also possible since all my LAN traffic (for these rules at least) enters through one LAN port so I could match both.

Or is there a better way of doing this? I like having the Mikrotik controlled by Sonar. We are a small shop so the less programming that needs to happen when customers signup, change packages, go delinquent, etc the better.


#7

Quick tip,
Change your Queue-Tree parent from ‘global’ to whatever your appropriate interface is.

Not sure this will help any of your other issues but it will save you a boatload of time when you start getting more than 450Mbps throughput at high usage times and your CPU will start spiking over 100% with no real explanation. (we were using a CCR1036-8G-2S+ with tons of memory and 36 CPUs)

Extremely hard to diagnose because simply hitting apply on any mangle rule or que-tree rule will mellow your CPU for about 4 minutes then it will start doing it again. Took us three days of troubleshooting at peak congestion times to solve such a simple problem when we first started using SONAR.

Just thought I’d give you the heads up.


#8

Which interface? WAN or LAN? Assuming the LAN interface these customers are on?


#9

“25 download” would be your LAN facing interface
“25 upload” would be your WAN facing interface

“50 download” would be your LAN facing interface
“50 upload” would be your WAN facing interface

etc,


#10

Got it. Keep wanting to reverse the up and down…

From another off-line conversation, it sounds like changing the packet mark rules as well to have them reference the interface will potentially fix the issue we are having with private IPs not working correctly.

So, our proposed packet mark rules for upload would be:

add action=mark-packet chain=forward comment=“mark packet 25 up” in-interface=WAN connection-mark=shaping_connection_25 new-packet-mark=shaping_packet_25_up passthrough=no

And for download packet mar:

add action=mark-packet chain=forward comment=“mark packet 25 down” in-interface=LAN connection-mark=shaping_connection_25 new-packet-mark=shaping_packet_25_down passthrough=no


#11

Looks like the interface tagging is redundant in both the mangle rule and the queue tree…

So, sounds like the best plan is to:

  1. Do a connection mark with the same name (ie: “connection-mark-25”)
  2. Take that connection mark and do a packet mark with the same name (ie: “packet-mark-25”)
  3. Set up the queue tree to use interfaces (not global) off those packet marks

This solves the CCR CPU issue and should solve the upload / download rules problem. Or so I think…


#12

Tried to post this yesterday but it failed due to too many replies to one topic. I already emailed it to the OP but submitting it here too for completeness.

No - you may think it is working but it is not. Currently the way you have things set up, it will work like this:

Customers on Private IPs: 100% of upload traffic and 100% of download traffic will get filed as “upload” and placed in the “upload” queue

Customers on Public IPs:80% of upload traffic and 80% of download traffic will get filed as “upload” and placed in the “upload” queue. About 20% of upload traffic and 20% of download traffic will get filed as “download” and placed in the “download” queue.

Because you see traffic for public IP customers in both the upload and download queues, you think it is working. It is not.

What is happening is, you are currently labeling the entire connection (bi-directional conversation) “25 up” or “25 down” based on who initiates the conversation. If the far side (on the Internet) is the first to send a packet in the conversation with your customer, everything in the connection thereafter, regardless of which direction it is sent in, is marked as “down”. If the near side (the customer) is the first to send a packet in the conversation with someone on the Internet, everything in that connection is marked as “up”.

However, the customers on private IPs are not reachable from people on the Internet. Therefore it is impossible, due to the nature of private IPs, for someone on the Internet to take the first step in initiating a connection to the customer - the customer must always take the first step. Because your rules are marking the packets based on whoever takes the first step in the conversation, it is impossible for any packets to private IPs be marked “down”, and only some packets to public IPs will be marked “down”.

You understand correctly - that is exactly what I was thinking.

I agree with the other poster that interface attached queue trees are preferable to global-attached queue trees with CCR, due to a single parent being on a single core. This is not always possible to roll out depending on configuration (ex. if there is more than one customer-facing interface). I use simple queues for this.


#13

Try changing you download connection mark rule to postrouting chain. I had same trouble after following the videos but after i used the rule generator i noticed it was using postrouting for download connection marking. Seemed to fix my problems. All lab tests are running perfectly.


#14

@Michael_Ducharme What is the difference between using “global” vs a bridge in the queue tree in terms of CPU? Many of our MDU buildings use a 16 port router where we bridge the LAN ports together and then assign the queue tree to that bridge. I want to avoid simple queues since that can’t be managed automatically in Sonar.

@Justin_Terwee I am hoping we get a response from @simon on the correct use of pre vs post routing for the mangle rule. I’ve moved one of mine to postrouting to test for now.


#15

There is no right or wrong answer, it depends on which chain you need to capture the traffic in. I always find these types of charts helpful:

Mostly in MikroTik, it comes down to NAT.


#16

Queues are often set up in parent-child structures. What happens is any queues that share the same top-level parent also share the same CPU. All queues that have parent of global and their children share the same CPU. Queues with a different parent that is unrelated can use different CPUs.

If you have queues on one interface for traffic going in one direction, and queues on another interface going in another direction, those two sets of queues will use two CPUs, because they have a top-level parent that differs from the other queues.

You can actually use simple queues managed by Sonar, we do this. You make them with the customer subnets as the target, set up in parent child relationships:
parent simple queue - all customers, target is (customer subnet) or multiple subnets
then under that a child for each package you have
child 1 - package 1, set for the same subnet but to also match the packet mark that you mangled, choose PCQ queue types
child 2 - package 2, set for the same subnet but also to match the packet mark you mangled, choose PCQ queue types

etc.

We do it that way because then we can still use queue trees for regular QoS purposes, like ensuring VoIP can go through in event of congestion etc.

For prerouting vs postrouting, first you need to know what input, forward, and output are:

input = traffic going to the router as a final destination (ex. you log in to winbox or the web interface on the router)
forward = traffic being routed by the router to some other final destination (ex. traffic to customers)
output = traffic being created by the router that is going out (ex. you use the ping tool on the router to ping some other router)

prerouting includes both input and forward chains but not output, postrouting includes both forward and output chains but not input.


#17

@Michael_Ducharme great description and help. So, in digesting the pre vs post routing, it sounds like we (I) should be using postrouting rules since I am intending on shaping both download (forward) and upload (output) traffic from each customer… If I use the current prerouting rule, I am shaping forward (download) traffic but not output (upload).


#18

Not exactly - forward chain handles all traffic from the customer to the Internet or from the Internet to the customer. So the forward chain will handle both directions and if this accounts for all traffic then there is no issue. In that case there is absolutely no difference whether you use a prerouting or postrouting chain rule since both include the forward chain.

The situation gets a little bit more complicated if you have the router itself providing services to the customer. For instance you may have the customer DNS server set to the router, and in that case, the DNS traffic from the customer to that router would hit that router’s input chain and the response back would hit the router’s output chain, but neither would hit the forward chain. If all you had was a prerouting rule then that would cover input+forward, so the customer’s DNS requests to the router would be rate limited, but the replies from the router to the customer would not since they are on the output chain. The same would be the case in the event that you use the web proxy feature on the router, since the traffic in that case is considered to be between the customer and the router itself (as a proxy server) with the connection to the Internet being a separate connection. In such cases you would need some combination of a prerouting chain rule plus an output chain rule to cover all three (IFO) chains (or an input chain rule plus a postrouting rule, but that would be less common).