58

MIKROTIK with SQUID/ZPH: how to bypass Squid Cache HIT object with Queues Tree in RouterOS 3+

aacable@hotmail.com

Using Mikrotik, we can redirect HTTP traffic to SQUID proxy Server, We can also control user bandwidth, but its a good idea to deliver the already cached content to user at full lan speed, that’s why we setup cache server for, to save bandwidth and have fast browsing experience , right :p , So how can we do it in mikrotik that cache content should be delivered to users at unlimited speed, no queue on cache content. Here we go.

By using ZPH directives , we will mark cache content, so that it can later pick by Mikrotik.

Basic requirement is that Squid  must be running in transparent mode, can be done via iptables and squid.conf directives.
I am using UBUNTU squid 2.7 , (in ubuntu , apt-get install squid will install squid 2.7 by default which is gr8 for our work)
Add these lines in SQUID.CONF

[sourcecode]#==============
#ZPH Syed Jahanzaib aacable@hotmail.com
#=======================
tcp_outgoing_tos 0×30 lanuser
zph_mode tos
zph_local 0×30
zph_parent 0
zph_option 136
#[lanuser is ACL for local network][/sourcecode]

That’s it for SQUID, Now moving on to Mikrotik box ,
Add following rules,

# Marking packets with DSCP (for MT 3+) for cache hit content coming from SQUID Proxy

[sourcecode]/ip firewall mangle add action=mark-packet chain=prerouting disabled=no dscp=12 new-packet-mark=proxy-hit passthrough=no comment="Mark Cache Hit Packets / aacable@hotmail.com"

/queue tree add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 max-limit=0 name=pmark packet-mark=proxy-hit parent=global-out priority=8 queue=default[/sourcecode]

Now every packet which is marked by SQUID CACHE_HIT, will be delivered to user at Full lan speed, rest of traffic will be restricted by user Queue.

 

TROUBLESHOOTING:

the above config is fully tested with UBUNTU SQUID 2.7 and FEDORA 10 with LUSCA

Make sure your squid is marking TOS for cache hit packets. You can check it via TCPDUMP

__________________________________________________________
tcpdump -vni eth0 | grep ‘tos 0×30′
(eht0 = LAN connected interface)

Can you see something like

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
20:25:07.961722 IP (tos 0×30, ttl 64, id 45167, offset 0, flags [DF], proto TCP (6), length 409)
20:25:07.962059 IP (tos 0×30, ttl 64, id 45168, offset 0, flags [DF], proto TCP (6), length 1480)
192 packets captured
195 packets received by filter
0 packets dropped by kernel
_________________________________

 

Regard’s
SYED JAHANZAIB

 

Filed in: Linux Related, Mikrotik Related Tags: , , , , , , , ,

Get Updates

Share This Post

Recent Posts

58 Responses to "MIKROTIK with SQUID/ZPH: how to bypass Squid Cache HIT object with Queues Tree in RouterOS 3+"

  1. whielyx says:

    how about proxy external with only one ethernet…???

    • Pinochio~:) says:

      Proxy can work with one ethernet too.
      But its better to have separate WAN for proxy. anyhow one ethernet will work , just set its default route / dns pointing to Mikrotik proxy interface.

      • whielyx says:

        nice…. This the best blog I have ever met who discuss mikrotik and external proxy.
        why use routing? why not use NAT to create a transparent proxy ..?

      • johan says:

        hi i like your work , i just got one problem i used your “yourtube” transparent proxy settings which you explained, but my problem is it works if i test it on squid x86 pc itself with ( proxy lan 127.0.0.1:8080) but when i put it in mikrotik it dont work ive tried all tricks already eg:

        ip – firewal l- nat – dstnat -protcol tcp6 – port80 -dst-nat8080 – dst-nat ip (squid ip)
        webproxy – parent ip & port = squid ip & port

        if you could help me out please

        • # Without knowing your network topologies , I can comment.
          # SQUID must be running in transparent mode via squid.conf and iptables directives.
          # Mikrotik may have 3 interfaces, one for lan, second for squid proxy, third for ISP/WAN. All must be running on different subnets.

          • Johan says:

            My Mikrotik:

            433UAH as follows:

            LAN 1 = master – 192.168.88.1

            LAN 2 = ADSL – 192.168.1.251 gateway – 192.168.1.1
            DNS – servers: 192.168.1.1
            allow-remote-requests: yes
            max-udp-packet-size: 512
            cache-size: 4096KiB
            cache-max-ttl: 1w
            cache-used: 538KiB

            LAN 3 = HOME PRIVATE INTERNET (LEAVE AS IS) with IP = 192.168.87.0/24

            Firewall NAT:
            0 ;;; Redirect Rediect Web-Proxy
            chain=dstnat action=redirect to-ports=8080 protocol=tcp dst-port=80

            1 ;;; Masquerade ADSL Network
            chain=srcnat action=masquerade out-interface=ADSL

            …………………………………………………………………………………………………………………………….

            My LAN 1 connects to 750G Router where all public clients connect to.

            My squid is setup on a X86 PC with default settings again as it was corrupt, i reconfigured ,
            reinstalled default “squid.conf”, ive plugged it directly in adsl with DHCP (192.168.1.1/24)
            and tested it in Mozilla Firefox with proxy IP & Port ( 127.0.0.1:8080 & 127.0.0.1:3128) results
            is it works.
            …………………………………………………………………………………………………………………………….

            What i need is for it to work through my Mikrotik router as a HTTP Proxy or Transparent & my clients PC’s to auto detect it without configurations

            Thank You

          • # There are couple of ways to accomplish this.

            # You can use dst-nat rule to redirect all http request to SQUID box.

            # You can mark HTTP packets and then route them to SQUID box, This is preferred way and squid can see users original source ip, not the mikrotik ip, and you can log users original ip address in squid logs. Which is sometimes requires for management purposes.

            Use the below links.

            http://aacable.wordpress.com/2011/12/30/howto-add-squid-proxy-server-with-mikrotik-short-reference-guide/

            http://aacable.wordpress.com/2011/07/21/mikrotik-howto-redirect-http-traffic-to-squid-with-original-source-client-ip/

  2. Pinochio~:) says:

    If you use NAT, SQUID will see only Mikrotik IP. so you cant log user ip in access.log

    Routing is used so that Proxy can log user ip for record purpose. and also it prevents double NATing, first at mikrotik , second on proxy.

  3. whielyx says:

    ok. so far, I use a dst-nat to make transparent proxy and I can still see the access.log contains client source ip.

    /ip firewall nat
    add action=dst-nat chain=dstnat comment=squid disabled=no dst-address=![squid-network] dst-port=80,81,8080,3128 in-interface=ether3-hotspot protocol=tcp src-address=[lan-network] to-addresses=[ip-squid] to-ports=[port-squid]

    cmiiw,
    best regards.

  4. Pinochio~:) says:

    Traffic shaping can be easily done via Mikrotik.

  5. whielyx says:

    thanks sir…

  6. SaFi says:

    Asalam Aliukom

    I have the following network topology (in summary)

    {internet}
    |
    [firewall & squid cache@pfsense]—[MT_AP]~[MT_STA with wds]—[hotspot@MT_router]—[clients]
    |
    |
    [radius@ubuntu]

    pfsense: LAN = 172.31.224.1/24 with transparent proxy cache
    MikroTik: WAN’s = dhcp client with 172.31.224.x/24

    this is the squid.conf on pfsense(freebsd)

    Code:
    http_port 172.31.224.1:3128 transparent
    http_port 127.0.0.1:80 transparent
    icp_port 0

    pid_filename /var/run/squid.pid
    cache_effective_user proxy
    cache_effective_group proxy
    error_directory /usr/local/etc/squid/errors/English
    icon_directory /usr/local/etc/squid/icons
    visible_hostname net4u1.com
    cache_mgr admin@net4u1.com
    access_log /var/squid/logs/access.log
    cache_log /var/squid/logs/cache.log
    cache_store_log none
    logfile_rotate 30
    shutdown_lifetime 0 seconds
    # Allow local network(s) on interface(s)
    acl localnet src 172.31.224.0/255.255.255.0
    forwarded_for transparent
    via off
    httpd_suppress_version_string on
    uri_whitespace strip
    dns_nameservers 127.0.0.1

    cache_mem 2048 MB
    maximum_object_size_in_memory 128 KB
    memory_replacement_policy heap GDSF
    cache_replacement_policy heap LFUDA
    cache_dir coss /var/squid/coss 8000 max-size=419840 block-size=512
    cache_dir aufs /var/squid/cache 320000 128 256 min-size=419840
    minimum_object_size 0 KB
    maximum_object_size 399 MB
    offline_mode off
    cache_swap_low 90
    cache_swap_high 95

    # No redirector configured

    # Setup some default acls
    acl all src 0.0.0.0/0.0.0.0
    acl localhost src 127.0.0.1/255.255.255.255
    acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 1337 3128 1025-65535
    acl sslports port 443 563 1337
    acl manager proto cache_object
    acl purge method PURGE
    acl connect method CONNECT
    acl partialcontent_req req_header Range .*
    acl mallware url_regex “/usr/local/etc/squid/mallware.url”
    acl conficker url_regex “/usr/local/etc/squid/conficker.url”
    #acl dynamic urlpath_regex cgi-bin ?
    #include /usr/local/etc/squid/include.conf
    include /usr/local/etc/squid/tunning.conf
    #cache deny dynamic
    http_access allow manager localhost
    http_access deny mallware
    http_access deny conficker
    http_access deny manager
    http_access allow purge localhost
    http_access deny purge
    http_access deny !safeports
    http_access deny CONNECT !sslports

    # Always allow localhost connections
    http_access allow localhost

    quick_abort_min 32 KB
    quick_abort_max 128 KB
    quick_abort_pct 75
    range_offset_limit 0 MB
    request_body_max_size 0 allow all
    reply_body_max_size 0 deny all

    delay_pools 1
    delay_class 1 2
    delay_parameters 1 -1/-1 -1/-1
    delay_initial_bucket_level 100
    # Throttle extensions matched in the url
    acl throttle_exts urlpath_regex -i “/var/squid/acl/throttle_exts.acl”
    delay_access 1 allow throttle_exts
    delay_access 1 deny all

    # Custom options
    zph_mode tos
    zph_local 0×30
    zph_parent 0
    zph_option 136

    # Allow local network(s) on interface(s)
    http_access allow localnet

    # Default block all to be sure
    http_access deny all

    and add this to mikrotik routers

    Code:
    /ip firewall mangle
    add chain=prerouting action=mark-packet dscp=12 new-packet-mark=proxy-hit passthrough=no
    /queue tree
    add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 max-limit=0 /
    name=pmark packet-mark=proxy-hit parent=global-out priority=1 queue=default

    I try many tests but no bypass cache hit it’s limit with hotspot client profile speed !
    and there is something wired, that is while monitoring the mangle rule I notice that the counter never increased and stay on 0 but while I change dscp value to 48 it’s began to count (that’s mean marking packets) I know that 30 in hex its equal to 48 in decimal (tos) and dscp its quarter of tos value 12

    so please I need your help me to make this situation work for me

    TIA
    SaFi

    • Pinochio~:) says:

      Q1# Make sure your squid is marking TOS for cache hit packets. You can check it via TCPDUMP
      What is the result of following command at Proxy server?
      __________________________________________________________
      # tcpdump -vni eth0 | grep ‘tos 0×30′
      (eht0 = LAN connected interface)

      Can you see somethign like

      tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
      20:25:07.961722 IP (tos 0×30, ttl 64, id 45167, offset 0, flags [DF], proto TCP (6), length 409)
      20:25:07.962059 IP (tos 0×30, ttl 64, id 45168, offset 0, flags [DF], proto TCP (6), length 1480)
      192 packets captured
      195 packets received by filter
      0 packets dropped by kernel
      __________________________________________________________

      Q2# What is your mikrotik version?

      Q3# What is your SQUID Version?

      Suggestion: Use Ubuntu instead, as it is being Labeled ‘Linux for human being’ :) Label is self explanatory.

      • SaFi says:

        Thank you Syed for your quick reply

        Answer #1: I run this command tcpdump -nnvvi rl0 | grep ‘tos 0x’ and I get lines like this
        21:02:33.760598 IP (tos 0×0, ttl 50, id 39136, offset 0, flags [DF], proto TCP (6), length 52)
        it’ seem that the marking doesn’t work …right?

        Answer #2 : MikroTik ver 5.5

        Answer #3 : My cache server is a package called “lusca-cache” from pfsense packages and it’s such version of squid 2.7.9 customized by chudy.fernandez :http://code.google.com/p/pfsense-cacheboy/wiki/Pfsense_Lusca

        salam

  7. Pinochio~:) says:

    # Version 2.7.STABLE6 is the default in ubuntu base installation and have builtin support for ZPH

    # If squid isn’t marking TOS packets, then mikrotik can’t recognize the packets. First diagnose it. I guess the syntax for ZPH in your squid.conf isn’t right. Correct it. Read my article @
    http://aacable.wordpress.com/2011/07/21/mikrotik-with-squidzph-unlimited-speed-for-cache-content-traffic/

    # to get working squid.conf , you can view my blog at http://aacable.wordpress.com/2011/06/01/working-squid-conf-example-fil/

    # Try Ubuntu, In past I have used many flavors of linux, But the most suited version I found is Ubuntu (10.4 Desktop Version, as you will get nice GUI for management, It can also act as a server base) , Try it on a test box. I have made very simple guides for ubuntu squid + zph + mikrotik rules. Its always succeed

  8. mr yaseen ansari says:

    asalam o alikum
    sir
    i have facing in some problems
    i had configure your articals but cache is to running perfectly and also flv for you tube isn,t run
    tell me some ideas how can i do please sir help me

  9. zaib bhai can we do cache youtube video’s on mikrotik without squid proxy??? how can we do this if possible?

  10. Miles says:

    Syed

    I follow yours instruction and boom squid working prefect bypassing the queue limit’s.
    I have little problem with hotspot mirkotik.
    When I redirect traffic via ubuntu server the login page won’t come up.
    My connection is client’s——–mikrotik hotpot—–ubuntu server—– intenret.
    In ubutu I use NAT and REDIRECT rules without Hotpot work prefect as soon as I turn on hotspot on interfaces the users won’t be able to surf on the internet.
    Any idea.
    Thanks Miles
    What else I can say to you, God bless you ..

  11. waqar hameed says:

    a/salam sir ma hotspot use kar rha hn ek probelm a rhe ha cache full speed ma nai deliver ho rhen wo as a bradwith use ho rhe hn. queue ma global in ma mention ho rhen hn global out ma nai deliver ho rhn plz help me out

  12. Hasan says:

    aoa brother
    I need to ask something if you are kind enough to answer, In my university there is squid 2.7 stable 21 wirelessproxy is installed on every router. Now I can not download anything using utorrent. Could you please help me out in this situation? Thanks

  13. Ma'el says:

    very useful information.
    i have to try this one….

    thanks pak syed

  14. Ma'el says:

    oops…im forgot to ask

    where should i put these command in squid.conf
    or i can put anywhere? :)

    #==============
    #ZPH Syed Jahanzaib aacable@hotmail.com
    #=======================
    tcp_outgoing_tos 0×30 lanuser
    zph_mode tos
    zph_local 0×30
    zph_parent 0
    zph_option 136
    #[lanuser is ACL for local network]

  15. karemm says:

    hey guys thanks for you all and any one can tell me how can i limit the output bandwidth to users they got now full band from the cache i need to limit that band coz im using wireless link and i don’t want to get it full capacity when users downloading some file from cache and i already limit the real internet band

  16. achmad says:

    when i try to test with tcpdump-VNI eth0 | grep ‘tos 0 × 30′
    grep: 0 * 30: ni Such file or directory
    tcp_outgoing_tos 0 × 30 lanuser
    zph_mode tos
    zph_local 0 × 30
    zph_parent 0
    zph_option 136

  17. Azma Yogi says:

    thanks for this great tutorial..

    is it possible if we marking packet by tcp_outgoing_tos that contains .exe packets and catch it by dscp mikrotik?

    i’ve tried but i failed. :(

    thanks in advanced…

  18. smn4all says:

    Asslam-o-alaikum bhai ! it is possible given bellow and how to

    3 DSL —-> Mt 5.18 —-> Squid 2.7 —–> Hotspot Mt 5.18——> clients

  19. muhammad azam says:

    i got error in freebsd+squid “kernel: negative sbsize uid = o”
    at that moment browsing getting slower.

    please help me out

  20. Patrick says:

    Hi,
    Pretty good doc!
    I’m almost there… but my pppoe users and queue are on another router, my network is routed(ospf) How to pass dscp to that router??
    Thanks!
    Patrick

  21. adda says:

    hello,
    Is it possible to use the internal mikrotik web cache and configure as parent a squid proxy, if an object is found from the squid cache and marked with th correct dscp value, how to passe this object at the lan speed with out limitation
    Regards

  22. Clive says:

    HI, can somebody help me with mikrotik config, i have an external cache device which is Appliansys Cachebox220, i want to connect it via the Mikrotik router, please give me the Mikrotik side configurations. thanks

  23. Ahmed Adel says:

    Dear Sir
    Is it differ if i put Squid between Mikrotik and Internet Modem , or it must be connected like the topology you figure above
    because i try to cache youtube videos but i fail

  24. Ahmed Adel says:

    Dear Sir

    i configure squid as your articles but it cannot save youtube videos , any suggestion please ?

  25. Vijay says:

    /ip firewall mangle add action=mark-packet chain=prerouting disabled=no dscp=12 new-packet-mark=proxy-hit passthrough=no comment=”Mark Cache Hit Packets / aacable@hotmail.com

    /queue tree add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 max-limit=0 name=pmark packet-mark=proxy-hit parent=global-out priority=8 queue=default
    ——————————————-
    Hi Syed,
    I found difficulty creating the above second rule. Since i am using the Mikrotik CCR-1036, which is the high end router. But i didn’t find the option “global-out” in the queue tree.
    Please do suggest me.

  26. Oluseyi says:

    Hi,
    i currently run a network on RB1100 and would like to know if the following is possible.
    I am currently trying out the above but would like to know the following

    I inherited a network and i have to allocate bandwidth pools for the clients and i tried using usermanager to allocate to each client a certain amount of bandwidth but i was not successful.

    What i discovered is that you have to allocate either to the up or downstream a certain amount, what i am looking for is a way to apportion to each client a certain amount of data bandwidth.

    I would like to be able to for instance allocate to a client for instance 3Gb worth of data and whether the client uploads or downloads i would like for it to be deducted from the 3gb data and also if the 3gb is not all consumed within 30days then the account should expire.

    Basically what i want to do here is to do data-capping.

    I would appreciate all the help i can get.

  27. Oluseyi says:

    Hi,
    Please in making this ubuntu proxy work do i enable webproxy in mikrotik or not?
    And also how what ipsettings should i use for the ubuntu?
    Thank you mightly

  28. Oluseyi says:

    hi,

    please let me explain me own network scenario to you.

    I have 2 sets of mikrotik the main one I use is a X86 and the backup is a RB1100, now the X86 has
    only 2 sets of network cards and can’t take any more than that. What do I do here?

    Should I use a cross cable and connect the X86 to the Ubuntu and then issue that nic on the Ubuntu 172.16.10.3 as I already have on the x86 and also on the rb1100 one other the ether being 172.16.10.2 as my gateway which is another machine is 172.16.10.1.

    I would really appreciate your step by step explanation here.

    Thanks
    My network is as follows:

    Gateway>>Mikrotik(X86 or RB1100)>> Switch>> Users

  29. OJ says:

    I tried this, and can see the mangle traffic matching, however this does not affect user download as users were still downloading HIT files within their respective bandwidth restriction. Is there anything i may be doing wrong?

    Thanks

  30. Oluseyi says:

    hello sir
    you have not answered my last two questions

  31. Oluseyi says:

    Hi,
    please what have i done wrong as you don’t want to answer me?
    please i really do need your help as i have just inherited a network as part of a new job and part of the conditions is to increase the speed of the network.
    I will appreciate all help possible.
    Thank you

  32. Ahmed Bello says:

    amin wa’alaikum salam
    Good day, I have just been employed in a company and I don’t know much about Mikrotik. Now I have an issue which invariably is a test as they have said that I have to ensure that I build a cache outside the Mikrotik and that if that improves the speed of browsing tremendously that means I have the job.
    Let me describe the network to you:
    Modem>Gateway(x86 PC)> Mikrotik (Rb1100 or x86)> switch>antennae

    I would like to incorporate both the cache option to do both web-pages and also videos, I would like to do both on the same machine. Also I would appreciate it if you could please give me detailed steps on what to do on each; both on the Linux package and the Mikrotik package. The mikrotik version is 5.22 and the Ubuntu version is 13.04 64 bits.

Leave a Reply

Submit Comment

© 2014 Syed Jahanzaib. All rights reserved.
Powered By Wifitech.