February 2024 Autobuild Statistics

ZBT-Z8102AX is the top of the list this month, but with a caveat – there’s a lot of fixing going on with this router. The manufacturer has released an early version and a late version, with very different bootloaders. The image currently on the site works with the early bootloader. Dairyman has been working with a few folks this month trying to get an image working with the second bootloader, which has a completely different flash layout.

Autobuild report for 2024-02
Compiled Fri 01 Mar 2024 08:00:01 AM UTC
Includes downloads from 2024-02-01 to 2024-02-29

Total router system downloads: 1598 by 845 unique users.

Top performing routers:
  117  ZBT-Z8102AX
   61  Arcadyan-AW1000
   42  ZBT-WG1608-16
   42  Beeline-SmartBox_TurboPlus
   41  x86-64
   33  ZBT-WG1608-32M
   32  Alwaylink-MK01K21
   30  RaspberryPi-4-SD
   29  ZBT-WE826T
   28  ZBT-Z8102AX-V2
   25  ZBT-WG3526
   24  Mikrotik-RBM33G
   22  ZBT-WE2802D
   20  Beeline-SmartBox_TurboModPlus
   18  ZBT-WR8305RT

Downloads by build system:
   78  AB_1800
  382  AB19
  588  AB21
  290  AB22
   64  AB23
   10  AB_BPIR4
   18  ABGL
   20  AB_RPI5
  148  ABSM

Routers with no downloads:
   GliNet_MT6000
   Linksys-EA9500v1
   Mikrotik-RBSXTR
   Mikrotik-RBwAPR-2nD
   Netgear-R6220
   Netgear-R7500-V1
   Netgear-R8000
   SamKnows-WhiteBox-V8
   TP-Link-Archer-WR1045ND
   TP-Link-WDR4300
   TP-Link-WDR4310
   TP-Link-WR703N-16meg
   UniElec-U7621-06

Lowest performing routers with downloads:
    1  Netgear-R6300-V2
    1  NanoPi-NEO-Plus2
    1  MYNET-N600
    1  Mikrotik-RBD52U
    1  Mikrotik-RB922UAGS
    1  Mikrotik-RB912UAG-5HPnD
    1  Mikrotik-RB750GR3
    1  Mikrotik-LHG-HB
    1  Mikrotik-LHG-2nD
    1  Linksys-EA8300
    1  Lenovo-Y1-V1
    1  IRZ-MT00
    1  GL-S1300
    1  Cudy-WR1300
    1  Asus-RT-AC56U

Highest individual downloaders (hashed):
  388  0d58b5d9c89b
   15  df443cf6a0cc
   15  805c760b077d
   14  801a97a16785
    8  76c38d950909
    8  47d593a83839
    8  44559ae5f771
    8  06135a944cba
    7  b13dcfcce625
    7  81feb3df44a9

Downloads by router name:
   10  Alfa-R36A
    3  Alfa-R36M-E4G
    3  Alfa-TUBE-E4G
    3  ALIX-2D13
   32  Alwaylink-MK01K21
    3  APU2C4
   61  Arcadyan-AW1000
    3  Archer-A7-V5
    3  Archer-C20-V1
    5  Archer-C2600
    3  Archer-C2-V1
    2  Archer-C5-V1
   14  Archer-C6Uv1
    7  Archer-C7-V2
    4  Archer-C7-V4
   12  Archer-C7-V5
    9  Asus-RT-AC51U
    1  Asus-RT-AC56U
   15  Asus-RT-AC58U
    4  Asus-RT-AC65P
   11  Asus-RT-AC68U
    3  Asus-RT-AC85P
    6  Asus-RT-AC87U
   10  Asus-RT-AX53U
    4  Asus-RT-N14U
    7  Asus-RT-N16
    9  Asus-RT-N56U
   13  Asus-RT-N66U
   10  Banana-Pi4
   11  BananaPi-BPi-R3
    6  Beeline-SmartBox_Pro
   20  Beeline-SmartBox_TurboModPlus
   42  Beeline-SmartBox_TurboPlus
    3  BT-HomeHub-5
    3  CheckPoint-L-50
    2  Comfast-CF-E3
    3  Comfast-CF-E5
    2  Compex-WPJ428
    1  Cudy-WR1300
    6  DIR505A1
    3  D-Link-DGL-5500-A1
    3  D-Link-DIR-825-C1
    3  D-Link-DIR-835-A1
    2  Dlink-DIR853-R1
    3  D-Link-DIR-860L-B1
    2  D-Link-DIR-882-A1
    3  D-Link-DIR-882-R1
    6  DualQ-H721-1907
    3  DualQ-H721-2102
    3  DualQ-H721-2203
    2  DualQ-H721AX
    4  Dynalink-DL-WRX36
    3  GL-AP1300
    6  GL-AR150
    6  GL-AR300M-16
    2  GL-AR300M-Lite
    8  GL-AR300M-nand
    2  GL-AR750
    3  GL-B1300
    9  GL-E750
    7  GLi-AR750S
    6  GL-iNet-AX1800
   11  GL-iNet-AXT1800
    3  Gl.iNet-GL6416
    8  Gl.iNet-MT1300
   12  GliNet_MT3000
    7  Gl.iNet-MV1000
    6  GliNet_X3000
    3  GL-MIFI
    4  GL-MT300A
    3  GL-MT300N-V1
   12  GL-MT300N-V2
    4  Globalscale-MOCHAbin
    1  GL-S1300
    4  GL-X1200
    8  GL-X750
    2  GL-XE300
   13  HiLink-HLK-7621a
    5  HiWiFi-HC5962
    6  Huasifei-WS1208V1
    4  Huasifei-WS1208v2-32
    4  Huasifei-WS1208V2
    3  Huasifei-WS1218
    6  Huasifei-WS1688AX-16
    4  Huasifei-WS1688AX-32
   13  Huasifei-WS1698AX
    3  Huasifei-WS7915AX
    4  Huastlink-HC851-HC841
    3  Huastlink-HC952
    8  Huawei-HG553
    7  Huawei-HG556a-A
    7  Huawei-HG556a-B
   11  Huawei-HG556a-C
    1  IRZ-MT00
    2  Lenovo-Y1S-V1
    1  Lenovo-Y1-V1
    4  Linksys-E4200-V2
    7  Linksys-E8450-UBI
    2  Linksys-EA3500
    2  Linksys-EA4500-V1
    3  Linksys-EA6350v3
    4  Linksys-EA7300
    3  Linksys-EA7500v1
    4  Linksys-EA7500v2
    3  Linksys-EA8100v1
    1  Linksys-EA8300
    2  Linksys-EA8500
    5  Linksys-MR8300
    4  Linksys-WRT1200AC
    8  Linksys-WRT1900ACS
    4  Linksys-WRT1900AC-V1
    4  Linksys-WRT1900AC-V2
   14  Linksys-WRT3200ACM
    9  Linksys-WRT32x
   17  M01K21
    3  Mediatek-MT7628-EVB
    3  Mercury-MW4530R
    1  Mikrotik-LHG-2nD
    1  Mikrotik-LHG-HB
    3  Mikrotik-RB433AH
    1  Mikrotik-RB750GR3
    2  Mikrotik-RB760iGS
    2  Mikrotik-RB912UAG-2HPnD
    1  Mikrotik-RB912UAG-5HPnD
    1  Mikrotik-RB922UAGS
    2  Mikrotik-RB951Ui-2nD
    1  Mikrotik-RBD52U
    8  Mikrotik-RBM11G
   24  Mikrotik-RBM33G
    3  Mofi-4500
    1  MYNET-N600
    4  MYNET-N750
    1  NanoPi-NEO-Plus2
    2  NanoPi-R2S
    5  NanoPi-R4S
    1  Netgear-R6300-V2
    1  Netgear-R7000
    1  Netgear-R7500-V2
    3  Netgear-R7800
    2  Netgear-WNDR3700-V1
   10  Netgear-WNDR3700-V2
    1  Netgear-WNDR3700-V4
    2  Netgear-WNDR3800
    5  Netgear-WNDR4300v1
    9  Newifi-D2
    3  Nexx-WT3020-16
    5  Nexx-WT3020-8
    3  O2-Box-6431
    6  OrangePi-PC
    3  OrangePi-Plus
    6  OrangePi-Zero-Plus
    5  P2W-R619AC
    9  RaspberryPi-1
   14  RaspberryPi-2
    8  RaspberryPi-3-SD
    1  RaspberryPi-3-USB
   30  RaspberryPi-4-SD
    9  RaspberryPi-4-USB
   15  RaspberryPi-5-SD
    5  RaspberryPi-5-USB
    6  RaspberryPi-CM4-SD
    1  RaspberryPi-CM4-USB
    1  Sanlinking-D240
    2  Seagate-Dockstar
   18  SmartBox_Giga
   13  SmartBox_ModGiga
    1  TP-Link-Archer-C59v1
    3  TP-Link-Archer-C59v2
    3  TP-Link-Archer-C6-V2
    4  TP-Link-Archer-C7v1
   12  TP-Link-MR3020v3
   12  TP-Link-MR3420v5
    4  TP-Link-WDR3500
    3  TP-Link-WDR3600
    1  TP-Link-WDR4900
    2  TP-Link-WR1043NDv2
    3  TP-Link-WR1043NDv3
    1  TP-Link-WR1043ND-V4
    1  TP-Link-WR1043N-V5
    4  TP-Link-WR842-V3
    1  TP-Link-WR902ACv1
    5  TP-Link-WR902ACv3
    6  TP-Link-WR942N
    3  Turris-Omnia-2102
    3  Turris-Omnia-2305
    1  UniElec-U7621-01
    1  UniElec-U7628-01
    3  WiFiX-NEXQ6GO
   41  x86-64
    4  x86-Generic
   13  x86-UEFI-64
   12  x86-UEFI-Generic
   10  Xiaomi-Mifi3G
    7  Xiaomi-Mifi3Pro
   11  Xiaomi-Mifi-Mini
    1  YouHua-WR1200JS
    6  YOUKU-YK1
   12  ZBT-WE1026-5G
   10  ZBT-WE1326
   22  ZBT-WE2802D
   13  ZBT-WE826-Q
   29  ZBT-WE826T
   11  ZBT-WE826-T-32meg
   10  ZBT-WE826WD
    8  ZBT-WG1602-16
    7  ZBT-WG1602-32
    3  ZBT-WG1602-32M-2102
   42  ZBT-WG1608-16
   33  ZBT-WG1608-32M
    4  ZBT-WG209
    3  ZBT-WG259
   25  ZBT-WG3526
    3  ZBT-WG827
   18  ZBT-WR8305RT
    9  ZBT-Z2101AX
    5  ZBT-Z2105AX
  117  ZBT-Z8102AX
   28  ZBT-Z8102AX-V2
    3  ZyXEL-Keenetic-Extra-II
   10  ZyXEL-Keenetic-Omni
    1  ZyXEL-NBG6817
    3  ZyZEL-NR7101

January 2024 Autobuild Statistics

The autobuild report is ready for last month. The new AW1000 has made it up to most popular router of the month already! We’re not seeing a lot of issue reports on the forum yet either, so with that many downloads, I have to think it’s working pretty well for folks so far.

Autobuild report for 2024-01
Compiled Fri 02 Feb 2024 01:07:32 PM UTC
Includes downloads from 2024-01-01 to 2024-01-31

Total router system downloads: 2315 by 1292 unique users.

Top performing routers:
  206  Arcadyan-AW1000
  127  ZBT-Z8102AX
   80  ZBT-WG3526
   71  x86-64
   57  Linksys-WRT1900ACS
   41  Alwaylink-MK01K21
   37  ZBT-WG1608-32M
   36  RaspberryPi-4-SD
   35  ZBT-WG1608-16
   35  x86-UEFI-64
   35  Archer-C7-V4
   34  Archer-C6Uv1
   32  Archer-C5-V1
   31  x86-UEFI-Generic
   30  Mikrotik-RB922UAGS

Downloads by build system:
  242  AB_1800
  452  AB19
  951  AB21
  398  AB22
   86  AB23
    4  AB_BPIR4
    4  ABGL
   24  AB_RPI5
  154  ABSM

Routers with no downloads:
   ALIX-2D13
   Archer-C2600
   Asus-RT-AC56U
   Asus-RT-N56U
   BT-HomeHub-5
   CheckPoint-L-50
   Comfast-CF-E5
   Compex-WPJ428
   D-Link-DIR-825-C1
   D-Link-DIR-860L-B1
   D-Link-DIR-882-R1
   DualQ-H721-1907
   GL-AP1300
   GL-AR300M-Lite
   GL-AR750
   Gl.iNet-GL6416
   Gl.iNet-MV1000
   GL-MT300A
   Globalscale-MOCHAbin
   GL-S1300
   GL-X1200
   Huasifei-WS1218
   IRZ-MT00
   Linksys-E4200-V2
   Linksys-EA3500
   Linksys-EA7500v1
   Linksys-EA7500v2
   Linksys-EA9500v1
   Linksys-MR8300
   Mercury-MW4530R
   Mikrotik-RB433AH
   Mikrotik-RB750GR3
   Mikrotik-RB912UAG-5HPnD
   Mikrotik-RB951Ui-2nD
   Mikrotik-RBwAPR-2nD
   Mofi-4500
   MYNET-N600
   MYNET-N750
   NanoPi-NEO-Plus2
   Netgear-R7500-V2
   Netgear-WNDR3700-V2
   Netgear-WNDR3700-V4
   Netgear-WNDR3800
   RaspberryPi-CM4-USB
   Sanlinking-D240
   Seagate-Dockstar
   TP-Link-Archer-C59v1
   TP-Link-Archer-C6-V2
   TP-Link-WDR4900
   TP-Link-WR1043ND-V4
   TP-Link-WR1043N-V5
   TP-Link-WR703N-16meg
   UniElec-U7621-06
   ZBT-WE826WD

Lowest performing routers with downloads:
    1  Linksys-EA8500
    1  Linksys-EA8300
    1  Linksys-EA8100v1
    1  Lenovo-Y1-V1
    1  Huastlink-HC952
    1  Huasifei-WS1688AX-16
    1  GliNet_X3000
    1  GLi-AR750S
    1  GL-AR300M-nand
    1  Dynalink-DL-WRX36
    1  D-Link-DIR-835-A1
    1  D-Link-DGL-5500-A1
    1  Asus-RT-AC85P
    1  Asus-RT-AC65P
    1  APU2C4

Highest individual downloaders (hashed):
   50  90b824fe9caf
   49  a72c06701e16
   49  7664df2981e9
   30  916219eddd8c
   29  7f37a96a8e33
   29  5212e2aba7c6
   28  162e4e63d161
   27  54543106c83d
   24  c0b6d9ef22da
   24  b48bd016f375

Downloads by router name:
    7  Alfa-R36A
    3  Alfa-R36M-E4G
    3  Alfa-TUBE-E4G
   41  Alwaylink-MK01K21
    1  APU2C4
  206  Arcadyan-AW1000
   25  Archer-A7-V5
   25  Archer-C20-V1
    2  Archer-C2-V1
   32  Archer-C5-V1
   34  Archer-C6Uv1
    6  Archer-C7-V2
   35  Archer-C7-V4
    9  Archer-C7-V5
    9  Asus-RT-AC51U
   10  Asus-RT-AC58U
    1  Asus-RT-AC65P
    4  Asus-RT-AC68U
    1  Asus-RT-AC85P
   15  Asus-RT-AC87U
   23  Asus-RT-AX53U
    3  Asus-RT-N14U
    2  Asus-RT-N16
    2  Asus-RT-N66U
    4  Banana-Pi4
    3  BananaPi-BPi-R3
    5  Beeline-SmartBox_Pro
   14  Beeline-SmartBox_TurboModPlus
   30  Beeline-SmartBox_TurboPlus
   23  Comfast-CF-E3
    2  Cudy-WR1300
    3  DIR505A1
    1  D-Link-DGL-5500-A1
    1  D-Link-DIR-835-A1
    3  Dlink-DIR853-R1
    3  D-Link-DIR-882-A1
    3  DualQ-H721-2102
   22  DualQ-H721
    5  DualQ-H721-2203
   13  DualQ-H721AX
    1  Dynalink-DL-WRX36
    5  GL-AR150
    5  GL-AR300M-16
    1  GL-AR300M-nand
    2  GL-B1300
    8  GL-E750
    1  GLi-AR750S
   19  GL-iNet-AX1800
   20  GL-iNet-AXT1800
   10  Gl.iNet-MT1300
   21  GliNet_MT3000
    1  GliNet_X3000
    3  GL-MIFI
   17  GL-MT300N-V1
   10  GL-MT300N-V2
    3  GL-X750
    2  GL-XE300
    7  HiLink-HLK-7621a
   26  HiWiFi-HC5962
    4  Huasifei-WS1208V1
   19  Huasifei-WS1208V2
    4  Huasifei-WS1208v2-32
    1  Huasifei-WS1688AX-16
    2  Huasifei-WS1688AX-32
    6  Huasifei-WS1698AX
    5  Huasifei-WS7915AX
   18  Huastlink-HC851-HC841
    1  Huastlink-HC952
    6  Huawei-HG553
    2  Huawei-HG556a-A
    2  Huawei-HG556a-B
    5  Huawei-HG556a-C
   18  Lenovo-Y1S-V1
    1  Lenovo-Y1-V1
    2  Linksys-E8450-UBI
    2  Linksys-EA4500-V1
    2  Linksys-EA6350v3
   15  Linksys-EA7300
    1  Linksys-EA8100v1
    1  Linksys-EA8300
    1  Linksys-EA8500
    7  Linksys-WRT1200AC
   57  Linksys-WRT1900ACS
   24  Linksys-WRT1900AC-V1
    1  Linksys-WRT1900AC-V2
   10  Linksys-WRT3200ACM
    3  Linksys-WRT32x
   23  M01K21
    2  Mediatek-MT7628-EVB
    1  Mikrotik-LHG-2nD
   28  Mikrotik-LHG-HB
   16  Mikrotik-RB760iGS
   17  Mikrotik-RB912UAG-2HPnD
   30  Mikrotik-RB922UAGS
    4  Mikrotik-RBD52U
    6  Mikrotik-RBM11G
    7  Mikrotik-RBM33G
    1  Mikrotik-RBSXTR
    4  NanoPi-R2S
    2  NanoPi-R4S
    2  Netgear-R6220
   16  Netgear-R6300-V2
    1  Netgear-R7000
   20  Netgear-R7500-V1
    1  Netgear-R7800
    2  Netgear-R8000
    1  Netgear-WNDR3700-V1
    2  Netgear-WNDR4300v1
   10  Newifi-D2
   22  Nexx-WT3020-16
    3  Nexx-WT3020-8
    3  O2-Box-6431
    3  OrangePi-PC
    3  OrangePi-Plus
   18  OrangePi-Zero-Plus
    2  P2W-R619AC
    2  RaspberryPi-1
    9  RaspberryPi-2
   26  RaspberryPi-3-SD
    2  RaspberryPi-3-USB
   36  RaspberryPi-4-SD
    5  RaspberryPi-4-USB
   19  RaspberryPi-5-SD
    5  RaspberryPi-5-USB
    1  RaspberryPi-CM4-SD
   17  SamKnows-WhiteBox-V8
   21  SmartBox_Giga
   22  SmartBox_ModGiga
   27  TP-Link-Archer-C59v2
   13  TP-Link-Archer-C7v1
    2  TP-Link-Archer-WR1045ND
   19  TP-Link-MR3020v3
   21  TP-Link-MR3420v5
    2  TP-Link-WDR3500
    5  TP-Link-WDR3600
    3  TP-Link-WDR4300
    1  TP-Link-WDR4310
    6  TP-Link-WR1043NDv2
    1  TP-Link-WR1043NDv3
   24  TP-Link-WR842-V3
   17  TP-Link-WR902ACv1
   21  TP-Link-WR902ACv3
   11  TP-Link-WR942N
    1  Turris-Omnia-2102
   17  Turris-Omnia-2305
   21  UniElec-U7621-01
    1  UniElec-U7628-01
   18  WiFiX-NEXQ6GO
   71  x86-64
   10  x86-Generic
   35  x86-UEFI-64
   31  x86-UEFI-Generic
   23  Xiaomi-Mifi3G
    6  Xiaomi-Mifi3Pro
   12  Xiaomi-Mifi-Mini
   26  YouHua-WR1200JS
    2  YOUKU-YK1
   18  ZBT-WE1026-5G
   12  ZBT-WE1326
   22  ZBT-WE2802D
   25  ZBT-WE826-Q
   27  ZBT-WE826T
    5  ZBT-WE826-T-32meg
   18  ZBT-WG1602-16
    2  ZBT-WG1602-32
    1  ZBT-WG1602-32M-2102
   35  ZBT-WG1608-16
   37  ZBT-WG1608-32M
    7  ZBT-WG209
    1  ZBT-WG259
   80  ZBT-WG3526
    1  ZBT-WG827
   12  ZBT-WR8305RT
    3  ZBT-Z2101AX
    2  ZBT-Z2105AX
  127  ZBT-Z8102AX
   11  ZBT-Z8102AX-V2
    4  ZyXEL-Keenetic-Extra-II
    3  ZyXEL-Keenetic-Omni
   15  ZyXEL-NBG6817
    2  ZyZEL-NR7101

December 2023 Autobuild statistics

And this is the new feature I just mentioned in the update. I’ve wanted to set up some log analysis for a year to see which routers were popular and how many images were being downloaded, but didn’t have the time. This weekend was rainy and I had a some mental energy and time, so I got it done. This analysis is automated, so I plan to post these up monthly, though there might not always be time to do it on the first of the month.

Some words on methodology – these are all based on my apache logs. This first report only covers from December 9 to the end of the month, because those were all I still had in my log rotation. My scripts are now grabbing the logs daily at the end of the day, and processing that one day immediately. Identifying info is not getting kept long term, that was a priorty for me – as soon as I process the raw log, I’m saving the first digits of a hash of the downloading IP address only. That’s enough to tell different downloaders apart, but can’t be reversed to an actual single address even by brute force methods thanks to only keeping part of the hash.

A few quirks this month. For one, you’ll see a couple of images noted under “unknown” build systems, that’s because one of the new systems wasn’t spitting out its proper “-ABxx” file tag on the initial test run. You’ll also see that there are only a handful of routers listed with zero downloads, but some of those are because the script also checks against routers available when the script is run (since routers that aren’t downloaded don’t show up in the log either, and I can’t check to see what files were available a month ago). So if a router is released on 1/1 and you run the report for December, that router will show up on this list as a December router with no downloads. Not a big deal, but it’s a thing.

The other one is that there are more routers with no real downloads than indicated. A handful of folks do full scrapes of the site every month (which is fine by me), so you see “false” downloads associated with those scrapes. I saw three full scrapes in the logs in this report period, and one partial. It’s easy to get an idea by looking at the list of top user hashes, when you see someone has 400+ downloads that’s an obvious candidate. So, a router that shows only 3 downloads has good odds of not being used by anyone that month.

Without further ado,

The results:

Autobuild report for 2023-12
Compiled Mon 08 Jan 2024 03:51:02 PM UTC
Includes downloads from 2023-12-09 to 2023-12-31

Total router system downloads: 2673 by 576 unique users.

Top performing routers:
   65  Mikrotik-RBM33G
   61  ZBT-WG1608
   56  ZBT-Z8102AX
   55  Huasifei-WS1208
   53  DualQ-H721
   44  ZBT-WG1602
   37  x86-64
   33  Alwaylink-MK01K21
   32  Beeline-SmartBox_TurboPlus
   30  TP-Link-MR3020v3
   29  Huasifei-WS1208V2
   28  ZBT-WG1608-32M
   28  RaspberryPi-4-SD
   27  Arcadyan-AW1000
   26  ZBT-WG1608-16

Downloads by build system:
    5  AB18
    3  AB_1800
  655  AB19
 1321  AB21
  488  AB22
  106  AB23
   34  ABGL
   57  ABSM
    4  unknown

Routers with no downloads:
   Banana-Pi4
   GL-iNet-AX1800
   GL-iNet-AXT1800
   GliNet_MT3000
   RaspberryPi-5-SD
   RaspberryPi-5-USB

Lowest performing routers with downloads:
    5  Huasifei-WS1698AX
    5  DualQ-H721AX
    5  Asus-RT-AC56U
    4  Netgear-R8000
    4  Linksys-EA9500v1
    4  Huastlink-HC841
    4  GliNet_X3000
    3  WiFiX-NEXQ6GO
    3  TP-Link-WR902ACv1
    3  TP-Link-Archer-C6-V2
    3  Mikrotik-LHG-HB
    3  Mikrotik-LHG-2nD
    3  Linksys-EA8300
    3  Huasifei-WS1208V2-32
    3  Dlink-DIR-853-R1

Highest individual downloaders (hashed):
  493  09a87510be6e
  488  6a199cc49538
  484  bb46aef0d4b2
  377  0d58b5d9c89b
   16  151a3d90824e
   12  a6396bc71c58
   11  4b4e08145894
   10  db803b2880f2
    8  a9f7c6baf2b0
    8  44559ae5f771

Downloads by router name:
   11  Alfa-R36A
    8  Alfa-R36M-E4G
    8  Alfa-TUBE-E4G
    8  ALIX-2D13
   33  Alwaylink-MK01K21
   10  APU2C4
   27  Arcadyan-AW1000
    9  Archer-A7-V5
    8  Archer-C20-V1
    9  Archer-C2600
    8  Archer-C2-V1
    8  Archer-C5-V1
   17  Archer-C6Uv1
   13  Archer-C7-V2
    9  Archer-C7-V4
   18  Archer-C7-V5
   10  Asus-RT-AC51U
    5  Asus-RT-AC56U
   13  Asus-RT-AC58U
    8  Asus-RT-AC65P
   16  Asus-RT-AC68U
    8  Asus-RT-AC85P
   10  Asus-RT-AC87U
   10  Asus-RT-AX53U
    8  Asus-RT-N14U
   10  Asus-RT-N16
    9  Asus-RT-N56U
   13  Asus-RT-N66U
   12  BananaPi-BPi-R3
   16  Beeline-SmartBox_Pro
   16  Beeline-SmartBox_TurboModPlus
   32  Beeline-SmartBox_TurboPlus
    8  BT-HomeHub-5
    8  CheckPoint-L-50
   10  Comfast-CF-E3
   11  Comfast-CF-E5
    8  Compex-WPJ428
    9  Cudy-WR1300
    9  DIR505A1
    7  D-Link-DGL-5500-A1
    8  D-Link-DIR-825-C1
    7  D-Link-DIR-835-A1
   11  Dlink-DIR853-R1
    3  Dlink-DIR-853-R1
    7  D-Link-DIR-860L-B1
   10  D-Link-DIR-882-A1
    8  D-Link-DIR-882-R1
   12  DualQ-H721-1907
   12  DualQ-H721-2102
   11  DualQ-H721-2203
   53  DualQ-H721
    5  DualQ-H721AX
    7  Dynalink-DL-WRX36
    7  GL-AP1300
    7  GL-AR150
    7  GL-AR300M-16
    7  GL-AR300M-Lite
    7  GL-AR300M-nand
   10  GL-AR750
    8  GL-B1300
   17  GL-E750
    8  GLi-AR750S
    8  Gl.iNet-GL6416
   19  Gl.iNet-MT1300
   15  Gl.iNet-MV1000
    4  GliNet_X3000
    9  GL-MIFI
    8  GL-MT300A
    8  GL-MT300N-V1
   20  GL-MT300N-V2
    7  Globalscale-MOCHAbin
   12  GL-S1300
    8  GL-X1200
   22  GL-X750
    9  GL-XE300
   18  HiLink-HLK-7621a
    8  HiWiFi-HC5962
   55  Huasifei-WS1208
   12  Huasifei-WS1208V1
   29  Huasifei-WS1208V2
    3  Huasifei-WS1208V2-32
    8  Huasifei-WS1208v2-32
    6  Huasifei-WS1218
    7  Huasifei-WS1688AX-16
    8  Huasifei-WS1688AX-32
    5  Huasifei-WS1698AX
   10  Huasifei-WS7915AX
    4  Huastlink-HC841
    9  Huastlink-HC851-HC841
    7  Huastlink-HC952
   17  Huawei-HG553
    8  Huawei-HG556a-A
   11  Huawei-HG556a-B
    7  Huawei-HG556a-C
    7  IRZ-MT00
    8  Lenovo-Y1S-V1
    9  Lenovo-Y1-V1
    7  Linksys-E4200-V2
   10  Linksys-E8450
    7  Linksys-E8450-UBI
    9  Linksys-EA3500
    9  Linksys-EA4500-V1
    8  Linksys-EA6350v3
    8  Linksys-EA7300
    7  Linksys-EA7500v1
    8  Linksys-EA7500v2
   10  Linksys-EA8100v1
    3  Linksys-EA8300
    7  Linksys-EA8500
    4  Linksys-EA9500v1
    9  Linksys-MR8300
   13  Linksys-WRT1200AC
   17  Linksys-WRT1900ACS
   19  Linksys-WRT1900AC-V1
   16  Linksys-WRT1900AC-V2
   17  Linksys-WRT3200ACM
   16  Linksys-WRT32x
   18  M01K21
    6  Mediatek-MT7628-EVB
    7  Mercury-MW4530R
    3  Mikrotik-LHG-2nD
    3  Mikrotik-LHG-HB
    6  Mikrotik-RB433AH
    7  Mikrotik-RB750GR3
    8  Mikrotik-RB760iGS
    7  Mikrotik-RB912UAG-2HPnD
    9  Mikrotik-RB912UAG-5HPnD
    7  Mikrotik-RB922UAGS
    6  Mikrotik-RB951Ui-2nD
    8  Mikrotik-RBD52U
   15  Mikrotik-RBM11G
   65  Mikrotik-RBM33G
    9  Mikrotik-RBSXTR
    9  Mikrotik-RBwAPR-2nD
   10  Mofi-4500
    8  MYNET-N600
    9  MYNET-N750
    8  NanoPi-NEO-Plus2
    9  NanoPi-R2S
   14  NanoPi-R4S
    8  Netgear-R6220
    7  Netgear-R6300-V2
   10  Netgear-R7000
    8  Netgear-R7500-V1
    8  Netgear-R7500-V2
   10  Netgear-R7800
    4  Netgear-R8000
    9  Netgear-WNDR3700-V1
    7  Netgear-WNDR3700-V2
    7  Netgear-WNDR3700-V4
    7  Netgear-WNDR3800
   18  Netgear-WNDR4300v1
   19  Newifi-D2
    8  Nexx-WT3020-16
    8  Nexx-WT3020-8
    7  O2-Box-6431
    9  OrangePi-PC
   10  OrangePi-Plus
    8  OrangePi-Zero-Plus
    8  P2W-R619AC
   22  RaspberryPi-1
   17  RaspberryPi-2
   21  RaspberryPi-3-SD
   16  RaspberryPi-3-USB
   28  RaspberryPi-4-SD
   23  RaspberryPi-4-USB
   22  RaspberryPi-CM4-SD
   17  RaspberryPi-CM4-USB
    9  SamKnows-WhiteBox-V8
    8  Sanlinking-D240
    7  Seagate-Dockstar
   23  SmartBox_Giga
   20  SmartBox_ModGiga
    8  TP-Link-Archer-C59v1
    7  TP-Link-Archer-C59v2
    3  TP-Link-Archer-C6-V2
    7  TP-Link-Archer-C7v1
    5  TP-Link-Archer-WR1045ND
   30  TP-Link-MR3020v3
   14  TP-Link-MR3420v5
   10  TP-Link-WDR3500
    9  TP-Link-WDR3600
    8  TP-Link-WDR4300
    8  TP-Link-WDR4310
    7  TP-Link-WDR4900
   10  TP-Link-WR1043NDv2
    8  TP-Link-WR1043NDv3
   10  TP-Link-WR1043ND-V4
    8  TP-Link-WR1043N-V5
    5  TP-Link-WR703N-16meg
   10  TP-Link-WR842-V3
    3  TP-Link-WR902ACv1
    9  TP-Link-WR902ACv3
   14  TP-Link-WR942N
   20  Turris-Omnia
    7  Turris-Omnia-2102
    7  Turris-Omnia-2305
    9  UniElec-U7621-01
    7  UniElec-U7621-06
    7  UniElec-U7628-01
    3  WiFiX-NEXQ6GO
   21  WS1688AX
   11  WS1688AX-32
   37  x86-64
   22  x86-Generic
   14  x86-UEFI-64
   21  x86-UEFI-Generic
   13  Xiaomi-Mifi3G
   10  Xiaomi-Mifi3Pro
   20  Xiaomi-Mifi-Mini
    9  YouHua-WR1200JS
   16  YOUKU-YK1
   13  Z2101AX
   10  ZBT-WE1026-5G
   18  ZBT-WE1326
    8  ZBT-WE2802D
   22  ZBT-WE826-Q
   23  ZBT-WE826T
   12  ZBT-WE826-T-32meg
   15  ZBT-WE826WD
   21  ZBT-WG1602-16
   19  ZBT-WG1602-32
   11  ZBT-WG1602-32M
    8  ZBT-WG1602-32M-2102
   44  ZBT-WG1602
   26  ZBT-WG1608-16
   28  ZBT-WG1608-32M
   61  ZBT-WG1608
    9  ZBT-WG209
    8  ZBT-WG259
   23  ZBT-WG3526
    8  ZBT-WG827
   11  ZBT-WR8305RT
   10  ZBT-Z2101AX
   13  ZBT-Z2105AX
   56  ZBT-Z8102AX
   12  ZyXEL-Keenetic-Extra-II
    7  ZyXEL-Keenetic-Omni
    7  ZyXEL-NBG6817
   10  ZyZEL-NR7101

January 2024 ROOter updates

Only a handful of news on my end right now, thanks to busy jobs on my part taking away from router fun. Here are the updates on the parts that I’m involved with:

New ROOter build systems

Dairyman has released four new build systems since my September update. All four are “special purpose” systems meant to cover a handful of routers. They are:

  • SourceMaster (link) to cover MT3000, WS1698, X3000, and Z8102. This is a fork off of the OpenWRT master as of roughly 27 Nov 2023.
  • SourceBPIR4 (link) to cover BPI4, the Banana Pi R4. Currently a single image system.
  • SourceRPI5 (link) to cover RASPPI5 and RASPPI5USB, which should be the Raspberry Pi 5 board with either the usual SD card or with USB mounted root file systems.
  • Source1800 (link) to cover AW1000, AX1800, and AXT1800. These are the Arcadyan AW1000 and two GL-iNet variants, all with ipq chipsets.

None of these are currently expected to become major build systems, but they will be used where necessary to support new systems not yet covered by one of our main systems. I’m still hearing 21.02 is likely to stay our primary system for a while, as 22.03 and 23.05 are producing much larger images that aren’t going to work on a lot of routers we currently support. Kernel growth has been a big issue in particular for several years now.

Build machine upgrades

The autobuild machine has also gotten an upgrade around New Years. With the current number of router images, the main system SSD was staying around 80-90% full. The build machine now has a second SSD of the same size, so we’re set for quite a bit of future growth.

This laptop has been running the autobuild systems for almost two years now. Not bad for a total investment of roughly $250 US – all this is done off of a salvaged Dell e6410 laptop with a 2 core (4 virtual) i7 chip and 8GB of RAM, with two 1TB SSD’s. It has cranked out roughly 250 images per week in recent builds, usually in 55h or less. Never underestimate the value of ancient hardware to do real work, as these laptops were released in 2010. With weekly builds only taking roughly 1/3 of the laptop’s possible duty cycle (in hours of the week), I expect this little guy should be up for the task for a lot longer barring catastropic failure.

One point of interest is that the CPU load is shown on the build system status page here the same as it is on a ROOter status page (which is the number given by the uptime command). In the first part of the build, it’s not unusual to see utilizations hovering in the upper 20’s. Since this is a two core machine, with hyperthreading, it can handle a load of roughly 1.4 per core depending on the type of load, so for anything over about 2.8, the CPU is 100% utilized. Something to remember when people tell you to tune your builds so that your CPU load stays under your core count – that’s absolute BS. For best builds, you want your load ABOVE your core count. The lower recommendations are to make sure your system stays responsive, but it’s much, much better to handle that using the nice command when you start your build. Despite my CPU being overloaded by an order of magnitude, the machine is still perfectly responsive, and the only way to tell how hard it’s working is by hearing the fan spun up or seeing that load number.

Docker Autobuild Image updates

The base image for the autobuild containers has been updated to Debian 11.8 from 11.7. This is because one of the new build systems needed a pair of new build tools added. This can be done directly within the container for testing, but really any change in the toolchain is work updating the base image instead. This keeps the resulting images a lot more consistent.

The new base image has been applied to all 9 build systems (which took half an evening), tested, gone through a week of full builds, and all of the build system changes have been pushed to github with the new images already pushed to Docker Hub.

Bonus: there’s one more feature announcement coming (which the folks on the forum today will have already seen), but it warrants its own post.

September 2023 ROOter updates

I’ve had a few chances over the last few weeks to make minor updates to the autobuild systems. Nothing major. This month’s news:

  • The autobuild status system has been reconfigured to update every minute, with an approximately one minute lag.
  • The status has also been added to a new Autobuilds headline page, easily accessible from the top menu. This is probably the quickest way to see the current system status. It doesn’t auto update, though, you’ll have to refresh occasionally.
  • A while back we were able to finish uploading all of the old autobuild images to carp4’s Autobuild archive space. Everything should be there, from just after the first runs, sorted by router name. You may find a few routers that are split into several folders due to name changes over time, but everything seems to be where it belongs. This archive also updates with new images within a day or so of them being built.
  • The old 18.06.7 system was retired about a week ago. For at least a year or more it had only been building the WR703N, which barely fit an image with the older kernel. New images no longer fit, so this one is finally wound down.

I have a few more tweaks on my wish list, but everything gets fit between the two real jobs, both of which have been hopping this year. Maybe more time will appear as things go on.

ROOter Autobuilds Archive

This has already been linked from the forum and the autobuilds page, but still great news. We only have space on the server to host the most recent two images for each router. However, user carp4 has donated a significant amount of space on a Google Drive to host all of the older images. Without further ado:

ROOter Autobuild Archive

It will take two to three months to upload all of the older images – there are almost half of a terabyte worth already. They’re all on a local drive now, so will trickle online during off-peak hours overnight. I rough estimate would be end of June, 2023 to have them all online, if all goes well.

My Samsung Galaxy S20 FE UW debloat list

I know I’m not the only one who’s driven nuts by all the extra stuff loaded on most new phones. I honestly mostly prefer a “stock” Android experience, but do prefer some of Samsung’s hardware. I just don’t like their software. Thankfully, ADB makes it pretty easy to rip most of their software out by the roots, even the things that are blocked from disabling via the user interface.

And that’s good, since this is the second one of these phones I’ve set up in about nine months (thanks to a catastrophe a few days ago, and decent insurance coverage). Unfortunately I didn’t save a list of what I removed to debloat last time, so this time once I got it workable I exported the list of disabled apps.

There are lots of other good references on how to set up ADB and how to disable individual packages, so the below is simply a copy of my own disable list. This rips out almost everything Samsung and a good bit of the garbage Verizon preloaded. It should be very similar to what I had removed from my last phone, which was very nice to use after the junk-ectomy.

Without further ado, the list:

com.diotek.sec.lookup.dictionary
com.google.android.apps.tachyon
com.LogiaGroup.LogiaDeck
com.microsoft.appmanager
com.microsoft.skydrive
com.netflix.mediaclient
com.netflix.partner.activation
com.samsung.android.app.camera.sticker.facearavatar.preload
com.samsung.android.app.contacts
com.samsung.android.app.routines
com.samsung.android.app.settings.bixby
com.samsung.android.app.spage
com.samsung.android.app.watchmanagerstub
com.samsung.android.ardrawing
com.samsung.android.aremoji
com.samsung.android.arzone
com.samsung.android.authfw
com.samsung.android.beaconmanager
com.samsung.android.bixby.agent
com.samsung.android.bixby.service
com.samsung.android.bixbyvision.framework
com.samsung.android.bixby.wakeup
com.samsung.android.calendar
com.samsung.android.da.daagent
com.samsung.android.forest
com.samsung.android.game.gamehome
com.samsung.android.game.gametools
com.samsung.android.livestickers
com.samsung.android.mateagent
com.samsung.android.mdx
com.samsung.android.messaging
com.samsung.android.samsungpass
com.samsung.android.samsungpassautofill
com.samsung.android.scloud
com.samsung.android.service.stplatform
com.samsung.android.stickercenter
com.samsung.android.tapack.authfw
com.samsung.android.themestore
com.samsung.android.visionintelligence
com.samsung.desktopsystemui
com.samsung.systemui.bixby2
com.sec.android.app.billing
com.sec.android.app.clockpackage
com.sec.android.app.desktoplauncher
com.sec.android.app.dexonpc
com.sec.android.app.samsungapps
com.sec.android.app.sbrowser
com.sec.android.autodoodle.service
com.sec.android.daemonapp
com.sec.android.desktopmode.uiservice
com.sec.spp.push
com.securityandprivacy.android.verizon.vms
com.vcast.mediamanager

Since I see far fewer explanations of how to conveniently generate this list of everything you’ve manually disabled, here’s the command you need to run (in Linux from outside adb shell, because you need to use command line tools to sift things out that aren’t available within most phone operating systems):

diff <(adb shell pm list packages) <(adb shell pm list packages -u) > ~/rawlist.txt

That command takes a list of all current packages, then takes a list of all packages including uninstalled, subtracts the good packages from the bad, and saves the results into a list in your home directory (in this case named rawlist.txt).

The results produced by that command are pretty ugly, so you can use the following command to trim out all the gibberish and leave just the clean package list above:

cat rawlist.txt | grep "> package" | sed 's/> package://' | sort > ~/cleanlist.txt

That leaves you a clean list of uninstalled packages from the stock image that you could even use as an input for a script to bulk-uninstall everything using ADB again later. That’s my real end goal here – if something large and steel falls on my poor phone again, I don’t want to have to spend another two hours remembering what junk I ripped out, when I could save the list, and whip up a script to remove them all in about ten minutes. Wish I’d thought of that the first time.

Filtering logd in OpenWRT

So I’ve been running into an odd problem with log spam in my ring buffer log, and just kludged together a solution when I couldn’t find one elsewhere. I use nft-qos on my main OpenWRT router, and some days it spams my logs like crazy with lines that look like this:

Aug 22 04:15:11 hexenbiest nft-qos-monitor: ACTION=update, MACADDR=a8:86:dd:ac:73:ad, IPADDR=192.168.1.191, HOSTNAME=
Aug 22 04:15:12 hexenbiest nft-qos-dynamic: ACTION=update, MACADDR=a8:86:dd:ac:73:ad, IPADDR=192.168.1.191, HOSTNAME=
Aug 22 04:26:46 hexenbiest nft-qos-monitor: ACTION=update, MACADDR=a4:08:01:3c:9e:bb, IPADDR=192.168.1.207, HOSTNAME=FireStick
Aug 22 04:26:47 hexenbiest nft-qos-dynamic: ACTION=update, MACADDR=a4:08:01:3c:9e:bb, IPADDR=192.168.1.207, HOSTNAME=FireStick
Aug 22 04:26:50 hexenbiest nft-qos-monitor: ACTION=update, MACADDR=a4:08:01:3c:9e:bb, IPADDR=192.168.1.207, HOSTNAME=FireStick
Aug 22 04:26:50 hexenbiest nft-qos-dynamic: ACTION=update, MACADDR=a4:08:01:3c:9e:bb, IPADDR=192.168.1.207, HOSTNAME=FireStick
Aug 22 04:26:59 hexenbiest nft-qos-monitor: ACTION=update, MACADDR=a4:08:01:3c:9e:bb, IPADDR=192.168.1.207, HOSTNAME=FireStick
Aug 22 04:26:59 hexenbiest nft-qos-dynamic: ACTION=update, MACADDR=a4:08:01:3c:9e:bb, IPADDR=192.168.1.207, HOSTNAME=FireStick
Aug 22 04:27:15 hexenbiest nft-qos-monitor: ACTION=update, MACADDR=a4:08:01:3c:9e:bb, IPADDR=192.168.1.207, HOSTNAME=FireStick
Aug 22 04:27:15 hexenbiest nft-qos-dynamic: ACTION=update, MACADDR=a4:08:01:3c:9e:bb, IPADDR=192.168.1.207, HOSTNAME=FireStick

That’s just a sampling – it’s not just the one device, more often it will spit out a list of every device on the network. Sometimes it does that multiple times per minute. Other times, it’ll go days and never do it once. I don’t have time to track down why nft-qos is logging this, or figure out how to turn it off, but I did have good luck with this kludge to filter it out.

Most people won’t need to filter this unless you’re using your logs for something else. I send my logs to an rsyslog server in my LAN and filter it there, so this isn’t an issue in my long term logs – only in the ring buffer copy that stays on the router itself. So why does it matter? I have a script that runs once a minute to count missed pings from mwan3 and update a conky dashboard on one of my monitoring servers, and it’s much more efficient if I have it parse the ring buffer, instead of asking the rsyslog server across the network once a minute for the whole day’s log!

In hunting around, I’ve seen a dozen or so people asking how to filter logs on OpenWRT, and the answer is universally to either implement rsyslog on the router, or send it to an external syslog server. My case is obviously a corner case, but it’s one that’s obviously come up enough that other people have asked, and I’ve never seen a solution that actually works with logd and logger on the box itself without an external server.

So I made one. And it turned out to be ridiculously simple. It’s also very easy for you to modify to filter out anything else you need to as well.

The Fix, without further ado

What you need to know: logger on OpenWRT isn’t a real program. Entering “type logger” in your shell will point to /usr/bin/logger, but that’s not a real file, just another link to /bin/busybox. When you type

logger -t testlog "Some test message"

into your terminal on OpenWRT, it just gets redirected into

/bin/busybox logger -t testlog "Some test message"

and busybox does the work. Filtering the log input is as simple as moving the link at /usr/bin/logger out of the way, and creating the following new ash script at /usr/bin/logger:

#!/bin/ash
# We are going to filter input to logger by intercepting it.

# Parse options if they're there, else use defaults.
STDERR=1
TAG=$USER
PRIO=0

while getopts st:p: flag ; do
        case "${flag}" in
                s) STDERR=0 ;;
                t) TAG=${OPTARG} ;;
                p) PRIO=${OPTARG} ;;
                *) exit 1 ;;
        esac
done

shift $((OPTIND -1))

# now filter the input string and exit early if we pick up a filtered string.

# Message filters (chain more if needed):
FILTERED=`echo "$*" | grep -v 'ACTION=update,'`

# Exit trigger lines (add more if needed):
[ -z "$FILTERED" ] && exit 0

# We have a good filtered string. Send it to the real logger.

if [ $STDERR -eq 1 ] ; then
        echo "$FILTERED" | /bin/busybox logger -t $TAG -p $PRIO
else
        echo "$FILTERED" | /bin/busybox logger -s -t $TAG -p $PRIO
fi

Now, here’s what that script does:

  • Parse any flags on the input using getopts. Busybox logger only accepts the three flags shown, so this is fairly simple. We’re setting defaults for all three, and then if a flag is actually given, replacing the defaults with what was given.
  • Shift the input over by the number of flags and flag parameters, leaving only the log message.
  • Using grep -v (think “grep inverse”) to filter according to the message content. This FILTERED line is either going to return exactly what you give it unchanged, or nothing at all if it matches your filter. It’s easy to add more filters here if needed, more info below.
  • Checking to see if the message has been filtered out (the code “[ -z “$FILTERED” ]” just returns success if the string you feed it is zero length) and exiting the script if so, basically discarding this log entry. This line is important to remember later if you want to filter on things like tags or priority, so remember it as an “exit trigger line.”
  • Finally, passing the message to the REAL logger facility, using either the default flag values or the values given in the input. The syslog flag is a true or false, and in doing this quickly I found it easiest to just split that using an if statement.

Modifying the script for your own purposes

Want to know how to modify this to make it your own, but not a bash scripting guru? It’s pretty simple, so I’ll give you the framework for the most common changes.

Changing it to filter based on different text is the easiest – just change what’s in the single quotes on the filter line. If you wanted to filter out messages where the content is “foobar” for example:

FILTERED=`echo "$*" | grep -v 'foobar'`

Changing it to filter multiple different things is also fairly easy. You just need to duplicate the filter line, and change any lines after the first to work with the output of the line before it instead of $*. You can chain together however many you want, at the end FILTERED is either going to contain your original input or just be a zero length “” string if a filter matched. If we wanted my original filter plus foo and bar:

FILTERED=`echo "$*" | grep -v 'ACTION=update,'`
FILTERED=`echo "$FILTERED" | grep -v 'foo'`
FILTERED=`echo "$FILTERED" | grep -v 'bar'`

You can change it to match anything in the message this way. One note, grep as it’s called there is also case sensitive, and all the other usual caveats around that kind of thing still apply – but overall, it’s easy to tweak these message filters any way you want.

What about filtering based on a whole tag? What if you want to filter out all log entries with the tag “testlog”? Now, instead of modifying the filter string, you need to modify the exit trigger line (if you only want to filter on the tag, not the message) or add additional exit trigger lines if you want to filter on both message and tags.

Example, filtering on one or more messages as well as tag “testlog”:

[ -z "$FILTERED" ] && exit 0
[ "$TAG" = "testlog" ] && exit 0

You can add as many exit triggers as you need – matching any one of them will cause the message to be discarded. What if you also want to filter by message priority, discarding any message with priority lower than (number higher than) 5? This is a functionality that’s been broken in OpenWRT since 2013, but it turns out it’s also trivial to fix by adding an exit trigger line:

[ -z "$FILTERED" ] && exit 0
[ "$TAG" = "testlog" ] && exit 0
[ "$TAG" = "foobar" ] && exit 0
[ "$PRIO" -gt 5 ] && exit 0

That one filters one one or more messages, plus two tags and priority. As you can see, you really can easily tweak this to do whatever you want.

In short, log filtering has been broken ever since the switch to logd, and it turns out it was always trivial to implement and no one has bothered. I’ve kludged it myself, but I don’t have the time to go implement a real replacement in the original source either – I wish I did! If you do, please feel free to take this framework and run with it. It would be almost as easy to add reading of filter settings from uci into this script so that we could actually make settings in /etc/config/system like conloglevel functional again. You could trivially include lots of message filter strings in /etc/config/system, and have the script loop through them instead of hard coding it like my examples above. Really, this is a trivial framework for fixing OpenWRT’s long-broken log filtering, and all it needs is someone with time and willingness to run with it.

But meanwhile, since you’re probably like me and just looking for a quick way to fix this – here you go, tweak as you see fit and enjoy your spam-free ring buffer logs!

Automatic Autossh Tunnel under Systemd

It took me half the afternoon to figure out how to do this with all the bells and whistles (mostly a learning experience on systemd), so I’d better write it down! Meanwhile, it took me at least a dozen reference docs to piece it together, because autossh has pretty brief documentation.

Edit 2021-03-27: One thing in the original version of this article that didn’t work as intended was keeping the systemd unit file in a different location and linking it where it needed to be. That turns out not to work right under reload. Further info below.

I had a remote server running a mail forwarder which is password protected but exposed, yet I was only using it from devices within my LAN. That’s not only not ideal, but a smidge less reliable due to various odd properties of my LAN (with regard to WAN failover behaviors), so I wanted to move this traffic into an SSH tunnel. My local tunnel endpoint would be a Debian 10 (Buster) machine that is fully within the LAN, not a perimeter device. I want this connection to come up fully automatically on boot/reboot, and give me simple control as a service (hence systemd).

Remote: an unprivileged, high port PPPPP on server RRR.RRR.RRR.RRR, whose ssh server listens on port xxxxx.

Local: the same unprivileged, high port PPPPP on server LLL.LLL.LLL.LLL

My remote machine was already appropriately set up, all I needed to do was add an unprivileged user account to add this connection. In the code below, you’ll see the account name (on both the local and remote servers) is raul, which is also the local server’s name on the network. Substitute your account name of choice wherever you see this. Before beginning the real work, you need this account set up on both machines, with key based authentication (with no pass phrase). Log into the remote machine from the local account at least once, to verify it works with the certificate, and to store the host key.

Install autossh on your local Debian machine with sudo apt install autossh.

Since everything will be run as your unprivileged user, it’s actually a bit easier to do all your initial editing from that account so that you don’t have to play with permissions later. That’s not how I started, but it would’ve saved me some steps. So, switch to your new account with su raul or equivalent. We’ll be keeping the three files necessary to run this in that account’s home directory at /home/raul/, but one of them is created automatically by your script, so we really only need to write the startup script and the systemd unit.

One thing to note beforehand – the formatting and word wrapping on this site can make it less than obvious what’s an actual newline in code snippets, and what’s just word wrap. Because of this, I’ve linked a copy of the maillink.sh script where you can get to it directly, and just change out your account name, addresses, and ports.

Startup Script

First, we’ll create the startup script, which is the meat of the work. Without further ado, create /home/raul/maillink.sh:

#!/bin/bash

# This script starts an ssh tunnel to matter to locally expose the mail port, PPPPP.                                                                           

logger -t maillink.sh "Begin autossh setup script for mail link."
S_TIME=`awk '{print int($1)}' /proc/uptime`
MAX_TIME="300"

# First, verify connection to outside world is working. This bit is optional.

while true ; do
        M_RESPONSE=`ping -c 5 -A RRR.RRR.RRR.RRR | grep -c "64 bytes from"`
        C_TIME=`awk '{print int($1)}' /proc/uptime`
        E_TIME=`expr $C_TIME - $S_TIME`
        [[ $M_RESPONSE == "5" ]] && break
        if  [ $E_TIME -gt $MAX_TIME ]
        then
                logger -t maillink.sh "Waiting for network, timed out after $E_TIME seconds."                                                                  
                exit 1
        fi
        sleep 10
done

logger -t maillink.sh "Network detected up after $E_TIME seconds."

# Now, start the tunnel in the background.

export AUTOSSH_PIDFILE="/home/raul/maillink.pid"

autossh -f -M 0 raul@RRR.RRR.RRR.RRR -p xxxxx -N \
        -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" \
        -L LLL.LLL.LLL.LLL:PPPPP:127.0.0.1:PPPPP

MAILPID=`cat /home/raul/maillink.pid`

logger -t maillink.sh "Autostart command run, pid is $MAILPID."

As you can see, this script has a few more bells and whistles than just the most basic. While systemd should take care of making sure we wait until the network is up, I want the script to verify that it can reach my actual remote server before it starts the tunnel. It will try every ten seconds for up to five minutes, until it gets 100% of its test pings back in one go.

I also like logs … putting a few log lines into your script sure makes it easier to figure out why things are going wrong.

The “AUTOSSH_PIDFILE” line is necessary for making this setup play pretty with systemd and give us a way to stop the tunnel the nice way (systemctl stop maillink) instead of manually killing it. That environment variable will cause autossh to store its pid once it starts up. Autossh responds to a nice, default kill by neatly shutting down the ssh tunnel and itself, so it makes that control easy. Of course, finding that and figuring out how to do it was … less easy, but it’s simple once you know. That pid file is the second important file in making this work, but this line creates it automatically whenever the tunnel is working.

Now, the meat of this file is the autossh command. Some of these are for autossh itself, and most get passed to ssh. Here’s a breakdown of each part of this command:

  • -f moves the process to background on startup (this is an autossh flag)
  • -M 0 turns off the built in autossh connection checking system – we’re relying on newer monitoring built into OpenSSH instead.
  • raul@RRR.RRR.RRR.RRR -p xxxxx are the username, address, and port number of the remote server, to be passed to ssh (along with all that follows). If your remote server uses the default port of 22, you can leave the port flag out. If your local and remote accounts for this service will be the same, you can also leave out the account name, but I find it clearer left in.
  • -N tells ssh not to execute any remote commands. The manpage for ssh mentions this is useful for just forwarding ports. What nothing tells you, is that in my experience this autossh tunnel simply won’t work without the -N flag. It failed silently until I added it in.
  • -o “ServerAliveInterval 60” -o “ServerAliveCountMax 3” are the flags for the built in connection monitoring we’ll be using. ServerAliveInterval causes the client to send a keep-alive null packet to the server at the specified interval in seconds, expecting a response. ServerAliveCountMax sets the limit of how many times in a row the response can fail to arrive before the client automatically disconnects. When it does disconnect, it will exit with a non-zero status (i.e. “something went wrong”), and autossh will restart the tunnel – that’s the main function of autossh. If you kill the ssh process intentionally, it returns 0 and autossh assumes that’s intentional, so it will end itself.
  • -L LLL.LLL.LLL.LLL:PPPPP:127.0.0.1:PPPPP is the real meat of the command, as this is the actual port forward. It translates to, “Take port PPPPP of the remote server’s loopback (127.0.0.1) interface, and expose it on port PPPPP of this local client’s interface at address LLL.LLL.LLL.LLL.” That local address is optional, but if you don’t put it in, it will default to exposing that port on the local client’s loopback interface, too. That’s great if you just need to access it from the client computer, but I needed this port exposed to the rest of my LAN.

One handy thing to note, is that you can forward multiple ports through this single tunnel. You can just keep repeating that -L line to forward however many ports you need. Or, if you’re forwarding for various different services that you might not want all active at the same time, it’s easy to duplicate the startup script and service file, tweak the names and contents, and have a second separate service to control.

Before you test this the first time, it’s important to make sure it’s executable!

chmod +x /home/raul/maillink.sh

At this point, if you aren’t looking to make this a systemd service, you’re done – you can actually just start your connection manually using “. /home/raul/maillink.sh” (note the . and space up front) and stop it manually using kill <pid>, where the pid is the number saved in the maillink.pid file. (If you’re planning to do this, it’s actually easiest to keep these in your main user’s home directory and modify the script for the pid location accordingly.) At this point, you should manually test the script to ensure everything is working the way you expected. You should see some helpful output in your syslog, and you should also see that port listening on your local machine if you run netstat -tunlp4.

Systemd Unit

However, with just a little more work, making this controllable turns out to be pretty simple. It took way longer to corral the info on how to do it, than it would’ve taken to do if I’d already known how.

Edit 2021-03-27: I originally tried placing this unit file in /home/raul/ and sym linking it into /etc/systemd/system/. That … well, doesn’t work. It works fine the first time, when you run systemctl daemon-reload to pull the unit into the system. The problem is, for whatever reason systemd will not find that file on reboot, even though the link is still there. You’d have to reload manually every time, which just defeats the purpose. Edits have been made accordingly below.

First, create the systemd unit file, which must be located in /etc/systemd/system/maillink.service:

[Unit]
Description=Keeps a tunnel to remote mailserver open
Wants=network-online.target
After=network-online.target

[Service]
Type=forking
RemainAfterExit=yes
User=raul
Group=raul
ExecStart=/home/raul/maillink.sh                                                                                                                               
ExecStop=kill `cat /home/raul/maillink.pid`

[Install]
WantedBy=multi-user.target

This file contains:

  • A description
  • Two lines that ensure it won’t run until after networking is up
  • Two lines that instruct the system to run it under your “raul” account
  • A command to be run to start the service
  • A command to be run to stop the service – this shuts it down clean using the pid file we saved on startup

Next, we need to update systemd to see the new service:

sudo systemctl daemon-reload

Now your systemd unit is loaded. You should be able to verify this by running systemctl status maillink, which should give you a bit of information on your service and its description.

Next, we can test the systemd setup out. First start the service once using sudo systemctl start maillink, and make sure it starts without error messages. Check it as well with systemctl status maillink, and verify the port is there using netstat -tnlp4.

If all went well, that status command should give you some nice output, including the process tree for the service, and an excerpt of relevant parts of the syslog.

Make sure you also verify the stop command, with systemctl stop maillink. This should turn off the port in netstat, and you should also no longer see autossh or ssh when you run ps -e.

If all looks good, you’re good to set this up and leave it! Enable the service to autostart using systemctl enable maillink, and if it’s not started already, start it back up with systemctl start maillink.

And, here’s hoping this was clear and helpful. If you catch any bugs, please let me know!

Quick and Dirty Live View of rsyslog Output

I mentioned in a post yesterday that I was watching the syslog of my router to see when it sent a boot image over tftp. However, OpenWRT does not have a “live updating” syslog view – so how was I doing that, just clicking refresh over and over with my third hand, while holding down a reset button with the other two? No, there’s a really easy way to do this with a stupidly simple bash script.

My routers use remote logging to an internal rsyslog server on my LAN, and you’ll see my script is set up to support that. However, this is very easy to modify to support OpenWRT’s default logging, as well.

Without further ado, here’s the script, which lives in my log folder:

#!/bin/sh

# Live log display

while true; do
        tail -n 52 $1
        sleep 5
done

My various consoles I log into this from have anywhere from a 40 to 50 line display set up, hence the “52” – it’s reading and displaying the last 52 lines of the log every five seconds. If you always use shorter consoles, you can easily trim this down.

By passing the name of the script you want to read, this script has also been made “universal” in that it can be used to monitor any log on your machine. I also use it on a couple of my other servers, with no modifications. If you want to monitor “hexenbiest.log” you simply cd into the appropriate log folder, and run:

./loglive hexenbiest.log

Stock OpenWRT doesn’t write the log to a real file, it uses a ring buffer in memory that may be accessed using the command logread. To modify this script to run on a stock OpenWRT router, place it in the home folder (/root/) instead, and modify it to read accordingly:

#!/bin/sh

# Live log display

while true; do
        logread | tail -n 52
        sleep 5
done

This way, instead of reading the last 52 lines of a file every five seconds, it’s reading the last 52 lines of the logread output instead.

You might think it would make sense to clear the terminal before each output, but I didn’t personally find that helpful. In fact, it resulted in a visible flicker every time the log updated. Helpful if you need to know each time it reads, but I didn’t find that useful myself.