[Discovery] How is dnat actually expressed?

[thediveo]

After starting a test container and exposing a port of it on the host, I'm trying to find the corresponding port forwarding expression(s) in a table/chain/rule dump ... or rather, not finding it. What am I missing either in terms of not properly understanding the low-level nft netlink expressions or maybe missing some decoding in the nftables module?

sudo nft list table nat turns up the following rule in the "DOCKER" chain:

table ip nat {
  chain DOCKER {
    iifname != "docker0" meta l4proto tcp tcp dport 49153 counter packets 0 bytes 0 dnat to 172.17.0.2:12345
  }
}

Now I would have expected to find kind of a dnat expression. However somehow the dnat eludes my eyes, because the while rule pos 35 handles iifname != "docker0" I don't see any signs of either the original port number 49153 nor the rewritten port 12345; additionally, I don't recognize an 172.17.0.2 byte signature, that is 0xac, 0x11, 0x0, 0x2. Am I even on the right track here?

  CHAIN "DOCKER" TYPE "" HOOK "PREROUTING"
    RULE POS 0
      EXPR &expr.Meta{Key:0x6, SourceRegister:false, Register:0x1}
      EXPR &expr.Cmp{Op:0x0, Register:0x1, Data:[]uint8{0x64, 0x6f, 0x63, 0x6b, 0x65, 0x72, 0x30, 0x0}}
      EXPR &expr.Counter{Bytes:0x0, Packets:0x0}
      EXPR &expr.Verdict{Kind:-5, Chain:""}
    RULE POS 35
      EXPR &expr.Meta{Key:0x6, SourceRegister:false, Register:0x1}
      EXPR &expr.Cmp{Op:0x1, Register:0x1, Data:[]uint8{0x64, 0x6f, 0x63, 0x6b, 0x65, 0x72, 0x30, 0x0}}
      EXPR &expr.Meta{Key:0x10, SourceRegister:false, Register:0x1}
      EXPR &expr.Cmp{Op:0x0, Register:0x1, Data:[]uint8{0x6}}
      EXPR &expr.Counter{Bytes:0x0, Packets:0x0}
    RULE POS 39
      EXPR &expr.Meta{Key:0x6, SourceRegister:false, Register:0x1}
      EXPR &expr.Cmp{Op:0x0, Register:0x1, Data:[]uint8{0x62, 0x72, 0x2d, 0x32, 0x36, 0x37, 0x36, 0x32, 0x39, 0x39, 0x66, 0x31, 0x36, 0x39, 0x38, 0x0}}
      EXPR &expr.Counter{Bytes:0x0, Packets:0x0}
      EXPR &expr.Verdict{Kind:-5, Chain:""}
    RULE POS 40
      EXPR &expr.Meta{Key:0x6, SourceRegister:false, Register:0x1}
      EXPR &expr.Cmp{Op:0x0, Register:0x1, Data:[]uint8{0x62, 0x72, 0x2d, 0x39, 0x38, 0x66, 0x65, 0x31, 0x32, 0x39, 0x66, 0x34, 0x30, 0x64, 0x37, 0x0}}
      EXPR &expr.Counter{Bytes:0x0, Packets:0x0}
      EXPR &expr.Verdict{Kind:-5, Chain:""}
    RULE POS 41
      EXPR &expr.Meta{Key:0x6, SourceRegister:false, Register:0x1}
      EXPR &expr.Cmp{Op:0x0, Register:0x1, Data:[]uint8{0x62, 0x72, 0x2d, 0x61, 0x31, 0x33, 0x34, 0x61, 0x61, 0x65, 0x39, 0x62, 0x35, 0x63, 0x33, 0x0}}
      EXPR &expr.Counter{Bytes:0x0, Packets:0x0}
      EXPR &expr.Verdict{Kind:-5, Chain:""}
    RULE POS 42
      EXPR &expr.Meta{Key:0x6, SourceRegister:false, Register:0x1}
      EXPR &expr.Cmp{Op:0x0, Register:0x1, Data:[]uint8{0x62, 0x72, 0x2d, 0x62, 0x66, 0x36, 0x63, 0x36, 0x63, 0x39, 0x66, 0x66, 0x38, 0x66, 0x62, 0x0}}
      EXPR &expr.Counter{Bytes:0x0, Packets:0x0}
      EXPR &expr.Verdict{Kind:-5, Chain:""}
    RULE POS 44
      EXPR &expr.Meta{Key:0x6, SourceRegister:false, Register:0x1}
      EXPR &expr.Cmp{Op:0x0, Register:0x1, Data:[]uint8{0x62, 0x72, 0x2d, 0x64, 0x64, 0x63, 0x64, 0x64, 0x32, 0x32, 0x39, 0x61, 0x66, 0x37, 0x34, 0x0}}
      EXPR &expr.Counter{Bytes:0x0, Packets:0x0}
      EXPR &expr.Verdict{Kind:-5, Chain:""}

[thediveo]

might this be related to issue #161 as I notice by adding PR #124 in my own dev branch that there are in fact missing match and target expressions?

[stapelberg]

When using nft --debug all with your rules, this is what it prints (among other details):

ip
  [ meta load iifname => reg 1 ]
  [ cmp neq reg 1 0x6b636f64 0x00307265 0x00000000 0x00000000 ]
  [ meta load l4proto => reg 1 ]
  [ cmp eq reg 1 0x00000006 ]
  [ payload load 2b @ transport header + 2 => reg 1 ]
  [ cmp eq reg 1 0x000001c0 ]
  [ counter pkts 0 bytes 0 ]
  [ immediate reg 1 0x020011ac ]
  [ immediate reg 2 0x00003930 ]
  [ nat dnat ip addr_min reg 1 proto_min reg 2 flags 0x2 ]

You should be able to find the IP address in an expr.Immediate, for which decoding is implemented.

Can you share the code you’re using? Otherwise it’s hard to see what you’re doing.

Note that the focus of this nftables package was on writing nftables config originally, not reading nftables config back. So, yes, there might be a few little details here and there that are missing.

[thediveo]

The code basically is just rules, err := GetRules(...) on the "nat" table of family "ip" and the "DOCKER" chain.

for _, rule := range rules {
	fmt.Printf("    RULE HANDLE %d POS %d \n", rule.Handle, rule.Position)
	for _, expr := range rule.Exprs {
		fmt.Printf("      EXPR %#v\n", expr)
	}
}

Unfortunately, nft --debug all list table nat isn't of too much help here, but at least shows that due to the xtables translation there are not immediate and nat expression in this situation. Instead, there's a target expression directed at the DNAT rev 2 extension.

ip nat DOCKER 45 35 
  [ meta load iifname => reg 1 ]
  [ cmp neq reg 1 0x6b636f64 0x00307265 ]
  [ meta load l4proto => reg 1 ]
  [ cmp eq reg 1 0x00000006 ]
  [ match name tcp rev 0 ]
  [ counter pkts 0 bytes 0 ]
  [ target name DNAT rev 2 ]

After implementing unmarshalling for the MATCH and TARGET expressions this now reveals that in the context of Docker (and several other larger projects) we actually deal with xtables translation. For brevity (hah!) here's just what the final target expression is in more detail.

EXPR &expr.Target{Name:"DNAT", Rev:0x2, Info:[]uint8{0x3, 0x0, 0x0, 0x0, 0xac, 0x11, 0x0, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xac, 0x11, 0x0, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x30, 0x39, 0x30, 0x39, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}

The Info field looks suspiciously like a struct nf_nat_range or struct nf_nat_range2 (from nf_nat.h), but there are additional bytes at beyond both of these types. Any idea as to where to find the proper definitions of the information passed between nft and the nft_nat extension?

[thediveo]

This is now looking more like something to further tackle on one of the netfilter mailing lists, I suppose?

[stapelberg]

Phew, yeah, that exceeds my knowledge about nft or xtables :)
Best to take it to the mailing list indeed.

[thediveo]

In case someone else finds this issue: it seems as if the Info []byte contents are padded to the max. alignment of the platform. xt_check_match in net/netfilter/x_tables.c contains this check, where match->matchsize is the registered size of a particular type of Info payload, and this registered size gets topped up for padding using XT_ALIGN:

if (XT_ALIGN(par->match->matchsize) != size &&